Squeez Ettin 32M Span Pruner
This model finds relevant text in coding-agent tool output. Give it a query and one tool observation. It returns verbatim character spans from that observation.
The model has 32M parameters. It is about one-fifth the size of the reproduced 150M baseline. It reaches 0.517 Word-F1 versus 0.588 for that baseline. This is about 88% of the baseline F1 score with a much smaller model. It does not beat the baseline.
Training code: finetune_squeez_ettin_span_pruner.py
Use the model
from transformers import pipeline
MODEL_ID = "lxyuan/Squeez-Ettin-32M-Span-Pruner"
REVISION = "972acbb8d79c477888148aacf78a697abc3f30c5"
pruner = pipeline(
"span-pruning",
model=MODEL_ID,
revision=REVISION,
trust_remote_code=True,
)
tool_output = """total 648
-rw-r--r-- 1 user staff 2256 Mar 9 21:07 __init__.py
-rw-r--r-- 1 user staff 30 Mar 9 21:07 __main__.py
drwxr-xr-x 7 user staff 224 Mar 9 21:07 __pycache__
-rw-r--r-- 1 user staff 82537 Mar 9 21:07 app.py
-rw-r--r-- 1 user staff 23781 Mar 9 21:07 blueprints.py
-rw-r--r-- 1 user staff 32036 Mar 9 21:07 cli.py
-rw-r--r-- 1 user staff 11285 Mar 9 21:07 config.py
-rw-r--r-- 1 user staff 17428 Mar 9 21:07 ctx.py
-rw-r--r-- 1 user staff 6171 Mar 9 21:07 debughelpers.py
-rw-r--r-- 1 user staff 1723 Mar 9 21:07 globals.py
-rw-r--r-- 1 user staff 30589 Mar 9 21:07 helpers.py
drwxr-xr-x 5 user staff 160 Mar 9 21:07 json
-rw-r--r-- 1 user staff 2273 Mar 9 21:07 logging.py
-rw-r--r-- 1 user staff 0 Mar 9 21:07 py.typed
-rw-r--r-- 1 user staff 32786 Mar 9 21:07 scaffold.py
-rw-r--r-- 1 user staff 15189 Mar 9 21:07 sessions.py
-rw-r--r-- 1 user staff 2126 Mar 9 21:07 signals.py
-rw-r--r-- 1 user staff 5626 Mar 9 21:07 templating.py
-rw-r--r-- 1 user staff 10281 Mar 9 21:07 testing.py
-rw-r--r-- 1 user staff 1982 Mar 9 21:07 typing.py
-rw-r--r-- 1 user staff 5948 Mar 9 21:07 views.py
-rw-r--r-- 1 user staff 5604 Mar 9 21:07 wrappers.py
"""
spans = pruner({
"query": (
"Find the file entry for the Flask CLI implementation, cli.py, "
"that contains the background loading exception handling."
),
"tool_output": tool_output,
})
print(spans)
Output:
[
{
"start": 302,
"end": 356,
"text": "-rw-r--r-- 1 user staff 32036 Mar 9 21:07 cli.py\n",
}
]
The input contains 23 directory entries. The model returns only the relevant cli.py line. start and end are character offsets in tool_output.
Pass one {"query", "tool_output"} mapping per call. You can also pass threshold=0.6. The default threshold is 0.50.
The pipeline uses code from this repository. trust_remote_code=True runs that code on your machine. Review the pinned span_pruner.py before use.
Results
The main comparison uses the same 9,595-example GLiNER test and the same Word-F1 scorer for both models.
| Model | Parameters | Word-F1 |
|---|---|---|
| This model | 32M | 0.517 |
| Reproduced baseline | 150M | 0.588 |
The 32M model is 0.071 F1 below the baseline. The paired 95% bootstrap interval is -0.102 to -0.042.
Separate line-level checks use a different evaluation protocol:
- Canonical 618-example test: 0.531 line-F1 and 0.936 line-count compression.
- Repository-held-out SWE subset: 0.463 line-F1 across 430 xarray and Flask examples.
These line-level scores are not direct comparisons with the 150M Word-F1 result.
Training
The base model is jhu-clsp/ettin-encoder-32m. Training uses every row in the GLiNER training split.
The query and tool output are a tokenizer pair. Only tool-output tokens contribute to weighted cross-entropy. Query, padding, and special tokens use label -100 and do not affect the loss.
The run used full fine-tuning for four epochs in float32. The best checkpoint was selected by validation token-F1. Validation Word-F1 selected the final threshold of 0.50.
Long observations use overlapping 4,096-token windows. The pipeline max-pools overlap scores before it creates character spans.
Run the training program with:
uv run --script scripts/finetune_squeez_ettin_span_pruner.py
Limits
- The training data contains English queries and coding-agent tool output.
- Very short or unnumbered snippets can give poor results.
- The model can remove useful context or keep irrelevant text.
- Keep the original observation until the downstream task succeeds.
- The held-out SWE test covers only xarray and Flask. It does not prove broad repository or language support.
- Downloads last month
- 16
Model tree for lxyuan/Squeez-Ettin-32M-Span-Pruner
Base model
jhu-clsp/ettin-encoder-32m