Squeez Ettin 32M Span Pruner

This model finds relevant text in coding-agent tool output. Give it a query and one tool observation. It returns verbatim character spans from that observation.

The model has 32M parameters. It is about one-fifth the size of the reproduced 150M baseline. It reaches 0.517 Word-F1 versus 0.588 for that baseline. This is about 88% of the baseline F1 score with a much smaller model. It does not beat the baseline.

Training code: finetune_squeez_ettin_span_pruner.py

Use the model

from transformers import pipeline

MODEL_ID = "lxyuan/Squeez-Ettin-32M-Span-Pruner"
REVISION = "972acbb8d79c477888148aacf78a697abc3f30c5"

pruner = pipeline(
    "span-pruning",
    model=MODEL_ID,
    revision=REVISION,
    trust_remote_code=True,
)

tool_output = """total 648
-rw-r--r--   1 user  staff   2256 Mar  9 21:07 __init__.py
-rw-r--r--   1 user  staff     30 Mar  9 21:07 __main__.py
drwxr-xr-x   7 user  staff    224 Mar  9 21:07 __pycache__
-rw-r--r--   1 user  staff  82537 Mar  9 21:07 app.py
-rw-r--r--   1 user  staff  23781 Mar  9 21:07 blueprints.py
-rw-r--r--   1 user  staff  32036 Mar  9 21:07 cli.py
-rw-r--r--   1 user  staff  11285 Mar  9 21:07 config.py
-rw-r--r--   1 user  staff  17428 Mar  9 21:07 ctx.py
-rw-r--r--   1 user  staff   6171 Mar  9 21:07 debughelpers.py
-rw-r--r--   1 user  staff   1723 Mar  9 21:07 globals.py
-rw-r--r--   1 user  staff  30589 Mar  9 21:07 helpers.py
drwxr-xr-x   5 user  staff    160 Mar  9 21:07 json
-rw-r--r--   1 user  staff   2273 Mar  9 21:07 logging.py
-rw-r--r--   1 user  staff      0 Mar  9 21:07 py.typed
-rw-r--r--   1 user  staff  32786 Mar  9 21:07 scaffold.py
-rw-r--r--   1 user  staff  15189 Mar  9 21:07 sessions.py
-rw-r--r--   1 user  staff   2126 Mar  9 21:07 signals.py
-rw-r--r--   1 user  staff   5626 Mar  9 21:07 templating.py
-rw-r--r--   1 user  staff  10281 Mar  9 21:07 testing.py
-rw-r--r--   1 user  staff   1982 Mar  9 21:07 typing.py
-rw-r--r--   1 user  staff   5948 Mar  9 21:07 views.py
-rw-r--r--   1 user  staff   5604 Mar  9 21:07 wrappers.py
"""

spans = pruner({
    "query": (
        "Find the file entry for the Flask CLI implementation, cli.py, "
        "that contains the background loading exception handling."
    ),
    "tool_output": tool_output,
})

print(spans)

Output:

[
    {
        "start": 302,
        "end": 356,
        "text": "-rw-r--r--   1 user  staff  32036 Mar  9 21:07 cli.py\n",
    }
]

The input contains 23 directory entries. The model returns only the relevant cli.py line. start and end are character offsets in tool_output.

Pass one {"query", "tool_output"} mapping per call. You can also pass threshold=0.6. The default threshold is 0.50.

The pipeline uses code from this repository. trust_remote_code=True runs that code on your machine. Review the pinned span_pruner.py before use.

Results

The main comparison uses the same 9,595-example GLiNER test and the same Word-F1 scorer for both models.

Model Parameters Word-F1
This model 32M 0.517
Reproduced baseline 150M 0.588

The 32M model is 0.071 F1 below the baseline. The paired 95% bootstrap interval is -0.102 to -0.042.

Separate line-level checks use a different evaluation protocol:

  • Canonical 618-example test: 0.531 line-F1 and 0.936 line-count compression.
  • Repository-held-out SWE subset: 0.463 line-F1 across 430 xarray and Flask examples.

These line-level scores are not direct comparisons with the 150M Word-F1 result.

Training

The base model is jhu-clsp/ettin-encoder-32m. Training uses every row in the GLiNER training split.

The query and tool output are a tokenizer pair. Only tool-output tokens contribute to weighted cross-entropy. Query, padding, and special tokens use label -100 and do not affect the loss.

The run used full fine-tuning for four epochs in float32. The best checkpoint was selected by validation token-F1. Validation Word-F1 selected the final threshold of 0.50.

Long observations use overlapping 4,096-token windows. The pipeline max-pools overlap scores before it creates character spans.

Run the training program with:

uv run --script scripts/finetune_squeez_ettin_span_pruner.py

Limits

  • The training data contains English queries and coding-agent tool output.
  • Very short or unnumbered snippets can give poor results.
  • The model can remove useful context or keep irrelevant text.
  • Keep the original observation until the downstream task succeeds.
  • The held-out SWE test covers only xarray and Flask. It does not prove broad repository or language support.
Downloads last month
16
Safetensors
Model size
32M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for lxyuan/Squeez-Ettin-32M-Span-Pruner

Finetuned
(48)
this model

Datasets used to train lxyuan/Squeez-Ettin-32M-Span-Pruner