Dipankar Sarkar PRO

The self-teaching job is the one that doesn't compress cleanly. On long-horizon agent trajectories the KL signal piles onto shallow tokens and leaves the deep decision turns under-trained. Does Class 2 weight supervision per-turn, or stay trajectory-level?

Your temp-0-vs-sampling split is the cleanest diagnostic in this whole thread.

It separates two failures that look identical from the outside.

One: the determinism trap. Greedy makes the loop a fixed point, re-entered token for token. Sampling breaks it, which is your 12 to 6.

Two: state collapse. At IQ1_M sampling cannot pull it out, because the model has lost the memory that it already ruled this branch out. A random escape just re-converges.

The tell that separates them is lexical. Exact n-gram repeat means determinism, and temperature or a repetition penalty fixes it. Same dead reasoning reworded means the working memory is gone, and nothing short of more bits helps.

So I would split loop rate into those two buckets rather than log it as one number. The ratio is the real bit-depth signal: exact-repeat share should fall and reworded-repeat share should rise as you drop toward 1.7-bit.

The 6 of 25 that survive sampling at 4-bit are already sitting in your prediction files. Do they repeat the exact tokens, or reword the same dead path?

The headline number hides the interesting part.

78.6% at 31.7M matching a 3B is real, but look at the one it missed: (21+27)(14-7). Both sub-expressions are easy, the model gets 48 and 7, then the final 487 is where it breaks. That is not an arithmetic gap, it is a magnitude gap: a two-digit times one-digit crossing into three.

So the number I would want from ArithMark-2 is accuracy stratified by the digit-length of intermediate results, not one aggregate. A 30M model can memorize the small-magnitude table and still fall off a cliff exactly where carries compound.

Does the score hold if you filter to problems whose intermediates all exceed two digits?

Reasoning breaking before retrieval is the tell.

Circular loops at short context are not a memory failure, they are a decisiveness failure. At IQ1_M the logit distribution flattens, so the model never commits hard enough to escape a branch it already tried. Chat survives that because any fluent continuation passes. Reasoning does not, because it has to remember it ruled a path out.

The number I would log next to RULER is loop rate: fraction of reasoning traces that repeat an n-gram window before terminating. If that climbs with bit depth faster than accuracy falls, you have isolated exactly what 1-bit breaks.

Does the collapse survive greedy decoding, or does temp 0 pull it back out?