build_corpus: lossless slot-mode identifier encoding

[InauguralPhysicist]

The defect

Frequency mode promotes the top-N identifiers and drops every other identifier onto one fallback token. At top_n=256 on the current ecosystem corpus that is 45.7% of all identifier occurrences.

The consequence isn't a quality loss, it's an expressiveness loss. total is total + items[i] encodes as:

<ident> is <ident> + <ident>[<ident>]

"Assign the variable I just read" — the property separating well-formed output from correct output — is not statable. No amount of training recovers it.

Measured identifier structure

379,321 identifier occurrences across the corpus:

class	occurrences	share	distinct
builtin/stdlib (must be exact)	26,904	7.1%	219
local/arbitrary	352,417	92.9%	25,355

93% are locals, where any consistent renaming is an equally correct program — only identity matters. The vocabulary was being spent memorising 25,355 arbitrary names while the 219 that must be exact were a rounding error.

The encoding

Builtins get exact tokens, decided by the runtime's own registry (env_get + VAL_BUILTIN), so the list cannot drift as builtins are added. Every other identifier becomes one of N reusable LRU slots — a name keeps its slot until evicted, so every occurrence in a window is the same token.

Slot count is measured, not guessed. Distinct locals per real 32-token window: median 5, p99 10, max 17 → 20 slots cover 100% of windows, and no eviction can occur inside a window.

Result

FREQUENCY  vocab 341  fallback <ident> = 91,161  (45.7% of identifier tokens)
SLOT       vocab 362  fallback <ident> =      0  ( 0.0%)

Zero identifier loss for 21 extra vocabulary tokens. Slot usage is uniform across all 20 (5.1–5.4% each) rather than concentrating — which prevents a slot becoming the new frequency attractor.

Honest scope — this does NOT raise the grade ceiling

Measured on 300 real corpus units, both encodings give an identical distribution:

incomplete=58  g0(no-parse)=14  g1(parses)=223  g2(runs)=5

Grade 2 requires the unit to run, and a 60-token window of real code almost never defines everything it references — so v1 is v1 + v2[v3] is coherent but still errors on undefined names. The ceiling is bounded by unit self-containment, not by identifier encoding.

The claim is only that a correct program is now representable. Whether a model reaches that is a separate question requiring a training run.

Compatibility

Opt-in via an optional 6th argument. Frequency mode and every number recorded from it are unchanged, so the two encodings can be A/B'd.

Testing

Suite 3107/3107. New [43a2b] pins zero-fallback, in-range slot ids, identity preservation (a repeated local keeps one slot), and that frequency mode is untouched.

Second commit adds EIGS_TERNARY_FLIPS, a gated QAT code-churn diagnostic that killed a Zeno-freezing hypothesis (flip rate rises 9× over training rather than collapsing).

🤖 Generated with Claude Code