Add eigen_eval_loss: forward-only held-out cross-entropy

[InauguralPhysicist]

Why

A language model should be selected on held-out loss. That metric didn't exist in the model extension, so iLambdaAi ranked checkpoints on a generate-and-parse rate — and that rate cannot see the model.

Three checkpoints scored an identical 60/300 on the greedy parse metric while their held-out loss spanned 3.679 to 1.751 (perplexity 39.6 vs 5.76):

checkpoint	greedy parse	val loss	perplexity
untrained	0/300	7.004	1102
unshuffled training	60/300	3.679	39.6
shuffled training	60/300	1.751	5.76

Greedy decoding collapses onto frequency attractors regardless of how good the model underneath is, so the metric separates trained from untrained and then flatlines across every trained candidate. Deterministic is not the same as informative.

Sample size compounded it: at n=30 a 20% rate carries a 95% interval of roughly 8%–39%, so an A/B whose arms differed by 14 points of accuracy read as "no difference."

What

eigen_eval_loss of [prompt_ids, target_id] → cross-entropy in nats.

Forward only — no backward, no weight update, no requantise, no observer state, and g_model_age/g_training_samples untouched, so scoring a checkpoint never mutates it.

Log-softmax subtracts the max before exponentiating; without it a large logit overflows float and returns inf, which would silently poison a mean taken over thousands of windows.

Testing

Suite [47d] pins it against the property that an untrained model must score ln(vocab_size) — a near-uniform output distribution has that cross-entropy whatever the target is:

• vocab=1109 → measured 7.004, predicted 7.011
• vocab=8 fixture → measured 2.016, predicted 2.079

That single anchor catches a missing max-subtraction, a mis-indexed target, a softmax summed over the wrong axis, and sign errors simultaneously. Also verifies out-of-range targets return the -1 sentinel rather than reading past the logits array.

Suite: 3168/3168 on the http+model build.

🤖 Generated with Claude Code