Add eval scenarios, reviewed and corrected

[jimutt]

Ten scenarios from tessl scenario generate . --count 10 --workspace dld-kit, reviewed against the skills they claim to test and corrected. Also adds tessl.json linking the directory to a Tessl project, which tessl eval run requires.

Read this before merging: the baseline is meaningless for this kit

A Tessl eval runs each task twice — without the plugin and with it — and reports the difference. That framing does not work here, and the scenarios should not be read as evidence that DLD "adds value."

The control is degenerate in both directions:

• Baseline gets the scripts (how the fixtures are built) — the scripts are part of the kit and largely self-documenting, so the baseline agent infers the workflow and matches the treatment.
• Baseline denied the scripts — it cannot conform to conventions it has never seen, so the delta only says the kit exists.

There is no configuration where the control is informative, because DLD's correct output is defined by DLD. The treatment supplies the standard being graded against.

Confirmed empirically. A probe run of scenario-4 — the most prohibition-heavy rubric in the set — scored 100/100 on every check in both variants, with the with-context run taking 3m33s / 666k input tokens against the baseline's 1m26s / 270k.

So run these with --skip-baseline and treat the result as a regression suite — "does v0.9.0 still produce correct DLD artifacts across ten workflows" — rather than a measure of the kit's worth:

tessl eval run . --skip-baseline --label "v0.9.0 regression"

The useful comparison is between kit versions, not against no kit: --context-commit v0.8.0 holds the scenarios fixed and varies the kit, which answers "did this change help."

Related consequence: the publish-time eval feeding the registry quality score runs the A/B, so that number will stay near zero-delta regardless of kit quality. Not worth chasing.

Corrections made to the generated set

scenario-9 rebuilt. Its setup.sh wrote reduced reimplementations of all eight reindex scripts inline (rename-decision.sh 119 lines → 31, commit-reindex.sh 177 → 52), so the eval measured the agent against a stub kit. Real scripts now vendored at .tessl/skills/, verified byte-identical. The rubric expected a commit subject of dld-reindex: rename colliding IDs, an invention of the stub; the real script emits reindex local decisions: DL-004 -> DL-006, .... Verified end-to-end.

Two rubrics contradicted their own skills. scenario-6 required annotations within the first 10 lines where dld-retrofit allows "top of file or on the main export/class"; scenario-8's highest-weighted item required two decisions on one annotation line where dld-implement says they can share one. Both would have penalized correct work.

Self-reported process checks: 15 items / 151 of 1000 weight → 2 / 26. Converted where observable (%% escaping is now checked by whether percentages survive intact in the record body), dropped where redundant, removed where unobservable. The two remaining score the audit report and reindex summary, which are the requested deliverables. Freed weight went to untested task requirements — amends/supersedes frontmatter, required body content, snapshot artifact state.

All 103 items had no category. All 95 now carry one.

Scenarios 2, 4 and 5 shipped embedded .git directories, which git can only store as gitlinks — a clone would get empty fixtures, and common.sh resolves the project root via git rev-parse. Replaced with setup.sh, matching scenarios 3, 7 and 9.

All ten rubrics total exactly 100.

🤖 Generated with Claude Code