test(framework): a diagnostic that proves the chain fires, and a period that has met a hundred sessions

[blafourcade]

As someone who turned measurement on
I want to know it is actually recording, and to trust a figure over a real stretch of time
So that an installation that reports itself healthy while producing nothing is impossible rather than merely unlikely

Why now

Everything shipped is verified and none of it has met load. 2614 CLI tests, 177 hook tests and two live probes — against three sessions at most, and a task tree of a handful of files.

"It scales" is currently a hope.

And #617 owns the diagnostic. It is deliberately sequenced after coverage: written before #681 and #680, it would spend its time reporting the two failures already known.

What this holds

A diagnostic — #617's substance, restated against what now exists. Not "is the switch on" but the four questions in order, each with a distinguishable answer:

Question	Broken looks like
Is the hook registered?	the switch is on, sessions run, and no run file appears
Is a session journalled?	a run file exists but carries only session_start
Can the tool's own files be read?	no session found for every tool while the journal names sessions
Do the two join?	records stored, every one of them unattributed

A period that has met a hundred sessions. Build a sink holding a year of day files and a hundred journalled sessions, then measure and write down: the period read, the sweep, and the turn-end walk over a real task tree. Nothing here has been run against more than three.

A budget for the turn-end walk. The observed pass walks aidd_docs/tasks once per turn, capped at 2000 entries and never measured on a real repository. A cap nobody has timed is a guess with a number on it.

A live multi-step flow. One skill gives two rows. A real SDLC chain gives several, and that is where interval closing, reconciliation across five steps, and interleaved skills stop being unit tests and become observations. Costs one real session on a small task.

Acceptance

• Each of the four failures above is deliberately induced and named as itself, never answered with a zero.
• A period holding a hundred sessions answers, and how long it takes is written down rather than assumed acceptable.
• The turn-end walk is timed on a repository with a real task tree, and its cap is set from that number.
• A live multi-step flow reports each step separately, and the breakdown reconciles to the total.
• The diagnostic never reports healthy while producing nothing — the mode the epic exists to make impossible.

Out of scope

Fixing what the diagnostic finds. Naming a failure is this ticket; #681, #680 and #676 own the failures already known.

Relations

Field	Value
parent	#631
depends_on	#681, #680
related	#617, #686

[blafourcade]

All three, done and written down

aidd_docs/tasks/2026_08/2026_08_21_telemetry-v1-close/measurements.md holds the numbers.

A period that has met a hundred sessions

365 day files, 100 sessions, 25 tasks, built through sink.js's own append() and the journal's own line builders — never hand-written JSON, because a hand-written fixture only proves the reader works on hand-written fixtures.

the period	55-77ms	every breakdown sums back to the total in whole micro-dollars, exactly
the sweep	55-75ms
one task's breakdown	55-70ms	assert.equal, no tolerance

A one-off at 1000 sessions and 1825 day files: 120ms and 976ms. Under a millisecond a session, amortised.

A finding that came out of it: the sweep's real cost is per-tool subprocess spawns, not the sink. With $PATH carrying an installed opencode binary the same 1000-session sweep is materially slower than with $PATH cleared.

The turn-end walk, timed

The premise in this ticket does not hold for this repository. It uses 172 of 2000 entries, 8.6% — not the hundreds of task folders assumed. The cap stays at 2000, now for a measured reason: ~13.5ms p95 to exhaust it, against a turn-end handler that budgets 200ms p95 and already spends ~9ms elsewhere.

Two bugs in the walk, both the exact shape this ticket exists to remove:

1. Off-by-one. if (++seen >= MAX) break counted the boundary entry as scanned without ever opening it.
2. A truncation false negative, found inside the first fix. Judging truncated by "is anything still queued" reads false when the cap lands mid-listing of one wide directory: the queue empties the moment that directory is popped, before its listing is read. Reproduced on a flat 2300-file tree, red then green against the buggy version.

And the deeper one: a wide enough directory makes the walk find zero files faster than the healthy case — indistinguishable from an empty period on a stopwatch. taskFilesModifiedSince now returns {found, truncated, scanned} and the run file carries a scan_truncated line, so reaching the cap says so.

A live multi-step flow

Three skills in sequence — 05-rule-generate, 07-command-generate, 01-commit — three step_start lines, one commit, 18 requests, 265.6s, $2.10.

Recomputed independently from report --json, all five token fields, integer-exact:

sum of the four by_step rows   18 / 42 / 6,865 / 925,799 / 42,289
totals in the same JSON        18 / 42 / 6,865 / 925,799 / 42,289

The fourth row is unattributed — the planning turn before the first skill call. It stays its own row with its own totals, never folded into the nearest step. That is the promise kept literally rather than approximately.

The diagnostic, run against the same session through a different code path, agrees exactly: records join ok — 17 of 18, 1 unattributed.

The first attempt produced nothing, and looked fine

Filed as #703. A project set up entirely through the CLI, every command reporting success, loaded none of its plugins under claude -p: Skipping orphaned enabledPlugins entry: marketplace not registered, on stderr, behind a debug flag. All three skills answered "Unknown skill" and the session hand-rolled the procedures off disk, which reads as success from outside. $1.27 for nothing.

The measurement layer was never at fault — it correctly reported nothing, because nothing happened. But it is the same failure this milestone exists to make impossible, one layer down.

One more thing measured

Codex sets CODEX_THREAD_ID in the shell a skill spawns, matching the value record.js already parses as its vendor_id, and confirmed not inherited from the launching shell. The diagnostic's session anchor now covers two tools instead of one. Where both variables are present that is the nested case by construction, and Codex's wins by ordering alone — proven with a fixture where the inherited Claude Code id matches a healthy old run file and the diagnostic still correctly reports this session as having left none.

Acceptance

• Each of the four failures induced and named as itself — plus a fifth, in feat(framework): a skill that proves the telemetry pipeline fires #617's skill.
• A hundred sessions answer, and the timings are written down.
• The walk is timed on a real tree and the cap set from that number.
• A live multi-step flow reports each step separately and reconciles to the total.
• The diagnostic never reports healthy while producing nothing.

Gate: plugin scripts 295 · CLI unit 1,893 · integration 567 · e2e 178 · biome 0 · tsc 0 · 0 broken links.

[blafourcade]

Done and awaiting review in #706, which closes this on merge.

The work is on claude/aidd-telemetry-layer-e403uf, eleven commits, targeting next. Gate at the time of push: 365 plugin specs, 1,931 CLI unit, 577 integration, 178 e2e, tsc and biome clean, no broken markdown links.

Read the eleven commits rather than the pull request's file count — the branch carries its own copy of the CLI migration that next has since received, so the diff counts it twice. The range and the real figures are in the first comment on #706: 142 files, +10,518 −418.

Every claim about a tool in this work rests on a session that was actually run. The negative results are kept in full in aidd_docs/tasks/2026_08/2026_08_22_telemetry-every-tool/measurements.md, including the two probes that disagreed about OpenCode and the measurement that reconciled them.