feat(framework): measurement that works on five tools, each proven by a session that ran - #706
Open
blafourcade wants to merge 69 commits into
Open
feat(framework): measurement that works on five tools, each proven by a session that ran#706blafourcade wants to merge 69 commits into
blafourcade wants to merge 69 commits into
Conversation
Contributor
Author
What to read, and why the file count liesGitHub says 1801 files. 142 of them are this work. The branch was cut on 16 July at The eleven commits that are this work, from Compare view for exactly those: 0509345...1015566 The base needs a decision before this leaves draftThree ways out, none of which I will take unasked because they all rewrite or reshape someone else's branch:
Everything else is readyEleven commits, each passing the pre-commit hook without modification. The pre-push run: 252 test files, 2680 tests. Plugin scripts 365, CLI unit 1931, integration 577, e2e 178, Every per-tool claim in the body rests on a session that was actually run, and the negative results are in |
Contributor
Author
Retargeted to
|
blafourcade
marked this pull request as ready for review
August 22, 2026 09:45
This was referenced Aug 22, 2026
Ground the telemetry design on what the five host tools actually expose, verified on their official docs. - No hook on any tool carries tokens or cost, so the framework joins vendor OTel exports instead of collecting its own. - Records the two join architectures (id mapping vs resource-attribute injection) and why they are complementary rather than exclusive. - Splits task identity from the session ledger, keeping the one-writer per file property that makes merge conflicts impossible. - Flags the contradiction inside #617 between its scope and its decisions, the undeclared #617 -> #620 ordering, and the ownership gaps on exporter config and sink. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uDp5FM1JJ6Yd2D9ZsfPH2
Second verification pass, primary sources only, per-cell [v]/[?] marks and a Sources annex naming every page read. Corrections to the first pass: - "no hook carries tokens" was too absolute. Claude Code PostToolUse on a foreground Agent call does carry totalTokens and usage, documented as covering the final request only. - Resource-attribute injection is verified, and splits in two: static keys land today through the settings env block, a per-session id still needs something that launches the tool. - Codex span attributes reach spans only, not the events carrying tokens. - Cardinality is now backed by vendor text and vendor code rather than asserted. Codex is verified from its otel crate because its docs host is blocked, and Cursor stays entirely unverified for the same reason. Both are stated as access gaps, not as findings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uDp5FM1JJ6Yd2D9ZsfPH2
Both vendors were unverified on the first pass because their doc hosts were unreachable. Read on their official documentation now. Cursor states outright that metric datapoints carry no correlation IDs, and documents the workaround: sum the log-side token fields grouped by conversation id. Tokens per session are reachable there, cost is not, because cost is metric-only. Its export is a team-level Enterprise beta, so the CLI can check it but never install it. Codex confirms SessionEnd with its 1s/3s timeout and its subagent gap, plus three lifecycle moments tool-paths.md does not list. Its metrics_exporter defaults to statsig, so enabling telemetry without setting that key ships metrics to a third party. Consequences reworked: identifiers-off-metrics is now backed by three vendors rather than asserted, cost per session is reachable on two tools out of five, and coverage is stated as a hierarchy so status can report it honestly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… execution The load-bearing assumption is now measured rather than assumed. Two real Claude Code sessions with OTLP captured locally: the session id a hook sees is the same value the export carries, skill.name rides on both the token and the cost counters, active_time is exported per session, and query_source separates main from subagent work. That shrinks what the framework has to build. On Claude Code the step journal is already emitted by the tool, so the only thing left to supply is the link to the work item. The spec records the folder layout, the two new files, and the field additions the existing templates need. Backlog artifacts already carry type and status; delivery artifacts do not, which is why the kanban's type filter returns nothing on this framework's own documents. Two measured limits are recorded as such: skill.name is sticky, so it over-attributes when skills interleave, and a Claude Code subagent has no identifier of its own where a Cursor subagent does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first pass had metadata.json carry a type and an issue reference. Both already live on the backlog artifact, in type, work_kind and source, so the spec was creating a second truth that would drift. metadata.json now holds one upward link and nothing more. Records the full chain from a run to its epic, and how each kind of work attaches. The existing relation model already covers every case: a bug fix is a Task whose parent is the Defect, a defect names the artifacts it broke through related_to, a spike names the artifacts it blocks. Nothing new was needed beyond task_id and backlog. States that nothing may point downward, so no one adds an inverse link later: readers index the delivery folders and group by backlog, which is what the relation reference already prescribes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ing quota The identifiers are minted client-side, before any model call, so a provider pointed at an address that answers nothing still starts a session, fires its hooks and emits its telemetry. Codex and Copilot were therefore verified on zero tokens and zero credits. Codex: hook session_id equals conversation.id on codex.sse_event. Copilot: hook sessionId equals gen_ai.conversation.id on the invoke_agent span. Cursor stays open, since its export is a team-level Enterprise setting and there is nothing to compare against without such an account. Two locks found by running the probes rather than by reading docs, both of which leave a hook installed and silent: Codex needs the hooks feature flag and persisted hook trust, and Copilot ignores repository-level hook files in a directory that was never trusted while honouring the same content at user scope. These are the states #617's status must report as broken rather than healthy. Because the probe is free on two tools, it belongs in continuous integration rather than in a one-off check: the equality is proven by a session, not by construction, and a tool update can break it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The probe was written and run. Cursor rejects a bogus key before opening a session, so no hook fires and the free-verification trick that works on Codex and Copilot does not apply. There is no login on this machine either, so nothing could be observed. More consequential than the missing measurement: the hooks documentation describes editor moments only, down to workspaceOpen, and nowhere states that the cursor-agent binary reads .cursor/hooks.json. Since a CLI install is the only mode the AIDD CLI has, Cursor coverage may have to be withdrawn rather than confirmed. Both documents now say so, and name what would unblock it, cheapest first: a login to learn whether the binary honours hooks at all, then an Enterprise account with team export to close the id equality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scope question is settled, and favourably. cursor-agent does read .cursor/hooks.json, so Cursor can sit in a layer installed by a CLI. The documentation describes editor moments only, down to workspaceOpen, which made the doubt reasonable; a real session settles it. Its payload carries session_id, conversation_id and generation_id where the documentation describes one. They hold the same value on a single-turn session, which is a trap rather than a reassurance: the ledger must store conversation_id, the only one documented as stable across turns. A two-turn probe would say whether the others drift. Getting there needed three refusals: Cursor validates the API key, then the model name, then workspace trust, all before opening a session. So the free-verification trick does not apply and checking Cursor costs a real turn. Which turns the earlier finding into a pattern: not one probe worked on the first try, and always for the same reason. Writing the hook file is not enough, a lock has to be lifted too, and each tool locks differently. A hook installed without lifting it is silent and raises nothing, which is the worst state a measurement layer can be in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Checked again after the upstream request for OpenTelemetry support was raised. Three findings, one of which corrects an earlier source. The binary does carry @opentelemetry/api and @opentelemetry/sdk-trace, along with the standard OTEL_* variable names. It carries no exporter package, and a session that completed successfully with OTEL_EXPORTER_OTLP_ENDPOINT set produced no OTLP request at all. The strings arrive as a transitive dependency, so their presence is not evidence of support - worth stating, because a table filled from a string search would have recorded the opposite. The upstream issue exists, is assigned, and has neither comment nor linked pull request. The repository also moved from sst/opencode to anomalyco/opencode. The first pass cited the old one: same conclusion, wrong source, now fixed in the annex. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wrong twice, and worth recording why. The switch is a config key rather than an environment variable, experimental.openTelemetry in opencode.json, described by its own schema as "Enable OpenTelemetry spans for AI SDK calls". And the second run that seemed to confirm the absence was void: the collector had failed to bind its port, so it recorded nothing whatever the tool did. With the flag set, a real session exports OTLP. Its ai.streamText spans carry gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and a detailed ai.usage breakdown, with session.id on the same span, so tokens per session are reachable. No cost is exported, so a price table is needed as on Codex. It also honours OTEL_RESOURCE_ATTRIBUTES, which makes the injection path available. Two reservations, both measured: the public documentation still says nothing and the upstream request has no reply, so the surface can move without notice; and one trivial session produced 495 spans across 348 KB, because everything down to file reads is instrumented. Sampling would not be optional. The reporter now checks the collector is listening before drawing any conclusion, and says a silent result is void rather than a finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The measurement campaign moved the risk, so the milestone plan has to move with it. The id join was the unknown the milestone was built around; it is now proven on four tools, two of them without spending quota. What replaces it is sharper: no probe worked on the first attempt, and never for a different reason. Codex needs a feature flag and persisted trust, Copilot ignores repository hooks in an untrusted folder, Cursor wants --trust. A hook installed without lifting its lock is silent and raises nothing, which is worse than having none, because it produces numbers that look right. So v1 proves the pipe flows before it proves what it carries: correct the per-tool facts, write the run journal, ship a status command that checks a hook actually fired rather than that a file exists, and read one number per task and per step. Claude Code only. Out of v1, with reasons: the commit trailer, the four other tools, the YAML config root, the backlog links, and both consumers. The plan also records not waiting on #585, since the CLI has no YAML parser and the one key it needs fits the JSON it already reads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…not mirroring The question has an answer already written in the framework's own persistence reference: never mirror one Story across supports. An artifact lives on exactly one support, so there is nothing to sync and no divergence to manage - the problem is removed rather than solved. The spec now says what backlog points at: an issue reference when the backlog lives with the ticket provider, a project-relative path when it lives in Markdown, which is what persistence.md already prescribes. The earlier draft assumed a Markdown backlog and hardcoded a path. It also records why the delivery folder and the run journal never compete with the remote: no ticket provider expresses which folder delivered which issue, which steps ran, or how long the sessions took. They add, they do not copy. This repository is the illustration. It has no aidd_docs/backlog, its GitHub issues are its backlog, and creating Markdown stories for the same subjects would break the rule and manufacture the drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on a real session with the AIDD plugins installed from their marketplace: skill.name reads "third-party" on both the token and the cost counters, and OTEL_LOG_TOOL_DETAILS does not lift it. The flag only un-redacts the skill_activated event, which then carries the real aidd-context:11-explore. The docs said so for anyone reading to the end - third-party plugin skill names are replaced - and AIDD ships from a third-party marketplace. The earlier probe missed it by testing a project-local skill, which the same rule exempts. Presence was measured, value was not. Three consequences. Claude Code stops being the exception: metric-grain joining holds for per-session totals only, and the per-step breakdown joins on logs like the other three tools, so the pipeline must ingest events from v1 rather than later. The breakdown becomes a correlation of skill_activated with api_request rather than a filter on an attribute. And a hard privacy trade appears, since the same flag also logs Bash commands and tool inputs, which makes collector-side attribute filtering a requirement rather than a convenience. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An earlier version of this plan described the owning plugin, the way a task resolves, the CLI surface and the join. All four were falsified by measurement within two days, and all four stayed readable as instructions while the issues said the opposite. The epic cites this directory as the plan of record, so anyone starting from it would have built the wrong thing. Phase files are removed rather than corrected: their content now lives in the issues, and a plan that restates its issues drifts from them silently. What remains is the ordering, the parallelism, and the three decisions that belong to no single issue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… is in scope Two contradictions the spec carried against itself. It said per-step cost could not come from the metrics, then, further down, that on Claude Code the join was direct and only needed filtering an attribute. An implementer reads the second and produces a report where every AIDD step reads third-party. Replaced with what was measured. api_request carries prompt.id, event.sequence, tokens, cost and model, and its own skill.name is redacted like the metrics. skill_activated carries the real name with the same correlation keys. So the rule is: order by event.sequence within a session and carry the last activated skill forward. That is exact rather than a time window, and it mirrors the provider's sticky behaviour instead of fighting it. The collector also stopped being a non-goal. Claude Code exposes no file exporter, so without a receiving endpoint nothing is readable once the session ends, and every reading issue depended on a component the design had excluded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two of #620's premises were unproven. Probed both before writing the plan. Installing the plugin does activate its hooks: no plugin.json in this repository declares a hooks key, so the mechanism might have reached users only through `aidd framework build`, which would make "do not install it" the wrong opt-out. It is discovered by convention; the premise holds. The same probe showed the hook fires on a session that ends "Not logged in", so verifying the journal costs nothing. Host detection cannot use field names: Claude Code and Codex hand a SessionStart hook the same five keys. It must not use environment either, since a Codex session launched from a Claude Code session inherits CLAUDECODE and CLAUDE_CODE_SESSION_ID from its parent, and nesting is the normal case here. The discriminator is the shape of transcript_path, with an unrecognised host degrading to writing nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ok field The journal's only consumer joins against telemetry, so the field name it carries has to be the one the export uses. The body's example already said session.id while the spec's prose described the hook-side name; they are different strings and one of them is unusable. The hook-side name needs no storage anyway, having already given its value in vendor_id. Three checks on the run-journal plan, recorded with it: the Codex path shape holds under a default ~/.codex and was not an artefact of the probe home; gitignoring .aidd/ does not re-open the coverage failure #620 flagged, since the pointer is ephemeral by design and the skills rewrite it; and the opt-in directory is not the destination, so status has to report "on, not yet materialised" rather than either "on" or "not wired". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two records one sequence apart share a millisecond in a captured export:
seq=45 api_request ts=2026-08-18T17:04:39.258Z
seq=46 assistant_response ts=2026-08-18T17:04:39.258Z
So a timestamp alone cannot order a session, and `event.sequence` was
being discarded on arrival because it was not on the allowlist - a value
that cannot be recovered later, since the payload is not kept.
Three lines: the field, its allowlist entry beside the timestamp, and its
place among the numeric fields. Nothing else about the stored shape
changes; the reader that consumes the order is #629.
The test asserts the collision from the raw capture rather than from a
hand-written record, and asserts the stored sequences sort into the
export's true order. It cannot assert the collision on stored lines: seq
46 carries no `cost_usd`, so the mapper drops it as not billed, by
design.
Refs #663
…tools Two things that had to land together, because the second has nowhere to write without the first. **The journal stops being single-host.** `journal.js` compared against the literal `"claude-code"`, so Codex, Cursor and Copilot wrote nothing at all - a step boundary had no run file to append to on three hosts out of four. Each host now declares how its session id is read (Copilot alone spells it `sessionId`) and how its working directory is read (Cursor delivers no `cwd` at all, only `workspace_roots`, of which the first entry that is genuinely a git repository is taken - a multi-root workspace lists several and not all are repositories). **A started step is recorded as a fact.** The already-declared `PostToolUse` is read twice, by two guard chains that share nothing else: `handleFileWritten` returns early unless the path looks like a task folder, and a skill call has no task path. One table per host, two extractor families - the name read from an argument on Claude Code and Copilot, derived from a `SKILL.md` path on Codex and Cursor. The path family scans every string in the payload rather than reading a named field on a named tool. That is not defensive coding: the Codex capture names the tool `Bash` where its own transcripts record `exec_command`, so an extractor keyed on the tool name would have matched nothing, silently, with every test still green. No end, no duration, no parent is written. A probe on each of the five tools established that not one exposes when a skill's work finishes; an end would be a conclusion stored as a fact, and the journal's shape exists to prevent exactly that. A step is a half-open interval, closed by the next marker or by the turn, and that derivation belongs to the reader. An emitted end marker was rejected because it would depend on the model choosing to emit it. The argv word `file-written` becomes `tool-used`, since the event now has two readings. A new test reads `hooks.json` and asserts every word it ships is one `journal.js` recognises - the journal was already dead on every real installation once, for a mismatch of exactly that shape. Two hosts stay silent for reasons outside this change, asserted as current behaviour rather than papered over: Copilot's payload shape defeats host detection (#681), and Cursor fires no turn-end headless (#680). Fixtures are captures from real sessions on all four hosts, redacted; a directory-scanning test fails on any leak, including in the four that predate it. Refs #663
Kept because the reasoning is the deliverable's other half: which tool exposes what, why a step is a half-open interval, and why an emitted end marker was rejected. The measurements behind them live on the ticket. Refs #663
…rote Decided in #684: the receiver stops being on the critical path. A project could be switched on, export correctly, and store nothing because nobody was listening, and nothing made that visible. The tools already write their own counters to their own files; this reads those. Claude Code's transcript, Codex's rollouts and OpenCode's export all yield token counts and a model. None yields a dollar amount, so nothing here computes one — #654 owns turning counters into money, and mixing the two would hide which half was measured. **Provenance is a required field, and the schema goes to 2.** A figure read locally and a figure received from an export are not interchangeable. Making it optional would have meant a default meaning "the old route", which is exactly the unreadable field this avoids. No migration is written: the sink is delivered but unmerged, so this is the one moment where bumping costs nothing. A reader cannot stamp its own provenance — `LocalCostCandidateRecord` omits the field, so it structurally cannot claim to be an export it is not. **OpenCode is asked, not queried.** Its counters live in SQLite, which would have meant a native dependency — a prebuild per platform and ABI, otherwise compiling at install time, so one tool's feature would break `npm i -g` for every user including the four fifths who never touch OpenCode. `opencode export --sanitize` returns the same figures on stdout. `package.json` is untouched. Four things measured against real files that a plausible implementation would have got wrong, silently: - **One Claude Code API call can log several transcript lines.** A `thinking` block and its `tool_use` share one `requestId`, one `message.id` and one identical `usage`. Four requestIds out of five in the sampled transcript carry more than one line, and one carries three. Mapping per line inflates every figure two- to threefold, and a synthetic one-line-per-call fixture passes. - **Codex's `input_tokens` is inclusive of `cached_input_tokens`**, where Claude Code's is exclusive. Confirmed by arithmetic on four consecutive events: `total_tokens` equals input plus output, not input plus cache plus output. Without the subtraction the same column means two different things depending on the tool. - **Codex's `token_count` event carries no model and no request id.** Those live on the `turn_context` that precedes the run, so records are emitted per turn, not per event. - **A rollout's `session_meta` carries both `id` and `session_id`, and the hook sees `id`.** On a fresh session they agree; on a resumed one `session_id` holds the parent thread. 124 of the 330 rollouts on the measuring machine are resumed, so this is not a corner case — and a reader keyed on the wrong one passes every test written against a fresh session. A tool that cannot be read says so, with its reason, and a covered tool carries any caveat on its own figures as data rather than a comment: a comment reaches nobody downstream. Copilot writes one counter per turn and nothing else; Cursor writes none at all and its export is an enterprise setting nobody here can enable. Both are measured facts, not gaps waiting to be filled. Refs #685, #684
Carries the reasoning the code cannot: why the receiver stops being on the critical path, why OpenCode is asked rather than queried, and why each tool is read at the granularity its own file offers rather than at one imposed on all three. Refs #685
A consumer had one clue and it was the wrong one. `vendor_field` reads `sessionId` for a locally-read Claude Code record, `session.id` for the same one exported, and `session_meta.id` for Codex — it encodes the route as much as the tool. Anyone aggregating across tools would have had to reverse that, and would have got it right until a sixth tool arrived. The answer already existed at both ends and was thrown away. The export mapper resolves which declared tool a payload belongs to, then keeps only the name of the attribute that matched. The local-read use-case asks a specific tool's reader and forgets which one it asked. Both now carry it through, and neither names a tool: the export path takes it from the declaration it matched, the read path from the entry it looked up. A reader still cannot name its own tool. `LocalCostCandidateRecord` omits it alongside `sink_schema_version` and `provenance` — a reader that could name itself could name another. The version stays at 2 although a required field is added. A version number tells a consumer what to expect from a line they hold, and nobody holds a version 2 line: the sink is delivered and unmerged. Bumping for a shape that exists only on this branch would encode its history into a wire format. The bump belongs to the first release. Two proofs worth keeping. The anti-literal test iterates `AI_TOOL_IDS` rather than listing the names to forbid, so a sixth tool is covered the day it is declared and not the day someone remembers. And the reader's inability to name itself is asserted at compile time — removing `tool` from the omitted set fails the build in four places, the assertion plus all three readers, which are then required to supply one. Verified against real data: reading this repository's own session stored 5247 records, every one naming its tool. Refs #687
Carries what the code cannot: why the tool is a field rather than an inference, why an absent step reads as unattributed and never as outside any step, and the measurement behind that — Claude Code omits its own attribution field both when no skill ran and when the version predates it, with no null to tell the two apart. Refs #687
Two sources, and they are not the same kind of claim. Claude Code writes the skill into its own transcript, per assistant message, on the line that already carries the counters — measured across 40 transcripts, 2267 attributed messages, 25 distinct skills. Real names, no flag, no journal, no interval. Where a tool states the step itself, that is what is stored. Everywhere else the run journal's boundaries apply: the export path, where the vendor's own attribute reads `third-party` for every framework skill, and Codex, Cursor and Copilot, none of which was measured to have an equivalent. A step covers the half-open interval from its own moment to the next boundary, and a record whose moment falls inside belongs to it. So every record says which answered: stated by the tool, derived from an interval, or unattributed. Collapsing the first two would let a consumer read an inference as a measurement, which is the failure this layer exists to prevent. **An absent step reads as unattributed and never as "outside any step".** Claude Code omits the field both when no skill ran and when the version predates it — it arrived around 2.1.220 — and across twelve versions there is not one null to tell the two apart. A report may say a figure is unattributed. It may not say the work happened outside a skill. The journal is not a requirement. A session without one yields the same counters, all unattributed, and a missing or truncated journal costs attribution rather than figures. Three things found while building this, none of them in the brief: - A journal boundary with an unparseable moment used to let the previous step's interval swallow everything after it — silent attribution to the wrong skill. Unparseable boundaries are dropped before pairing. - The export path is deliberately not journal-attributed. The mapper runs at receive time, while the journal is still being written: the live turn has no end yet, so its last boundary would read as an open interval and claim whatever arrived next. Inferring once, from an incomplete file, into a stored record is worse than leaving it unattributed. - A Codex record carried no moment at all, so no interval could ever reach it — the journal is its only step source. The rollout carries one on `turn_context`; it is now taken. Deliberately the turn's start and not a moment from a counted event inside it, since the record covers the whole turn. Refs #687
Everything the layer stores was described only by a TypeScript interface, which a pricing service or an aggregator cannot import. This is the shape in a form they can implement against: every field with its meaning, its presence condition, and what its absence means — an absent counter and a stored zero being different facts, evidenced by Codex omitting `cache_write_input_tokens` rather than sending it as zero. Four things a consumer cannot discover and would get wrong, each stated with the numbers rather than a sentence: - **The two kinds overlap.** `request` records carry per-request figures; `session` records carry ten-second deltas of the same quantities. On one captured session the request lines total $0.1605 and the session lines $0.0151 — a whole session against one flush, not a contradiction. Cost and the four counters come from `request` only; `active_time_s` from `session` only, since no log record carries it. - **`session` is one line per datapoint, never merged.** Six for one session. A consumer expecting one reads a fifth of the truth. - **A re-read appends unless matched on `turn_id`.** Local reading re-reads a growing file by design. The worked example shows 30 stored rather than 60, and the counterfactual without a `turn_id` that does reach 60. - **`unattributed` is not "outside any step"**, in the same words the records use, because that is where a consumer would look for permission to collapse them. The coverage table keeps each tool's own declared vocabulary rather than flattening it: measured, unmeasured and unsupported mean different things, and Cursor is unmeasured on export while unsupported on local read for two unrelated reasons. The document cannot drift. A test parses the field names from the interface and from the document's own headings — neither list is hand-copied — and fails in both directions naming the offending field. The worked examples' totals are recomputed from their samples, because an example that no longer adds up is how a reader stops trusting the rest. One claim was flagged as unsourced by its author and is now sourced: the Copilot `cost` field is denominated in premium requests rather than currency. Across fourteen local sessions it reads 0.33 for every single-request `claude-haiku-4.5` session while consumption ranges from 2.04 to 2.95 billion nano-AIU, and 0 for a five-request `gpt-5-mini` one. It tracks request count times a per-model multiplier, invariant to what was consumed. Refs #687
Something reads what #687 wrote down. `aidd telemetry report` answers what a period, or one task inside it, consumed - broken down by step, model and tool, with every attribution's strength printed as a number rather than gestured at in a caveat. Three deliverables interleave in these files and are committed together because they touch the same lines: the reporter itself, one tool's reader no longer failing every other tool's read, and one object a program can consume. A period means when the work ran. A session read locally days later is appended to today's day file while its records carry their own, older moments - measured on a real Codex rollout, two records stamped 2026-07-29 living in 2026-08-21.jsonl. Selecting by day file would have put July's work in August's total and looked right doing it. Every route now carries a moment taken from what it already writes, and a record with none belongs to no period rather than being placed by the day we heard about it. Determinism is asserted against record order, not only against repetition. A re-read appends, so the sink's line order genuinely differs between machines, and that check found `attributionMix` carrying insertion order. All three strengths now emit every time, in a fixed order, zero where zero is what was measured. What a tool can supply is declared per route rather than discovered from a missing number - Claude Code carries an amount on its export and not on its local read, and states its own step on the local read and not on the export. Each declaration is checked against what its reader actually produces from a captured file, and a declared route with no capture may claim nothing. That check found a real error on its first run. Closes #629, #689, #690. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Installing the plugin is the whole installation. Two self-contained scripts, each inside the skill that owns it, running under plain node with nothing else on the machine. skills/00-init/scripts/telemetry-switch.js 1.6 KB on | off skills/01-cost/scripts/telemetry-report.js 102.3 KB read | report CommonJS `.js`, matching the hooks beside them: the plugin directory carries no package.json, so one module system across every executable file it ships. Where they live was not a preference. A top-level `bin/` is dropped - installing a plugin translates its files into each tool's own layout, and that translation carries skills, agents, commands, rules and hooks and nothing else. A script anywhere else is silently never installed. Moving them was not enough either. Every installed file went through `rewriteContent`, which edits paths inside prose: measured, it changed the bundle by six bytes on Codex and one on Copilot - a script that no longer parses, shipped without a word. The translator now separates prose from artefact, on the native path and the flat one. The hook scripts had been escaping that by luck rather than by rule. Two guards keep two builds one contract: the plugin's script and the CLI answer byte-identical JSON for the same inputs, and the committed bundles are rebuilt and compared so a source change without a rebuild fails. The whole life of measurement is exercised in one sequence rather than in a test per step - reporting before enabling answers nothing rather than failing, disabling stops the recording without erasing what was measured, and re-enabling resumes into the journal that already exists. Closes #691. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
A live headless session exposed the gap: asked to create a file, the model reached for Bash and wrote it with a shell redirect. That payload carries a command string and no path, so nothing was recorded and the session belonged to no task - while the tokens it consumed were measured in full. No fixture could have shown it. Every captured payload uses a file tool, so every test asserted the path that works. The command is still never read. A command mentioning a path is not a command that wrote it, and attributing on a mention would invent a task. What changed is that the hook observes instead: at the end of every turn it walks the task tree and records what changed, marked `source: "observed"` beside the exact `source: "tool-stated"` a payload gives. At turn end, not at every tool call. A turn ends once per prompt while tools fire dozens of times, so this costs one walk per turn - and it keeps a guarantee a test already pinned, which caught a first attempt that ran on every PostToolUse: a tool call that wrote nothing still shells out to git zero times. The recorded moment is the end of the turn rather than the write. Nothing here observed when a file changed, only that it had, and a task is derived from the path. It also stops task attribution being Claude Code's alone: the pass reads the repository rather than a payload, so it runs identically on every host the journal covers - Codex included, whose apply_patch writes were the other half of this gap. Closes #692. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…goes next The published FAQ said the framework collects nothing while it collects. That sentence is the one people quote when asking whether AIDD watches them, and leaving it while shipping the opposite is worse than never having written it. Measurement now has its own section rather than a cramped bullet: off unless you turn it on, what is recorded and where, what is never recorded, that nothing leaves the machine today, and that turning it off keeps what was already measured. The remaining "no hosted service" claim is still true and stays. The plugin's own README stopped describing a plugin that ships hooks only and does nothing yet. It now says what a person gets, in the words they would use to explain it, with a coverage table naming what each tool cannot do and the ticket that would close it. `docs/telemetry-limits.md` collects every limit with the measurement behind it, so a missing figure is explained rather than rediscovered. And a plan to a clean v1 in four milestones, each worth stopping at: what exists reaches someone, every declared tool records, it cannot lie quietly, the figures leave the machine. Merging comes before any new work - nine tickets on one branch is the largest risk in it, and it grows every hour. Closes #658. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
A minified blob in someone else's repository is a reason not to trust it, before it is anything else. Both scripts were unreadable; neither had to be. The switch is now hand-written plain CommonJS, like the hooks beside it. It is the file someone reads before allowing anything to be recorded, and sixty commented lines answer "what does `on` do" better than any build artefact could. Nothing generates it, so nothing can drift from it. The reporter stays generated - it bundles the whole domain and could not be hand-written - but no longer minified. Unminified it keeps real function names and a `// src/...` marker above every block, so a reader sees what it does and where each part came from. A header at the top says what it is, what generates it, and what fails if it goes stale. It costs forty percent in size, on a file that is copied rather than downloaded, and buys an auditable one. Both properties are pinned: the bundle must still carry its source markers and its real names, and the switch must stay short enough to read in full. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…it is not Milestone 1 begins with the host that records nothing at all. #681 read the breakage out of a bundle two minor versions behind what is installed, and every link in it is sound and untested against a payload. So the plan starts by watching rather than fixing: a hook that writes down what it is given, one session, a fixture. A detector fixed against a shape nobody has seen would be guessing with extra steps, and a wrong guess there is silent - the journal simply stays empty, with no error and no line. Two things this machine already settles for free. Another tool registers Copilot hooks in PascalCase and receives payloads, so PascalCase hooks do fire on 1.0.80 - the open question is only which key set arrives. And its own hooks file mixes both spellings, so both payload builders are reachable and handling one is not enough. Also filed, and not planned here: the reporter the plugin ships is 144 KB because reading a session asks for a tool's whole definition. Marketplace entry shapes are sixteen percent of a script that reads a transcript, and over half of it is machinery for installing plugins. It is not a copy of the CLI by design; it is one by accident, through a dependency edge nobody meant to draw. #696. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The reporter was a 4,183-line bundle. It is now 1,281 lines of plain CommonJS across seven files, each doing one thing, readable by whoever installs the plugin into their repository. The size was never the code. Reading a session asked for a tool's whole definition, so marketplace entry shapes were sixteen percent of a script that reads a transcript and over half of it was machinery for installing plugins. A table of five tools replaces the registry; a file replaces a port and its adapter. Two implementations of one contract is the risk this takes on, and the answer is to check it rather than to hope. Both are run against the same files and compared byte for byte - the printed answer, the records that land on disk, the JSON, and the error messages, since two tools that disagree about why something failed are two tools nobody can reason about together. That check found two real divergences before it was even a test: `agent_name`, which the transcript spells `attributionAgent` and only on a sidechain, and `effort`, which the reader dropped entirely. A third was key order, which changes no meaning and does change bytes - the counters are now added in one order everywhere, so equivalence is assertable rather than approximate. The measured facts kept their comments: input inclusive of cache on Codex and exclusive on Claude Code, increments rather than running totals, deduplication on a request id, and an absent skill meaning nothing rather than none. Nothing else has one. Fifty-eight tests beside the scripts, named for the behaviour they hold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…ild strategy Each tool that runs hooks declares which root variable it expands, so any install route can read it from one place. The build route now reads the declaration rather than keeping its own copy, eliminating the point where the two spellings can drift. A tool that runs no hooks declares none, making absence a statement rather than a default value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
… says why Hook support no longer defaults to false; every tool now declares what is true of it. Tools that run hooks receive them with commands naming their own declared variable. When a tool cannot host plugins or has no hook support, the reason is stated alongside the falsehood, not inherited from a silent default. A hook is checked for resolving on arrival, not merely for installing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…ends The journal reads a payload shape that differed from what a bundled version showed, and that gap kept the chain from running. We now capture Copilot's real payload as fixtures, holding its key set exactly as it arrived. The host recognition logic extends to the actual shape, with a test that fails if the shape regresses. An unrecognised payload is now distinguishable from no payload at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…rding A new skill diagnostic answers four questions independently: is the hook registered, is the session journalled, are the tool's files readable, and do the journal and files join. Each question has its own check so failures name themselves. The chain stops failing silently and starts naming the way it breaks, every time. Tests prove the checks detect their specific failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…t it dropped The turn-end walk processes the task tree, capped at 2000 entries based on timing measured on a real repository. When the cap is reached, the report now says what was dropped instead of silently truncating. A period holding a hundred journalled sessions answers fully, with timings written down so the next person can compare against them rather than guessing at scale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
OpenCode now journals its own session id. Codex says when it is holding a hook back. Copilot's step boundaries are recognised from its own payload. Cursor is known not to run plugin hooks. Every claim about a tool comes from that tool running, never from reading its bundle, and what it cannot measure is stated as itself rather than guessed. A script that a hook loads now resolves to the installed plugin on every tool it was delivered to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Architecture and telemetry limits are documented for each of five tools: Claude Code measures everything and writes it out; Codex runs hooks but they may be skipped in silence; Copilot's compat payload is now recognised and its steps recorded; Cursor is known not to run plugin hooks; OpenCode journals its own sessions but misses nothing else. Each tool's facts come from sessions that ran, and every limitation is written down so a consumer can act on it instead of hoping for a number that cannot come. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
A script identifies which tests a working-tree change can break, running only those instead of the full suite. This keeps feedback loops tight during telemetry work, where the change footprint is large but the affected tests are concentrated. Tests run in seconds, not minutes, so a developer can verify a change before committing rather than waiting for a full suite to finish. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The journal resolves project_id and project_remote for the repository a session ran in. Carry that fact forward to the stored record instead of losing it - a machine-level sink can then separate costs by project. Phase 1 of measurement breakdown: the session-project record type lives in the CLI domain, the journal surfaces what session_start already holds, and the report script threads it through to each record. By-project reporting comes next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The envelope bumps to version 2. Add two new groupings at the top level: every day the period spans (including gaps, since omitting a row reads as continuity), and every project a record named. A record with no project is its own row, never folded, so the figure stays honest. Both groupings sum to the period total exactly. The CLI's cost-report-display mirrors the plugin's render functions; both have tests including the hundred-session reconciliation that confirms the byte identity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…efact A person asking what last month cost does not know which axis answers them. Phase 3 moves the burden from the command-line flags onto the question the person came to ask: what did this cost (total), what changed (by day), where did it go (by step/model/tool/project)? The skill's SKILL.md offers these in plain language; actions/03-report.md names the axis that answers each. The plugin computes nothing - it reads the envelope exactly as it does now. The artefact rendered matches the axis: a line for a total, a table for a breakdown, a markdown series for a timeline. Every figure in an artefact reads identically from the envelope. The cost-report-contract documents the envelope shape and what each axis can deliver. Test coverage includes the full artefact set and the byte-identity validation across plugin and CLI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The rebase onto next kept this branch's extracted writeSkillTree, which had no transform parameter, beside next's call site that passes one. The type error was the only thing that noticed; without it a tool's SKILL.md transform would have been dropped in silence on every marketplace build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…he test git exports GIT_DIR into every process it starts, so under a pre-push hook `git init` on a temp directory left it pointing at the outer repository. The journal then wrote somewhere else and four tests failed — only ever inside a git hook, which is the one place nobody runs them by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
blafourcade
force-pushed
the
claude/aidd-telemetry-layer-e403uf
branch
from
August 22, 2026 10:02
1015566 to
2867585
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement that works on five tools, each proven by a session that was actually run.
What a person gets
Three skills, no CLI dependency, no endpoint, nothing leaving the machine:
00-initturns measurement on for a repository,01-costanswers what a period or a task consumed,02-checksays whether the chain is actually recording.The first citable figure this layer has produced, from a real three-skill chain:
Integer-exact on all five token fields. The fourth row is
unattributed— the planning turn before the first skill call — and it stays its own row rather than being folded into the nearest step.Per tool, without rounding up
Two tools give time and steps without an amount. That is a property of those tools, and the report names it beside the figure instead of printing a zero.
What was found by running rather than reading
Every one of these was invisible to a green test suite, and each was the same shape: something that looks installed, does nothing, and says nothing.
hooks.json. Across three probes, headless and interactive, auto-discovered and explicitly loaded with a valid manifest, not one of seven declared events fired. The project's own.cursor/hooks.jsonfires normally — so the obstacle was never Cursor, it was where we installed. It also names its repository rootworkspace_rootswhere every other tool sayscwd, and closes a turn onstopinteractively andsessionEndheadlessly, disjointly.tool_name: "skill"its own way,tool_input: {skill}Claude Code's way. Neither value followed from the other; guessing either would have failed silently.hooks/died at load on the one install shape that has nohooks/. 310 tests passed over it, because every one ran from the source tree.Delivered, not only proven
Every mechanism above is delivered by an install, not by a file placed by hand — including the routes that were still skipping it: OpenCode's build and setup paths, and Cursor's marketplace route.
aidd plugin removeunmerges what it merged.Guards added, each proven by breaking it
journalAttributablederivation is pinned against the plugin's own declaration, per tool. It caught two real drifts.pnpm test:changedruns only what a change can break: a CLI-only change went from 2m17s to 25s, without skipping anything reachable.Known limits, written down
docs/telemetry-limits.mdcarries each with the measurement behind it. The three that will bite first: Codex needs an interactive approval once and a headless run never sees the prompt; OpenCode misses the first session of a server process, and a one-shotopencode runis always a first session; only Claude Code's writes name a task, so the others report by period and by step and not by ticket.Closes
Closes #617
Closes #676
Closes #680
Closes #681
Closes #694
Closes #698
Closes #699
Closes #701
Opened by this work and left open: #700 (setup writes a model a ChatGPT account rejects), #702 (the tool declarations exist in several copies), #703 (a CLI-set-up project loads no plugins headless), #704 and #705 (per-project and per-day reporting).
🤖 Generated with Claude Code
https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp