feat(pricing): monitor the rate table against two public sources

[c-1k]

What

PRICING_TABLE ships to customers in the published usertrust package and had no drift monitoring of any kind. PRICING_TABLE_VERSION is a hand-typed constant; nothing fetched, compared, or expired.

This matters beyond hygiene. receipt.pricing.appliedRates + tableVersion are what let a third party recompute a metered cost from the record alone — check out the stamped table version, recompute ceil(sum(counts × rates / 1000)), floored at 1. That the table is Apache-2.0 public source is what makes independent verifiability real rather than rhetorical. So a stale rate produces receipts that are internally consistent, cryptographically sound, and externally wrong — and the hash chain preserves them faithfully.

Measured against the shipped table

Outcome	Count
Exact agreement, all four tiers	16
Expected deviation (ours higher)	1 — claude-sonnet-5
Sources conflict with each other	2 — deepseek-chat, deepseek-reasoner
No upstream source at all	7
Total	26

Current exit status is 0. Nothing in the table is wrong today.

Four rules carry the weight

Vendor pinning. models.dev carries 190 providers, most of them resellers. An unpinned matcher answered claude-fable-5 at 30/185 against a true 100/500, and claude-opus-5 at 55/275 (a regional premium) — every one well-formed and wrong. Both sources are pinned to the vendor; a non-vendor row is treated as absent, never as an answer. Mapping is explicit per model, because command-a would otherwise fuzzy-match command-a-translate-08-2025.

Deviations only in the safe direction. Both sources publish claude-sonnet-5 at $2/$10 — the introductory rate through 2026-08-31 that pricing.ts deliberately does not carry. So "differs from upstream" must not mean "we are wrong", or the monitor's first act would be to undo a deliberate conservative choice. An allowlist entry suppresses failure only while our rate is ≥ upstream; if upstream ever rises above ours it fails despite the entry. Understatement is never allowlistable — which enforces the D1 invariant rather than restating it.

Exit 2 is never 0. "Could not check" and "checked, found nothing" are different states. A coverage floor derived from the map (not a transcribed constant, so it self-maintains) turns a silent upstream schema change into a failure instead of a clean sweep of nothing.

Only agree is silent. Every model gets exactly one outcome, the counts must sum to the table size, and every other outcome reports by construction — including the 7 models with no external check, which is the most useful thing the report says.

Testing

27 tests, no network. The suite carries a positive control (a rate mutated below upstream that must be reported) beside a negative control — an all-pass suite cannot detect an instrument that is disconnected. The exit-2 path is tested by stubbing fetch, not by blocking the network: Node's fetch ignores https_proxy, so a proxy-based probe succeeds and proves nothing.

The unit conversion is pinned to a real table entry (PRICING_TABLE["claude-opus-5"].inputPer1k) rather than a literal, because assert.equal(convert(5), 5) passes against a broken identity conversion and locks the error in.

Scope

Report-only: contents: read means GitHub refuses a push, so no upstream value can reach pricing.ts through this path. Does not touch the receipt shape, TrustReceipt, or shared/types.ts. No cross-repo token.

Pure decision-rule tests run in ci.yml; the live comparison stays weekly so no network fetch can flake a PR.

Deferred findings (not in this PR)

1. resolveRates prefix-matching understates silently. claude-opus-5-fast matches claude-opus-5, prices at standard rates (2× understatement — fast is $10/$50), and returns rateSource: "table", indistinguishable on the receipt from an exact match. Regional variants are safe only by luck: us.anthropic.… doesn't prefix-match, so it lands on FALLBACK_RATE with unknown: true — visible, and it fires unknownModelPolicy. The suffix case is the silent one.
2. No regional dimension. LiteLLM carries us./eu./au./jp. at a 10% premium.
3. 7 of 26 shipped rates have no external corroboration. Reported from day one; closing the gap is separate work.

[c-1k]

Codex review — round 1

codex review --base origin/master
model: gpt-5.6-sol (max effort)
REVIEWED_SHA: 378988e219b117012edce1d97282946f6870ee92
EXIT: 0

Verdict: REVISE — 4 P1, 1 P2.

Several comparison paths can report a clean result despite proven or possible understatement, including cache-write premiums and source conflicts. Cache schema loss and cache-gap reporting can also disappear silently, undermining the monitor's core contract.

The review named the changed files (103 references), so this is a real pass, not a no-op against an empty diff.

Findings

	Finding	File
P1	Fail omitted cache tiers that underprice premiums	scripts/pricing-drift/compare.mts:221-226
P1	Preserve understatement checks across source conflicts	scripts/pricing-drift/compare.mts:188
P1	Merge nonconflicting tiers from every source	scripts/pricing-drift/compare.mts:212
P1	Validate cache fields against upstream schema changes	scripts/pricing-drift/sources.mts:112-113
P2	Render cache gaps for otherwise agreeing models	scripts/pricing-drift/report.mts:125

All five accepted as real and fixed in 86929ec. Three of the four P1s were one underlying mistake — electing a single source as "the" consensus rather than resolving each tier across all sources — so the fix was structural rather than five patches.

The two worth calling out

The monitor had, in its own instrument, the defect it exists to detect. The "schema pin" test asserted LITELLM_FIELDS against a hardcoded copy of itself. A constant compared to itself says nothing about upstream. Had LiteLLM renamed cache_creation_input_token_cost, input and output would still have resolved, every row would still have counted as answered, coverage would have stayed at 19/19, and the run would have exited 0 with cache comparison silently disabled. Replaced with runtime sentinels that validate both fetched corpora by name at ingest and exit 2; the tautological test was deleted rather than left reading as coverage.

An assumption carried over from the design was wrong. I had written that an omitted cache tier is always conservative, because it meters at inputPer1k through the D1 fallback. That holds for a cache-read discount (~0.1x input). It is false for a cache-write premium (1.25x input), where the fallback understates by 20%. Understatement is now judged on the effective metered rate rather than on field presence. Probed the live table: no model currently omits a cache tier that upstream prices above its input rate, so this was latent — it would have passed silently the day one appeared.

Note packages/core/src/ledger/pricing.ts:58 makes the same overbroad claim in its own comment ("priced at inputPer1k — conservative overstatement"). Not touched here; flagged as a follow-up.

Live verdict unchanged after the fixes: 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. Round 2 running against 86929ec.

[c-1k]

Codex review — round 2

codex review --base origin/master
model: gpt-5.6-sol (max effort)
REVIEWED_SHA: 86929ec2466bd3e9d9708c33672339f14c332e9f
EXIT: 0

Verdict: REVISE — 3 P1, 4 P2. No repeats from round 1.

The checker has false-clean paths and workflow failures that bypass its tracking issue. It can also silently stop validating cache tiers after certain schema changes.

211 references to the changed files, so a real pass.

Findings

	Finding	File
P1	Evaluate definitive diffs before source conflicts	compare.mts:276
P1	File an issue when the checker tests fail	pricing-drift.yml:75
P1	Validate schema values, not just property names	sources.mts:125
P2	Handle pre-report process failures as unchecked runs	pricing-drift.yml:83
P2	Reuse the canonical cache-rate resolver	compare.mts:143
P2	Preserve the full source range for conflict reports	compare.mts:157
P2	Include orphan failures in JSON output	run.mts:101

All seven accepted and fixed in 2cd264b.

The one worth dwelling on

The first P1 is round 1's fix reappearing one abstraction higher. Round 1 stopped a source conflict from masking understatement. It did not stop a conflict from masking plain disagreement: ours 75/250 against sources 50/200 and 50/250 returned a non-failing source-conflict while input was a unanimous, unallowlisted mismatch at 50. Same defect, same file, one level up — the abstraction introduced to fix a defect class was where that class reappeared. Definitive diffs now classify before conflict; conflict wins only when every diff is conflicted.

Two others were the monitor failing to monitor itself. If drift:test failed, every later step was implicitly gated on success(), so the tracking-issue step was skipped and the run merely turned red — recording nothing, which is precisely the "fail the job only" option this design rejected up front. The reporting step is now always()-gated and names which stage broke. That is not the always() anti-pattern it resembles: that rule governs steps reporting success (a healthcheck ping gated always() reads healthy for failed runs). This step reports failure, so success() is what breaks it. Called out in a comment so it doesn't get "fixed" later.

The schema sentinel counted property names, so an upstream keeping a deprecated key holding null while moving the price elsewhere would have cleared the floor while the normalizer dropped every value. It now counts usable typed values.

effectiveRate also duplicated D1's fallback rather than calling resolveAppliedRates, and had already diverged — a negative finite cache rate meters at inputPer1k in the SDK but compared as negative here. AGENTS.md forbids a second resolution site, and a monitor that resolves rates differently from the thing it monitors is measuring the wrong quantity.

Live verdict unchanged across all three rounds: 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. 44 tests, no network. Round 3 running against 2cd264b.

[c-1k]

Codex review — rounds 3 and 4

round 3  REVIEWED_SHA: 2cd264b33d39e0cc3d94b98e2a7bc371c7f8c576  EXIT: 0  → 1 P1, 2 P2
round 4  REVIEWED_SHA: 23009a8cddc88f6a944abb2226e21e40292b9e4a  EXIT: 0  → 3 P1, 2 P2

Both passes named the changed files (113 and 213 references). All seven findings accepted and fixed — round 3 in 23009a8, round 4 in 4b36ce5.

Round 3 — 1 P1, 2 P2

The checker can silently lose an explicitly mapped source while retaining full coverage, and two conflict paths produce incomplete or inaccurate reports.

	Finding
P1	Enforce coverage for every mapped source
P2	Avoid claiming an in-range rate exceeds every source
P2	Preserve the range for conservative cache conflicts

The P1 was the coverage floor at the wrong granularity. It counted models with at least one answering source, so if LiteLLM renamed gpt-4o while models.dev still answered, the model stayed corroborated, coverage held at 19/19, and the run came back clean — with one of two independent checks silently gone. The entire value of two sources is independence; a gate that cannot notice one disappearing is not guarding that. Coverage is now counted over (source, model) pairs: 36/36 today, where the model-level view showed 19/19.

Round 4 — 3 P1, 2 P2

The monitor can emit a clean result for definite rate drift and can silently redefine or skip portions of its coverage.

	Finding
P1	Flag rates above every conflicting source
P1	Keep a fixed coverage floor outside MODEL_MAP
P1	Scope schema sentinels to mapped vendor rows
P2	Report which source-model mappings disappeared
P2	Set GH_REPO for failure-reporting commands

The second P1 inverts a claim I made in the round-3 commit. I described the derived coverage floor as self-maintaining and treated that as a virtue. It is the weakness: a gate computed from the thing it guards is not a gate. Setting one model's litellm to null lowers the actual and the expected count together, so 35/35 passes and the lost check closes the issue. Both now run — the derived expectation catches a source dropping a model, and checked-in MIN_MAPPINGS / MIN_CORROBORATED_MODELS catch the map itself being weakened. Lowering either is an explicit, reviewable edit.

The sentinel scoping is the same shape one level down: counting fields across the whole 3,040-row corpus is satisfied by rows this tool never reads, so a mapped provider could rename a cache field while a handful of reseller rows kept the old name. Now counted over the 17 mapped LiteLLM rows.

Where this stands

Five rounds, 20 findings, every one accepted as real. The live verdict has been byte-identical through all of them — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0. Every fix closed a path by which a future run could report clean while something was wrong; none changed today's answer. That is the expected shape for this kind of tool, and it is why the review rounds were worth spending.

52 tests, no network in the suite. Round 5 running against 4b36ce5.

Recurring theme worth recording: three separate rounds found the same defect at successively higher levels of abstraction — conflict masking understatement (r1), conflict masking disagreement (r2), then a rate above every source still classified as conflict (r4). And twice the monitor carried, in its own instrument, the failure mode it exists to detect: a schema test that compared a constant to itself, and a coverage floor derived from the map it was guarding.

[c-1k]

Codex review — round 5

REVIEWED_SHA: 4b36ce5d8179077c1958d32e1b5cdf4adb91df68
EXIT: 0   →  1 P1, 3 P2   (163 references to the changed files)

The checker can silently lose cache-tier coverage and still exit successfully.

	Finding
P1	Pin cache-field coverage at the mapped-row baseline
P2	Classify conflicts using the effective cache rate
P2	Expose absolute-floor breaches in the report
P2	Keep non-default dispatches from rewriting the tracking issue

All four fixed in df416db.

The P1 is the schema sentinel's floor being shared across fields. One number is cleared by unaffected rows while a partial rename — one provider, or one field — strips the tier from every other mapped row. Those rows still answer, because input and output survive, so coverage stays full and a changed cache rate reads as agree. Worst on a one-source model like kimi-k3, where nothing else can contradict it.

Now per-source, per-field baselines measured against the mapped rows (litellm 17/17/17/17/8, models.dev 19/19/18/8), as a floor rather than an equality — upstream adding cache rates is fine, losing them is not.

Mutation-tested against the live feeds rather than only fixtures. Raising one baseline from 8 to 99 produces:

EXIT=2
LiteLLM: field coverage fell below the recorded baseline — schema likely changed:
  cache_creation_input_token_cost (8 of 99 mapped rows). Refusing to report a
  comparison that silently skipped these tiers.

That is a positive control on the real path, not just in the suite.

---

Stopping point

Six rounds, 24 findings, every one accepted as real: 5 → 7 → 3 → 5 → 4. Round 6 is running against df416db. My intent is to treat it as the last gate: fix anything P1, and ledger any remaining P2/P3 as residuals in this PR rather than continue indefinitely. This is a report-only tool — contents: read means it cannot write to pricing.ts under any failure mode — so the residual risk of a narrow edge case is a missed alarm, never a wrong rate.

The live verdict has been byte-identical through all six rounds — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, exit 0, 36/36 mappings. Not one fix changed today's answer; each closed a path by which a future run could report clean while something was wrong. For a monitor, that is the whole product, and it is why the rounds were worth spending.

Two patterns are worth recording beyond this PR:

The same defect kept reappearing one abstraction higher. Conflict masking understatement (r1) → conflict masking disagreement (r2) → a rate above every source still classified as conflict (r4) → a malformed rate compared raw instead of effective (r5). Four rounds, one idea: a comparison that cannot conclude something must not thereby conclude nothing is wrong.

Twice the monitor carried the exact failure mode it exists to detect. A schema test that compared a constant to itself, and a coverage floor derived from the map it was guarding. Both read as coverage while providing none — which is the definition of the thing this tool was built to catch.

[c-1k]

Codex review — round 6, and the gate is now blocked

REVIEWED_SHA: df416db40ad597d90043980dc1295ab448f5582e
EXIT: 0   →  1 P1, 2 P2

The checker can silently lose cache-tier coverage and can classify non-finite required rates as agreement. The workflow's default-branch state guard also admits an identically named tag.

All three fixed in 99afd36.

This round took three attempts. The first died on a malformed shell command Codex generated for itself; the second was cut off mid-investigation. Neither produced a findings block or a verdict sentence, and neither was read as clean — an absent verdict is a dead review, not a passing one. Only the third attempt concluded.

The P1 is the third narrowing of the same check, and the first two were both still too coarse

• A single shared occurrence floor is cleared by unaffected rows.
• Per-field totals are only a lower bound, so an upstream addition creates headroom: once one row gains cache_write, a later loss on a different row keeps the total at the baseline and passes — while that row still answers from input/output and the comparison silently skips its missing tier.

Now a per-(source, model, tier) expectation, 51 entries measured against the live feeds. Mutation-tested end to end:

EXIT=2
models.dev: schema/coverage regression — kimi-k3 lost cache_write.

A named model and a named tier, where the aggregate version returned a total that still summed correctly.

Also: a non-finite required rate reported agree. rawTier rejects Infinity/NaN, so inputPer1k: Infinity fell through the cache-absence path — required tiers are only excluded from cacheGaps — and emitted no diff at all, for a model whose metered cost is not a number. Now a malformed-rate outcome. And github.ref_name omits the ref namespace, so a dispatch on a tag sharing master's short name could rewrite master's tracking issue; both issue steps now require github.ref_type == 'branch'.

---

⚠️ The final SHA has no Codex verdict, and that is not a pass

Round 7 against 99afd36 terminated with:

ERROR: stream disconnected before completion: You have no credits remaining.
codex: Review was interrupted. Please re-run /review and wait for it to complete.

The Codex account is out of OpenAI API credits. That is also the likely cause of the two round-6 deaths. 99afd36's changes are therefore covered by 60 local tests and every local gate, but not by an independent review. Recording that plainly rather than letting six green rounds imply a seventh.

Re-run codex review --base origin/master once credits are restored.

Where this landed

Seven rounds attempted, six concluded, 28 findings, every one accepted as real: 5 → 7 → 3 → 5 → 4 → 3.

The live verdict never moved — 16 agree, 1 expected deviation, 2 source conflicts, 7 uncorroborated, 0 malformed, exit 0, 36/36 mappings. Not one of the 28 fixes changed today's answer; each closed a path by which a future run could report clean while something was wrong. For a monitor that is the entire product.

Local gates on 99afd36: 60 drift tests, 4367 repo tests, biome at the 39-warning master baseline, typecheck clean for both the repo and the new scripts.

Two patterns worth keeping

The same defect reappeared four times, each time one abstraction higher. Conflict masking understatement (r1) → conflict masking disagreement (r2) → a rate above every source still called conflict (r4) → a malformed rate compared raw instead of effective (r5). One idea underneath all of them: a comparison that cannot conclude something must not thereby conclude that nothing is wrong.

Three times the monitor carried the very failure mode it exists to detect. A schema test comparing a constant to itself; a coverage floor derived from the map it was guarding; and a cache-coverage baseline whose totals could stay satisfied while a specific model lost a specific tier. Each read as coverage while providing none.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99afd36efb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject wholesale disappearance of mapped source rows

When LiteLLM returns an object but its table shape changes (for example, {} or a new wrapper around all rows), every mapped lookup reaches this continue, problems remains empty, and the schema sentinel accepts the response. The run then classifies the resulting coverage loss as ordinary drift with exit 1, despite main() reserving exit 2 and the “could not run” issue path for schema mismatches. Record missing/provider-mismatched mapped rows as schema problems, at least when the response yields no usable mapped rows; assertModelsDevSchema has the same skip.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Attribute each tier difference only to its publishing sources

When both sources answer a model's required input/output rates but only one publishes the cache tier that differs, sourceNames still contains both sources and the Markdown renders both beside that cache comparison. This falsely attributes the displayed upstream value to a source that supplied no value for that tier, which can mislead the human expected to adjudicate the rate change. Preserve source names per TierDiff rather than attaching every model-level responder to every displayed difference.

Useful? React with 👍 / 👎.

[c-1k]

Round 7 re-attempted — still blocked, and a live finding from the sweep

Gate: re-ran codex review --base origin/master on 99afd36 after a report that credits were restored. Still fails:

ERROR: stream disconnected before completion: You have no credits remaining.
warning: Falling back from WebSockets to HTTPS transport. stream disconnected...
Review was interrupted. Please re-run /review and wait for it to complete.
EXIT=1

It now dies in 1.3 KB within seconds, where earlier deaths produced 250–350 KB before dropping — the API rejects at connect rather than mid-stream. Diagnosed before reporting, since auth misrouting reads almost identically to a spend limit: codex is on API-key auth, this session's env also carries OPENAI_API_KEY, but the two fingerprints match — same key, no second org, balance simply at zero.

99afd36 still has no independent verdict. Six concluded rounds, 28 findings, all fixed; the final SHA is covered by 60 local tests and every local gate only.

---

🔴 Live billing exposure in resolveRates — measured, not theoretical

The prefix-match issue I deferred as a follow-up turns out to be live and considerably worse than the -fast case that surfaced it. Swept getModelRates against both sources:

20 uncataloged OpenAI/Anthropic models are silently underbilled, worst 16.7×.

model	meters at	actual	underbill
gpt-4o-mini-tts	1.5 / 6	25 / 100	16.7×
gpt-5.4-pro	25 / 150	300 / 1800	12×
o3-pro	20 / 80	200 / 800	10×
gpt-4o-mini-transcribe	1.5 / 6	12.5 / 50	8.3×
o3-deep-research	20 / 80	100 / 400	5×
gpt-4o-realtime-preview	25 / 100	50 / 200	2×
o4-mini-deep-research	11 / 44	20 / 80	1.8×

Longest-prefix-first means any suffix that raises the price collapses onto the cheaper base. Every model above returns rateSource: "table" — indistinguishable on a receipt from an exact match, so the receipt is internally consistent, chain-verifiable, and wrong.

Models landing on FALLBACK_RATE are excluded and are fine: that path sets unknown: true and warns. The bug is specifically that these do not.

20 is a floor. The sweep only sees models published upstream as distinct ids, so claude-opus-5-fast is not counted — models.dev carries fast mode under experimental.modes.fast.cost rather than as its own id. The suffix family that first surfaced the class is invisible to the sweep that confirms it.

Fix direction: an allowlist of suffixes known to preserve pricing, everything else routed to the loud unknown fallback. A denylist would enumerate -fast, -pro, -tts, -realtime-preview, … and fail open on whatever ships next — and this class is defined by "a suffix we have not seen yet", so failing open is the one behaviour it must not have.

Not fixed in this PR. It is packages/core money-path (tier 0/1) and belongs in its own change with its own gates, not folded into a CI script. Raised here because this PR is what found it.