fix(spec,cli): govern the QA testing domain and enforce TestSuiteSchema at the os test load site (#6247) - #7255
Merged
Conversation
…ma at the `os test` load site (#6247) #6247 filed `packages/spec/src/qa/testing.zod.ts` as declared-but-inert on a grep that scanned only `*Schema` identifiers. Every consumer here reads the TYPE names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`), so the search matched nothing and a complete execution chain read as zero consumers: core's `TestRunner` + `HttpTestAdapter` (whose `action.type` switch labels ARE the `TestActionTypeSchema` values), published through `export * as QA`, driven by the shipped, documented CLI command `os test`. The 2026-08-07 retire ruling rested on that reading and was withdrawn on 2026-08-08 in favour of enforce. The real gap was narrower and genuine: the type was the contract and the schema had no `parse` site anywhere, so `os test` loaded suites with `JSON.parse(content) as QA.TestSuite` beside the author's own `// Should validate with Zod`. - `packages/spec/liveness/qa.json` — seed the ledger, governed via the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation` (a QA suite is an authored file, not stack metadata). 4 live rows with file:line evidence into the runner, 5 dead recorded honestly; step/action/assertion keys sit below the one-level walk and are measured in the notes rather than fanned into rows the gate would not check. No `authorWarn` anywhere, deliberately: the lint walks stack collections and a QA suite belongs to no stack, so the flag would be a silent no-op inside the mechanism built to catch silent no-ops. - `packages/cli/src/commands/test.ts` — `loadTestSuite()` parses with `TestSuiteSchema.safeParse` at the load boundary and refuses a bad suite there, naming the file, listing the issues and quoting the expected shape. A refusal counts as one failed suite instead of killing the run. - pin `packages/cli/test/qa-suite-schema-load.test.ts` — the three shapes the cast admitted (missing `scenarios` → TypeError inside the runner; misspelled `steps` → scenario reports PASSED having executed nothing; bad `action.type` → dies mid-run after earlier steps wrote records) are now refused at load. Closes #6247 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up7rAGwREEy754haLKVtZH
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
📓 Docs Drift CheckThis PR changes 2 package(s): 111 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
⛔ 7 release-owned page(s) also reference the affected code. These are read-only:
|
github-actions
Bot
added
documentation
#6247) The load-site parse is user-visible behaviour: a suite that does not match `TestSuiteSchema` is now refused before it runs, named, and counted as one failed suite while the rest of the glob continues. The `os test` section listed the flags and said nothing about what happens to a bad file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up7rAGwREEy754haLKVtZH
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31357039709 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Aug 10, 2026
Contributor
Author
|
Queue ejection triaged: not this PR — it is the known watcher flake, #7282. Checklist item 2, not item 1. Recording the measurement rather than re-queueing blindly, since every blind re-queue rebuilds everything behind it. The failure. Queue run 31357039709, Why it cannot be this PR.
What this ejection adds to #7282, which already diagnosed this family as "the event is suppressed, not late" — this is a fourth occurrence and it sharpens two of that issue's open points, so I've posted the detail there:
Not re-queueing on my own initiative, per the triage bot's guidance: nothing in this PR changed, so a re-queue is a coin flip on the same flake rather than a fix. Re-queue whenever convenient — the checks all passed on the PR-side run (26 checks, 25 success + 1 skipped) and the branch is conflict-free. I'll keep watching and will re-triage if the next attempt fails on a different signature. Generated by Claude Code |
This was referenced Aug 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executes the standing maintainer ruling on #6247, verbatim:
Premise re-measurement (fresh
origin/main@f40c5b4)Every premise the ruling rests on was re-measured in a clean worktree before any edit:
packages/spec/liveness/qa.jsonexistsls packages/spec/liveness/— 29 ledgers, noqa.jsonpackages/cli/src/commands/test.ts:98—JSON.parse(content) as QA.TestSuite; // Should validate with Zodpackages/core/src/qa/runner.ts:27, 41, 62, 86, 101, 112, 118, 125action.type=TestActionTypeSchemavaluespackages/core/src/qa/http-adapter.ts:21-38export * as QApackages/core/src/index.ts:25os testcommandpackages/cli/src/commands/test.ts,content/docs/deployment/cli.mdx:987,1012-1020,packages/cli/README.md:104deployment/cli.mdx, notreferences/cli.mdxas the re-ruling request cited — same content, same line numbers)qais not a registered metadata typelistMetadataTypeSchemaTypes(); the gate reports noungovernedentry for itSPEC_ONLY_SCHEMASroute, likequery/webhook/validationThe whole finding turns on one thing worth repeating: the original grep scanned
*Schemaidentifiers, and every consumer in this domain reads the TYPE names (QA.TestSuite,QA.TestScenario,QA.TestStep,QA.TestAction,QA.TestAssertion). The full reference set is four files and the search matched none of them. TheevidenceScopesection of the liveness README already says no amount of specifier matching is sufficient for a negative claim (#4895 case 4); this is the same lesson one level over, for identifier matching.Changes
packages/spec/liveness/qa.jsonlivewithfile:lineevidence into the runner, 5dead.scenariosis DRILLED viachildren, so noundrilled-containers.baseline.jsonrow is needed.packages/spec/scripts/liveness/check-liveness.mtsqaadded toGOVERNED+SPEC_ONLY_SCHEMAS(TestSuiteSchema), with the rationale comment the other three overrides carry. Without this the ledger file would be inert —loadLedgeris only called for governed types, so an ungovernedqa.jsonwould be documentation nothing checks, which is the exact sin the issue reports.packages/cli/src/commands/test.tsloadTestSuite()—TestSuiteSchema.safeParseat the load boundary, replacing the cast. Bad JSON and schema failures both throw an error naming the file, listing the issues and quoting the expected shape. Wired ahead of the run branch so a refusal is reported as a refusal, not as a run failure; a broken suite is one failed suite, not a dead command.packages/cli/test/qa-suite-schema-load.test.tspackages/spec/liveness/README.mdqarow in the state table + heading count..changeset/qa-testing-liveness-enforce.mdspecminor,clipatch).What the ledger actually says
live(4):scenarios.id,scenarios.setup,scenarios.steps,scenarios.teardown— each citing the runner line that reads it.dead(5), measured, not assumed:name(suite) —runSuitetouches onlysuite.scenarios; the CLI printspath.basename(file). The file name is the suite identity today.scenarios.name— itsdescribe()says "Scenario name for test reports" and no report carries it:TestResulthasscenarioIdand no name field. An author who writes a carefulnameand a terseidgets the terse one in every failure line.scenarios.description— docs-shaped, kept (ADR-0033), same disposition asflow.description.scenarios.tags— promises "Tags for filtering and categorization" and nothing filters:os testhas exactly two flags (--url,--token). Enforce-or-remove worklist.scenarios.requires— declaresparams/pluginspreconditions nothing checks. A suite naming an absent plugin runs anyway and fails later as an unexplained HTTP error. Enforce-or-remove worklist.Three deliberate calls a reviewer should check rather than skim:
authorWarnanywhere, on purpose.tagsandrequiresare exactly the misleading shape the flag exists for — but the author-side lint (packages/lint/src/lint-liveness-properties.ts) walks stack collections (stack.flows,stack.views, …) and a QA suite is a loose file in no stack. Marking them would produce a warning nothing can emit: a silent no-op inside the mechanism built to catch silent no-ops. Recorded as_authorWarnSkippedin the file note, with the correction carried in each row'snoteinstead.scenarios.setupnote rather than fanned into rows the gate would never check — thequeryprecedent for sub-walk verdicts. Two value-level gaps are noted there and are loud, not silent (run_scripthas no adapter branch → throws;not_contains/gt/gte/lt/lte/error→ throw "Unknown assertion operator"), so neither changes a key verdict — theapi.jsontypeprecedent.evidenceScope: "in-repo"on every row. Honest rather than optimistic:objectuiwas not walked. The structural argument for why that is sufficient here — the surface is a JSON file read by an oclif command in a Node process, with no renderer path to reach it — is stated in the ledger, not assumed.Reverse verification — direction predicted first
Predicted before running: the reject pins must be RED against the cast (a type assertion checks nothing at runtime, so every malformed document loads fine), and the valid-suite pin must be GREEN in both states (proving the RED is caused by the defect, not by the harness).
safeParse)scenariosstepskeyaction.typeUnexpected end of JSON inputMeasured, not asserted:
5 failed | 1 passed (6)before,6 passed (6)after. To take the pre-change measurement against the real defect rather than a missing import, the loader was first extracted with the cast body preserved verbatim, the pin run, then the body swapped for the parse — so the only variable between the two runs isJSON.parse(…) as QA.TestSuiteversusTestSuiteSchema.safeParse(…).Each rejected shape is one the cast really shipped, which is why the pin uses these three and not synthetic garbage:
scenarios→TypeErrorinsideTestRunner.runSuite, attributed to the runner with no idea which file it came from;steps→ the scenario reports PASSED having executed nothing (the dangerous one: indistinguishable from green in CI);action.type→ survives load and runner, dies in the adapter'sdefault:branch mid-run, after earlier steps have already written records.End-to-end smoke through the built CLI, since the pin covers the loader and not the loop wiring:
Gates
check:livenessqa 9 classified (dead 5, live 4); 349/349 repo-local evidence paths resolve; no orphan rows, no undeclared container inheritancecheck:empty-stateall classified (1 closed, 2 open, 4 output, 9 scope)check:generatedAll 11 generated artifacts are up to date(spec built first, per #7122)scripts/liveness/*.test.ts)@objectstack/spectests@objectstack/coretests@objectstack/clitestsspec,cli,core)check:scripts-typecheckandcheck:test-typecheckTwo suites failed on the first pass for an environmental reason and are recorded rather than quietly re-run:
core(1 file) andcli(76 files) could not resolve unbuilt workspace packages (@objectstack/metadata-core,@objectstack/lint,@objectstack/runtime, …). Both are green afterpnpm build; neither failure touched any file in this diff. Heavy runs held/tmp/os-heavy-verify.lock; no wait was observed.Dual-snapshot: not applicable — no package exports change. The gate agrees:
check:api-surfaceandcheck:export-originsare green with no regeneration.TestSuiteSchemawas already exported from@objectstack/spec/qa; this PR adds its firstparsecall site.Special-inspection items
GOVERNED+SPEC_ONLY_SCHEMASis load-bearing, not incidental. Droppingqa.jsonintopackages/spec/liveness/alone would produce a file nothing reads. Please confirm you agreeqabelongs on the spec-only route (it is a file surface an author writes, likequeryis a request surface a caller writes) rather than being registered as a metadata type — registering it would switch on Studio CRUD andsaveMetaItemoverlay acceptance for a type that has no business there.authorWarnomission (point 1 above) — the strongest candidate for a reviewer disagreeing. The alternative is registering aqacollection in the lint'sTYPE_COLLECTIONS, which would be false: there is nostack.qa.result.data as QA.TestSuitein the loader is a narrowing cast between zod'sinputandoutputtypes of the same schema (the exportedQA.TestSuiteisz.input<…>), not a re-introduction of the unchecked assertion — the value has been parsed by then. Flagging it because it looks like the thing this PR removes.containsassertion silently PASSES when the actual value is neither an array nor a string (runner.ts:171-177) — the one genuinely silent path in the assertion engine; and the liveness README's state table has no rows for the already-governedapi/capabilitytypes. The README heading now says 30 with a short note admitting the table carries 28, rather than back-filling two Notes cells from changes somebody else measured.Closes #6247
Generated by Claude Code