docs(qa): map the qa capability to a real checklist item, not a waiver (#7347) - #7786
Merged
Merged
Conversation
…ver (#7347) `check:platform-checklist` has been red on `main` since the `qa` liveness ledger was seeded (#6247 / PR #7255): the coverage ratchet derives its universe of capabilities from `packages/spec/liveness/*.json`, and `coverage.json` carried no `qa` key. Two dispatched devs reproduced the same single problem in one day and each paid the read-and-explain tax, which is what promoted it. The exit is the one the card argued for: `os test` is a shipped, documented CLI command, so a waiver would have had to claim the capability is out of the checklist's scope. Instead this authors the mapping: - `cli.qa-suite-execution` (surface: cli) — the `os test` load/execute/verdict contract. Every clause was measured on a booted showcase app before it was written: the green path and its exit 0, `capture` + `{{var}}` threading state between steps, the #6247 load-time TestSuiteSchema refusal (named file, offending paths, siblings still run), the #7256 unevaluable-`contains` failure, teardown running after a failed step, the 8-member action-type sweep and the #7363 lazy glob. `enumSource` pins TestActionTypeSchema at 8 members so a new action type makes the matrix stale instead of drifting. - `examples/app-showcase/qa/platform-smoke.test.json` — the repo's first real Quality Protocol suite, and the fixture that item drives. It sits where the command's DEFAULT glob finds it with no argument, and its teardown leaves the fixture exactly as it found it, so the run repeats. - `coverage.json` gains the `qa` row; the validator is now green (28 kinds mapped, 2 waived). No `automated` entry: the three existing unit pins cover the load boundary, the glob resolver and one assertion operator, and none of them proves a suite reaches a real server — which is the whole point of the item. Recorded, deliberately NOT fixed here (out of scope, reported on #7347): five of the eight action types answer HTTP 404 against a stock server because `HttpTestAdapter` builds `${baseUrl}/api/data/:object` while the server serves `/api/v1/data/:object`; and a zero-match glob exits 0, so a run that loaded no suite looks green. Both are carried as `negative` clauses so a run records them instead of rediscovering them. Fixes #7347 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NzRhAgKEGiKCWe9tN5Ww3H
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
huangyiirene
added
the
skip-changeset
Claude
huangyiirene
marked this pull request as ready for review
August 11, 2026 16:07
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31510771208 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31513733632 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31514484754 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31515155984 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31516020568 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Aug 11, 2026
Contributor
⛔ merge queue 构建失败 — 先分诊，再决定要不要重排队列构建 31516761124 红了。队列跑的是全量套件（PR 侧 CI 只跑 affected 子集）， 失败的 job（日志抽取，best effort）：
历史信号：
分诊清单：
Generated by Claude Code · merge-queue-triage workflow (#4859) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7347
Premise, re-verified first
On a clean
origin/main@7a8476f, before touching anything:premise_still_valid: true— the red reproduces on a pristine tree, unchanged since the card was filed.After this PR:
The direction, per the 2026-08-11T14:08Z triage ruling
A waiver was not taken.
os testis a shipped, documented CLI command, so a waiver would have had to argue the capability is out of the checklist's scope — which is exactly what the card said looks wrong. No fork to report: the authored-item route worked.What landed
docs/qa/platform-checklist/areas/cli.json— new itemcli.qa-suite-execution(surface: cli, P1,since: v17).Area choice, per README's sharding logic: the
qadomain's user-facing surface is the shippedos testcommand, andcli.jsonis the area that already owns per-command CLI contracts (dev,build,migrate,verify, scaffold, flag/command error UX).cli.flag-command-error-uxalready liststestamong its variants for error UX only; this item owns the command's own load/execute/verdict contract. The area title was extended to nametest. No new area file — a one-command area would have been thinner than the shard it belongs in.Eight acceptance clauses, each measured on a booted showcase app before it was written (not derived from reading the source):
SUCCESS: All 2 scenarios passed.,$? = 0capture+{{var}}thread state between stepsAssertion failed: data.status expected degraded, got ok,$? = 1containsFAILScannot be evaluated by 'contains' … got undefined+ theis_nullhintTestSuiteSchemarefusal✗ scenarios.0.steps: …, the expected shape; the two sibling suites still ran; counted as a failed suiteTestActionTypeSchemamembers get their own verdictos testfound the suite on the default glob;qa/**/*.test.jsonreturned in ~3senumSourcepinsTestActionTypeSchemaat 8 members, so a ninth action type makes the variants matrix stale at the next check instead of drifting silently.No
automatedentry — deliberately. The three existing unit pins (qa-suite-schema-load.test.ts,resolve-glob-lazy-walk.test.ts,runner.test.ts) cover the load boundary, the glob resolver and one operator; none proves a suite reaches a real server. They are cited insourceso a run knows what is already covered, and anegativeclause says executing them is not a run of this item.examples/app-showcase/qa/platform-smoke.test.json— the repo's first real Quality Protocol suite. Two scenarios: the unauthenticated health probe (so a failing run can tell "wrong--url" from "broken assertions"), and a create → capture id → read-back-by-{{id}}→ list roundtrip againstshowcase_category(public_read_write, one required field). It sits where the command's defaultqa/*.test.jsonglob finds it with no argument, and its teardown deletes the record it made — verified by listing the object before and after, and by running it twice.The deliberately broken suites (misspelled
steps, false assertion,containsover a missing path) are staged per run in a scratch dir, never committed, so the repo's own suite stays green.docs/qa/platform-checklist/coverage.json— theqarow, mapped to that item, inserted alphabetically.Out-of-scope findings — recorded, not fixed
Both are carried as
negativeclauses on the item so a run records them rather than rediscovering them, and neither is patched here.HttpTestAdapterbuilds${baseUrl}/api/data/:object; the server serves{basePath}/data/:objectwith basePath/api/v1. Measured verbatim —create_record,read_record,update_record,delete_record,query_recordsall answerHTTP Error 404: {"error":"Not found"};update_recordadditionally issues PUT where the route is PATCH. Onlyapi_callandwaitexecute (run_scripthas no adapter branch and throws loudly, which the liveness ledger already records). This is a product fix inpackages/core, not a checklist edit.os test 'qa/nothing-matches-*.test.json'printsNo test files found matching: …and returns 0 — a green exit from a run that loaded no suite. Whether that is a defect or a deliberate posture is a product call; the item's evidence rule works around it by requiring a run to quote theFound N test suites.line with N > 0.Also noted, not touched:
docs/qa/platform-checklist/README.md's closing paragraph still saysqa"currently has no runtime consumer …qahas no liveness ledger entry" — stale since #6247 seeded the ledger and enforced the parse. README is held by a parallel unit this wave, so it is left alone.Changeset
None. Docs/QA ledger + an example-app fixture (
examples/*isprivate: true, nothing published changes).skip-changesethandling is the PM's per the dispatch.Generated by Claude Code