QA run · automation (FULL area) · a86db175 · 2026-08-11 · 5 PASS / 3 PARTIAL / 2 FAIL

[baozhoutao]

Full automation area run of the checklist-test skill — all 10 runnable items driven against a live showcase (2 opus subagents, isolated boot each). 1 item pre-blocked on fixtures (rollup-summary-filter). Labels: qa-run + bug.

Result: 5 PASS · 3 PARTIAL · 2 FAIL. Text-only per RUNNER.md. Both FAILs are checklist-vs-implementation contract disagreements with documented implementation-side rationale — neither is a regression. They need a maintainer decision, not a fix ticket.

Environment — framework a86db175 (PR #7304) · showcase · isolated file DB + port per batch · no outbound egress.

🟠 The two FAILs are contract decisions

1. flow-node-type-matrix — unknown node types are not refused at registration

POST /api/v1/automation with a node of type:'bogus_node' returns 200 and really registers the flow. It is not silent, which is the #1887 anti-goal: registration logs a WARN listing unknownTypes + knownTypes, and triggering answers success:false "No executor registered for node type bogus_node" with the run recorded failed. Implementation-side rationale is explicit — engine.ts registerFlow() / ADR-0018 §M1 / #4771: "membership is checked at that seam instead, and stays soft-fail — a flow authored against a currently-absent plugin must still register."

Worth an explicit ruling: the same endpoint hard-refuses an undeclared node config key (#4277, with a located message listing the declared keys) while accepting an unknown node type. One seam, two strictness levels.

Otherwise strong: 20 of 21 node types proven executed success from a 235-step sweep across ~100 runs; region tagging (loop-body/try/catch/parallel-branch + iteration + parentNodeId) all correct; every step names its nodeType (0 missing). Gap: end never appears as an executed step, only as a skipped branch target.

2. flow-error-handling — a failing try-region node produces no step at all

On a caught failure the steps are exactly [start, guarded_push(try_catch,success), record_failure(catch)] — nothing with regionKind:'try', nothing with status:'failure' — and the persistence layer agrees. Reproduced 2×. Deliberate per engine.ts runRegion() ("a failed attempt's partial steps are not surfaced") and try-catch-node.ts, which returns childSteps only from a successful region.

Operator consequence worth weighing: after a try_catch run you cannot tell what failed, how many attempts ran, or which node threw — only the catch's side effects.

Everything else in the item passes: $error interpolation into the record, the retry ladder (7.13s/7.16s matching 0+1+2+4s), unhandled failure terminating status=failed with both run-level and step-level errors, the designer Runs panel rendering region nesting, and the loud ERROR log for trigger-fired unheard failures. The "catch must not run when try succeeds" negative was proven with a purpose-built probe.

✅ PASS — 5 items

• durable-suspend-restart (5/5) — 6 paused runs byte-identical across a real cold restart on the same file DB; a PT1M wait timer elapsed on the new process with no manual resume and completed end-to-end (notification + receipt written); a nested subflow→approval pair both resumed from one /approve with the child's output bubbling to the parent.
• flow-toggle-kill-switch (4/4) — toggle OFF flips /_status enabled+bound to false (the trigger is genuinely unbound, not merely guarded) and a matching write produces no new run, asserted on bracketing reads.
• screen-flow-roundtrip (7/7) — and both negatives are the good kind: an empty-inputs resume is refused 400 naming the required field with the run left paused, and Cancel issues zero requests, leaves the run paused, and it remains resumable.
• flow-run-step-nesting (5/5) — the Studio panel renders a real indentation ladder (0/12/24px) with 1-based ITERATION headers, and the rendered tree reconciles exactly with the API step log.
• flow-runs-page-test-trigger (5/5) — Console: screen-flow Submit never calls the resume endpoint — every screen flow is un-completable from the UI #3528 refuted: a screen flow's paused envelope is handed to FlowRunner on the page, Submit resumes and the downstream write lands; dismissing instead leaves a working "Continue run" affordance, with the paused-row tally 0 before and after.

🟡 PARTIAL — 3 items

• trigger-type-matrix — all 6 trigger kinds fire and record their runtime kind; webhook HMAC intake 202 with payload interpolation, and both bad-signature and missing-signature refused 401 with no task created; anonymous /trigger denied 401 with no run row; schedule fired 12 consecutive runs exactly 60s apart. Two real gaps: trigger.recordId is never populated on record_change runs (so runs cannot be correlated back to the triggering record from the run log), and the persisted sys_automation_run history row carries no trigger block at all, so trigger kinds are lost across a process restart.
• time-relative-trigger — the sweep itself is solid (5 sweeps → exactly one run each, with the out-of-window and filter-excluded rows never firing, proven by the runs-per-sweep count rather than by inference; the both-windowing-modes build gate refuses at os lint and os compile with a located message and no artifact). Clause 0 blocked(fixture) — see below.
• connector-dispatch-matrix — MCP dispatch fully proven (run output upper=OBJECTSTACK); the unregistered-connector negative gives a named refusal, no silent no-op. Partial because the rest/declarative variants dispatch but capture nothing (run.output empty) — an authoring gap, not an engine one: those flows declare no isOutput variable while the MCP one does. Slack is blocked(environment).

Fixture / authoring issues to fix (they block clauses, or mislead)

1. Showcase REST connectors hard-wire baseUrl to http://127.0.0.1:3000 (StatusApi/StatusOpenApi providerConfig literals, not env-overridable), so self-ping flows fail fetch failed on any isolated instance — a port mismatch, not the sandbox egress block. Worked around with a throwaway TCP forwarder, giving clean before/after causal evidence. Needs an env-overridable baseUrl or a knownGaps entry.
2. Package-provided flows are read-only in Studio (no design mode, no node inspector for any node), so every clause needing a node panel on a shipped flow is unreachable. Affects time-relative-trigger c0 and connector-dispatch-matrix c3. A writable draft does open the inspector — verified, so this is a fixture limit, not a product defect.
3. A source comment overstates behaviour: TaskCompletedRestPingFlow / ShowcaseDeclarativeConnectorPingFlow claim "the call and its {status:'ok'} response are captured on the flow run" — they aren't, because neither declares an isOutput variable. Either add it (as the MCP flow has) or correct the comment and the checklist clause.
4. POST /api/v1/automation/<unknown>/toggle answers 500 INTERNAL_ERROR instead of 404 NOT_FOUND (P3 error-class mismatch).

[huangyiirene]

Extraction ledger — COMPLETE

Every item in this run now has a card. This record stays open as the aggregation node and carries tracking, so the triage seat skips it and it is not itself a dispatch candidate.

Run item	Finding	Card	Lane
flow-node-type-matrix (FAIL)	Unknown node types register with 200 while an undeclared node config key is hard-refused — one seam, two strictness levels	#7545	needs-user-decision
flow-error-handling (FAIL)	A failed try-region produces no step at all — after a caught failure an operator cannot tell what failed, how many attempts ran, or which node threw	#7546	needs-user-decision
trigger-type-matrix (PARTIAL)	trigger.recordId never populated on record_change runs; persisted sys_automation_run carries no trigger block, so trigger kinds are lost across restart	#7533	domain:services
connector-dispatch-matrix (PARTIAL)	rest/declarative variants dispatch but capture nothing — neither declares an isOutput variable	#7542	domain:services
time-relative-trigger (PARTIAL)	Clause 0 blocked(fixture) — package-provided flows are read-only in Studio	#7571	⚠️ no lane (see below)
Fixture 1	Showcase REST connectors hard-wire baseUrl to http://127.0.0.1:3000 — a port mismatch, not the sandbox egress block	#7538	domain:services
Fixture 3	TaskCompletedRestPingFlow / ShowcaseDeclarativeConnectorPingFlow comments claim the response is captured on the run; it is not	#7542	domain:services
Fixture 4	POST /api/v1/automation/:name/toggle answers 500 INTERNAL_ERROR instead of 404 NOT_FOUND	#7535	domain:cli

The 5 PASS items produced no cards, as expected.

The two FAILs are decisions, not bugs — and are filed as such

Both carry documented implementation-side rationale (engine.ts registerFlow() / ADR-0018 §M1 / #4771 for the first; engine.ts runRegion() and try-catch-node.ts for the second), and the run said so plainly: "Both FAILs are checklist-vs-implementation contract disagreements with documented implementation-side rationale — neither is a regression. They need a maintainer decision, not a fix ticket."

So #7545 and #7546 carry only needs-user-decision — deliberately no pm:queue, so they sit in the maintainer's inbox rather than any lane's dispatch queue. Each presents options with a recommendation, analysed along the three axes: real measured business need, long-term platform coherence, and whether the option makes it structurally harder for an AI agent to author this wrong.

Whichever way each is decided, the losing side implies a checklist edit rather than a code fix — worth remembering when they are answered, so the resolution does not stop at the ruling.

#7571 carries no lane on purpose

"Package-provided flows are read-only in Studio" has three candidate landing places — the QA checklist (domain:devx), the showcase fixtures, or objectui's Studio — and labelling it now would pre-judge the very ruling the card requests. It records both readings honestly: the run's classification as a fixture limit (a writable draft does open the inspector — verified control), and the product question that "you cannot inspect a shipped flow's nodes in Studio" is itself questionable, since inspection is a read.

Noting it here so a later sweep reads the absent lane as a deliberate hold rather than an oversight.

Stale-premise status

Checked against PRs merged since 2026-08-10; none of these appears already fixed. Worth stating because a sibling card in this batch (#7524, approvals run) was closed precisely because PR #7395 had fixed its defect hours after build a86db175 was cut.

---

Generated by Claude Code

[huangyiirene]

Closing — extraction is complete (see the ledger comment above; the two contract disagreements sit in the maintainer's decision inbox as #7545 / #7546, everything else is in lane queues). A closed run record stays fully readable; nothing actionable remains anchored here.

Provenance: maintainer instruction of 2026-08-11 (chat, verbatim): 「QA run 相关的 issue 如果任务已经分拆出去了，是不是应该直接关闭」 — same exit as the all-pass run records #7399 / #7326.

---

Generated by Claude Code