Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .changeset/qa-assertion-field-path-body-root.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
"@objectstack/spec": patch
---

docs(spec): fix the QA `field`/`capture` path convention — no `body.` prefix (#7365)

`TestAssertionSchema.field`'s `.describe()` text read `'Field path in the
result to check (e.g. "body.data.0.status")'`, and `TestStepSchema.capture`'s
sibling `.describe()` repeated the same `body.*` example. Neither convention
ever matched the runtime: `TestRunner.assert`/`runStep` resolve both paths
against `result` directly — the value `HttpTestAdapter.handleResponse`
returns, which is the parsed response body itself with no `body` wrapper (nor
does the platform's own response envelope, `data`/`meta`, ever nest under a
`body` key). A suite written to the documented convention resolved every path
to `undefined`.

Filed as #7365 (observation-class finding from #7256's blast radius): with
`equals`-class operators a `body.*` path already failed loudly, so an author
worked the real convention out by trial; with `contains`, `undefined` fell out
of the switch and the assertion silently passed, so a `body.*` `contains`
reported green forever. #7256 (PR #7348) turned that silent pass into a loud
failure, which is correct, but it meant an author following the schema's own
example now hits a error that never says the *documentation* is wrong.

Maintainer ruling, 2026-08-11 (issue comment 5248467805): "docs follow the
adapter" — fix the `describe()` text (and the regenerated reference) to the
real convention, root-relative, no `body.` wrapper. `body.*` never worked, so
there is no stored-suite compatibility to preserve. The acceptance face is
unchanged — `field` and `capture` both stay their original Zod types; only the
description text moves.

Both faces now read, in the file's existing one-sentence-plus-example style:

- `field`: `'Field path in the result to check, resolved against the parsed
response body root — no "body." prefix (e.g. "data.0.status")'`
- `capture`: `'Map result fields to context variables, paths resolved against
the response body root (e.g. { "newId": "data.id" })'`

`content/docs/references/qa/testing.mdx` is regenerated to match
(`pnpm --filter @objectstack/spec gen:docs`); `check:docs` reports all 231
generated files in sync.
4 changes: 2 additions & 2 deletions content/docs/references/qa/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ A test assertion that validates the result of a test action

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| **field** | `string` | ✅ | Field path in the result to check (e.g. "body.data.0.status") |
| **field** | `string` | ✅ | Field path in the result to check, resolved against the parsed response body root — no "body." prefix (e.g. "data.0.status") |
| **operator** | `Enum<'equals' \| 'not_equals' \| 'contains' \| 'not_contains' \| 'is_null' \| 'not_null' \| 'gt' \| 'gte' \| 'lt' \| 'lte' \| 'error'>` | ✅ | Comparison operator to use |
| **expectedValue** | `any` | ✅ | Expected value to compare against |

Expand Down Expand Up @@ -126,7 +126,7 @@ A single step in a test scenario, consisting of an action and optional assertion
| **description** | `string` | optional | Human-readable description of what this step tests |
| **action** | `{ type: Enum<'create_record' \| 'update_record' \| 'delete_record' \| 'read_record' \| … +4 more>; target: string; payload?: Record<string, any>; user?: string }` | ✅ | The action to execute in this step |
| **assertions** | `{ field: string; operator: Enum<'equals' \| 'not_equals' \| 'contains' \| 'not_contains' \| 'is_null' \| 'not_null' \| … +5 more>; expectedValue: any }[]` | optional | Assertions to validate after the action completes |
| **capture** | `Record<string, string>` | optional | Map result fields to context variables: `{ "newId": "body.id" }` |
| **capture** | `Record<string, string>` | optional | Map result fields to context variables, paths resolved against the response body root (e.g. `{ "newId": "data.id" }`) |


---
Expand Down
4 changes: 2 additions & 2 deletions packages/spec/src/qa/testing.zod.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ export const TestAssertionTypeSchema = lazySchema(() => z.enum([
]).describe('Comparison operator for test assertions'));

export const TestAssertionSchema = lazySchema(() => z.object({
field: z.string().describe('Field path in the result to check (e.g. "body.data.0.status")'),
field: z.string().describe('Field path in the result to check, resolved against the parsed response body root — no "body." prefix (e.g. "data.0.status")'),
operator: TestAssertionTypeSchema.describe('Comparison operator to use'),
expectedValue: z.unknown().describe('Expected value to compare against')
}).describe('A test assertion that validates the result of a test action'));
Expand All @@ -55,7 +55,7 @@ export const TestStepSchema = lazySchema(() => z.object({
action: TestActionSchema.describe('The action to execute in this step'),
assertions: z.array(TestAssertionSchema).optional().describe('Assertions to validate after the action completes'),
// Capture outputs to variables for subsequent steps
capture: z.record(z.string(), z.string()).optional().describe('Map result fields to context variables: { "newId": "body.id" }')
capture: z.record(z.string(), z.string()).optional().describe('Map result fields to context variables, paths resolved against the response body root (e.g. { "newId": "data.id" })')
}).describe('A single step in a test scenario, consisting of an action and optional assertions'));

export const TestScenarioSchema = lazySchema(() => z.object({
Expand Down
Loading