fix(metadata-protocol,spec): a stopped bulk write reports every record — NOT_ATTEMPTED tail and reconciling counters (#7539) (#7581)

* fix(metadata-protocol,spec): report NOT_ATTEMPTED for a stopped bulk write's tail, and make the counters reconcile (#7539)

A non-atomic `/batch` that stopped at the first failure answered with a
truncated `results` array and counters that did not add up: two results for
three records, no entry for the un-attempted record, and
`succeeded + failed != total`. The skipped record was invisible twice over —
no `results[]` entry and counted in neither bucket — so the arithmetic
mismatch was its only trace.

`buildBatchDataResponse` read `total` from the request while `results`,
`succeeded` and `failed` came from a loop that had stopped early.
`buildUpdateManyResponse` and `buildDeleteManyResponse` under-reported the
same way. All three now share one reconciler that pads the outcome out to the
request length with `NOT_ATTEMPTED` rows — the registered ADR-0112 code the
atomic arm has emitted since #4793 — and returns `failed` as the count of
every non-success row, so `succeeded + failed === total === results.length`
on both arms.

The stop itself is unchanged, per `BatchOptionsSchema.continueOnError` ("If
true (and atomic=false), continue processing remaining records after errors")
and ADR-0119 D4, whose test plan holds non-atomic batches to "behave exactly
as before". This is a reporting fix; `continueOnError` remains the flag that
buys continuation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016gd2bypaK4KYs78q8RP38G

* docs(api): describe the stopped-batch NOT_ATTEMPTED tail on the non-atomic arm (#7539)

`data-api.mdx` described `atomic: false` as "sequential best-effort, stopping
at the first failure" without saying what the response contains for the
records it never reached — the shape the fix makes explicit. `batch.mdx` is
regenerated from the two `.describe()` strings this change touches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016gd2bypaK4KYs78q8RP38G

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: claude[bot]

.changeset/batch-not-attempted-tail-reporting.md

@@ -0,0 +1,58 @@
+---
+"@objectstack/spec": patch
+"@objectstack/metadata-protocol": patch
+---
+
+fix(metadata-protocol,spec): a bulk write that STOPS now reports every record — `NOT_ATTEMPTED` rows instead of a truncated `results` array, and counters that reconcile (#7539)
+
+`POST /data/:object/batch` with no `options` (so `atomic` defaults `false`,
+ADR-0119 D4) and three records — valid, failing, valid — answered:
+
+```
+200 { "total": 3, "succeeded": 1, "failed": 1,
+      "results": [ { idx 0: ok }, { idx 1: VALIDATION_FAILED } ] }
+```
+
+Two results for three records, no entry for idx 2, and `succeeded + failed` (2)
+`!= total` (3). The un-attempted record was invisible **twice over**: it
+produced no `results[]` entry and was counted in neither bucket, so the only
+trace of it was an arithmetic mismatch a client had to notice and interpret.
+
+`buildBatchDataResponse` read `total` from the REQUEST (`records.length`) while
+`results` / `succeeded` / `failed` came from a loop that had stopped early. Its
+two siblings under-reported identically — the same defect on `updateManyData`
+and `deleteManyData`, whose per-object bulk counters lost the tail whenever a
+row failed without `continueOnError`. All three now go through one shared
+reconciler rather than a fourth copy of the same arithmetic.
+
+**What changed is the REPORT, not the semantics.** Every record now gets a row
+saying what happened to it: records after the failure carry
+`errors[0].code === 'NOT_ATTEMPTED'` — the same registered ADR-0112 code the
+atomic arm has emitted since #4793, because "never ran" means the same thing to
+a client whether the batch stopped to roll back or stopped because it was told
+to. The message names the causal row index and `continueOnError`, since on this
+arm the caller's next action is a flag rather than a fixed row. `results` now
+always covers all `total` records, and `succeeded` / `failed` partition it, so
+`succeeded + failed === total === results.length` on both arms.
+
+**The stop itself is unchanged, deliberately.** Without `continueOnError` the
+first failure still ends the run, records written before it stay written
+(nothing is rolled back on this arm), and the tail is still not attempted.
+That is the declared contract, not an accident:
+`BatchOptionsSchema.continueOnError` reads *"If true (and atomic=false),
+continue processing remaining records after errors"*, ADR-0119 D4 scopes the
+flag to exactly `atomic=false`, and D4's test plan holds non-atomic batches to
+"behave exactly as before". If `atomic: false` alone continued past a failure,
+`continueOnError` would be inert. Callers who want every valid row to land
+should send `continueOnError: true` — unchanged, and now the only difference
+between the two is whether the tail is attempted, not whether it is reported.
+
+**Upgrade note.** A non-atomic batch that stops now returns more `results` rows
+and a larger `failed` count than before, for the same request and the same
+writes. `failed` counts every row that is not a success — matching the atomic
+rollback response, which has always counted never-reached rows this way. A
+client that summed `succeeded + failed` and compared it to `total` to detect
+truncation no longer needs to; one that treated `failed` as "rows the server
+tried and could not write" should branch on `errors[0].code` instead, where
+`NOT_ATTEMPTED` distinguishes "skipped" from "attempted and failed". No schema
+field was added or removed.

content/docs/api/data-api.mdx

@@ -241,7 +241,7 @@ Execute a batch operation (create / update / upsert / delete) on multiple record
 
 **Response**: `BatchUpdateResponse` with `succeeded`, `failed`, `total`, and a per-record `results` array. Each entry in `results` has `id`, `success`, `index` (the row's position in the request array), an optional `errors` array (`ApiError[]` — read `errors[0].message`, branch on `errors[0].code`), and optional `data` (the full record, present when `returnRecords` is `true`).
 
-`options.atomic` defaults to `false` (sequential best-effort, stopping at the first failure). Set it to `true` and the whole batch runs inside one transaction: the first failure rolls back every prior write, and the response reports `succeeded: 0` — each row's `errors[0].code` says what happened: `ROLLED_BACK` (written, then undone), the causal row's own error code, or `NOT_ATTEMPTED` (never reached). A deployment whose driver cannot roll back rejects an atomic request with `501 NOT_IMPLEMENTED` instead of running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` first. `atomic` takes precedence over `continueOnError`.
+`options.atomic` defaults to `false`: sequential best-effort that stops at the first failure. Records written before the failure stay written — nothing is rolled back on this arm — and every record after it is reported with `errors[0].code` `NOT_ATTEMPTED` rather than omitted, so `results` always covers all `total` records and `succeeded + failed === total` (#7539). Send `continueOnError: true` to process the remaining records instead of stopping. Set `atomic` to `true` and the whole batch runs inside one transaction: the first failure rolls back every prior write, and the response reports `succeeded: 0` — each row's `errors[0].code` says what happened: `ROLLED_BACK` (written, then undone), the causal row's own error code, or `NOT_ATTEMPTED` (never reached). A deployment whose driver cannot roll back rejects an atomic request with `501 NOT_IMPLEMENTED` instead of running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` first. `atomic` takes precedence over `continueOnError`.
 
 ### `POST /data/:object/createMany`
 
@@ -284,7 +284,9 @@ selects rows, so no body key can widen the delete into a filter.
 deleted one at a time by primary key, so each honours `deleteBehavior`
 (`cascade` / `set_null` / `restrict`) on relations pointing at it. The run stops
 at the first failure; `continueOnError: true` processes the remaining ids and
-reports the failures instead.
+reports the failures instead. Either way every id gets a `results` entry — the
+ids a stopped run never reached carry `errors[0].code` `NOT_ATTEMPTED`, so the
+counters reconcile against `total` (#7539).
 
 `options.atomic: true` is honoured here the same way as on `/batch` (#4620): the
 whole id list runs inside one transaction, the first failure rolls back every

content/docs/references/api/batch.mdx

@@ -55,7 +55,7 @@ const result = BatchConfigSchema.parse(data);
 | :--- | :--- | :--- | :--- |
 | **id** | `string` | optional | Record ID if operation succeeded |
 | **success** | `boolean` | ✅ | Whether this record was processed successfully |
-| **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| … +259 more>; message: string; category?: string; httpStatus?: integer; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error (#4793). |
+| **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| … +259 more>; message: string; category?: string; httpStatus?: integer; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error (#4793). A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back (#7539). |
 | **data** | `Record<string, any>` | optional | Full record data (if returnRecords=true) |
 | **index** | `number` | optional | Index of the record in the request array |
 | **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key'> }[]` | optional | Write-observability (#3407/#3431/#3455): caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` (#2948) / TRUE `readonlyWhen` (#3042) on update, or the #3043 create-ingress strip. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. |
@@ -83,7 +83,7 @@ const result = BatchConfigSchema.parse(data);
 | :--- | :--- | :--- | :--- |
 | **atomic** | `boolean` | ✅ | Opt-in all-or-nothing. When explicitly true the whole batch runs inside ONE engine transaction: the first failure rolls back every prior write, and the response reports zero successes — each row carries `errors[0].code` ROLLED_BACK (written, then undone), the causal row its own error, and rows never reached NOT_ATTEMPTED. A runtime that cannot roll back REFUSES the request (501 NOT_IMPLEMENTED) rather than silently degrading to best-effort — probe `capabilities.transactionalBatch` on /discovery first. Takes precedence over continueOnError. Default false: sequential best-effort. |
 | **returnRecords** | `boolean` | ✅ | If true, return full record data in response |
-| **continueOnError** | `boolean` | ✅ | If true (and atomic=false), continue processing remaining records after errors |
+| **continueOnError** | `boolean` | ✅ | If true (and atomic=false), continue processing remaining records after errors. Default false: the first failure ENDS the run — records before it stay written (nothing is rolled back on this arm), and every record after it is reported `errors[0].code` NOT_ATTEMPTED rather than omitted, so `results` always covers all `total` records and `succeeded + failed === total` (#7539). |
 | **validateOnly** | `never` | optional | [REMOVED] `options.validateOnly` was removed from BatchOptions in @objectstack/spec (#4052). It was never implemented: the batch surfaces persisted regardless, so a "dry-run" would have silently executed. There is no dry-run today — drop the key. If you need to preview a batch without writing, open an issue so it can be designed (no-commit cascade / constraint semantics) and reintroduced as a flag that actually holds. |
 
 

packages/metadata-protocol/src/protocol.batch-atomic.test.ts

@@ -255,10 +255,19 @@ describe('batchData non-atomic — unchanged (ADR-0119 D4 regression net)', () =
 
         expect(t.engine.transaction).not.toHaveBeenCalled();
         expect(res.succeeded).toBe(1);
-        expect(res.failed).toBe(1);
         expect(res.results[0].success).toBe(true);   // committed, and honestly reported
         expect(res.results[1].success).toBe(false);
-        expect(res.results).toHaveLength(2);          // stops without continueOnError
+        // Still stops without `continueOnError` — two inserts, never three.
+        expect(t.insert).toHaveBeenCalledTimes(2);
+        // [#7539] But the STOP is now reported rather than inferred from a
+        // counter mismatch. This block used to assert `failed: 1` and
+        // `results.length === 2` against `total: 3` — the truncated `results`
+        // array and the `succeeded + failed != total` arithmetic that were the
+        // card's entire symptom.
+        expect(res.failed).toBe(2);
+        expect(res.results).toHaveLength(3);
+        expect(res.succeeded + res.failed).toBe(res.total);
+        expect(res.results[2].errors[0].code).toBe('NOT_ATTEMPTED');
     });
 
     it('atomic: false is best-effort, not a refusal, even on a non-transactional engine', async () => {