fix(objectql,runtime): stop platform stamps failing spec validation (#7561) (#7591)

`GET /api/v1/meta/diagnostics` reported 94 of 94 registry entries INVALID.
Both error shapes were self-inflicted — the platform reporting defects about
columns it wrote itself, on documents no author wrote or could fix — so the
endpoint carried no signal at all.

- `provisionSearchCompanion` stamped `index: true` on `__search`. Field-level
  `index` was removed from `FieldSchema` in the 16.x line (#2377, ADR-0049)
  and `FieldSchema` is a `strictObject`, so the key was rejected by name. Same
  mechanism as #6810 (`applySystemFields`/`indexed` on `organization_id`), same
  retired key, one field over. Stamp and the false "It IS `index`ed" docblock
  claim both removed.

  Unlike #6810 no `indexes[]` entry replaces it: the column's only reader is
  `{ __search: { $contains: term } }`, a leading-wildcard LIKE no B-tree can
  serve, and `IndexSchema` cannot express a trigram/GIN index. Behaviour-neutral
  either way — nothing read the flag.

- `DefaultDatasourcePlugin.registerVisibility` published the `default` row
  without `config`, which `DatasourceSchema` requires. Now stamped `{}` —
  deliberately empty rather than the host's real config, which carries
  connection credentials that would otherwise reach every metadata reader.
  Fixed at the producer, not by widening the spec.

Two pins land with the fix, since patching one key at a time is what turned
#6810 into this card: a class pin walking every platform-stamped field through
`FieldSchema` across all stamper branches, and a baseline pin asserting a
realistically-built registry sweeps clean, naming both error shapes. Both
reverse-verified red against the unfixed producers.

`search-companion.test.ts` asserted `col.index === true` — it pinned the defect
rather than the contract; it now pins the key's absence.

Fixes #7561


Claude-Session: https://claude.ai/code/session_019MfMR4enbnPgoH6efeKmDJ

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-help

.changeset/diagnostics-clean-baseline.md

@@ -0,0 +1,58 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/runtime": patch
+---
+
+fix(objectql,runtime): stop the platform's own stamps from failing spec validation — `/meta/diagnostics` reads clean again (#7561)
+
+`GET /api/v1/meta/diagnostics` reported **94 of 94** registry entries INVALID —
+every entry, `sys_*` and `showcase_*` alike. The endpoint reports entries that
+fail their registered Zod schema, so at a 94/94 baseline it carried **no
+signal**: a genuinely broken object was indistinguishable from a healthy one,
+and any gate or dashboard built on it read permanently red.
+
+Both error shapes behind the 94 were self-inflicted — the platform reporting
+defects about columns it wrote itself, on documents no author wrote or could
+fix.
+
+**`fields.__search: Unrecognized key 'index'`.** `provisionSearchCompanion`
+stamped `index: true` on the hidden `__search` companion column. Field-level
+`index` was removed from `FieldSchema` in the 16.x line (#2377, ADR-0049)
+because a field-level index flag built no index, and `FieldSchema` is a
+`strictObject`, so the key was rejected by name. The companion is provisioned
+before the document is stored and `/meta` re-parses the served body, so the
+stamp badged `_diagnostics: { valid: false }` onto every object the platform
+provisions a companion for. This is the #6810 mechanism one field over
+(`applySystemFields` stamping `indexed` on `organization_id`), and the same
+retired key. The stamp is gone, along with the docblock claim that the column
+"IS `index`ed".
+
+Unlike #6810 the index is **not** re-declared in the object's `indexes[]`, and
+that difference is measured rather than overlooked. #6810's predicate is
+`organization_id = ?` — equality, which a B-tree serves. This column's only
+reader is `buildSearchFilter`, which emits `{ __search: { $contains: term } }`
+— a leading-wildcard `LIKE '%term%'` no B-tree can answer — and `IndexSchema`
+spells nothing else (`name` / `fields` / `unique`; no trigram/GIN method).
+Declaring one would buy write amplification on every row for a read path that
+cannot use it. Search behaviour is unchanged either way: nothing read the flag.
+
+**`config: expected record, received undefined`.** The datasource-visibility
+registration in `DefaultDatasourcePlugin` published the `default` row without
+`config`, which `DatasourceSchema` requires. It is now stamped `{}` —
+deliberately empty, not the host's real config, which carries connection
+credentials that would otherwise land on `GET /api/v1/meta/datasources` for
+every metadata reader. No information is lost versus the omitted key; only the
+spelling changes to the one the contract accepts. Fixed at the producer rather
+than by widening the spec: a real datasource document genuinely needs its
+config, so relaxing the schema would trade one honest verdict for a permanently
+weaker one.
+
+**So it stops recurring.** Two pins land with the fix, because patching one key
+at a time is what turned #6810 into this card. A class pin walks every field
+the platform stamps — `applySystemFields` and `provisionSearchCompanion`,
+across every ownership / tenancy / `systemFields` branch — through
+`FieldSchema`, so the next retired-key stamp turns a suite red instead of
+poisoning diagnostics. A baseline pin asserts a realistically-built registry
+sweeps clean, naming both of this card's error shapes explicitly; the 94/94
+state survived undetected until a human read the endpoint by hand, because
+nothing asserted the baseline.

packages/objectql/src/diagnostics-clean-baseline.test.ts

@@ -0,0 +1,151 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#7561] THE BASELINE PIN — a healthy registry validates clean.
+ *
+ * ## Why this file exists
+ *
+ * `GET /api/v1/meta/diagnostics` reports every metadata entry that fails its
+ * registered Zod schema. That verdict is only worth reading if the baseline is
+ * ZERO: at 94 of 94 entries INVALID — where the platform's own
+ * `applySystemFields`/`provisionSearchCompanion` stamps supplied both error
+ * shapes — a genuinely broken object is indistinguishable from the baseline,
+ * and any gate or dashboard built on the endpoint reads permanently red.
+ *
+ * The 94/94 state was not caught by any suite. It survived until a human read
+ * the endpoint by hand during a QA run (#7514), which is the actual failure
+ * this file addresses: nothing asserted the baseline. So this pin walks a
+ * registry built the way the platform builds one — every stamper live, the
+ * tenancy branch that #6810 broke armed — and asserts the sweep finds NOTHING.
+ *
+ * Companion to `stamped-system-fields-spec-conformance.test.ts`: that one pins
+ * the PRODUCERS (whatever a stamper writes, `FieldSchema` accepts); this one
+ * pins the OBSERVABLE the card was filed against (the served documents sweep
+ * clean). A regression that slipped past the first — a bad key on a document
+ * assembled somewhere other than a stamper — still fails here.
+ */
+
+import { computeMetadataDiagnostics } from '@objectstack/metadata-protocol';
+import { describe, it, expect } from 'vitest';
+
+import { SchemaRegistry } from './registry.js';
+
+const PKG = 'showcase';
+
+/**
+ * A baseline shaped like the registry the card was filed against: `sys_*` and
+ * `showcase_*` alike, spanning the branches that decide which system columns
+ * get stamped, and every one carrying a title-eligible field so the `__search`
+ * companion is actually provisioned (an object with no companion could not
+ * reproduce #7561 and would pass vacuously).
+ */
+const OBJECTS: Array<Record<string, unknown>> = [
+  { name: 'showcase_account', label: 'Account', fields: { name: { type: 'text' }, revenue: { type: 'number' } } },
+  {
+    name: 'showcase_contact',
+    label: 'Contact',
+    ownership: 'user',
+    fields: { name: { type: 'text' }, email: { type: 'email' } },
+  },
+  {
+    name: 'showcase_order',
+    label: 'Order',
+    ownership: 'org',
+    fields: { name: { type: 'text' }, total: { type: 'currency' } },
+  },
+  { name: 'sys_thing', label: 'Thing', fields: { name: { type: 'text' } } },
+  {
+    name: 'showcase_note',
+    label: 'Note',
+    managedBy: 'platform',
+    fields: { name: { type: 'text' }, body: { type: 'textarea' } },
+  },
+];
+
+/**
+ * The `datasource` row `DefaultDatasourcePlugin.registerVisibility` publishes.
+ * Reproduced as a literal rather than imported because the pin is about the
+ * SHAPE that reaches the metadata list — importing `@objectstack/runtime` here
+ * would invert the package dependency (runtime depends on objectql).
+ * `default-datasource-plugin.ts` carries the matching `[#7561]` note.
+ */
+const DEFAULT_DATASOURCE_ROW = {
+  name: 'default',
+  label: 'Default',
+  driver: 'sqlite',
+  config: {},
+  origin: 'code',
+};
+
+function buildBaseline(multiTenant: boolean): SchemaRegistry {
+  // `searchCompanion: true` explicitly: the flag defaults off the environment
+  // (`OS_SEARCH_PINYIN_ENABLED`), and a pin that silently skipped the companion
+  // would be green on exactly the deployments #7561 was reported from.
+  const registry = new SchemaRegistry({ multiTenant, searchCompanion: true });
+  for (const def of OBJECTS) registry.registerObject(structuredClone(def) as any, PKG);
+  return registry;
+}
+
+describe('[#7561] the baseline registry sweeps clean through /meta/diagnostics', () => {
+  describe.each([true, false])('multiTenant: %s', (multiTenant) => {
+    it('every served object document is spec-valid', () => {
+      const registry = buildBaseline(multiTenant);
+      const served = registry.getAllObjects(PKG);
+      expect(served.length, 'no objects registered — pin would pass vacuously').toBe(OBJECTS.length);
+
+      // Report the SUBSTANCE — which entry, which path, which code — so a
+      // regression names itself instead of asserting a bare boolean.
+      const invalid: string[] = [];
+      for (const doc of served) {
+        const diag = computeMetadataDiagnostics('object', doc);
+        expect(diag, `object/${(doc as any).name}: no schema registered`).toBeDefined();
+        for (const err of diag!.errors ?? []) {
+          invalid.push(`object/${(doc as any).name} → ${err.path}: ${err.code}`);
+        }
+      }
+      expect(invalid).toEqual([]);
+    });
+
+    it('the platform stamps the companion, and it is one of the entries swept', () => {
+      // Guards the guard: the sweep above is only meaningful while `__search`
+      // is actually present on the documents it validates.
+      const registry = buildBaseline(multiTenant);
+      const withCompanion = registry
+        .getAllObjects(PKG)
+        .filter((o: any) => o.fields?.__search !== undefined);
+      expect(withCompanion.length).toBe(OBJECTS.length);
+    });
+  });
+
+  it('the default datasource row is spec-valid as registered', () => {
+    // The second of the card's two error shapes: `config: expected record,
+    // received undefined`, produced by the datasource-visibility registration
+    // omitting a key `DatasourceSchema` requires.
+    const diag = computeMetadataDiagnostics('datasource', DEFAULT_DATASOURCE_ROW);
+    expect(diag, 'datasource has no registered schema').toBeDefined();
+    expect(
+      (diag!.errors ?? []).map((e) => `${e.path}: ${e.code}`),
+    ).toEqual([]);
+  });
+
+  it("neither of the card's two error shapes appears anywhere in the sweep", () => {
+    // Named explicitly so a REINTRODUCTION of either fails on the shape the
+    // card reported, not on a generic count. These are the two strings a human
+    // read off the endpoint during #7514.
+    const all: Array<{ entry: string; path: string; message: string }> = [];
+    for (const multiTenant of [true, false]) {
+      for (const doc of buildBaseline(multiTenant).getAllObjects(PKG)) {
+        for (const err of computeMetadataDiagnostics('object', doc)?.errors ?? []) {
+          all.push({ entry: `object/${(doc as any).name}`, path: err.path ?? '', message: err.message ?? '' });
+        }
+      }
+    }
+    for (const err of computeMetadataDiagnostics('datasource', DEFAULT_DATASOURCE_ROW)?.errors ?? []) {
+      all.push({ entry: 'datasource/default', path: err.path ?? '', message: err.message ?? '' });
+    }
+
+    expect(all.filter((e) => e.path === 'fields.__search')).toEqual([]);
+    expect(all.filter((e) => /Unrecognized key/i.test(e.message) && /`index`/.test(e.message))).toEqual([]);
+    expect(all.filter((e) => e.path === 'config' && /expected record/i.test(e.message))).toEqual([]);
+  });
+});

packages/objectql/src/search-companion.test.ts

@@ -85,7 +85,18 @@ describe('provisionSearchCompanion', () => {
     expect(col.readonly).toBe(true);
     expect(col.system).toBe(true);
     expect(col.searchable).toBe(false);
-    expect(col.index).toBe(true);
+    // [#7561] Was `expect(col.index).toBe(true)` — this line pinned the DEFECT
+    // rather than the contract. `index` is not a `FieldSchema` key (removed in
+    // the 16.x line, #2377 / ADR-0049, because a field-level index flag built no
+    // index), and `FieldSchema` is a `strictObject`, so stamping it badged every
+    // object carrying a companion `_diagnostics: { valid: false }` and drove
+    // `GET /api/v1/meta/diagnostics` to 94/94 INVALID. The key is now absent,
+    // and no index is declared in its place: this column's only reader is a
+    // `$contains`, which no B-tree serves. See the docblock in
+    // `search-companion.ts` and the pins in
+    // `stamped-system-fields-spec-conformance.test.ts`.
+    expect(col.index).toBeUndefined();
+    expect(Object.keys(col)).not.toContain('index');
   });
 
   it('is idempotent and skips ineligible / opted-out objects unchanged', () => {

packages/objectql/src/search-companion.ts

@@ -128,8 +128,31 @@ export function resolveSearchCompanionSources(schema: CompanionObjectMeta | unde
  * it never appears in auto-generated views/forms, is excluded from the
  * `$search` auto-default (hidden fields are skipped) and from `$searchFields`
  * overrides (the override intersects with the allowed set), and non-system
- * callers cannot forge it on update (#2948 readonly write guard). It IS
- * `index`ed — every search touches it.
+ * callers cannot forge it on update (#2948 readonly write guard).
+ *
+ * [#7561] It carries NO index — and the stamp that claimed otherwise is gone.
+ * This block used to append `index: true` and the paragraph above used to read
+ * "It IS `index`ed — every search touches it". Both were false, in the #6810
+ * shape one field over (`applySystemFields` stamping `indexed` on
+ * `organization_id`): `index` was removed from `FieldSchema` in the 16.x line
+ * (#2377, ADR-0049) because a field-level index flag built no index, and
+ * `FieldSchema` is a `strictObject`, so the key was rejected BY NAME. The
+ * companion is provisioned BEFORE the document is stored and `/meta` re-parses
+ * the served body, so the stamp put `_diagnostics: { valid: false, errors:
+ * [{ path: 'fields.__search', code: 'unrecognized_keys' }] }` on every object
+ * the platform provisions a companion for — a defect the platform reported
+ * about its own column, on a document no author wrote or could fix.
+ *
+ * Unlike #6810 the index is NOT re-declared in the object's `indexes[]`, and
+ * that is a measured difference rather than an omission. #6810's predicate is
+ * `organization_id = ?` — equality, which a B-tree serves. This column's ONLY
+ * reader is `buildSearchFilter`, which emits `{ __search: { $contains: term } }`
+ * (`search-filter.ts`) — a leading-wildcard `LIKE '%term%'` that no B-tree can
+ * answer, and `IndexSchema` spells nothing else (`name` / `fields` / `unique`;
+ * no trigram/GIN method). Declaring one would buy write amplification on every
+ * row for a read path that cannot use it. If the companion ever warrants a
+ * real substring index, it needs an `IndexSchema` that can express one — a
+ * separate change, not a dead index declared here.
  *
  * Objects that opt out of search entirely (`searchable: false`, ADR-0061 D2)
  * are skipped: a companion no query will ever read is dead weight.
@@ -152,7 +175,6 @@ export function provisionSearchCompanion<T extends CompanionObjectMeta>(schema:
         readonly: true,
         system: true,
         searchable: false,
-        index: true,
         description:
           `Search-normalized forms of the display/name field (normalizers: ${SEARCH_COMPANION_NORMALIZERS.join(', ')}) — ` +
           'e.g. full pinyin + initials for CJK names. Maintained by plugin-pinyin-search; never hand-edited. See #2486.',