Skip to content

Commit bf972e7

Browse files
os-zhuangos-zhuang
os-zhuang
and
os-zhuang
authored
docs(search): make the mirror field the one prescription for related-title search (#6670)
`$search` scans the queried object's own columns; a dotted path (`project_id.name`) in `searchableFields` / `$searchFields` is refused by the #4254 ingress gate and by `validate-searchable-fields`. Both errors already pointed at "mirror the related title onto a local field", but nothing in the authoring corpus said it, so every project invented its own convention. Writes ONE prescription, told the same way in every place authors and AI look: to search by a related record's title, mirror that title into a STORED text field on this object and declare that field searchable. - skills/objectstack-data — new `## Search Fields (searchableFields)` section with the worked `task` → `project_name` example, the maintenance hooks, and both error messages quoted verbatim so either is greppable back to the fix. - skills/objectstack-query — mirror-field subsection under Full-Text Search (query-side shape + expansion), plus a row in the cross-object tool table. - content/docs/data-modeling/schema-design — canonical long form. - content/docs/data-modeling/queries, protocol/objectql/query-syntax, api/data-api, ui/views — short form pointing at the canonical section. Records one correctness point the existing hints get loose: the mirror must be a STORED field. A `formula` field is virtual (no driver materializes a column), so a `$contains` predicate against one has nothing to scan, and CEL only reads `record.<own field>` so it cannot fetch the related title anyway — yet `searchableFields` admits any declared field, so the mistake clears both lint and the ingress gate and then silently never matches. Cross-object search paths stay rejected by design; routes 2/3 are not written in as roadmap. Fixes #4840 Co-authored-by: os-zhuang <steve@objectstack.ai>
1 parent bb7cb41 commit bf972e7

7 files changed

Lines changed: 294 additions & 1 deletion

File tree

‎content/docs/api/data-api.mdx‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,7 @@ corrupts something the earlier axes do not:
128128
| `?search=alpha&searchFields=title` | scans only `title` |
129129
| `?search=alpha&searchFields=no_such_field` | `400 INVALID_FIELD` |
130130
| `?search=alpha&searchFields=amount` | `400 INVALID_FIELD` — real field, but not searchable |
131+
| `?search=alpha&searchFields=project_id.name` | `400 INVALID_FIELD` — search scans this object's own columns; mirror the related title instead (see below) |
131132
| `groupBy: ["status"]` | one bucket per status value |
132133
| `groupBy: ["no_such_field"]` | `400 INVALID_FIELD` |
133134
| `aggregations: [{function:"sum", field:"amount", alias:"total"}]` | the real total |
@@ -143,6 +144,18 @@ corrupts something the earlier axes do not:
143144
`searchableFields`), and a `searchableFields` entry that names no field (a
144145
stale declaration — the bug is on the object, and clients that echo the
145146
declaration verbatim are told so).
147+
148+
A **dotted path** (`project_id.name`) is the typo case with its own hint:
149+
`search` scans this object's own columns, so a related record's column can
150+
never be a search target, and the search axis does not resolve traversal the
151+
way `$select` / `$orderby` / `$filter` do. To search by a related record's
152+
title, **mirror** that title into a stored field on this object and declare
153+
*that* field searchable — a task list searched by project name carries a
154+
`project_name` text column on `task`, maintained on write and listed in
155+
`task.searchableFields`. It must be a stored field: a `formula` field is
156+
virtual, so no driver has a column for `$contains` to scan. Cross-object
157+
search paths are rejected by design, not pending — see
158+
[Schema Design → Searching by a related record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value).
146159
- **`groupBy`** — an unknown column projected `null` for every row, so all
147160
rows fell into **one bucket** whose count is the true row count:
148161
structurally perfect, indistinguishable from a column that really holds a

‎content/docs/data-modeling/queries.mdx‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -513,6 +513,30 @@ that exists to narrow a search, silently widening it. Internal callers reaching
513513
`engine.find()` directly are unaffected.
514514
</Callout>
515515

516+
### Searching by a related record's title — mirror the value
517+
518+
`search` scans **the queried object's own columns**. A dotted path
519+
(`project_id.name`) is not a search target — unlike `fields` / `sort` / `filters`,
520+
the search axis does not resolve traversal, and a dotted entry is refused, not
521+
silently dropped:
522+
523+
```text
524+
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
525+
columns 'search' scans, so a name the object does not declare cannot narrow
526+
anything — and the engine used to drop it and scan the default columns instead,
527+
answering a NARROWER search with a WIDER one. 'search' scans this object's own
528+
columns; a related record's column cannot be a search target.
529+
```
530+
531+
The answer is a **mirror field**: copy the related record's title into a stored
532+
field on this object and declare *that* field searchable. A task list searched by
533+
project name gets a `project_name` text column on `task`, maintained on write and
534+
listed in `task.searchableFields`. It has to be a **stored** field — a `formula`
535+
field is virtual, so no driver has a column for `$contains` to scan. Cross-object
536+
search paths are rejected by design, not pending. Full recipe (the hooks that keep
537+
the mirror fresh, and the lint wording) in [Schema Design → Searching by a related
538+
record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value).
539+
516540
### Pinyin recall (Chinese deployments)
517541

518542
When pinyin search is enabled (`OS_SEARCH_PINYIN_ENABLED` — auto-on when the stack's

‎content/docs/data-modeling/schema-design.mdx‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,95 @@ Queries then pass a top-level `$search` parameter to match across `searchableFie
103103
`$searchFields`. When `searchableFields` is unset, search falls back to the
104104
name/title field plus short-text fields.
105105

106+
#### Searching by a related record's title — mirror the value
107+
108+
`$search` scans **the queried object's own columns**. A dotted path such as
109+
`project_id.name` is not a search target: unlike `$select` / `$orderby` /
110+
`$filter`, the search axis does not resolve traversal, and a dotted entry is
111+
refused rather than silently dropped (#4254). That refusal is deliberate, not a
112+
missing feature — cross-object search paths are rejected by design.
113+
114+
The declarative answer is a **mirror field**: copy the related record's title
115+
into a stored field on *this* object, and make that field the search target.
116+
To let users search a task list by project name:
117+
118+
```typescript
119+
// `project_name` is a stored, denormalized copy of the parent's title.
120+
{
121+
name: 'task',
122+
enable: { searchable: true },
123+
fields: {
124+
name: { type: 'text', required: true },
125+
project_id: { type: 'lookup', reference: 'project' },
126+
project_name: { type: 'text', label: 'Project Name' }, // ← the mirror
127+
},
128+
searchableFields: ['name', 'project_name'],
129+
}
130+
```
131+
132+
`?search=apollo` now expands to `name $contains 'apollo' OR project_name
133+
$contains 'apollo'` — one single-table scan, on every driver, with no traversal.
134+
If the object declares no `searchableFields` at all, a `text` mirror is picked up
135+
by the auto-default anyway; declare the set explicitly when you want to pin it.
136+
137+
<Callout type="warn">
138+
**The mirror must be a stored field — a `formula` field does not work.** A
139+
`formula` field is *virtual*: no driver materializes a column for it, so a
140+
`$contains` predicate against one has nothing to scan (the SQL driver would emit
141+
a `WHERE` over a column that does not exist). A CEL formula also only reads this
142+
record's own fields (`record.<field>`), so it cannot fetch the related title in
143+
the first place. Nothing catches the mistake for you — `searchableFields` admits
144+
any field the object declares, so a formula entry passes both lint and the
145+
ingress gate and then just never matches.
146+
</Callout>
147+
148+
**Keeping the mirror fresh.** A mirror is denormalized data, only as current as
149+
whatever maintains it. Two write paths have to be covered:
150+
151+
| When | What maintains the mirror |
152+
|:-----|:--------------------------|
153+
| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id` and stamp `project_name` |
154+
| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks |
155+
156+
Rows written by a path that bypasses hooks (bulk import, direct SQL) need a
157+
one-off backfill. See [Hooks](/docs/automation/hooks) for the hook shapes.
158+
159+
**The errors you get if you try the dotted path.** Both the lint and the runtime
160+
send you to the same fix, so either message is greppable back to this section.
161+
162+
`os validate` reports `searchable-field-unknown`:
163+
164+
```text
165+
searchableFields entry "project_id.name" is not a field on object "task". The
166+
declaration is stale: searching it can never match, and the engine silently
167+
drops it — leaving a narrower search than declared, or the auto-default set once
168+
every entry is dropped.
169+
170+
hint: 'search' scans this object's own columns, so a related record's column
171+
cannot be a search target — expand the relation and search the related object,
172+
or copy the value onto a formula field here. Clients echo this declaration
173+
verbatim as the '$searchFields' override, so a stale entry becomes a 400
174+
INVALID_FIELD on list search (#4254), not just a quietly narrowed one.
175+
```
176+
177+
(That hint's "text/formula" family of wording is loose — only the **stored**
178+
half works; see the callout above.)
179+
180+
A request that sends the dotted path is `400 INVALID_FIELD`:
181+
182+
```text
183+
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
184+
columns 'search' scans, so a name the object does not declare cannot narrow
185+
anything — and the engine used to drop it and scan the default columns instead,
186+
answering a NARROWER search with a WIDER one. 'search' scans this object's own
187+
columns; a related record's column cannot be a search target.
188+
```
189+
190+
If the dotted path is in the object's own `searchableFields` (so clients echo it
191+
back verbatim), the same 400 arrives under its stale-declaration wording
192+
instead: `Field 'project_id.name' on object 'task' is declared in
193+
'searchableFields' but does not exist.`
194+
106195
---
107196

108197
## Field Types & Configuration

‎content/docs/protocol/objectql/query-syntax.mdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -887,6 +887,32 @@ a user typing `acme` does not find `ACME Corp`. Only `select` / `status` option
887887
`[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the
888888
expansion ignores them.
889889

890+
#### Searching by a related record's title — mirror the value
891+
892+
Search targets are **this object's own columns**. A dotted path is not one of
893+
them: `searchFields: ['project_id.name']` is refused at the ingress rather than
894+
dropped, because the search axis does not resolve traversal the way `fields`,
895+
`sort` and `filters` do:
896+
897+
```text
898+
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
899+
columns 'search' scans, so a name the object does not declare cannot narrow
900+
anything — and the engine used to drop it and scan the default columns instead,
901+
answering a NARROWER search with a WIDER one. 'search' scans this object's own
902+
columns; a related record's column cannot be a search target.
903+
```
904+
905+
The declarative answer is a **mirror field**: copy the related record's title
906+
into a stored field on this object and declare *that* field searchable — a task
907+
list searched by project name carries a `project_name` text column on `task`,
908+
maintained on write and listed in `task.searchableFields`, so the expansion stays
909+
a single-table `$or` of `$contains`. The mirror must be **stored**: a `formula`
910+
field is virtual, no driver materializes a column for it, and a `$contains`
911+
against one has nothing to scan. Cross-object search paths are rejected by
912+
design, not pending — see [Schema Design → Searching by a related record's
913+
title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value)
914+
for the maintenance hooks.
915+
890916
### Joins — removed (#4286)
891917

892918
`query.joins` was **removed in `@objectstack/spec` 17** (#4286, ADR-0049

‎content/docs/ui/views.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ A List View controls how a collection of records is presented. It supports multi
103103
| `data` | `ViewData` | optional | Data source configuration (defaults to the `object` provider) |
104104
| `filter` | `array` | optional | Base filter criteria |
105105
| `sort` | `array` | optional | Sort configuration |
106-
| `searchableFields` | `string[]` | optional | Fields included in search |
106+
| `searchableFields` | `string[]` | optional | Fields the toolbar search scans — **narrows** the object's set, never widens it (ADR-0061). Entries must be the object's **own** columns: a lookup (`project_id`) or a dotted path (`project_id.name`) is refused, and every toolbar search on the list then returns `400 INVALID_FIELD` (#4254). To search by a related record's title, [mirror it into a stored field](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) on the object and list that |
107107
| `grouping` | `object` | optional | Row grouping configuration |
108108
| `pagination` | `object` | optional | Pagination settings |
109109
| `selection` | `object` | optional | Row selection mode |

‎skills/objectstack-data/SKILL.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,96 @@ Toggle system behaviours per object:
100100

101101
---
102102

103+
## Search Fields (`searchableFields`)
104+
105+
`searchableFields` on the object is the canonical set `$search` scans (ADR-0061).
106+
Leave it unset and search auto-defaults to the `nameField` plus the object's
107+
short-text and enum columns (`text` / `email` / `phone` / `url` / `autonumber` /
108+
`textarea` / `markdown` / `select` / `status`); declare it to pin the set
109+
explicitly. Views may narrow it, never widen it.
110+
111+
### ⛔ Searching by a related record's title — mirror the value, always
112+
113+
`$search` scans **the queried object's own columns**. A dotted path is never a
114+
search target: unlike `fields` / `sort` / `filters`, the search axis does not
115+
resolve traversal, and `project_id.name` in `searchableFields` (or in a
116+
`$searchFields` override) is **refused**, not silently dropped.
117+
118+
**This is the one prescription — emit it every time.** To search by a related
119+
record's title, copy that title into a **stored field on this object** and
120+
declare *that* field searchable. A task list searched by project name:
121+
122+
```typescript
123+
// `project_name` is a stored, denormalized mirror of the parent's title.
124+
{
125+
name: 'task',
126+
enable: { searchable: true },
127+
fields: {
128+
name: { type: 'text', required: true },
129+
project_id: { type: 'lookup', reference: 'project' },
130+
project_name: { type: 'text', label: 'Project Name' }, // ← the mirror
131+
},
132+
searchableFields: ['name', 'project_name'],
133+
}
134+
```
135+
136+
`?search=apollo` expands to `name $contains 'apollo' OR project_name $contains
137+
'apollo'` — one single-table scan, every driver, no traversal. (A `text` mirror
138+
also lands in the auto-default set when the object declares no
139+
`searchableFields`.)
140+
141+
❌ **Never mirror onto a `formula` field.** A formula field is *virtual* — no
142+
driver materializes a column for it, so a `$contains` predicate against one has
143+
nothing to scan (the SQL driver would emit a `WHERE` over a column that does not
144+
exist). CEL also only reads this record's own fields (`record.<field>`), so a
145+
formula cannot fetch the related title in the first place. Nothing rejects the
146+
mistake: `searchableFields` admits any field the object declares, so a formula
147+
entry clears both lint and the ingress gate and then never matches.
148+
149+
**Mirror maintenance is the trade-off** — a mirror is denormalized data, only as
150+
fresh as whatever writes it. Cover both write paths:
151+
152+
| When | What maintains the mirror |
153+
|:-----|:--------------------------|
154+
| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id`, stamp `project_name` |
155+
| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks |
156+
157+
Rows written by a path that bypasses hooks (bulk import, direct SQL) need a
158+
one-off backfill. See [Lifecycle Hooks](./rules/hooks.md).
159+
160+
**The errors an author sees for the dotted path** (grep either back to here).
161+
`os validate` → `searchable-field-unknown`:
162+
163+
```text
164+
searchableFields entry "project_id.name" is not a field on object "task". The
165+
declaration is stale: searching it can never match, and the engine silently
166+
drops it — leaving a narrower search than declared, or the auto-default set once
167+
every entry is dropped.
168+
169+
hint: 'search' scans this object's own columns, so a related record's column
170+
cannot be a search target — expand the relation and search the related object,
171+
or copy the value onto a formula field here. Clients echo this declaration
172+
verbatim as the '$searchFields' override, so a stale entry becomes a 400
173+
INVALID_FIELD on list search (#4254), not just a quietly narrowed one.
174+
```
175+
176+
(The hint's "formula field" wording is loose — only a **stored** mirror works.)
177+
178+
A request carrying the dotted path is `400 INVALID_FIELD`:
179+
180+
```text
181+
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
182+
columns 'search' scans, so a name the object does not declare cannot narrow
183+
anything — and the engine used to drop it and scan the default columns instead,
184+
answering a NARROWER search with a WIDER one. 'search' scans this object's own
185+
columns; a related record's column cannot be a search target.
186+
```
187+
188+
Cross-object search paths are rejected by design, not pending. Do not invent a
189+
per-project convention for this — the mirror field is the answer.
190+
191+
---
192+
103193
## Field Groups (MVP)
104194

105195
Organize fields into logical groups (e.g., "Contact Information", "Billing",

‎skills/objectstack-query/SKILL.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -499,6 +499,56 @@ fields match by option *label*, mapped to stored values.
499499
Omit `fields` to search the object's declared `searchableFields` (or an
500500
auto-default of name/title + short-text fields), resolved server-side.
501501

502+
`fields` can only **narrow** that set, never widen it: over the REST/protocol
503+
ingress a name outside it is `400 INVALID_FIELD` (#4254), not a silent
504+
fall-back to the full scan.
505+
506+
### ⛔ Searching by a related record's title — mirror the value, always
507+
508+
`search` scans **the queried object's own columns**. A dotted path is never a
509+
search target: unlike `fields` (projection) / `sort` / `filters`, the search axis
510+
does not resolve traversal, so `searchFields: ['project_id.name']` is **refused**:
511+
512+
```text
513+
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
514+
columns 'search' scans, so a name the object does not declare cannot narrow
515+
anything — and the engine used to drop it and scan the default columns instead,
516+
answering a NARROWER search with a WIDER one. 'search' scans this object's own
517+
columns; a related record's column cannot be a search target.
518+
```
519+
520+
**This is the one prescription — emit it every time.** Copy the related record's
521+
title into a **stored field on the queried object** and search that field. A task
522+
list searched by project name gets a `project_name` text column on `task`,
523+
maintained on write and listed in `task.searchableFields`:
524+
525+
```typescript
526+
{
527+
object: 'task',
528+
search: { query: 'apollo', fields: ['name', 'project_name'] },
529+
limit: 20,
530+
}
531+
// Expands to a single-table scan — no traversal, every driver:
532+
// { $and: [{ $or: [
533+
// { name: { $contains: 'apollo' } },
534+
// { project_name: { $contains: 'apollo' } },
535+
// ]}]}
536+
```
537+
538+
❌ The mirror must be a **stored** field — a `formula` field is virtual, no
539+
driver materializes a column for it, so a `$contains` predicate against one has
540+
nothing to scan. Nothing rejects the mistake for you: `searchableFields` admits
541+
any field the object declares, so a formula entry clears both lint and the
542+
ingress gate and then never matches. The trade-off is mirror maintenance — hooks
543+
on both write paths (child re-parented, parent renamed) plus a backfill for rows
544+
written around the hooks.
545+
546+
Cross-object search paths are rejected by design, not pending. Modelling side of
547+
this (the field, the hooks, the lint wording): **objectstack-data → Search Fields
548+
(`searchableFields`)**. To *filter* by a related record's column — a different
549+
axis — use a [nested relation filter](#nested-relation-filters); to *display* it,
550+
use [`expand`](#expand-related-records).
551+
502552
> ⚠️ **`[EXPERIMENTAL — not enforced]` (#4286):** `fuzzy`, `boost`,
503553
> `operator`, `minScore`, `language`, and `highlight` validate against the
504554
> schema but are never read — their `.describe()` markers now say so. Terms
@@ -535,6 +585,7 @@ auto-default of name/title + short-text fields), resolved server-side.
535585
|:---------|:----|
536586
| Load lookup fields for display | `expand` |
537587
| Filter parent by child conditions | Nested relation filter |
588+
| **Keyword-search by a related record's title** | **Mirror the title into a stored field on this object and search that** — `search` never traverses (see **Full-Text Search** above) |
538589
| Simple parent→child navigation | `expand` |
539590
| Paginate/sort a parent's related records | Query the related object directly |
540591
| Analytical queries across objects | Report/dashboard metadata, or separate queries combined in app code (`joins` was removed in #4286 — see above) |

0 commit comments

Comments
 (0)