path query: SQLite step index (path-cli 0.18.0)

CHANGELOG.md

@@ -2,6 +2,84 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## SQLite step index for `path query` — 2026-07-21
+
+`path query` now keeps a disposable SQLite accelerator at
+`$CONFIG_DIR/index.db`: every wrapped step as a row (exactly what the
+jaq engine consumes), with generated columns over the hot fields and a
+stat-level freshness gate per document. The cache files stay canonical
+— delete the index and the next query rebuilds it lazily; a schema
+version bump rebuilds it automatically. `TOOLPATH_QUERY_NO_INDEX=1`
+bypasses it.
+
+What it accelerates (real 97-doc / 114 MB / 46k-step cache, medians;
+"rayon" = 0.17.0's parallel scan):
+
+- **Absorbed counts** — a filter that is exactly `length` or
+  `map(select(P)) | length` with a recognized `P` becomes
+  `SELECT count(*)`: 0.90 s → **30 ms** (~30×; no step is parsed at
+  all). Recognized `P` atoms: `.dead_end` (and `| not`),
+  `.step.actor == "…"`, `.step.actor | startswith("…")`,
+  `.path.meta.source == "…"`, and `and`-conjunctions of those.
+- **Predicated scans** — a leading `map(select(P))` / `.[] | select(P)`
+  prefilters rows in SQL before parsing: `map(select(.step.actor |
+  startswith("agent:"))) | length` 0.98 s → 169 ms (~6×; 310 ms under
+  rayon alone).
+- Unrecognized/slurp queries are unchanged (they ride 0.17.0's
+  parallel scan); a predicated stream lands at parity with it
+  (row-fetch + per-row parse ≈ parallel whole-file parse).
+
+Correctness: recognition is exact (whole-predicate or nothing) and the
+prefilter feeds the *unchanged* original filter, so pruned rows are
+ones the filter's own first stage would drop — the integration suite
+asserts index-served output is byte-identical to the no-index path
+across every plan, staleness self-healing included. Content-scoped
+queries (`--kind`/`--project`/`--parent-dir`) bypass the index.
+
+- **`path-cli`** (0.18.0):
+  - `query/index.rs`: the store — WAL, per-thread connections,
+    `(mtime_ns, size)` stamps, write-through from `cache::write_cached`
+    (sync/import keep the index warm), purge on `p cache rm`, orphan
+    pruning on full scans, version-gated self-rebuild. One-time build
+    cost on first query: ~1.7 s for the 114 MB cache; index size ~250 MB
+    (rows + hot-field indexes).
+  - `query/plan.rs`: `RowPredicate` — conservative recognizer for the
+    leading-`select` predicate and the absorbable-count shape, with an
+    in-code `matches` mirror used when a stale doc is reparsed.
+  - `TOOLPATH_QUERY_EXPLAIN=1` now also prints the index strategy
+    (`count absorbed into SQL (dead_end=true)`, `serving fresh docs,
+    prefilter …`, or `off`).
+
+## Parallel `path query` execution — 2026-07-21
+
+`path query` now evaluates cache documents on a thread pool. For the
+plans that treat every file independently (`PerFileStream`,
+`Decompose`), the whole per-file pipeline — read, parse, wrap, filter,
+render — runs on rayon workers, with only ordered output assembly on
+the main thread; `Slurp` plans parallelize the parse/wrap phase only
+(the merged jaq array is `Rc`-based and must be built on one thread).
+Output is byte-identical to the sequential scan — same ordering, same
+warnings, same error precedence — enforced by tests pinning the
+parallel building blocks against the sequential engine.
+
+Measured on a real 97-doc / 114 MB cache (M-series, medians of 5):
+streamed and decomposed queries drop from ~0.95 s to ~0.31 s (~3.1×),
+slurped queries improve ~1.2×. On an even 60-doc / 56 MB synthetic
+cache the same queries run ~4.7× faster (e.g. `length` 966 ms →
+206 ms).
+
+- **`path-cli`** (0.17.0):
+  - `query/mod.rs`: plan dispatch (`execute_plan`) and a chunked
+    parallel driver (`for_each_file`) that preserves selection order,
+    per-file warning order, and explicit-file error positions.
+  - `query/filter.rs`: per-file worker evaluation (`render_file`,
+    `partials_file` — partials cross threads as compact JSON bytes and
+    are reparsed on the consuming thread), a per-thread compiled-filter
+    cache, and `finish_decompose` shared by the sequential and parallel
+    drivers so the zero-file rule lives once.
+  - The emscripten (playground) build keeps the sequential engine —
+    no threads there.
+
 ## `path p cache sync` — incremental session ingestion — 2026-07-16
 
 Adds `path p cache sync [types…]`, the first step toward a cache that
@@ -101,6 +179,20 @@ hand.
     and `--no-sync` opts out. This is the piece that makes the cache
     an implementation detail: a new user can run `path query` with no
     setup and get their sessions.
+  - `--parent-dir <dir>` / `-d` on `p cache sync` and `path query`
+    restricts ingestion (and the query's reads) to artifacts under a
+    directory. Stat gate first, always: unchanged+cached artifacts skip
+    before any scope check; only artifacts that would cost a derive get
+    the constraint, with a one-line peek for codex and copilot (whose
+    cwd lives inside the session file) memoized into the manifest so it
+    happens at most once per artifact — and a derive never clobbers the
+    memoized peeked cwd. The copilot peek tolerates `session.start`
+    anywhere in the first lines and top-level cwd keys. Claude project
+    matching happens in slug space (its dir slugs are lossy), and pi
+    project scoping compares in its dir-encoded space so hyphenated
+    paths match. Out-of-scope artifacts are noted in the manifest
+    ("known, not materialized") but not derived, tallied separately
+    (`N out of scope`), and never touch a materialized record's stamp.
 - **`toolpath-codex`** (0.6.1, extends the bump below):
   `session_id_from_stem` is public — the trailing UUID of a rollout
   filename stem (or the whole stem when it has none), the same