perf(semantic-search): index from manifest projection

[aryasaatvik]

Summary

Carries the tool manifest projection into semantic-search scan jobs by retaining each manifest source revision alongside the fingerprint. This makes index work traceable to the source catalog version that produced it without adding schema reads to scan.

Changes

• Add optional sourceRevision to semantic-search index jobs.
• Copy ToolSchemaManifest.sourceRevision into both skipped and changed scan jobs.
• Add regression coverage that manifest source revisions survive scan materialization.

Call Stack

ToolSearchIndex.create()
  -> executor.tools.manifest()
  -> write manifest snapshot

ToolSearchIndex.scan()
  -> read manifest projection
  -> compare manifest.indexFingerprint with stored fingerprint
  -> write IndexJob(sourceRevision, fingerprint, path, status)

Tests

• bun run --cwd packages/plugins/semantic-search test -- src/sdk/tool-search-index.test.ts
• bun run --cwd packages/plugins/semantic-search typecheck
• bunx oxfmt --check packages/plugins/semantic-search/src/sdk/collections.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts
• bunx oxlint -c .oxlintrc.jsonc --deny-warnings packages/plugins/semantic-search/src/sdk/collections.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts

Stack

Base: #63

Stack

1. feat(sdk): harden tool manifest projection for indexing #63
2. perf(semantic-search): index from manifest projection #64 👈 current
3. perf(semantic-search): snapshot manifest partitions #65

[greptile-apps]

Greptile Summary

This PR threads sourceRevision from the ToolSchemaManifest projection into both branches of scan() — skipped and pendingChunk — so every emitted IndexJob can be traced back to the catalog version that produced it, without any additional schema reads at runtime.

• collections.ts: sourceRevision: Schema.optional(Schema.String) is appended to IndexJob; the field is optional so existing persisted records deserialize cleanly without a migration.
• tool-search-index.ts: Both job-construction branches use the same conditional spread (manifest.sourceRevision === undefined ? {} : { sourceRevision }) to omit the field when the manifest does not carry one.
• tool-search-index.test.ts: Adds a single test covering both the changed and skipped paths with a manifest that carries sourceRevision: "spec-hash-v1", plus a negative assertion in an existing test confirming the field stays absent when the manifest omits it.

Confidence Score: 5/5

Safe to merge — the change is additive and backward-compatible, both scan branches are symmetric, and the test suite now covers all four cases (changed with revision, skipped with revision, changed without revision, and the negative undefined assertion).

The field is optional so no persisted records break on read. The conditional spread pattern is identical in both branches, and the PR addresses the previously flagged gap by testing both the skipped and changed paths explicitly.

No files require special attention.

Important Files Changed

Filename	Overview
packages/plugins/semantic-search/src/sdk/collections.ts	Adds optional sourceRevision: Schema.optional(Schema.String) field to the IndexJob struct; backward-compatible schema change with no migration required.
packages/plugins/semantic-search/src/sdk/tool-search-index.ts	Spreads sourceRevision from the manifest into both the skipped and pendingChunk job branches using a conditional spread; logic is symmetric and correct in both paths.
packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts	Extends manifestForTool helper to accept an optional sourceRevision; adds a comprehensive regression test covering both the changed and skipped scan branches, plus a negative assertion for the no-revision case.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["scan()"] --> B["Load manifest projections"]
    B --> C{"For each manifest:\nstoredRow?.fingerprint\n== indexFingerprint?"}
    C -- "Yes (fingerprint match)" --> D["Build skipped IndexJob\n+ sourceRevision spread"]
    C -- "No (new/changed)" --> E["Build pendingChunk IndexJob\n+ sourceRevision spread"]
    D --> F["Increment skipped counter"]
    E --> G["Increment changed counter"]
    F --> H["putJobs()"]
    G --> H
    H --> I["Return scan result\n(processed, changed, skipped, …)"]

    style D fill:#d4edda
    style E fill:#d4edda

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["scan()"] --> B["Load manifest projections"]
    B --> C{"For each manifest:\nstoredRow?.fingerprint\n== indexFingerprint?"}
    C -- "Yes (fingerprint match)" --> D["Build skipped IndexJob\n+ sourceRevision spread"]
    C -- "No (new/changed)" --> E["Build pendingChunk IndexJob\n+ sourceRevision spread"]
    D --> F["Increment skipped counter"]
    E --> G["Increment changed counter"]
    F --> H["putJobs()"]
    G --> H
    H --> I["Return scan result\n(processed, changed, skipped, …)"]

    style D fill:#d4edda
    style E fill:#d4edda

Loading

_{Reviews (3): Last reviewed commit: "test(semantic-search): cover skipped man..." | Re-trigger Greptile}