feat(semantic): [merge candidate build] FTS5 index and search tools, provider-aware typed embeddings, reranking, diagnostics, and eval harness

[Zireael]

Summary

This PR hardens AFT’s semantic retrieval layer and adds the surrounding machinery needed to configure, inspect, rerank, and benchmark it.

The main changes are:

• provider-aware semantic indexing;
• typed vector storage with correct metric selection;
• fingerprint-based index invalidation;
• optional local model2vec embeddings;
• optional reranking through chat-completion or /v1/rerank endpoints;
• semantic diagnostics, doctor output, and JSONL eval support;
• Semble-inspired benchmark scripts for local regression testing;
• opt-in FTS5 search graph.

Default behavior should remain conservative. Users can continue using the existing semantic path, while advanced users can opt into local model2vec, OpenAI-compatible embedding servers, rerankers, and FTS5 benchmark paths.

Why this PR exists

AFT’s previous semantic search path was too thin for serious agent use. It could embed and rank, but it lacked enough structure around provider capabilities, index invalidation, vector format handling, diagnostics, and repeatable evaluation.

This PR addresses those missing pieces. The goal is to make semantic retrieval easier to configure, safer to change, easier to debug, and measurable enough to improve without guessing.

Key user-facing improvements

Provider-aware semantic configuration

Embedding backends now carry explicit capability information such as:

• supported output encoding;
• vector kind;
• distance metric;
• dimension expectations;
• batch-size limits;
• provider/model fingerprint fields.

This prevents invalid combinations from silently producing a bad index. Switching meaningful provider settings triggers the appropriate rebuild or cache invalidation path.

Typed vector storage

Stored vectors are no longer treated as undifferentiated f32 arrays. The search path can distinguish dense vectors, decoded int8 vectors, and binary-packed vectors.

That matters because binary vectors need Hamming-style comparison, not cosine scoring. The PR adds the plumbing needed for correct metric selection instead of relying on one ranking method for every provider output shape.

Local model2vec support

The branch adds an optional local model2vec backend, intended for fast local code retrieval. The current recommended model for this workstream is Potion Code 16M:

{
  "semantic_search": true,
  "semantic": {
    "backend": "model2vec",
    "model": "minishlab/potion-code-16M",
    "model_path": "D:/AI/LLM_models/potion-code-16M"
  }
}

This backend is local-only and feature-gated with semantic-model2vec.

Optional reranking

Search results can be reranked after initial retrieval.

Supported reranker API modes:

• chat: OpenAI-compatible /v1/chat/completions;
• rerank: cross-encoder-style /v1/rerank.

Example cross-encoder configuration:

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "CodeRankEmbed",
    "base_url": "http://127.0.0.1:8090/v1",
    "api_key_env": "OPENAI_API_KEY",

    "rerank_enabled": true,
    "rerank_api_type": "rerank",
    "rerank_model": "GTE-Reranker-Modernbert",
    "rerank_base_url": "http://127.0.0.1:8090/v1",
    "rerank_max_candidates": 30,
    "rerank_max_candidate_chars_cross_encoder": 512
  }
}

Reranking is fail-safe. If the reranker fails, returns malformed output, or exceeds safety limits, search falls back to the original retrieval order.

Diagnostics and doctor output

The PR adds semantic health-reporting surfaces for:

• active config;
• index state;
• provider/model state;
• model2vec health;
• warning conditions;
• suggested fixes.

This is meant to make retrieval failures debuggable. Examples: wrong model path, unavailable provider, stale index, failed reranker, or feature-gate mismatch.

Eval and benchmark support

The semantic eval harness accepts JSONL cases with expected paths and symbols, then reports retrieval metrics such as recall@k and MRR.

Example shape:

{"query":"where is JWT validation handled","expected_paths":["src/auth/session.ts"],"expected_symbols":["validateJwt"],"top_k":10}

The benchmark scripts are intended for regression testing and local backend comparison. They should not be treated as publishable model-quality claims yet.

How to try it

Minimal semantic search

{
  "semantic_search": true
}

Then use the normal AFT search path:

aft_search({ "query": "authentication middleware" })

OpenAI-compatible embedding backend

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "CodeRankEmbed",
    "base_url": "http://127.0.0.1:8090/v1",
    "api_key_env": "OPENAI_API_KEY",
    "max_batch_size": 16
  }
}

Local model2vec backend

{
  "semantic_search": true,
  "semantic": {
    "backend": "model2vec",
    "model": "minishlab/potion-code-16M",
    "model_path": "/path/to/potion-code-16M"
  }
}

Reranking with /v1/rerank

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "CodeRankEmbed",
    "base_url": "http://127.0.0.1:8090/v1",

    "rerank_enabled": true,
    "rerank_api_type": "rerank",
    "rerank_model": "GTE-Reranker-Modernbert",
    "rerank_base_url": "http://127.0.0.1:8090/v1",
    "rerank_max_candidates": 30
  }
}

Benchmark quick start

Build with the semantic features you want to test:

cargo build --release -p agent-file-tools --features semantic-model2vec,semantic-fts5

Clone the small pinned pilot corpus:

bun run benchmarks/semble/corpus.ts sync --pilot

Run the pilot comparison:

bun run benchmarks/semble/pilot.ts --binary ./target/release/aft --k 10

The current pilot corpus covers:

• axum;
• serde;
• express;
• pydantic;
• gin.

For embedding and reranker server comparisons:

AFT_BENCH_EMBED_BASE_URL=http://127.0.0.1:8090/v1 \
AFT_BENCH_EMBED_MODEL=CodeRankEmbed \
AFT_BENCH_RERANK_BASE_URL=http://127.0.0.1:8090/v1 \
AFT_BENCH_RERANK_MODEL=GTE-Reranker-Modernbert \
bun run benchmarks/semble/run-semble-bench.ts \
  --profile c \
  --k 10 \
  --rerank \
  --rerank-candidates 30 \
  --binary ./target/release/aft

Recommended benchmark interpretation:

• use the pilot for fast regression checks;
• use rerank candidate sweeps to detect candidate starvation;
• compare semantic-only, hybrid, FTS5, and reranked variants separately;
• do not use the pilot as a final claim about model quality.

Review guide

This PR is easier to review by subsystem than by file order.

1. Config and trust boundary

Review:

• crates/aft/src/config.rs
• packages/opencode-plugin/src/config.ts
• packages/pi-plugin/src/config.ts

Check:

• Rust and TypeScript schemas match;
• user-only provider fields are not accepted from untrusted project config;
• prompt template fields are handled consistently;
• defaults match docs and tests;
• feature-gated settings fail clearly when the feature is unavailable.

2. Semantic index and vector storage

Review:

• crates/aft/src/semantic_index.rs
• crates/aft/src/vector_store.rs
• crates/aft/src/local_embed.rs
• crates/aft/src/commands/semantic_search.rs

Check:

• vector kind selects the correct metric;
• fingerprint changes trigger correct rebuild/cache behavior;
• stale vectors are pruned;
• changed/deleted/excluded files do not leave searchable old entries;
• fallback paths are explicit and observable.

3. Reranking

Review:

• crates/aft/src/semantic_rerank.rs
• crates/aft/src/commands/semantic_search.rs

Check:

• chat and /v1/rerank request shapes;
• sorting behavior for provider response formats;
• duplicate and out-of-bounds reranker indices;
• fallback behavior;
• response-size caps;
• candidate count defaults;
• cross-encoder snippet limits.

4. Diagnostics, doctor, and eval

Review:

• crates/aft/src/semantic_diagnostics.rs
• crates/aft/src/semantic_doctor.rs
• crates/aft/src/semantic_eval.rs
• crates/aft/src/commands/semantic_doctor.rs
• crates/aft/src/commands/semantic_eval.rs

Check:

• diagnostics expose useful failures without leaking secrets;
• minimal diagnostic mode does not hide important reranker/provider failures;
• eval metrics match documented behavior;
• JSONL parsing is strict enough to avoid misleading benchmark output.

5. Benchmark scripts

Review:

• benchmarks/SEMBLE-BENCHMARKS.md
• benchmarks/semble/run-semble-bench.ts
• benchmarks/semble/pilot.ts
• benchmarks/semble/repos-pilot.json
• benchmarks/semble/annotations/*

Check:

• failed backends are reported instead of silently skipped;
• benchmark profiles are reproducible;
• local endpoint assumptions are documented;
• FTS5 results are labeled experimental unless the FTS5 e2e path is complete;
• pilot results are not overstated.

Suggested validation before merge

cargo fmt --check
cargo clippy -p agent-file-tools --all-targets --features semantic-model2vec,semantic-fts5 -- -D warnings
cargo test -p agent-file-tools --features semantic-model2vec,semantic-fts5
bun install
bun run typecheck
bun test

Optional benchmark smoke:

cargo build --release -p agent-file-tools --features semantic-model2vec,semantic-fts5
bun run benchmarks/semble/corpus.ts sync --pilot
bun run benchmarks/semble/pilot.ts --binary ./target/release/aft --k 10

Optional local reranker comparison:

AFT_BENCH_EMBED_BASE_URL=http://127.0.0.1:8090/v1 \
AFT_BENCH_EMBED_MODEL=CodeRankEmbed \
AFT_BENCH_RERANK_BASE_URL=http://127.0.0.1:8090/v1 \
AFT_BENCH_RERANK_MODEL=GTE-Reranker-Modernbert \
bun run benchmarks/semble/run-semble-bench.ts \
  --profile c \
  --k 10 \
  --rerank \
  --rerank-candidates 30 \
  --binary ./target/release/aft

Known limits

The following should remain explicit:

• FTS5 is opt-in and should not be presented as the default search backend.
• Benchmark tooling is useful for local comparison and regression detection, not final external claims.
• The five-repo pilot is intentionally small.
• Full Semble, RepoQA, SWE-Explore, and CORE-Bench adapters are follow-up work.
• Reranking quality depends heavily on candidate pool size. A weak rerank delta may indicate candidate starvation rather than a bad reranker.
• Semantic search should complement exact symbol and lexical search, not replace them.

Feature traceability

Relevant workstreams:

• aft-t6p — semantic search upgrade;
• aft-t6p.m2v — model2vec / Potion Code 16M backend;
• aft-t6p.tok — tokenizer fixture compatibility;
• aft-t6p.bench — benchmark adaptation;
• aft-t6p.bench.quick — targeted quick benchmark mode;
• aft-fts5e2e — FTS5 end-to-end feature work.

Merge posture

I would merge this when the semantic retrieval foundation is safe, tested, and diagnosable.

I would not block the PR on full benchmark coverage or publishable external comparisons. Those are useful follow-up work, but they should not be required for merging the retrieval substrate.

I would block the PR if any of these are true:

• untrusted project config can redirect embedding or reranker endpoints;
• switching vector format/provider/model can reuse an incompatible index;
• reranker failures are silent in normal diagnostics;
• benchmark scripts skip failed backends without surfacing that failure;
• FTS5 is documented as complete before its e2e path is actually validated.

FTS5 index will allow introducing more advanced symbol operations.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Production‑ready semantic retrieval with provider‑aware typed embeddings, safe reranking, contextualized doc‑chunk indexing, full diagnostics/doctor/eval, and an optional FTS5 path. CI/release build with semantic-model2vec and semantic-fts5 by default, plus a cross‑platform build-aft workflow and Docker‑based Rust validation.

• New Features

  • Reranking: overfetch → rerank → truncate; chat and /v1/rerank; 2 MiB body cap; cross‑encoder support; fence stripping; overflow chunking (max_embed_tokens, chunk_overlap_tokens).
  • Contextualized embeddings: Perplexity document‑chunk mode with split/retry diagnostics; model prompt profiles (use_model_profiles), document_prompt_template, deterministic ordering.
  • Typed vectors/storage: correct metrics per kind (incl. Hamming for BinaryPacked), file manifest + chunk_hash, fingerprint invalidation, include/exclude file policy, per‑file caps.
  • Backends/tooling: local model2vec with HF auto‑download, catalog/health/version checks; OpenAI‑compatible embeddings; experimental FTS5 search/symbol commands; Semble benchmark suite (pilot, runners, reports); eval harness runs real searches with per‑case k.
  • Build/CI: default‑feature release builds (semantic-model2vec, semantic-fts5), manual multi‑platform build-aft workflow, and Docker‑based Rust validation for fmt/check/clippy/test.

• Bug Fixes

  • Trust/safety: strip rerank_prompt_template from untrusted configs; SSRF guard for rerank_base_url; traversal‑safe jsonl_path.
  • Rerank correctness/UX: score‑sorted and de‑duplicated outputs; out‑of‑bounds index warnings; fix candidate starvation; more_available signals correctly after truncation.
  • Stability/diagnostics: clamp cosine NaNs; bounded streaming reads; warning dedup; surface RerankerFailure in minimal mode; accurate query cache‑hit.
  • Indexing/eval: hardened contextualized split/retry mapping; fixed chunk line bounds; configure parses all semantic/JSONL fields; eval harness wired to real search with per‑case k.

^{Written for commit 92744a1. Summary will update on new commits.}

Greptile Summary

This alpha PR replaces the prototype semantic search subsystem with a production-oriented retrieval pipeline: typed vectors (DenseF32, Int8SourceDecoded, BinaryPacked), provider capability profiles, fingerprint-driven index invalidation, background build with cooperative cancellation, optional reranking, diagnostics/doctor/eval tooling, and a security trust boundary that prevents project configs from redirecting embeddings to attacker-controlled endpoints.

• Core pipeline overhaul (semantic_index.rs, vector_store.rs): typed vectors with correct distance metrics (Hamming for binary, cosine for f32), stale-vector pruning, priority-ordered cold-start build, and fingerprint-based rebuild decisions.
• Reranking pipeline (semantic_rerank.rs): supports chat-completions and cross-encoder /v1/rerank endpoints with multiple response-format parsers, body-size cap, and safe fallback to original cosine order on any failure.
• Diagnostics, doctor, and eval (semantic_diagnostics.rs, semantic_doctor.rs, semantic_eval.rs): JSONL diagnostic logging, health-check command, and JSONL-based eval harness with recall@k and MRR scoring.

Confidence Score: 4/5

Safe to merge with awareness that several known issues from the previous review cycle remain open (reranker failures silent in minimal mode, more_available undercount, data-format rerank ordering, non-ASCII eval file corruption), plus the new gap where project configs can supply an adversarial rerank prompt template.

The new rerank_prompt_template stripping gap is the only genuinely new finding in this update; all other open issues were already identified in earlier review rounds. The change is an alpha build with documented limitations and strong test coverage (~93 tests). The trust boundary is mostly correct, the TypeScript enum mismatches are now fixed, and the vector-store and Hamming-distance math are sound.

packages/opencode-plugin/src/config.ts — add rerank_prompt_template to the stripProjectSemanticFields function. crates/aft/src/semantic_diagnostics.rs — surface RerankerFailure in minimal output mode.

Security Review

• Prompt injection via rerank_prompt_template in project config (packages/opencode-plugin/src/config.ts:1481): query_prompt_template and document_prompt_template are correctly stripped from project-level configs; rerank_prompt_template is not. A hostile repository can supply an adversarial reranker prompt that manipulates search result ordering for the user. Data exfiltration is not possible (the reranker endpoint URL is user-only), but result integrity can be silently degraded.
• No new credential-leakage, injection, or authentication bypass issues introduced by this PR. The trust-boundary stripping of backend, base_url, api_key_env, rerank_base_url, and rerank_api_key_env from project configs is correctly implemented.

Important Files Changed

Filename	Overview
crates/aft/src/semantic_rerank.rs	New reranking pipeline with body-size cap, fence stripping, and multiple response-format parsers. The build_rerank_endpoint trailing-slash fix is present. The data/results formats in extract_indices_from_rerank_results return insertion order rather than score-sorted order (flagged in previous review cycle).
crates/aft/src/semantic_eval.rs	New eval harness with JSONL parsing, recall@k, and MRR scoring. strip_trailing_commas is implemented but corrupts non-ASCII bytes (previously flagged). The score_case docstring says hits beyond k don't affect first_hit_rank, but the code does set it, so MRR includes full-list ranks, not @k ranks.
crates/aft/src/semantic_diagnostics.rs	New diagnostic subsystem. format_warning_minimal returns None for RerankerFailure, silently suppressing reranker failures in the default Minimal output mode (flagged in previous review cycle).
crates/aft/src/commands/semantic_search.rs	Reranking, diagnostics, and fusion-limit integration. more_available is computed against fusion_limit rather than top_k, so candidates between top_k and fusion_limit are silently dropped after reranking without setting more_available = true (flagged in previous cycle). OOB reranker-index warning still gated on diagnostics_enabled.
packages/opencode-plugin/src/config.ts	TypeScript Zod schema updated with 16+ new fields including corrected enum values (base64_binary, binary_packed, dot_product, perplexity). Trust-boundary stripping function added. rerank_prompt_template is exposed in the schema but not stripped from project configs, unlike query_prompt_template and document_prompt_template.
crates/aft/src/vector_store.rs	New VectorStore trait with FlatF32 (cosine) and FlatBinaryHamming (Hamming distance) implementations. Score normalization, orphan pruning, and stale-vector pruning look correct. Hamming similarity computation is sound.
crates/aft/src/config.rs	Provider capability profiles, RerankApiType enum, trust-boundary doc comments, and expanded SemanticBackendConfig. RerankApiType uses snake_case serde, matching the TypeScript ["chat", "rerank"] values.
crates/aft/src/compress/trust.rs	Added security-focused tests: atomic writes, multi-project trust, idempotent untrust, and reload survival. No logic changes.

Comments Outside Diff (2)

1. packages/opencode-plugin/src/config.ts, line 37-54 (link)

   TypeScript enum values don't match the Rust serde strings — config will fail to deserialize

   Several new enum schemas use values that don't align with the Rust serde representation:

   • SemanticOutputEncodingEnum allows "binary", "ubinary", "int8", "uint8" but Rust OutputEncoding deserializes from "base64_binary" and "base64_int8".
   • SemanticStorageStrategyEnum allows "flat" and "binary_pack" but Rust StorageStrategy expects "native_f32" and "binary_packed".
   • SemanticInputModeEnum includes "chunk_extracts" and "contextualized" but Rust InputMode only has "flat_texts" and "document_chunks".
   • SemanticDistanceMetricEnum uses "dot" but Rust DistanceMetric expects "dot_product".
   • SemanticBackendEnum is missing the new "perplexity" variant added to Rust.

   A user who follows the TypeScript autocomplete and picks output_encoding: "int8" will pass TypeScript validation but receive a deserialization error (or silent fallback to default) from the Rust binary at runtime.

2. crates/aft/src/commands/semantic_search.rs, line 116-119 (link)

   more_available understates available results when reranking is active

   fused_more_available is now computed as results.len() > fusion_limit (e.g., > 20) rather than > top_k (e.g., > 10). After reranking, results.truncate(top_k) discards any candidates between positions top_k and fusion_limit, but more_available has already been set and stays false. Concretely: if the fused pool yields 15 candidates (top_k=10, rerank_max_candidates=20), fused_more_available = 15 > 20 = false, more_available = false, and the 5 reranked-but-discarded candidates are silently dropped with no "more results" hint surfaced to the agent.

   Capture the pool size before truncation and fold it into more_available after the rerank block, before results.truncate(top_k).

_{Reviews (29): Last reviewed commit: "feat(semantic): add model prompt profile..." | Re-trigger Greptile}