Default Retriever to weighted hybrid search

[jioffe502]

Summary

This PR makes weighted hybrid retrieval the default for new NeMo Retriever indexes.

The product behavior is intentionally simple:

• New indexes use hybrid retrieval automatically.
• Existing tables keep their current physical mode when users append.
• Reranking remains available as an optional maximum-accuracy layer.
• Users do not need to choose or tune a retrieval configuration for the common case.

Each hybrid query combines dense semantic search with BM25 full-text search (FTS) for exact terms, then merges the two ranked lists with weighted Reciprocal Rank Fusion (RRF). RRF is a rank-list merge: dense remains the primary signal, while FTS can rescue identifiers, numbers, names, and other lexical matches.

Why weighted hybrid

Dense retrieval is strong for paraphrases and semantic similarity, but it can miss an exact identifier or phrase. FTS has the opposite profile. Combining both signals is a better general-purpose default, but equal weighting can allow a lexical match to displace a stronger semantic result.

The product paths therefore use one fixed, dense-favoring policy:

• 50 candidates from each retrieval leg
• dense weight 0.8
• FTS weight 0.2
• RRF k=10

This policy is intentionally not exposed as a runtime tuning surface. Low-level LanceDB callers can still opt into explicit behavior.

What the evaluation says

BO767 showed the clearest positive movement in the nightly validation, particularly for lexical/document matching. The full ViDoRe suite was used as a broad check against overfitting:

Primary no-reranker ablation

Evaluation	Queries	Recall@5: dense / equal / weighted	nDCG@10: dense / equal / weighted
ViDoRe V3	14,514	0.4623 / 0.4157 / 0.4661	0.5161 / 0.4559 / 0.5197
BO767	991	0.8486 / 0.8496 / 0.8597	0.7480 / 0.7509 / 0.7632
Earnings/Consulting	628	0.6545 / 0.6720 / 0.6736	0.5666 / 0.5791 / 0.5807
FinanceBench	150	0.8467 / 0.8133 / 0.8400	0.7359 / 0.6728 / 0.7332

This is the no-model-reranker policy-selection run, comparing two different fusion policies against dense retrieval. Equal-weight RRF reduced quality on ViDoRe and FinanceBench. The selected dense-favoring weighted RRF improved ViDoRe, BO767, and Earnings/Consulting, and removed the large FinanceBench regression, leaving FinanceBench effectively tied with dense. The four-arm table below separately measures how the model reranker interacts with dense and weighted hybrid retrieval.

Arm	Retrieval	Reranker	Recall@5	Recall@10	nDCG@10
A	Dense	No	46.26%	57.01%	51.58%
B	Dense	Yes	50.69%	61.78%	57.35%
C	Weighted hybrid	No	46.57%	57.50%	51.92%
D	Weighted hybrid	Yes	50.83%	61.92%	57.45%

These are query-weighted results over all 14,514 queries in the eight ViDoRe V3 datasets.

The important conclusions are:

• Reranking remains valuable: it adds about 4.4 Recall@5 points and 5.5 nDCG@10 points on either retrieval strategy.
• Weighted hybrid is not a replacement for reranking. It is a small aggregate improvement over dense retrieval, with its strongest rationale being lexical coverage and protection against exact-term misses.
• With reranking enabled, weighted hybrid is slightly ahead of dense (+0.14 Recall@5 points, +0.10 nDCG@10 points), but this is best described as preserving or modestly improving quality—not as a large benchmark gain.

This is a deliberate, evidence-based default change: hybrid does not materially regress dense behavior, and it improves the query classes where semantic-only retrieval is least reliable.

Index-mode behavior

index_mode=auto is now the default for SDK, CLI, harness, and service paths:

Operation	auto behavior
New or overwritten table	Creates vector plus FTS indexes and resolves to hybrid
Append to dense table	Preserves dense
Append to hybrid table	Preserves hybrid
Append to sparse table	Preserves sparse
Explicit index_mode=hybrid on dense table	Performs the explicit dense-to-hybrid upgrade
Destructive or mismatched transition	Rejected

Query retrieval_mode=auto resolves from the table's physical vector and FTS capabilities, so stale schema metadata cannot override the actual storage state.

Service behavior

• Fresh hybrid service storage creates and waits for the FTS index after the first write.
• Rows added before indexing completes remain searchable through LanceDB's unindexed-tail scan.
• The service runs optimize() after 20 writes or 100,000 added rows.
• Health reports effective mode, FTS presence, unindexed rows, and the last optimization status.
• Helm exposes serviceConfig.vectordb.indexMode, defaulting to auto.

Scope and non-goals

This PR does not add a query router, model reranker, approximate vector index, fusion-learning loop, or new runtime tuning surface. Reranking remains an existing opt-in capability; this change only makes hybrid retrieval the default retrieval strategy for new indexes.

Validation

• uvx pre-commit run --all-files: passed.
• Focused retrieval, index-policy, CLI, Helm, and service tests: 155 passed, 4 skipped.
• VectorDB service and collection regression tests: 103 passed.
• Full ViDoRe V3 four-arm replay completed for all 14,514 queries.

Recommendation

Merge this as a default-behavior improvement. New users get the best tested general-purpose retrieval configuration without having to select a mode. Existing dense deployments remain safe and unchanged until an explicit hybrid upgrade is requested. Users who need maximum accuracy can continue to enable the reranker, while future routing work can target the query slices where the evidence shows the largest lexical benefit.

[greptile-apps]

Greptile Summary

This PR makes weighted dense-favoring hybrid retrieval the default for newly created indexes while preserving the physical retrieval mode of existing tables on append.

• Adds automatic ingest-mode resolution and physical LanceDB capability detection.
• Adds weighted RRF fusion for hybrid queries.
• Adds service-managed FTS creation, incremental optimization, health telemetry, and Helm configuration.
• Updates CLI defaults, documentation, and retrieval/service tests.

Confidence Score: 4/5

The PR is not yet safe to merge because transient FTS telemetry failures can still cause the VectorDB health probe to report the service unavailable.

The current health path catches backend telemetry exceptions only at the endpoint boundary and converts them to HTTP 503; because _fts_unindexed_rows() still performs an unguarded list_indices() call, a diagnostic failure can continue to fail Kubernetes probes even when retrieval remains functional.

Files Needing Attention: nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py, nemo_retriever/src/nemo_retriever/service/vectordb_app.py

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py	Adds weighted hybrid retrieval and service-side FTS maintenance, but FTS telemetry can still propagate transient index-listing failures into an unhealthy service response.
nemo_retriever/src/nemo_retriever/common/vdb/hybrid_fusion.py	Introduces a validated weighted-RRF policy and LanceDB reranker implementation.
nemo_retriever/src/nemo_retriever/ingest/index_mode.py	Centralizes validation and resolution of automatic, dense, hybrid, and sparse ingest modes.
nemo_retriever/src/nemo_retriever/ingest/plan.py	Resolves automatic index mode against existing physical table capabilities before constructing the ingest plan.
nemo_retriever/src/nemo_retriever/service/vectordb_app.py	Enables automatic hybrid service storage and query fusion, while the prior health-probe availability issue remains reachable through backend telemetry.
nemo_retriever/src/nemo_retriever/common/vdb/lancedb_capabilities.py	Makes physical vector and FTS capabilities authoritative over stale schema metadata.
nemo_retriever/helm/templates/deployment-vectordb.yaml	Validates the configured VectorDB index mode and forwards it to the service process.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Ingest request] --> B{Table exists and append?}
    B -- No --> C[Resolve auto to hybrid]
    B -- Yes --> D[Inspect physical indexes]
    D --> E[Preserve dense, hybrid, or sparse mode]
    C --> F[Write vectors and text]
    E --> F
    F --> G[Create or maintain FTS index]
    H[Query with mode auto] --> I[Inspect vector and FTS capabilities]
    I --> J{Hybrid capable?}
    J -- Yes --> K[Dense search plus BM25 search]
    K --> L[Weighted RRF fusion]
    J -- No --> M[Use supported physical mode]

Loading

_{Reviews (9): Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." | Re-trigger Greptile}