Persist LanceDB index metadata and resolve query embedding models automatically

[jioffe502]

Problem

The split local/batch retriever ingest and non-service retriever query workflows currently require callers to repeat embedding configuration. A query can therefore silently use a different embedding model from the one that produced the selected LanceDB table.

There is also a prefix-only routing bug: when --embed-model-provider-prefix is supplied without --embed-model-name, the prefix can be discarded before the effective model is resolved.

Service deployments have a related model-identity problem, but their VectorDB ownership and configuration flow are separate from the local/batch path. This issue tracks that work as a distinct follow-up phase; Phase 1 does not modify service ingest, the VectorDB service, or MCP behavior.

Metadata contract

Store these non-secret, namespaced values in the LanceDB Arrow schema:

• nemo_retriever.index_format_version = "1"
• nemo_retriever.producer_version = get_version_info()["full_version"]
• nemo_retriever.embedding_model_name = <canonical logical model ID>
• nemo_retriever.retrieval_mode = dense|hybrid|sparse

The embedding identity is the canonical logical model used for ingest, resolved from known aliases before provider routing. Endpoint URLs, provider prefixes, API keys, and other credentials must never be stored.

If a caller supplies an already-routed model name without the separate prefix option, retain it as supplied; do not heuristically strip path segments from arbitrary model identifiers.

The shared LanceDB capability inspector should expose optional index-format, producer-version, and embedding-model fields. Legacy and third-party tables without these fields surface None.

Delivery phases

Phase 1: local/batch persistence and non-service query resolution

This is the scope of the initial implementation:

• Local and batch ingest persist the metadata contract.
• Root non-service query, direct Retriever callers, and agentic callers resolve the embedding model from the selected LanceDB table.
• Existing CLI flags and query JSON output remain unchanged.
• Service ingest, VectorDBState, service health/introspection, service query behavior, and the MCP tool schema remain untouched.

A Phase 1 pull request should reference this issue but should not close it.

Phase 2: service and MCP follow-up

After reviewing ownership in the service VDB path:

• Persist and inspect the same non-secret metadata at the service's LanceDB boundary.
• Define how an unconfigured service adopts the table model and how an explicitly configured conflicting model is rejected before embedding or append.
• Expose non-secret index metadata through the existing redacted service introspection surface.
• Keep the MCP query tool free of embedding-model parameters; the service should own model selection and validation.

This issue remains open until the service follow-up is complete, or that work is moved to a separately linked issue.

Phase 1 behavior

Ingest

• The resolved ingest plan owns the canonical effective embedding model, including the default.
• Sparse ingest owns no embedding model and writes no embedding-model field.
• Create and overwrite write the metadata contract.
• Append accepts matching metadata, rejects a known embedding-model mismatch before writing, and warns while preserving behavior for legacy tables without metadata.

Query

Resolve the embedding model inside Retriever, after table inspection and before embedding, using this precedence:

1. Per-call or explicit CLI/SDK model
2. EMBED_MODEL_NAME
3. Selected-table metadata
4. Built-in default

Explicit and environment overrides retain precedence. Warn when their canonical identity conflicts with table metadata.

Apply EMBED_MODEL_PROVIDER_PREFIX only after the effective model is resolved. Support EMBED_INVOKE_URL, EMBED_MODEL_NAME, and EMBED_MODEL_PROVIDER_PREFIX consistently across local/batch ingest and non-service query.

Sparse retrieval remains embedding-free. Metadata-free legacy tables retain the built-in fallback. Reject unsupported non-null index-format versions.

Phase 1 acceptance

• Local and batch ingest persist canonical model identity and non-secret provenance metadata.
• Dense, hybrid, and sparse schemas expose the expected metadata; sparse schemas omit the embedding model.
• Known model aliases are canonicalized before provider routing.
• Prefix-only endpoint configuration routes the model resolved from table metadata.
• Matching append succeeds, known model mismatch fails before writing, and legacy append remains supported with a warning.
• Model-omitted root and direct Retriever queries use selected-table metadata.
• Explicit/environment precedence and mismatch warnings are preserved.
• Live dense and hybrid ingest-to-query checks succeed without a query-side model flag.
• Metadata-free tables still query through the built-in fallback.
• Existing CLI flags and query JSON output remain unchanged.
• No service or MCP behavior changes in Phase 1.

Phase 2 acceptance

• Service ingest persists or preserves the metadata contract at the service VDB boundary.
• Service query adopts or validates the selected table's model before embedding.
• Conflicting explicit service configuration produces a clear compatibility error.
• Non-secret index metadata is available through redacted service introspection.
• MCP query behavior uses the service-owned model without adding embedding parameters to the tool schema.

Probe experiment

Versioned probe string: nemo-retriever embedding identity probe v1.

Local Hugging Face runs were deterministic within each model, with repeat cosine similarity of 1.0 across three runs. The VL and text embedding models produced a cross-model cosine similarity of approximately 0.9581 for this single probe. Build and Inference Hub comparisons were unavailable because the development environment did not have remote endpoint credentials.

Exact output hashes are therefore unsuitable across providers, and the local-only result does not support a sufficiently discriminative tolerant signature. Keep the probe as a possible future verification mechanism; do not gate v1 model selection on it.

Future manifest

A future working-directory manifest may assist URI/table discovery only. It must never override metadata from the selected LanceDB table.