diff --git a/README.md b/README.md index c80093c2aa..6fe8b82e74 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,6 @@ Cat is the animal whose activity (jumping onto a laptop) matches the location of - **[Official Documentation](https://docs.nvidia.com/nemo/retriever/extraction/)** - Complete user guides, API references, and deployment instructions - **[Getting Started Guide](https://docs.nvidia.com/nemo/retriever/extraction/overview/)** - Overview and prerequisites for production deployments -- **[Retriever Harness](nemo_retriever/harness/README.md)** - Repeatable end-to-end ingest and retrieval benchmarks - **[Stage Benchmarking](nemo_retriever/docs/cli/benchmarking.md)** - Internal per-stage throughput measurements - **[MIG Deployment](nemo_retriever/helm/README.md)** - Multi-Instance GPU configurations for Kubernetes - **[API Documentation](docs/docs/extraction/nemo-retriever-api-reference.md)** - Python client and API reference diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index 0ae72592b1..4bbfbafb8a 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -181,7 +181,7 @@ When you call [NVIDIA-hosted NIMs](deployment-options.md#when-to-use-nvidia-host | nemotron-3-nano-omni-30b-a3b-reasoning | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | Optional image captioning. Also a supported configurable `/v1/answer` VLM backend when you point `serviceConfig.llm` at this endpoint. Enabling the Omni caption Helm key does not enable `/v1/answer`. | | llama-3.3-nemotron-super-49b-v1.5 | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Default optional `/v1/answer` LLM (Helm `answer_llm`) and OpenAI-compatible agentic RAG endpoint mode. Not part of the default extraction pipeline. Agentic query and harness runs default to local in-process vLLM instead. Helm auto-wires to the in-cluster NIM when `nimOperator.answer_llm` is enabled. Refer to [Answer generation](#answer-generation). | | nemotron-3-nano-omni-30b-a3b-reasoning | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | Optional image captioning | -| llama-3.3-nemotron-super-49b-v1.5 | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Optional `/v1/answer` (Helm `answer_llm`) and OpenAI-compatible agentic RAG endpoint mode; not part of the default extraction pipeline. Agentic query/harness runs default to local in-process vLLM instead. Helm auto-wires to the in-cluster NIM when `nimOperator.answer_llm` is enabled. Refer to [Workflow: Agentic retrieval](workflow-agentic-retrieval.md). | +| llama-3.3-nemotron-super-49b-v1.5 | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Optional `/v1/answer` (Helm `answer_llm`) and OpenAI-compatible agentic RAG endpoint mode; not part of the default extraction pipeline. Agentic query and NRB benchmark runs default to local in-process vLLM instead. Helm auto-wires to the in-cluster NIM when `nimOperator.answer_llm` is enabled. Refer to [Workflow: Agentic retrieval](workflow-agentic-retrieval.md). | | parakeet-1-1b-ctc-en-us | `grpc.nvcf.nvidia.com:443` (function ID from [build.nvidia.com](https://build.nvidia.com/)) | Optional ASR; refer to [Parakeet hosted inference](audio-video.md#parakeet-hosted-inference-build-nvidia) | diff --git a/docs/docs/extraction/releasenotes.md b/docs/docs/extraction/releasenotes.md index 4676ce364d..a2aea35efd 100644 --- a/docs/docs/extraction/releasenotes.md +++ b/docs/docs/extraction/releasenotes.md @@ -67,8 +67,9 @@ The following sections summarize user-visible changes introduced in 26.08. Capab ### CLI { #cli } -- `retriever ingest` and `retriever query` replace the retired compatibility pipeline command. Other top-level subcommands, including `eval`, `benchmark`, `harness`, and `skill-eval`, are development and experimental. -- `retriever query --agentic` is the CLI entry point for agentic retrieval. Refer to [Workflow: Agentic retrieval](workflow-agentic-retrieval.md). +- Root CLI adds first-class `retriever ingest` and `retriever query` commands with NIM URL flags, batch tuning, and LanceDB overwrite/append controls +- `retriever query --agentic` runs an LLM-driven ReAct retrieval loop over the same LanceDB table as one-pass retrieval. Local CLI and NRB benchmark runs default to in-process vLLM (`nemotron-8b`). Remote OpenAI-compatible NIM or NVIDIA-hosted endpoints use `--agentic-invoke-url`. Refer to [Workflow: Agentic retrieval](workflow-agentic-retrieval.md). +- `retriever ingest` and `retriever query` replace the retired compatibility pipeline command. Other top-level subcommands—including `eval`, `benchmark`, and `skill-eval`—are development and experimental. ### Retriever Service and deployment { #retriever-service-and-deployment } diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index 144820baec..95c24f2573 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -10,7 +10,10 @@ Use this workflow after you have ingested documents into a LanceDB table. Agenti ### Local in-process vLLM { #local-in-process-vllm } -The CLI and harness BEIR agentic paths default to an in-process local vLLM agent LLM. If you omit `--agentic-llm-model` and `--agentic-invoke-url`, the library loads `nemotron-8b` (`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`) on the local CUDA host. This requires a Linux CUDA GPU and the `[local]` extra. +The CLI and NRB agentic benchmark paths default to an in-process local vLLM agent +LLM. If you omit `--agentic-llm-model` and `--agentic-invoke-url`, the library +loads `nemotron-8b` (`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`) on the local CUDA +host. This requires a Linux CUDA GPU and the `[local]` extra. GPU placement follows process-level vLLM behavior. Set `CUDA_VISIBLE_DEVICES` before you start the command. diff --git a/evaluation/bo767_recall.md b/evaluation/bo767_recall.md deleted file mode 100644 index ff0f1b2dd2..0000000000 --- a/evaluation/bo767_recall.md +++ /dev/null @@ -1,27 +0,0 @@ -# Evaluate BO767 Retrieval - -BO767 is a checked-in batch benchmark. Configure the host's dataset paths, dry -run the exact request, and then execute it. - -```bash -cp nemo_retriever/harness/dataset_paths.example.yaml \ - /local/path/to/dataset_paths.yaml -${EDITOR:-vi} /local/path/to/dataset_paths.yaml - -uv run --project nemo_retriever retriever harness run-files \ - --session-name bo767_beir \ - --output-dir /local/path/to/retriever-artifacts/bo767-beir \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - --dry-run \ - nemo_retriever/harness/runfiles/bo767_beir.json -``` - -Confirm that `session_summary.json` succeeds and inspect the child -`resolved_benchmark.json`. Remove `--dry-run` to execute the benchmark. - -Read `session_summary.json` first and the child `results.json` for terminal -metrics. Use `run.log` only when deeper diagnostics are needed. - -- [Library harness guide](../nemo_retriever/harness/docs/library.md) -- [BO767 dataset facts and observations](../nemo_retriever/harness/docs/expected-results.md#bo767-observations) -- [Shared artifact contract](../nemo_retriever/harness/README.md#results-and-artifacts) diff --git a/examples/README.md b/examples/README.md index 79117823ea..24e87b7070 100644 --- a/examples/README.md +++ b/examples/README.md @@ -18,6 +18,5 @@ For advanced scenarios, use these guides and notebooks: - [Build a Custom Vector Database Operator](building_vdb_operator.ipynb) - [Try Enterprise RAG Blueprint](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) -- [Evaluate BO767 retrieval with the Retriever harness](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.md) - [Multimodal RAG with LangChain](langchain_multimodal_rag.ipynb) - [Multimodal RAG with LlamaIndex](llama_index_multimodal_rag.ipynb) diff --git a/nemo_retriever/README.md b/nemo_retriever/README.md index 31ba19a53f..c0776cbcca 100644 --- a/nemo_retriever/README.md +++ b/nemo_retriever/README.md @@ -829,14 +829,12 @@ sudo apt install python3.12-dev After installing the headers, restart the pipeline. -## Retriever Harness - -The developer harness runs registered ingest and retrieval benchmarks through -`retriever harness`. Start with the -[harness guide](harness/README.md), then choose -[library execution](harness/docs/library.md) or -[service execution](harness/docs/service.md). Recurring workstation runs use the -[nightly launcher](../ops/retriever-nightly/README.md). +## Benchmarking + +End-to-end Retriever experiments and benchmark orchestration are maintained in +the [NeMo Retriever Benchmark (NRB) repository](https://gitlab-master.nvidia.com/charlesb/nemo-retriever-benchmark/). +This repository continues to provide the library, CLI workflows, service +implementation, and Helm chart that NRB benchmarks. ### Ingest image storage diff --git a/nemo_retriever/developer_docs/README.md b/nemo_retriever/developer_docs/README.md index e76f1ec990..96e2935fc4 100644 --- a/nemo_retriever/developer_docs/README.md +++ b/nemo_retriever/developer_docs/README.md @@ -9,6 +9,4 @@ architecture, subsystems, and developer-facing tools. |-------|-------------| | [Graph Pipeline Registry](graph_pipeline_registry.md) | Central registry for managing, inspecting, comparing, and serializing golden pipeline graphs. | | [NimClient and Custom NIM Endpoints](nimclient.md) | Developer guide for custom NIM integrations with `NimClient`, `ModelInterface`, and UDFs. | -| [Retriever Harness](../harness/README.md) | Current user and agent guide for registered Retriever benchmarks. | -| [Retriever Harness Design History](harness_retriever_ingest_query_prd.md) | Implemented design decisions and product boundaries. | | [Root Ingest CLI Design](root_ingest_cli_design.md) | Reviewer guide for the `retriever ingest` local, batch, and service CLI ownership split. | diff --git a/nemo_retriever/developer_docs/harness_retriever_ingest_query_prd.md b/nemo_retriever/developer_docs/harness_retriever_ingest_query_prd.md deleted file mode 100644 index 4fc69440a4..0000000000 --- a/nemo_retriever/developer_docs/harness_retriever_ingest_query_prd.md +++ /dev/null @@ -1,45 +0,0 @@ -# Retriever Harness Design History - -> This design has been implemented. The -> [Retriever Harness README](../harness/README.md) is the current user and agent -> contract. - -The harness was rebuilt as an internal end-to-end benchmark runner for NeMo -Retriever engineers. It ingests registered datasets, queries the resulting -index, evaluates retrieval quality, and writes durable artifacts. - -## Decisions That Still Apply - -| Decision | Consequence | -| --- | --- | -| Keep benchmark definitions in a typed Python registry. | Recurring benchmarks are reviewed code, not an open-ended YAML system. | -| Treat runfiles as concrete requests. | A runfile selects a registered benchmark; it cannot define a new one. | -| Keep one harness contract across execution targets. | Library and service runs emit the same results, gates, and artifacts. | -| Treat local and batch as library ingest modes. | Batch changes how ingest runs; it is not a separate harness. | -| Treat service as a system-under-test mode. | The harness uses service ingest and query APIs while retaining the same evaluation contract. | -| Keep Helm outside benchmark semantics. | `run-helm` provisions around `run-files`; Helm is not a runfile mode. | -| Make artifacts the API. | Callers use exit codes, `status.json`, `results.json`, and `session_summary.json`, not stdout. | -| Keep execution separate from reporting. | `post-slack` reads completed artifacts and never reruns or mutates them. | -| Keep scheduling outside the harness. | The nightly launcher and its caller own recurrence, locking, Git selection, and secrets. | -| Require explicit gates. | The harness records quality and performance but does not impose a global score. | - -## Product Boundary - -Use `retriever ingest` and `retriever query` for direct product workflows. Use -`retriever harness` for registered benchmark and evaluation work. - -The harness calls the same library workflow code or corresponding service APIs -used by the product. It owns benchmark resolution, run lifecycle, evaluation, -metric gates, and artifacts. Retriever owns ingest and query behavior. - -## Non-Goals - -The harness is not: - -- a public supported product API -- a scheduler, retry system, or secret distributor -- a benchmark history database or regression-policy engine -- a general deployment manager -- a compatibility wrapper for retired pipeline or sweep commands - -Git history retains the original implementation PRD and its research notes. diff --git a/nemo_retriever/developer_docs/root_ingest_cli_design.md b/nemo_retriever/developer_docs/root_ingest_cli_design.md index db00913b88..7d60cce1ae 100644 --- a/nemo_retriever/developer_docs/root_ingest_cli_design.md +++ b/nemo_retriever/developer_docs/root_ingest_cli_design.md @@ -2,7 +2,7 @@ This note explains the `retriever ingest` redesign for reviewers and future CLI maintainers. It is intentionally scoped to the root ingest CLI. It does not -change `GraphIngestor`, `Retriever.query`, eval, harness logic, BEIR/QA/audio +change `GraphIngestor`, `Retriever.query`, eval, benchmark logic, BEIR/QA/audio recall, or pipeline reporting. ## Summary diff --git a/nemo_retriever/docs/cli/README.md b/nemo_retriever/docs/cli/README.md index 875e317c43..0e2a043434 100644 --- a/nemo_retriever/docs/cli/README.md +++ b/nemo_retriever/docs/cli/README.md @@ -8,7 +8,6 @@ For product-facing examples, prefer these commands: - `retriever ingest` - ingest supported documents and media into a Retriever index. - `retriever query` - query a local LanceDB table written by local or batch ingest. - `retriever query service` - query a Retriever service deployment. -- `retriever harness run` - run a named, code-owned benchmark. - `retriever service` - operate a Retriever service deployment. Format names and internal stages are not root commands. Use `retriever ingest` @@ -48,7 +47,6 @@ configuration, local embed backend selection, or local media controls. > Use `retriever ingest` and `retriever query` for product-facing workflows. -> Use `retriever harness run` only for benchmark execution. ## Quick start diff --git a/nemo_retriever/docs/cli/benchmarking.md b/nemo_retriever/docs/cli/benchmarking.md index 078358d3ea..a610623ec9 100644 --- a/nemo_retriever/docs/cli/benchmarking.md +++ b/nemo_retriever/docs/cli/benchmarking.md @@ -1,33 +1,10 @@ # Benchmarking with the Retriever CLI -Retriever has two development benchmarking surfaces: - -| Goal | Command | Documentation | -| --- | --- | --- | -| End-to-end ingest, query, and retrieval evaluation | `retriever harness` | [Retriever Harness](../../harness/README.md) | -| Throughput for one internal pipeline stage | `retriever benchmark` | [Stage micro-benchmarks](#stage-micro-benchmarks) | - -For product workflows on your own inputs, use `retriever ingest` and -`retriever query` instead. - -## End-to-End Benchmarks - -The harness owns registered datasets, repeatable runfiles, metric gates, and -stable artifacts. Start with: - -```bash -retriever harness list --runsets -retriever harness show jp20_beir -``` - -Then choose the execution guide: - -- [Local and batch library runs](../../harness/docs/library.md) -- [Existing-service and managed-Helm runs](../../harness/docs/service.md) -- [Recurring workstation nightly](../../../ops/retriever-nightly/README.md) - -Agentic BEIR evaluation is documented under -[library harness runs](../../harness/docs/library.md#evaluate-agentic-retrieval). +End-to-end experiments are maintained in the [NeMo Retriever Benchmark (NRB) +repository](https://gitlab-master.nvidia.com/charlesb/nemo-retriever-benchmark/). +The product CLI retains internal stage micro-benchmarks for focused development +measurements. For product workflows on your own inputs, use +`retriever ingest` and `retriever query`. ## Stage Micro-Benchmarks @@ -54,5 +31,5 @@ retriever benchmark extract ./data/pdf_corpus \ ``` Stage commands report rows per second, or chunk rows per second for audio. They -do not produce the harness artifact contract and should not be used as retrieval +do not produce the NRB artifact contract and should not be used as retrieval quality evidence. diff --git a/nemo_retriever/harness/README.md b/nemo_retriever/harness/README.md deleted file mode 100644 index 6ebaa8d25a..0000000000 --- a/nemo_retriever/harness/README.md +++ /dev/null @@ -1,223 +0,0 @@ - - - -# Retriever Harness - -The Retriever Harness runs registered ingest and retrieval benchmarks with -repeatable configuration and machine-readable results. Use `retriever ingest` -and `retriever query` for your own data; use `retriever harness` for benchmark -and evaluation work. - -One harness supports two execution paths: - -- **Library execution** runs Retriever directly in `local` or Ray-backed - `batch` mode. -- **Service execution** tests an existing Retriever endpoint or a temporary - Helm deployment. - -Both paths use the same runfiles, metric gates, and artifact contract. - -## Start Here - -| Goal | Guide | -| --- | --- | -| Run one small benchmark | [Library execution](docs/library.md#run-one-benchmark) | -| Run a larger or multi-GPU benchmark | [Library execution](docs/library.md#run-in-batch) | -| Test an existing Retriever service | [Service execution](docs/service.md#test-an-existing-service) | -| Provision and test a service with Helm | [Service execution](docs/service.md#provision-a-service-with-helm) | -| Run the workstation suite every day | [Nightly launcher](../../ops/retriever-nightly/README.md) | -| Review known dataset facts and observed results | [Expected results](docs/expected-results.md) | - -Run harness commands from the repository root: - -```bash -uv run --project nemo_retriever retriever harness list --runsets -uv run --project nemo_retriever retriever harness show jp20_beir --json -``` - -Use `retriever harness --help` for the complete option list. - -Service-mode benchmarks wait for remote document completion without downloading -retained result payloads. Their `rows_processed` value is the sum of -`result_rows` from successful document-completion events; the independent -VectorDB coverage checks remain the authoritative end-to-end validation. As a -result, service `ingest_secs` measures remote ingestion completion rather than -ingestion plus client-side result materialization. Service timing baselines -recorded before this behavior changed are not directly comparable. - -## Commands - -| Command | Purpose | -| --- | --- | -| `list` | List registered benchmarks and optional runsets. | -| `show` | Show one resolved benchmark definition. | -| `run` | Run one registered benchmark. | -| `run-set` | Run a registered benchmark group using registry paths. | -| `run-files` | Run one or more portable runfiles as a session. | -| `run-helm` | Provision a service, run one session, collect failure logs, and tear it down. | -| `check-vidore-access` | Check remote ViDoRe queries, qrels, and corpora without downloading them. | -| `post-slack` | Preview or post completed artifacts without rerunning a benchmark. | -| `diff` | Compare summary metrics from two completed runs. | - -`run-files` is the normal portable entrypoint. It accepts one runfile for one -benchmark or several runfiles for a suite. Real children run sequentially in -fresh processes so each run releases its model and Ray resources. - -## Configuration - -The Python registry owns benchmark definitions. A runfile selects one registered -benchmark and may set its mode, narrow overrides, and explicit metric gates: - -```json -{ - "schema_version": 1, - "benchmark": "jp20_beir", - "mode": "local", - "require": ["files==20", "pages==1940"] -} -``` - -Runfiles may be JSON or YAML. They cannot define new benchmarks. For a -`run-files` session, set `--output-dir` and `--dry-run` on the command rather -than in individual runfiles. - -Dataset locations are machine-specific. Copy -[`dataset_paths.example.yaml`](dataset_paths.example.yaml) outside the -repository, edit it, and pass the file with `--dataset-paths`: - -```bash -cp nemo_retriever/harness/dataset_paths.example.yaml \ - /local/path/to/dataset_paths.yaml -``` - -Relative paths in that file resolve from the file's directory. Settings resolve -from lowest to highest precedence: - -1. Benchmark registry defaults. -2. Runfile values. -3. Machine-local dataset paths. -4. CLI `--set` values. - -Keep credentials, webhooks, and machine-local paths out of runfiles and source -control. - -## Results and Artifacts - -Use the process exit code and terminal JSON files—not console output—to decide -whether a run succeeded: - -| File | Read it when | -| --- | --- | -| `status.json` | A run is active and you need its current phase. | -| `results.json` | One run is terminal. | -| `session_summary.json` | A `run-files` or `run-set` session is terminal. | - -Terminal files contain `success`, `exit_code`, summary metrics, and relative -paths to detailed evidence. Follow those paths only when needed: - -- `run.log` for full errors and lower-level output -- `resolved_benchmark.json`, `ingest_plan.json`, and `query_plan.json` for the - effective configuration -- `environment.json` for the source revision, GPU inventory, workload-visible - GPU count, and runtime context -- `beir_metrics.json`, `beir_run.trec`, and `query_results.jsonl` for evaluation - details -- `service_logs/` for a failed managed Helm deployment - -A multi-run session has this stable layout: - -```text -/ - expanded_runs.json - session_summary.json - 001_/ - 002_/ -``` - -Dry-runs resolve configuration and write planning artifacts, but they are not -execution evidence. - -## Gates and Exit Codes - -Metric gates compare values in `results.json.summary_metrics`: - -```bash ---require 'files==20' ---require 'recall_5>=0.85' -``` - -During a dry-run, static facts can be evaluated; execution metrics are recorded -as skipped. The harness has no implicit quality or performance threshold. -[`expected-results.md`](docs/expected-results.md) contains human-readable observations -that can inform explicit gates. - -| Code | Meaning | -| ---: | --- | -| `0` | Success | -| `2` | Invalid benchmark, configuration, override, or gate | -| `3` | Missing dataset or input | -| `4` | Managed Helm lifecycle failure | -| `10` | Ingest failure | -| `11` | Query failure | -| `12` | Evaluation failure | -| `20` | Metric gate failure | -| `30` | Artifact write failure | -| `70` | Unexpected internal error | - -## Report Completed Results - -Execution and reporting are separate. Preview a Slack report without a webhook: - -```bash -uv run --project nemo_retriever retriever harness post-slack \ - --preview \ - --title "Retriever benchmark results" \ - /path/to/session -``` - -To post, export `SLACK_WEBHOOK_URL` and remove `--preview`. Each invocation -creates a new message and never changes the completed run. - -A nightly can display the current release beside matching observations. Keep -one release snapshot outside the repository: - -```json -{ - "baselines": [ - { - "name": "RC26.05 Perflab", - "dataset": "bo767", - "environment": {"gpu_sku": "NVIDIA H100 80GB HBM3", "gpu_count": 8}, - "metrics": {"ingest_secs": 4036.85, "pages_per_sec_ingest": 13.56} - } - ] -} -``` - -Pass it with `--reference-file` or set -`RETRIEVER_HARNESS_REFERENCE_FILE`. The report shows the two observations with -their GPU context; it does not assign a verdict, update the file, or maintain -history. - -Compare two local runs without Slack: - -```bash -uv run --project nemo_retriever retriever harness diff \ - /path/to/left/results.json \ - /path/to/right/results.json -``` - -## Automation Contract - -For agents and other callers: - -1. Discover benchmarks with `list --runsets --json`. -2. Inspect the target with `show --json`. -3. Supply machine-local dataset paths outside the repository. -4. Dry-run the exact command before expensive GPU work. -5. Use the exit code and terminal JSON file as the result. -6. Read detailed artifacts only when the terminal result points to them. - -Do not parse progress bars, human CLI formatting, or raw stdout. Scheduling, -retry policy, secret distribution, and result history belong to the caller. The -[nightly launcher](../../ops/retriever-nightly/README.md) is one such caller. diff --git a/nemo_retriever/harness/docs/expected-results.md b/nemo_retriever/harness/docs/expected-results.md deleted file mode 100644 index f110447bf1..0000000000 --- a/nemo_retriever/harness/docs/expected-results.md +++ /dev/null @@ -1,132 +0,0 @@ - - - -# Harness Expected Results - -This page records dataset facts and observed benchmark results. It is not an -executable baseline or universal pass/fail policy. - -Checked-in runfiles enforce only portable integrity facts such as file, page, -and query counts. Quality and performance vary with the Retriever revision, -hardware, runtime, and model configuration. Use those observations for review, -then add an explicit `--require` gate only when the environment and intended -policy justify it. - -## Dataset Facts - -Registry paths describe the standard internal mount. Use -[`dataset_paths.example.yaml`](../dataset_paths.example.yaml) to map these datasets -on another host. - -| Dataset | Benchmark | Standard path | Files | Pages | Queries | -| --- | --- | --- | ---: | ---: | ---: | -| JP20 | `jp20_beir` | `/datasets/nv-ingest/jp20` | 20 | 1,940 | 115 | -| BO20 | — | `/datasets/nv-ingest/bo20` | 20 | — | — | -| BO767 | `bo767_beir` | `/datasets/nv-ingest/bo767` | 767 | 54,730 | 991 | -| FinanceBench | `financebench_beir` | `/datasets/nv-ingest/foundation_rag/financebench` | 369 | 54,057 | 150 | -| BO10K | Not yet defined | `/datasets/nv-ingest/bo10k` | 10,000 | — | — | -| Earnings Consulting | `earnings_beir` | `/datasets/nv-ingest/earnings_consulting_flattened` | 514 | 12,988 | 628 | - -The query files are: - -| Dataset | Query or qrels file | -| --- | --- | -| JP20 | `data/jp20_query_gt.csv` | -| BO767 | `data/bo767_query_gt.csv` | -| FinanceBench | `data/financebench_train.json` | -| BO10K | `data/digital_corpora_10k_annotations.csv` | -| Earnings Consulting | `data/earnings_consulting_multimodal.csv` | - -## JP20 Observations - -Recent `jp20_beir` runs on local hardware: - -| Metric | Observed | -| --- | ---: | -| Rows processed | 3,154 | -| Ingest seconds | 215–223 | -| Query p50 | 909–915 ms | -| Query p95 | 953–1,003 ms | -| Recall@5 | 0.878–0.887 | -| Recall@10 | 0.930–0.948 | -| nDCG@10 | 0.793–0.802 | - -The original observations did not record the GPU SKU or workload-visible GPU -count. Treat their latency and throughput as context only. - -## BO767 Observations - -Runs on an eight-H100 80GB HBM3 host: - -| Configuration | Workload GPUs | Rows | Ingest seconds | Pages/s | Recall@5 | Recall@10 | nDCG@10 | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| RC26.05 Perflab | Not recorded | 79,221 | 4,036.847 | 13.56 | — | — | — | -| Automatic batch | 1 | 79,229 | 1,594.339 | 34.328 | 0.848638 | 0.897074 | 0.750110 | -| Automatic batch | 8 | 79,230 | 764.946 | 71.548 | 0.849647 | 0.895055 | 0.748583 | -| Legacy worker-capped batch | 8 visible, effectively 1 used | 79,230 | about 2,265 | about 24.16 | 0.850656 | 0.896065 | 0.751507 | - -The RC26.05 artifact recorded eight physical GPUs but not the number visible to -the workload. Current runfiles leave worker counts and batch sizes automatic so -the planner can scale to available GPUs. The one- and eight-GPU automatic runs -differed by one output row; keep row counts and quality visible when comparing -throughput. - -## ViDoRe V3 - -The eight public ViDoRe v3 benchmarks use original PDFs with: - -- `nvidia/llama-nemotron-embed-vl-1b-v2` -- `text_image` page embeddings -- page-image and infographic extraction -- page-level BEIR document IDs - -Integrity facts: - -| Dataset | Files | Pages | Queries | -| --- | ---: | ---: | ---: | -| Computer Science | 2 | 1,360 | 1,290 | -| Energy | 41 | 2,225 | 1,848 | -| Finance EN | 6 | 2,942 | 1,854 | -| Finance FR | 5 | 2,384 | 1,920 | -| HR | 14 | 1,110 | 1,908 | -| Industrial | 27 | 5,244 | 1,698 | -| Pharmaceuticals | 52 | 2,313 | 2,184 | -| Physics | 42 | 1,674 | 1,812 | - -Observed complete batch runs on an eight-H100 DGX: - -| Dataset | Indexed rows | Ingest seconds | Pages/s | Recall@5 | Recall@10 | nDCG@10 | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| Computer Science | 1,358 | 100.2–123.5 | 11.0–13.6 | 0.599–0.600 | 0.729–0.730 | 0.708–0.709 | -| Energy | 2,211 | 116.7 | 19.1 | 0.575 | 0.674 | 0.581 | -| Finance EN | 2,927 | 149.4 | 19.7 | 0.496 | 0.609 | 0.547 | -| Finance FR | 2,149 | 106.4 | 22.4 | 0.324 | 0.426 | 0.345 | -| HR | 1,091 | 82.6 | 13.4 | 0.452 | 0.574 | 0.530 | -| Industrial | 5,039 | 137.5 | 38.1 | 0.348 | 0.426 | 0.381 | -| Pharmaceuticals | 2,290 | 93.7 | 24.7 | 0.547 | 0.647 | 0.607 | -| Physics | 1,674 | 89.2 | 18.8 | 0.369 | 0.485 | 0.451 | - -Computer Science was run twice; the other domains have one complete observation. -The observed all-domain macro-average nDCG@10 was about `0.519`. The observations -predate separate physical and workload-visible GPU counts, so do not infer -scaling behavior from their throughput. - -### RC26.05 Recall Comparison - -| Dataset | RC26.05 Recall@5 | Observed Recall@5 | -| --- | ---: | ---: | -| Finance EN | 0.499 | 0.496 | -| Industrial | 0.348 | 0.348 | -| Computer Science | 0.600 | 0.599 | -| Pharmaceuticals | 0.549 | 0.547 | -| HR | 0.453 | 0.452 | -| Energy | 0.577 | 0.575 | -| Physics | 0.367 | 0.369 | -| Finance FR | 0.324 | 0.324 | - -The observed macro-average Recall@5 was `0.464` across all eight domains. - -Finance FR omitted 235 empty-text pages, including 69 judged image-only pages, -because the current dense LanceDB path drops records without text even when an -image embedding exists. Its Recall@5 still matched RC26.05, but preserving those -judged image-only pages remains correctness work. diff --git a/nemo_retriever/harness/docs/library.md b/nemo_retriever/harness/docs/library.md deleted file mode 100644 index 4d204e39d2..0000000000 --- a/nemo_retriever/harness/docs/library.md +++ /dev/null @@ -1,159 +0,0 @@ - - - -# Library Harness Runs - -Use library execution to benchmark the current NeMo Retriever checkout directly. -Choose the mode by workload size: - -| Mode | Use it for | Execution | -| --- | --- | --- | -| `local` | Small smoke tests such as JP20 | In-process ingest | -| `batch` | Larger corpora and multi-GPU runs | Ray-backed batch ingest | - -Both modes write the [same artifacts](../README.md#results-and-artifacts). - -## Prepare the Host - -You need the repository dependencies, NVIDIA drivers, the selected datasets, -and enough storage for artifacts. Run commands from the repository root through -the `nemo_retriever` project. - -Dataset mounts vary by machine: - -```bash -cp nemo_retriever/harness/dataset_paths.example.yaml \ - /local/path/to/dataset_paths.yaml -${EDITOR:-vi} /local/path/to/dataset_paths.yaml -``` - -Keep this file outside the repository. It maps registered dataset names to local -corpus paths and any annotations that are not checked in. JP20 uses the -repository's `data/jp20_query_gt.csv` unless `query_file` is explicitly -overridden. - -## Run One Benchmark - -Start with the checked-in JP20 runfile: - -```bash -uv run --project nemo_retriever retriever harness run-files \ - --session-name jp20_check \ - --output-dir /tmp/retriever-harness-jp20-check \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - --dry-run \ - nemo_retriever/harness/runfiles/jp20_beir.json -``` - -Confirm that `session_summary.json` succeeds and inspect the child -`resolved_benchmark.json`. Then run the same command without `--dry-run`. - -If the registry paths already exist on the host, `run` is a shorter single-run -form: - -```bash -uv run --project nemo_retriever retriever harness run jp20_beir \ - --output-dir /tmp/retriever-harness-jp20 \ - --require 'files==20' \ - --require 'pages==1940' -``` - -## Run in Batch - -Checked-in runfiles for BO767, FinanceBench, Earnings, and ViDoRe already select -`batch`. Run one of them exactly as you ran JP20: - -```bash -uv run --project nemo_retriever retriever harness run-files \ - --session-name bo767_beir \ - --output-dir /local/path/to/retriever-artifacts/bo767-beir \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - nemo_retriever/harness/runfiles/bo767_beir.json -``` - -Pass several runfiles to create one sequential session: - -```bash -uv run --project nemo_retriever retriever harness run-files \ - --session-name library_beir \ - --output-dir /local/path/to/retriever-artifacts/library-beir \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - nemo_retriever/harness/runfiles/jp20_beir.json \ - nemo_retriever/harness/runfiles/bo767_beir.json \ - nemo_retriever/harness/runfiles/earnings_beir.json \ - nemo_retriever/harness/runfiles/financebench_beir.json -``` - -Children run sequentially in fresh processes. Ray and model resources are -released between datasets, while the parent writes one -`session_summary.json`. - -## Run ViDoRe - -ViDoRe queries, qrels, and corpus metadata require Hugging Face access. Check it -before GPU work: - -```bash -export HF_TOKEN=... -uv run --project nemo_retriever retriever harness check-vidore-access -``` - -Then run one domain with its checked-in runfile: - -```bash -uv run --project nemo_retriever retriever harness run-files \ - --session-name vidore_v3_computer_science \ - --output-dir /local/path/to/retriever-artifacts/vidore-v3-computer-science \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - nemo_retriever/harness/runfiles/vidore_v3_computer_science_beir.json -``` - -The [nightly launcher](../../../ops/retriever-nightly/README.md) runs all eight -ViDoRe domains plus the four library benchmarks. - -## Apply a One-Off Override - -Use repeated `--set KEY=VALUE` options for an experiment: - -```bash -uv run --project nemo_retriever retriever harness run jp20_beir \ - --set query.top_k=20 \ - --set query.rerank=true -``` - -Supported namespaces are `dataset.*`, `ingest.*`, `query.*`, and -`evaluation.*`. Unknown keys fail before execution. Use `show --json` and -`run --dry-run --json` to inspect the resolved configuration. - -For reproducible recurring changes, put the values in a reviewed runfile or -benchmark definition rather than a shell command. - -## Evaluate Agentic Retrieval - -Agentic BEIR evaluation uses the same retrieval graph as `retriever query ---agentic` after ingest: - -```bash -uv run --project nemo_retriever retriever harness run jp20_beir \ - --set query.agentic=true -``` - -The default agent LLM runs locally. To use an OpenAI-compatible endpoint: - -```bash -uv run --project nemo_retriever retriever harness run jp20_beir \ - --set query.agentic=true \ - --set query.agentic_llm_model=custom-remote-model \ - --set query.agentic_invoke_url=http://localhost:9000/v1/chat/completions -``` - -Use `query.agentic_llm_client`, `query.agentic_react_max_steps`, -`query.agentic_num_concurrent`, and the local vLLM resource overrides only when -the experiment requires them. The resolved benchmark records every value. - -## Run on a Schedule - -The harness itself is one-shot. Use the -[nightly launcher](../../../ops/retriever-nightly/README.md) for the checked-in -twelve-benchmark suite, Git selection, Slack reporting, and a transparent daily -`tmux` loop. diff --git a/nemo_retriever/harness/docs/service.md b/nemo_retriever/harness/docs/service.md deleted file mode 100644 index ea191c6e41..0000000000 --- a/nemo_retriever/harness/docs/service.md +++ /dev/null @@ -1,108 +0,0 @@ - - - -# Service Harness Runs - -Service execution runs the same registered benchmarks through Retriever service -ingest and query APIs. It writes the same artifacts and evaluates the same -metric gates as library execution. - -Choose how the service is provided: - -| Target | Command | -| --- | --- | -| An already-running service | `retriever harness run-files --mode service` | -| A temporary service provisioned by the harness | `retriever harness run-helm` | - -Helm is a provisioning wrapper, not a fourth benchmark mode. - -## Test an Existing Service - -The service must be reachable from the harness host. Export its bearer token -only when authentication is enabled: - -```bash -export HARNESS_SERVICE_API_TOKEN=... -``` - -Run a checked-in benchmark against the endpoint: - -```bash -uv run --project nemo_retriever retriever harness run-files \ - --mode service \ - --service-endpoint http://localhost:7670 \ - --session-name service_jp20 \ - --output-dir /local/path/to/retriever-artifacts/service-jp20 \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - nemo_retriever/harness/runfiles/jp20_beir.json -``` - -`--mode service` overrides the runfile mode for the session. -`--service-endpoint` applies only to service-mode children, so mixed sessions -remain possible. If omitted, the endpoint defaults to -`http://localhost:7670`. - -The dataset paths identify documents and queries available to the harness -client. Service storage and infrastructure remain owned by the service. - -## Provision a Service with Helm - -`run-helm` deploys one immutable service image, waits for readiness, establishes -a local port-forward, runs the shared `run-files` session in service mode, -collects logs on failure, and tears the release down. - -Prerequisites: - -- working `helm` and `kubectl` commands, or equivalents selected in the config -- a Kubernetes cluster with the required image-pull and NGC secrets -- an immutable service image built from the checkout -- a non-secret Helm deployment file - -Start from -[`examples/managed-helm-main.yaml`](../examples/managed-helm-main.yaml). Select a -checked-in profile such as [`helm-profiles/core.yaml`](../helm-profiles/core.yaml) -with `helm_values_file`. - -Set the immutable image: - -```bash -export HARNESS_HELM_SERVICE_IMAGE_REPOSITORY=nvcr.io/example/nrl-service -export HARNESS_HELM_SERVICE_IMAGE_TAG= -``` - -Then run: - -```bash -uv run --project nemo_retriever retriever harness run-helm \ - --config nemo_retriever/harness/examples/managed-helm-main.yaml \ - --output-dir /local/path/to/retriever-artifacts/helm-jp20 \ - --session-name helm_jp20 \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - nemo_retriever/harness/runfiles/jp20_beir.json -``` - -The deployment file may configure the chart, release, namespace, values file, -`helm_set` overrides, timeouts, local service port, command paths, and whether -Helm or kubectl requires `sudo`. Keep credentials out of it. - -Image tags named `latest`, `main`, or `nightly` are rejected because they are -not reproducible. `run-helm` returns the benchmark session status unless -deployment or teardown fails, in which case it returns `4`. - -## Inspect and Report Results - -Read `session_summary.json` first. Failed managed sessions may also contain -`service_logs/`. The artifact tree records the benchmark configuration and -runtime environment. Keep the immutable image reference and non-secret Helm -configuration with the surrounding job record. - -Reporting is a separate step: - -```bash -uv run --project nemo_retriever retriever harness post-slack \ - --preview \ - /local/path/to/retriever-artifacts/helm-jp20 -``` - -See the [shared artifact and reporting contract](../README.md#results-and-artifacts) -for exit codes, Slack posting, and release comparisons. diff --git a/nemo_retriever/src/nemo_retriever/harness/cli.py b/nemo_retriever/src/nemo_retriever/harness/cli.py index 5a0aa0369c..e9d3c830e6 100644 --- a/nemo_retriever/src/nemo_retriever/harness/cli.py +++ b/nemo_retriever/src/nemo_retriever/harness/cli.py @@ -5,7 +5,6 @@ from __future__ import annotations import json -import os from pathlib import Path from typing import Annotated @@ -24,7 +23,6 @@ ) from nemo_retriever.harness.diff import diff_artifact_dirs from nemo_retriever.harness.resolution import make_run_id -from nemo_retriever.harness.release_reference import load_release_references from nemo_retriever.harness.runfile import load_runfile from nemo_retriever.harness.runsets import run_runfiles, run_runset from nemo_retriever.harness.slack import ( @@ -378,13 +376,6 @@ def post_slack_command( bool, typer.Option("--artifact-paths/--no-artifact-paths", help="Include local artifact paths in the Slack post."), ] = False, - reference_file: Annotated[ - Path | None, - typer.Option( - "--reference-file", - help="Current release reference snapshot to show beside observed results.", - ), - ] = None, preview: Annotated[ bool, typer.Option("--preview", help="Render the Slack payload as JSON without reading a webhook or posting."), @@ -399,11 +390,7 @@ def post_slack_command( "metric_keys": metric_keys or DEFAULT_SLACK_METRIC_KEYS, "post_artifact_paths": post_artifact_paths, } - configured_reference = reference_file - if configured_reference is None and (reference_path := os.environ.get("RETRIEVER_HARNESS_REFERENCE_FILE")): - configured_reference = Path(reference_path) - release_references = load_release_references(configured_reference) if configured_reference else None - payload = build_slack_payload(report, slack_config, release_references=release_references) + payload = build_slack_payload(report, slack_config) if not preview: post_slack_payload(payload, resolve_slack_webhook_url()) except Exception as exc: diff --git a/nemo_retriever/src/nemo_retriever/harness/release_reference.py b/nemo_retriever/src/nemo_retriever/harness/release_reference.py deleted file mode 100644 index fa0045bbc7..0000000000 --- a/nemo_retriever/src/nemo_retriever/harness/release_reference.py +++ /dev/null @@ -1,54 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024-26, NVIDIA CORPORATION & AFFILIATES. -# All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -from __future__ import annotations - -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from nemo_retriever.harness.json_io import read_json_object - - -@dataclass(frozen=True) -class ReleaseReference: - release: str - dataset: str - environment: dict[str, Any] - metrics: dict[str, int | float] - - -def load_release_references(path: Path) -> list[ReleaseReference]: - payload = read_json_object(path.expanduser().resolve()) - results = payload.get("baselines") - if not isinstance(results, list) or not results: - raise ValueError("Release reference must define a non-empty 'baselines' list") - - references: list[ReleaseReference] = [] - for index, result in enumerate(results): - if not isinstance(result, dict): - raise ValueError(f"Release result at index {index} must be an object") - release = result.get("name") - dataset = result.get("dataset") - environment = result.get("environment", {}) - metrics = result.get("metrics") - if not isinstance(release, str) or not release.strip(): - raise ValueError(f"Release result at index {index} must define non-empty 'name' text") - if not isinstance(dataset, str) or not dataset.strip(): - raise ValueError(f"Release result at index {index} must define non-empty 'dataset' text") - if not isinstance(environment, dict): - raise ValueError(f"Release result at index {index} 'environment' must be an object") - if not isinstance(metrics, dict) or not metrics: - raise ValueError(f"Release result at index {index} must define non-empty 'metrics'") - if any(isinstance(value, bool) or not isinstance(value, (int, float)) for value in metrics.values()): - raise ValueError(f"Release result at index {index} metrics must be numeric") - references.append( - ReleaseReference( - release=release.strip(), - dataset=dataset.strip(), - environment=dict(environment), - metrics=dict(metrics), - ) - ) - return references diff --git a/nemo_retriever/src/nemo_retriever/harness/slack.py b/nemo_retriever/src/nemo_retriever/harness/slack.py index 180c54ee22..91f124c277 100644 --- a/nemo_retriever/src/nemo_retriever/harness/slack.py +++ b/nemo_retriever/src/nemo_retriever/harness/slack.py @@ -6,7 +6,6 @@ from typing import Any from nemo_retriever.harness.json_io import read_json_object -from nemo_retriever.harness.release_reference import ReleaseReference DEFAULT_USERNAME = "nemo_retriever Harness" DEFAULT_ICON_EMOJI = ":satellite:" @@ -484,66 +483,9 @@ def _run_display_label(run: HarnessRunReport, repeated_datasets: set[str]) -> st return f"{run.dataset} ({workload_gpu_count} workload {gpu_label})" -def _release_reference_blocks( - report: HarnessSessionReport, - references: list[ReleaseReference], -) -> list[dict[str, Any]]: - blocks: list[dict[str, Any]] = [] - for reference in references: - for run in report.results: - if not run.success or run.dataset != reference.dataset: - continue - rows = [_three_column_row("METRIC", "CURRENT", reference.release.upper(), bold=True)] - for key in ("gpu_sku", "gpu_count", "workload_gpu_count"): - current_value = run.run_metadata.get(key) - reference_value = reference.environment.get(key) - if current_value is None and reference_value is None: - continue - rows.append( - _three_column_row( - _format_metric_label(key), - _format_metric_value(key, current_value), - _format_metric_value(key, reference_value), - ) - ) - for metric_name, reference_value in reference.metrics.items(): - rows.append( - _three_column_row( - _format_metric_label(metric_name), - _format_metric_value(metric_name, run.metrics.get(metric_name)), - _format_metric_value(metric_name, reference_value), - ) - ) - workload_gpu_count = _gpu_count(run.run_metadata.get("workload_gpu_count")) - if workload_gpu_count is None: - workload_label = "" - else: - gpu_label = "GPU" if workload_gpu_count == 1 else "GPUs" - workload_label = f" ({workload_gpu_count} workload {gpu_label})" - blocks.extend( - [ - {"type": "divider"}, - { - "type": "section", - "text": { - "type": "mrkdwn", - "text": ( - f"*{reference.release} reference — {run.dataset}{workload_label}*\n" - "Observed values only; hardware may differ and no pass/fail threshold is applied." - ), - }, - }, - {"type": "table", "rows": rows[:MAX_SLACK_TABLE_ROWS]}, - ] - ) - return blocks - - def build_slack_payload( report: HarnessSessionReport, slack_config: dict[str, Any], - *, - release_references: list[ReleaseReference] | None = None, ) -> dict[str, Any]: metric_keys = [str(key) for key in slack_config.get("metric_keys", [])] post_artifact_paths = bool(slack_config.get("post_artifact_paths", False)) @@ -682,8 +624,6 @@ def build_slack_payload( {"type": "table", "rows": _vidore_v3_accuracy_rows(vidore_v3_runs)}, ] ) - blocks.extend(_release_reference_blocks(report, release_references or [])) - return { "username": DEFAULT_USERNAME, "icon_emoji": DEFAULT_ICON_EMOJI, @@ -722,8 +662,7 @@ def post_report_to_slack( slack_config: dict[str, Any], *, webhook_url: str | None = None, - release_references: list[ReleaseReference] | None = None, ) -> dict[str, Any]: - payload = build_slack_payload(report, slack_config, release_references=release_references) + payload = build_slack_payload(report, slack_config) post_slack_payload(payload, resolve_slack_webhook_url(webhook_url)) return payload diff --git a/nemo_retriever/tests/test_harness_nightly_launcher.py b/nemo_retriever/tests/test_harness_nightly_launcher.py index e00fe101dd..72ba16808a 100644 --- a/nemo_retriever/tests/test_harness_nightly_launcher.py +++ b/nemo_retriever/tests/test_harness_nightly_launcher.py @@ -70,12 +70,6 @@ def nightly_launcher(tmp_path: Path): "expected_hf_token = os.environ.get('EXPECT_HF_TOKEN')", "if expected_hf_token is not None and os.environ.get('HF_TOKEN') != expected_hf_token:", " raise SystemExit(95)", - "expected_reference = os.environ.get('EXPECT_REFERENCE_FILE')", - ( - "if expected_reference is not None and " - "os.environ.get('RETRIEVER_HARNESS_REFERENCE_FILE') != expected_reference:" - ), - " raise SystemExit(94)", "with Path(os.environ['FAKE_UV_CALLS']).open('a', encoding='utf-8') as stream:", " stream.write(json.dumps(args) + '\\n')", "if 'run-files' in args:", @@ -258,17 +252,12 @@ def test_configured_webhook_posts_terminal_session_to_slack(nightly_launcher, tm config_dir.mkdir(parents=True) config_file = config_dir / "nightly.env" config_file.write_text( - f"SLACK_WEBHOOK_URL={SLACK_WEBHOOK_URL}\nRETRIEVER_HARNESS_REFERENCE_FILE=/configured-reference.json\n", + f"SLACK_WEBHOOK_URL={SLACK_WEBHOOK_URL}\n", encoding="utf-8", ) config_file.chmod(0o600) - result = run( - extra_env={ - "RETRIEVER_HARNESS_REFERENCE_FILE": "/exported-reference.json", - "EXPECT_REFERENCE_FILE": "/exported-reference.json", - } - ) + result = run() assert result.returncode == 0, result.stderr assert len(calls()) == 2 diff --git a/nemo_retriever/tests/test_harness_slack.py b/nemo_retriever/tests/test_harness_slack.py index 2255b16931..238794298d 100644 --- a/nemo_retriever/tests/test_harness_slack.py +++ b/nemo_retriever/tests/test_harness_slack.py @@ -357,38 +357,3 @@ def capture_post(payload, webhook_url): assert preview_payload == posted[0][0] assert preview_payload == json.loads(post_result.stdout) assert str(session_dir) not in json.dumps(preview_payload) - - -def test_post_slack_shows_current_release_reference_without_a_verdict(monkeypatch, tmp_path): - session_dir = _write_session(tmp_path) - reference_file = tmp_path / "current-release.json" - _write_json( - reference_file, - { - "baselines": [ - { - "name": "RC26.05 Perflab", - "dataset": "jp20", - "environment": {"gpu_sku": "NVIDIA H100 80GB HBM3", "gpu_count": 8}, - "metrics": {"pages": 1940}, - } - ], - }, - ) - monkeypatch.setenv("RETRIEVER_HARNESS_REFERENCE_FILE", str(reference_file)) - - result = CliRunner().invoke(app, ["post-slack", "--preview", str(session_dir)]) - - assert result.exit_code == 0 - payload = json.loads(result.stdout) - assert payload["blocks"][-2]["text"]["text"] == ( - "*RC26.05 Perflab reference — jp20 (8 workload GPUs)*\n" - "Observed values only; hardware may differ and no pass/fail threshold is applied." - ) - assert _table_rows(payload["blocks"][-1]) == [ - ["METRIC", "CURRENT", "RC26.05 PERFLAB"], - ["physical GPU SKU", "NVIDIA H100 NVL", "NVIDIA H100 80GB HBM3"], - ["physical GPU count", "8", "8"], - ["GPUs available to workload", "8", "N/A"], - ["pages", "1940", "1940"], - ] diff --git a/ops/retriever-nightly/README.md b/ops/retriever-nightly/README.md deleted file mode 100644 index aaff56ed96..0000000000 --- a/ops/retriever-nightly/README.md +++ /dev/null @@ -1,214 +0,0 @@ - - - -# Retriever Nightly Launcher - -`run-nightly.sh` runs the checked-in library and ViDoRe benchmark suite once. -It adds host locking, Git selection, standard dataset paths, artifact placement, -and optional Slack reporting around -[`retriever harness run-files`](../../nemo_retriever/harness/README.md). - -It does not install a scheduler or system service. - -## Choose the Source - -| Goal | Command | Source used | -| --- | --- | --- | -| Test the current checkout | `./ops/retriever-nightly/run-nightly.sh` | Current branch and local changes | -| Run the newest upstream code | `./ops/retriever-nightly/run-nightly.sh --ref upstream/main` | Freshly fetched `upstream/main` | -| Reproduce one commit | `./ops/retriever-nightly/run-nightly.sh --ref ` | Clean detached worktree | - -With no positional runfiles, the launcher runs twelve benchmarks: JP20, BO767, -Earnings, FinanceBench, and all eight ViDoRe v3 domains. Pass one or more -runfiles to run a smaller selection. - -## First Run - -On a standard host with `/datasets/nv-ingest` and writable `/raid/$USER`: - -```bash -export HF_TOKEN=... -export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... # optional -``` - -Validate access and configuration before starting GPU work: - -```bash -./ops/retriever-nightly/run-nightly.sh --check-vidore-access -./ops/retriever-nightly/run-nightly.sh --dry-run -./ops/retriever-nightly/run-nightly.sh \ - --no-slack \ - nemo_retriever/harness/runfiles/jp20_beir.json -``` - -All three commands should exit zero. Then run the full current checkout: - -```bash -./ops/retriever-nightly/run-nightly.sh -``` - -Use `--ref upstream/main` instead when the result must represent the newest -clean upstream commit. - -## Host Requirements - -The launcher expects: - -- Linux with NVIDIA drivers -- Git, Bash, `uv`, `flock`, and `realpath` -- local access to the selected datasets -- `HF_TOKEN` and outbound Hugging Face access for real ViDoRe runs -- enough RAM, model cache, and artifact storage for the selected runfiles - -The full batch suite is not validated on 128 GiB hosts. It starts models locally -and does not require a model-provider API key. - -Common settings: - -| Setting | When needed | -| --- | --- | -| `HF_TOKEN` | Every real launcher run; a read token is sufficient | -| `SLACK_WEBHOOK_URL` | Only when the terminal result should post to Slack | -| `RETRIEVER_HARNESS_REFERENCE_FILE` | When Slack should show the current RC beside matching results | -| `RETRIEVER_DATASET_PATHS` | Hosts without the standard dataset layout | -| `RETRIEVER_NIGHTLY_ROOT` | Hosts that need a nondefault config, artifact, and checkout root | - -Direct exports are the smallest setup. Persistent values may be copied from -[`nightly.env.example`](nightly.env.example) into: - -```text -/.config/nemo-retriever/nightly/nightly.env -``` - -The launcher uses `/raid/$USER` as the nightly root when it is writable and -`$HOME` otherwise. The optional file must be owned by the current user with mode -`600`. Existing exported values take precedence. The launcher never loads a -repository `.env`. - -## Use Nonstandard Dataset Paths - -The default map, -[`dataset_paths.datasets.yaml`](dataset_paths.datasets.yaml), uses -`/datasets/nv-ingest` for benchmark corpora. JP20 evaluation uses the checked-in -`data/jp20_query_gt.csv`. On another host, copy the portable template outside -the repository: - -```bash -cp nemo_retriever/harness/dataset_paths.example.yaml \ - /local/path/to/dataset_paths.yaml -${EDITOR:-vi} /local/path/to/dataset_paths.yaml -``` - -Pass the YAML file—not a dataset directory: - -```bash -./ops/retriever-nightly/run-nightly.sh \ - --dataset-paths /local/path/to/dataset_paths.yaml \ - --dry-run -``` - -For repeated runs, export the same path as `RETRIEVER_DATASET_PATHS`. - -## Git Selection - -Without `--ref`, the launcher runs the checkout containing the script exactly -as it exists. It records the commit and dirty state in the session. Dirty Slack -reports are prefixed with `[LOCAL CHANGES]`. - -With `--ref`: - -1. Local branches, tags, and SHAs resolve without fetching. -2. Remote branches such as `upstream/main` are fetched first. -3. Fetch failure stops the run instead of using a stale remote-tracking commit. -4. The selected commit runs in a clean detached worktree. - -Managed worktrees and a shared `uv` environment live under the nightly root. -The launcher retains the seven most recently used clean worktrees and never -moves or merges the invoking checkout. - -Configure `upstream` once if needed: - -```bash -git remote get-url upstream >/dev/null 2>&1 || \ - git remote add upstream https://github.com/NVIDIA/NeMo-Retriever.git -``` - -Use `--ref HEAD` to ignore local changes and run only the current commit. - -## Run Daily with `tmux` - -Start a session: - -```bash -tmux new -s retriever-nightly -``` - -Inside it, export the environment and run a serial 24-hour loop: - -```bash -export HF_TOKEN=... -export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... -export RETRIEVER_HARNESS_REFERENCE_FILE=/path/to/current-release.json - -interval=86400 -while true; do - started="$(date +%s)" - ./ops/retriever-nightly/run-nightly.sh --ref upstream/main - elapsed=$(( $(date +%s) - started )) - if (( elapsed < interval )); then - sleep "$(( interval - elapsed ))" - fi -done -``` - -Each iteration fetches the newest `upstream/main`. Runs do not overlap. If one -exceeds 24 hours, the next starts after it finishes. - -Detach with `Ctrl-b d`, reconnect with `tmux attach -t retriever-nightly`, and -stop the loop with `tmux kill-session -t retriever-nightly`. The session -survives an SSH disconnect but not a host reboot. - -To exercise an unmerged branch repeatedly, omit `--ref upstream/main`. - -## Slack and the Current Release - -When `SLACK_WEBHOOK_URL` is set, a real run posts once after -`session_summary.json` exists. Dry-runs and access checks never post. Use -`--no-slack` for a real canary. - -Set `RETRIEVER_HARNESS_REFERENCE_FILE` to show the current release beside -matching nightly results. The -[harness reporting guide](../../nemo_retriever/harness/README.md#report-completed-results) -defines the small external JSON format. - -The report presents observed values with their GPU context. It does not enforce -a score, assign a verdict, append history, or modify the release file. To move -to a new RC, replace the external file's label and values. - -## Runtime Behavior - -- A nonblocking host-local lock prevents overlapping launcher processes. -- Every run uses batch mode; each benchmark runs in a fresh child process. -- A failed child is recorded and later benchmarks still run. -- Each child has a six-hour wall-time limit. -- A configured Slack report is attempted once after a terminal session exists. -- `VLLM_DEEP_GEMM_WARMUP` defaults to `skip` unless the caller sets it. - -The command returns the harness status. If the harness succeeds but Slack -posting fails, it returns the Slack command's nonzero status. - -## Troubleshooting - -**ViDoRe access fails:** rerun `--check-vidore-access`. A final `403` from a -Hugging Face CAS host can indicate proxy, firewall, or egress policy rather than -an invalid token. Compare from another network before rotating credentials. - -**A large run stalls with high system memory:** batch ingest can materialize -page payloads in Python. Capture the child `run.log`, `status.json`, process RSS, -and Ray task summary, then reproduce only that runfile. Do not classify it as a -GPU OOM without GPU or kernel allocation evidence. - -**Configuration is unclear:** run -`./ops/retriever-nightly/run-nightly.sh --help`. CLI flags override exported -values, which override the optional config file, which overrides launcher -defaults. diff --git a/ops/retriever-nightly/nightly.env.example b/ops/retriever-nightly/nightly.env.example index 36380216a7..8194123fcf 100644 --- a/ops/retriever-nightly/nightly.env.example +++ b/ops/retriever-nightly/nightly.env.example @@ -18,7 +18,3 @@ # /datasets/nv-ingest path map. This value must name a YAML file, not a dataset # directory: # RETRIEVER_DATASET_PATHS=/path/to/dataset_paths.yaml - -# Optional: current release snapshot rendered beside nightly results in Slack. -# Updating a release means replacing this file's label and observed values: -# RETRIEVER_HARNESS_REFERENCE_FILE=/path/to/current-release.json diff --git a/ops/retriever-nightly/run-nightly.sh b/ops/retriever-nightly/run-nightly.sh index 692d07bee0..017c924c3f 100755 --- a/ops/retriever-nightly/run-nightly.sh +++ b/ops/retriever-nightly/run-nightly.sh @@ -45,7 +45,6 @@ load_config_defaults() { RETRIEVER_ARTIFACT_ROOT RETRIEVER_CHECKOUT RETRIEVER_DATASET_PATHS - RETRIEVER_HARNESS_REFERENCE_FILE RETRIEVER_LATEST_CHECKOUT_ROOT RETRIEVER_LATEST_KEEP_CHECKOUTS RETRIEVER_MODE