Skip to content

Remove NRL docs and Perflab baseline reporting - #2432

Open
jioffe502 wants to merge 4 commits into
mainfrom
jioffe502/remove-legacy-benchmark-harness
Open

Remove NRL docs and Perflab baseline reporting#2432
jioffe502 wants to merge 4 commits into
mainfrom
jioffe502/remove-legacy-benchmark-harness

Conversation

@jioffe502

@jioffe502 jioffe502 commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the NRL harness README, guides, design-history page, nightly README, and BO767 walkthrough
  • remove the Perflab/release-baseline loader and Slack side-by-side comparison block
  • remove the --reference-file and release_references compatibility surfaces entirely
  • remove the associated baseline test fixture and nightly configuration example
  • keep the harness runner, benchmark runfiles, artifact/history/portal flows, nightly launcher, and ordinary Slack posting intact

Why

Perflab is the canonical home for performance results and comparisons. This removes the ad hoc baseline presentation from NRL while preserving the harness implementation for the separate code decision.

Validation

  • git diff --check passes
  • focused Slack tests pass: 12 passed
  • focused Slack tests pass: 12 passed
  • no Perflab, release-reference, or reference-file integration remains in the repository
  • replacement benchmarking docs link to the internal NRB repository
  • the harness runner and nightly orchestration remain in the tree
  • the full nightly launcher test file cannot run on this macOS host because its unchanged Bash 4+ [[ -v "$name" ]] construct is rejected by system Bash 3.2

@jioffe502
jioffe502 force-pushed the jioffe502/remove-legacy-benchmark-harness branch from 3df166a to f6ffc92 Compare August 18, 2026 18:39
@jioffe502 jioffe502 changed the title Remove legacy benchmark harness after NRB migration Remove legacy NRL harness documentation Aug 18, 2026
@jioffe502 jioffe502 changed the title Remove legacy NRL harness documentation Remove NRL docs and Perflab baseline reporting Aug 18, 2026
@jioffe502
jioffe502 marked this pull request as ready for review August 18, 2026 19:20
@jioffe502
jioffe502 requested review from a team as code owners August 18, 2026 19:20
@greptile-apps

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR removes the local NRL harness documentation and Perflab baseline comparison implementation while redirecting benchmark guidance to NRB.

  • Deletes harness guides, expected-result documentation, and the BO767 walkthrough.
  • Removes release-reference loading and Slack side-by-side baseline rendering.
  • Updates nightly configuration, launcher coverage, and benchmark documentation.
  • The final HEAD does not preserve the stated compatibility handling for deprecated reference inputs.

Confidence Score: 4/5

The PR is not yet safe to merge because existing CLI and Python callers using the deprecated reference-reporting inputs will fail.

The final command and Python signatures remove the compatibility inputs that the PR description says remain accepted and ignored, so callers still passing them encounter an unknown-option error or unexpected-keyword exception.

Files Needing Attention: nemo_retriever/src/nemo_retriever/harness/cli.py; nemo_retriever/src/nemo_retriever/harness/slack.py

Important Files Changed

Filename Overview
nemo_retriever/src/nemo_retriever/harness/cli.py Removes baseline loading but also removes the deprecated CLI compatibility option, leaving existing automation unable to invoke post-slack.
nemo_retriever/src/nemo_retriever/harness/slack.py Removes baseline rendering and the compatibility keyword accepted by existing Python callers.
nemo_retriever/src/nemo_retriever/harness/release_reference.py Deletes the obsolete release-reference parser as part of removing Perflab comparison support.
ops/retriever-nightly/run-nightly.sh Removes release-reference propagation while retaining ordinary nightly execution and Slack posting.
nemo_retriever/README.md Replaces deleted harness guidance with a concrete hyperlink to the NRB repository.
nemo_retriever/docs/cli/benchmarking.md Redirects end-to-end benchmarking guidance to NRB while retaining local stage-microbenchmark documentation.

Comments Outside Diff (1)

  1. nemo_retriever/src/nemo_retriever/harness/cli.py, line 376-383 (link)

    P1 Deprecated reference inputs rejected

    When existing automation passes post-slack --reference-file, Typer rejects the removed option; existing Python callers passing release_references to build_slack_payload or post_report_to_slack receive an unexpected-keyword TypeError. This breaks the compatibility contract for inputs that are intended to remain accepted while their data is ignored.

    Rule Used: Changes to public API surfaces (FastAPI endpoints,... (source)

    Knowledge Base Used: Benchmark Harness

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: nemo_retriever/src/nemo_retriever/harness/cli.py
    Line: 376-383
    
    Comment:
    **Deprecated reference inputs rejected**
    
    When existing automation passes `post-slack --reference-file`, Typer rejects the removed option; existing Python callers passing `release_references` to `build_slack_payload` or `post_report_to_slack` receive an unexpected-keyword `TypeError`. This breaks the compatibility contract for inputs that are intended to remain accepted while their data is ignored.
    
    **Rule Used:** Changes to public API surfaces (FastAPI endpoints,... ([source](.greptile))
    
    **Knowledge Base Used:** [Benchmark Harness](https://app.greptile.com/nvidia-public-github/-/custom-context/knowledge-base/nvidia/nemo-retriever/-/docs/benchmark-harness.md)
    
    ---
    
    For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Prompt To Fix All With AI
### Issue 1
nemo_retriever/src/nemo_retriever/harness/cli.py:376-383
**Deprecated reference inputs rejected**

When existing automation passes `post-slack --reference-file`, Typer rejects the removed option; existing Python callers passing `release_references` to `build_slack_payload` or `post_report_to_slack` receive an unexpected-keyword `TypeError`. This breaks the compatibility contract for inputs that are intended to remain accepted while their data is ignored.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (3): Last reviewed commit: "Purge deprecated reference compatibility" | Re-trigger Greptile

Comment thread nemo_retriever/src/nemo_retriever/harness/cli.py
Comment thread nemo_retriever/README.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants