Skip to content

Fix generation error precedence in failure classification - #2541

Merged
jperez999 merged 2 commits into
NVIDIA:mainfrom
edknv:nvbug_6622224
Aug 18, 2026
Merged

Fix generation error precedence in failure classification#2541
jperez999 merged 2 commits into
NVIDIA:mainfrom
edknv:nvbug_6622224

Conversation

@edknv

@edknv edknv commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Description

NVBug 6622224.

Batch Live RAG evaluation currently classifies generic generation failures such as transport_error and request_error as judge_error when generation produces an empty candidate and no judge score is available. This incorrectly attributes the failure to the judge stage even though gen_error identifies generation as the originating stage.

This change gives generation errors precedence when deriving failure_mode. The existing thinking_truncated classification remains unchanged, while other non-null generation errors now use the stable generation_error bucket. judge_error is reserved for cases where generation succeeded but judging did not produce a score. The DataFrame scoring path also normalizes missing generation-error values before classification.

This affects only batch/operator evaluation and reporting. Single-query Retriever.answer() already returns before judging or scoring after a generation failure.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@edknv
edknv requested review from a team as code owners August 18, 2026 15:51
@edknv
edknv requested a review from ChrisJar August 18, 2026 15:51
@greptile-apps

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates batch RAG evaluation to prioritize generation failures over missing judge scores and normalizes missing generation-error values in DataFrame scoring.

  • Maps non-null generation errors to the stable generation_error failure bucket while preserving thinking_truncated.
  • Adds pipeline-level coverage for generation, truncation, judge, and successful classifications.

Confidence Score: 5/5

The PR appears safe to merge because no eligible or previously reported blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
nemo_retriever/src/nemo_retriever/tools/evaluation/scoring.py Adjusts failure-mode precedence and normalizes DataFrame generation-error values before classification.
nemo_retriever/tests/test_live_rag.py Adds parametrized pipeline coverage for originating-stage failure classification.

Reviews (2): Last reviewed commit: "Merge branch 'main' into nvbug_6622224" | Re-trigger Greptile

@jperez999
jperez999 merged commit 6d55e58 into NVIDIA:main Aug 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants