Fix generation error precedence in failure classification - #2541
Merged
Conversation
Contributor
Greptile SummaryThis PR updates batch RAG evaluation to prioritize generation failures over missing judge scores and normalizes missing generation-error values in DataFrame scoring.
|
| Filename | Overview |
|---|---|
| nemo_retriever/src/nemo_retriever/tools/evaluation/scoring.py | Adjusts failure-mode precedence and normalizes DataFrame generation-error values before classification. |
| nemo_retriever/tests/test_live_rag.py | Adds parametrized pipeline coverage for originating-stage failure classification. |
Reviews (2): Last reviewed commit: "Merge branch 'main' into nvbug_6622224" | Re-trigger Greptile
jperez999
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NVBug 6622224.
Batch Live RAG evaluation currently classifies generic generation failures such as
transport_errorandrequest_errorasjudge_errorwhen generation produces an empty candidate and no judge score is available. This incorrectly attributes the failure to the judge stage even thoughgen_erroridentifies generation as the originating stage.This change gives generation errors precedence when deriving
failure_mode. The existingthinking_truncatedclassification remains unchanged, while other non-null generation errors now use the stablegeneration_errorbucket.judge_erroris reserved for cases where generation succeeded but judging did not produce a score. The DataFrame scoring path also normalizes missing generation-error values before classification.This affects only batch/operator evaluation and reporting. Single-query
Retriever.answer()already returns before judging or scoring after a generation failure.Checklist