fix(ingest): enable OCR text extraction in default pipeline - #2524
fix(ingest): enable OCR text extraction in default pipeline#2524iamdevdhanush wants to merge 5 commits into
Conversation
Greptile SummaryThe PR enables OCR text extraction for the default pdfium pipeline and adds full-page OCR fallback handling for scanned pages lacking text-region detections.
|
| Filename | Overview |
|---|---|
| nemo_retriever/src/nemo_retriever/common/modality/ocr/shared.py | Adds full-page OCR crop handling and routes its predictions into page text. |
| nemo_retriever/src/nemo_retriever/graph/ingestor_runtime.py | Enables OCR graph construction whenever text extraction is requested. |
| nemo_retriever/src/nemo_retriever/operators/extract/pdf/extract.py | Marks scanned pages using the default pdfium method as requiring OCR text extraction. |
| nemo_retriever/src/nemo_retriever/operators/graph_ops/multi_type_extract_operator.py | Enables OCR stage selection and text kwargs independently of the extraction method. |
| nemo_retriever/tests/test_ocr_version_selection.py | Adds regression tests for default pdfium OCR scheduling, scanned-page eligibility, and full-page crop creation. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[PDF or image ingest] --> B{extract_text enabled?}
B -- No --> C[Continue without text OCR]
B -- Yes --> D[Schedule OCR stage]
D --> E{Text-region crop available?}
E -- Yes --> F[OCR text crops]
E -- No --> G[OCR full-page fallback]
F --> H[Populate page text]
G --> H
Reviews (5): Last reviewed commit: "Merge branch 'main' into main" | Re-trigger Greptile
| extract_text=False, | ||
| extract_tables=False, | ||
| extract_charts=False, | ||
| extract_infographics=False, |
There was a problem hiding this comment.
The new test_detection_pipeline_includes_ocr_for_default_method function leaves monkeypatch untyped and omits its -> None return annotation, weakening static checking and violating the repository's public-function annotation standard.
Rule Used: All public functions, methods, and class attribute... (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: nemo_retriever/tests/test_ocr_version_selection.py
Line: 410
Comment:
**New test lacks annotations**
The new `test_detection_pipeline_includes_ocr_for_default_method` function leaves `monkeypatch` untyped and omits its `-> None` return annotation, weakening static checking and violating the repository's public-function annotation standard.
**Rule Used:** All public functions, methods, and class attribute... ([source](.greptile))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Description
Fixes the default CLI ingest pipeline to activate OCR text extraction for all extraction methods, not just pdfium_hybrid and ocr.
The default extract_text=True flag was ignored for method="pdfium" (the default) because three code paths gated OCR activation on:
if method in ("pdfium_hybrid", "ocr") and extract_text:
This condition was always False when method="pdfium", causing:
Remove the method restriction in all three locations. OCR text extraction now activates whenever extract_params.extract_text is True, regardless of extraction method:
Five new regression tests in test_ocr_version_selection.py verify the fix.
Closes #2443
Checklist