fix: preserve heterogeneous Ray Data rows across batch ingestion

[jioffe502]

Description

PR #2474 is the canonical Arrow-boundary fix for the reported page embedding, text chunking, bbox, and LanceDB vector issues. The StoreOperator issue remains out of scope.

The fix has three layers:

• Original 171918de: preserve pandas output for explode/collapse UDFs, normalize bbox arrays to lists, and convert embeddings with .tolist() at the LanceDB record boundary.
• Rebuild collapsed page rows in a new DataFrame instead of mutating the Arrow-backed frame.
• Opt TextChunkCPUActor into the same pandas-output contract for extract-only chunking paths where explode does not run.

This PR does not change the legacy pipeline/content.py implementation, StoreOperator or _stored_image_uri, Arrow representation of images, pickle warning policy, dedup bbox handling, or BrowseComp storage behavior.

Bug mapping

Bug	Fix owner
6597993 page embed / reshape on result	collapse rebuild plus original preserve_pandas_output
6597580 text chunk	TextChunkCPUActor.PRESERVE_PANDAS_OUTPUT
6598030 explode bbox	original bbox list normalization, locked by the text_image regression
6604117 zero LanceDB rows	original embedding list normalization
6597862 store	out of scope; leave open

Regression coverage

• test_explode_text_image_does_not_boolean_evaluate_numpy_bbox exercises the failing --embed-modality text_image path.
• test_collapse_returns_iterrows_safe_page_rows verifies rebuilt page rows remain iterable after the Arrow-backed collapse boundary.
• Small executor assertions verify the TextChunk class opt-in participates in stable-pandas graph handling.

Validation

Tested with the QA stack overlay using Ray 2.57.0, PyArrow 24.0.0, and Pandas 2.3.3:

• Focused regression and CI-order check: 16 passed
• Arrow, graph, text chunking, ingest, and VDB suites: 176 passed, 3 skipped
• Full retriever unit suite: 3,268 passed, 23 skipped, 12 deselected, 41 subtests passed
• Pre-commit hooks: passed

[greptile-apps]

Greptile Summary

The PR preserves heterogeneous pandas rows across Ray Data ingestion boundaries and normalizes bbox and embedding containers before downstream processing.

• Rebuilds exploded and collapsed content rows into stable pandas DataFrames.
• Keeps pandas blocks stable after content reshaping and text chunking stages.
• Converts bbox values and embedding vectors into ordinary Python lists at persistence boundaries.
• Adds regression coverage for Arrow-backed rows, text chunking, bbox handling, and VDB records.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/common/modality/content_transforms.py	Rebuilds transformed rows and normalizes bbox containers to preserve heterogeneous pandas-compatible values.
nemo_retriever/src/nemo_retriever/common/vdb/records.py	Normalizes array-like embeddings into Python lists before constructing VDB records.
nemo_retriever/src/nemo_retriever/graph/executor.py	Detects pandas-preserving operators and maintains stable pandas blocks through downstream Ray stages.
nemo_retriever/src/nemo_retriever/graph/ingestor_runtime.py	Marks content reshape UDFs as requiring preserved pandas output.
nemo_retriever/src/nemo_retriever/operators/extract/txt/ray_data.py	Annotates the text chunk actor capability flag and opts chunk expansion into stable pandas handling.
nemo_retriever/tests/test_executor_arrow_pandas.py	Adds regression coverage for Arrow conversion, pandas preservation, tensor promotion, and content reshaping.
nemo_retriever/tests/test_pipeline_graph.py	Verifies ingestion graph stages opt into pandas-preserving execution.
nemo_retriever/tests/test_vdb_records.py	Covers conversion of array-like embedding values at the VDB record boundary.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Extracted Ray Data rows] --> B{Reshaping stage}
    B -->|Explode content| C[Heterogeneous pandas rows]
    B -->|Collapse pages| C
    B -->|Text chunking| C
    C --> D[Stable pandas blocks]
    D --> E[Embedding operator]
    E --> F[Normalize vectors to lists]
    F --> G[LanceDB ingestion]

Loading

_{Reviews (2): Last reviewed commit: "Update nemo_retriever/src/nemo_retriever..." | Re-trigger Greptile}