fix: repair Ray Arrow-to-pandas batch ingestion

[jioffe502]

Summary

Fix the QA-reported SDK batch workflow for default element-granularity PDF ingestion with local embedding, Sidecar metadata, and LanceDB.

The fix stays at two existing production boundaries:

• Ray Arrow-to-pandas operator adaptation.
• VDB actor graph reconstruction.

No planner, routing, harness, public API, dependency, or service behavior changes are included.

Root cause

The QA workflow exposed three consequences of losing DataFrame contracts across Ray boundaries:

1. A sliced nested Arrow struct with an inferred null child retained a nonzero parent offset. Ray converted that layout to Arrow-backed pandas, and conversion of the UDF result back to Arrow failed validation with Struct child array ... (1 < 2).
2. Once that was repaired, Ray pickled-object extension columns containing empty arrays materialized as NumPy shape (3, 0). Pandas row operations expected block shape (1, 3) and failed before the embedding model call.
3. Sidecar keys were consumed before IngestVdbOperator recorded its graph constructor kwargs. Ray rebuilt the VDB actor without meta_dataframe, meta_source_field, and meta_fields, so rows were stored without the requested Sidecar fields.

Fix

• Keep pandas-format graph batches as Arrow until the executor boundary.
• Compact only sliced nested columns whose type contains a null child.
• Delegate conversion to Ray BlockAccessor.to_pandas() so Ray conversion policy remains authoritative.
• Convert only Ray arrow_pickled_object extension columns to ordinary pandas object columns before invoking the NRL operator.
• Preserve the original VDB kwargs for actor reconstruction while continuing to pass cleaned backend kwargs to LanceDB.

Validation

• End-to-end QA fixture on the repository lock: data/multimodal_test.pdf, batch mode, default element granularity, local vLLM embedder, in-memory Sidecar DataFrame, local LanceDB.
  • 3 result rows.
  • 3 successful 2048-dimensional embeddings.
  • 3 stored LanceDB rows.
  • category, department, and timestamp persisted on every row.
  • Warm-cache pipeline execution: 37.34 seconds; acceptance script: 43.72 seconds.
• QA dependency matrix boundary check: Python 3.12.3, Ray 2.57.0, Pandas 2.3.3, PyArrow 24.0.0.
  • 34 focused executor/VDB tests passed.
  • Captured real PDF Arrow batch embedded 3/3 rows with a deterministic model.
• Final GitHub unit suite at commit 96531854: 3,220 passed, 23 skipped, 12 deselected, 41 subtests passed.
• Black, Flake8, and git diff --check: passed.

Artifacts are preserved under /datasets/nv-ingest/qa-arrow-sidecar-2456/20260812-c5371840.

Scope

Four files: the Ray executor boundary, VDB graph-constructor preservation, and focused regression tests. The broader graph-aware planner work in #2454 remains separate.

[greptile-apps]

Greptile Summary

The PR repairs DataFrame contracts at Ray’s Arrow boundary and preserves Sidecar settings when VDB operators are reconstructed.

• Routes pandas-oriented Ray stages through a shared Arrow-to-pandas adapter.
• Compacts vulnerable sliced nested Arrow columns and normalizes Ray pickled-object extension columns.
• Preserves VDB Sidecar constructor parameters and normalizes Arrow-backed content and bounding-box values.
• Adds focused regression coverage for executor conversion, ingestion diagnostics, multimodal content, Sidecar reconstruction, and VDB records.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/graph/executor.py	Adds the shared Arrow-to-pandas conversion boundary and applies it to pandas-format Ray graph operators.
nemo_retriever/src/nemo_retriever/ingestor/branch_extraction.py	Moves branch-schema normalization to the safe pyarrow batch conversion path.
nemo_retriever/src/nemo_retriever/ingestor/graph_ingestor.py	Uses safe Arrow conversion for batch iteration and error-row extraction.
nemo_retriever/src/nemo_retriever/operators/extract/pdf/split.py	Routes PDF splitting through the shared Arrow-to-pandas boundary.
nemo_retriever/src/nemo_retriever/operators/vdb.py	Preserves original Sidecar-bearing VDB kwargs for graph reconstruction while retaining cleaned backend kwargs internally.
nemo_retriever/src/nemo_retriever/common/modality/content_transforms.py	Treats one-dimensional object NumPy arrays as extracted-content collections.
nemo_retriever/src/nemo_retriever/common/vdb/records.py	Converts array-backed bounding boxes into JSON-safe lists without ambiguous array truth testing.
nemo_retriever/tests/test_executor_arrow_pandas.py	Covers sliced nested Arrow batches, Ray conversion policy, pickled-object columns, numeric arrays, and multimodal content semantics.

Sequence Diagram

sequenceDiagram
    participant RD as Ray Dataset
    participant Adapter as Arrow/Pandas Adapter
    participant Op as NRL Operator
    participant VDB as VDB Actor
    RD->>Adapter: pyarrow.Table batch
    Adapter->>Adapter: Compact vulnerable nested columns
    Adapter->>Adapter: Ray BlockAccessor.to_pandas()
    Adapter->>Adapter: Normalize pickled-object columns
    Adapter->>Op: pandas.DataFrame
    Op-->>RD: Processed batch
    RD->>VDB: Global VDB batch
    VDB->>VDB: Reconstruct with preserved Sidecar kwargs
    VDB-->>RD: Persist records with Sidecar metadata

Loading

_{Reviews (7): Last reviewed commit: "Merge branch 'main' into jioffe502/fix-r..." | Re-trigger Greptile}