fix: reserve Ray task capacity in batch admission

[jioffe502]

Summary

Current batch admission reserves CPU for filesystem source reads, but two scheduling gaps remain:

• multi-branch ingestion can consume every remaining CPU before schema-normalization tasks run; and
• file-backed text extraction defaults to one worker even when the input is already well sharded.

This PR reserves the remaining known task capacity after the complete execution graph is resolved and gives file-backed text a bounded automatic worker pool.

What changes

• Reserves 1 CPU per extraction dataset when multiple datasets may require schema normalization.
• Sizes file-backed text extraction to at most 8 CPU workers.
• Reduces only automatically derived actor pools when task reservations or other resolved stages need capacity.
• Treats the maximum of (min, max) and (min, max, initial) Ray actor-pool tuples as requested capacity.
• Keeps explicit BatchTuningParams and node_overrides authoritative; infeasible explicit plans fail before execution.
• Logs actor CPUs, source-read CPUs, other task CPUs, GPU allocation, and final pool sizes separately.

Filesystem-reader capacity remains owned by the existing source_cpu_reservation contract. This PR adds normalization capacity without counting source readers twice.

Evidence

Check	Result
Current-main 4-CPU production text run	Admitted 3 text actors plus 1 source-reader CPU; 100 files produced exactly 675 rows with 0 errors in 7.40 s
Explicit 4-worker text request on 4 CPUs	Rejected before execution: 5 CPUs required, including 1 source-reader CPU
BrowseComp-Plus 10k extraction ablation	1 to 8 text workers reduced extraction from ~377 s to ~66 s while preserving all 75,065 rows
Mixed text/PDF workload on 32 CPUs	Shared admission completed 268 files in 80.914 s and stored 3,145 valid rows
BRIGHT full ingest	1,333,166 files produced 1,376,750 stored rows across all 12 domains in 11,297.082 summed seconds
Classic acceptance set	All 12 ingests completed: BO767, Earnings, FinanceBench, JP20, and 8 ViDoRe domains

The text-worker change improves extraction throughput, not total BrowseComp wall time: embedding and the final store remain the dominant stages after extraction is parallelized.

Scope

• Production library run_mode="batch" only.
• Service-mode actor residency is unchanged.
• The deprecated NRL benchmark harness is unchanged; workload construction and telemetry remain NRB-owned.
• The LanceDB sink is unchanged. BrowseComp streaming writes remain tracked separately in Bound LanceDB ingestion memory with an Arrow-native transactional sink #2476.

Validation

• Focused graph, planner, manifest, and ingest-interface suites: 237 passed, 3 skipped, 2 deselected
• Full current-main NRL non-integration suite: 3,202 passed, 142 skipped, 12 deselected, 26 subtests passed
• Pre-commit: all hooks passed
• Bounded current-main production runtime: 100 files / 675 rows / 0 errors
• Runtime acceptance evidence and harness changes: NRB MR !13

Reviewer focus

• Is normalization capacity added to source-reader capacity exactly once?
• Are only automatically derived pools reduced while explicit concurrency remains authoritative?
• Do the changes remain confined to production batch execution?

[greptile-apps]

Greptile Summary

This PR updates batch-mode Ray admission to reserve capacity for schema-normalization tasks and automatically scales file-backed text extraction.

• Adds shared non-actor CPU reservations to executor preflight.
• Introduces a bounded automatic text-extraction actor pool.
• Validates actor-pool tuples using their maximum requested capacity.
• Documents the revised admission and override behavior.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/graph/executor.py	Extends shared and per-executor preflight with non-actor CPU reservations, tuple-aware capacity accounting, and resource-allocation logging.
nemo_retriever/src/nemo_retriever/graph/ingestor_runtime.py	Adds a bounded automatic CPU-only actor pool for batch text extraction and marks it eligible for preflight reduction.
nemo_retriever/src/nemo_retriever/ingestor/branch_extraction.py	Passes effective extraction mode into tuning and reserves normalization-task capacity during multi-dataset branch execution.
nemo_retriever/src/nemo_retriever/ingestor/graph_ingestor.py	Propagates the effective extraction mode into batch override derivation.
docs/docs/extraction/performance_guide.md	Documents normalization reservations, bounded text-worker sizing, explicit-plan authority, and actor-pool tuple semantics.
nemo_retriever/tests/test_ingest_manifest.py	Updates branch preflight tests to verify normalization CPU reservations.
nemo_retriever/tests/test_ingest_plans.py	Tests automatic text-pool creation and reduction for filesystem-reader capacity.
nemo_retriever/tests/test_pipeline_graph.py	Tests non-actor CPU reservation and infeasible two-element actor-pool tuples.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Resolve batch execution graphs] --> B[Count source-read CPUs]
    A --> C[Count normalization task CPUs]
    A --> D[Resolve actor CPU and GPU requests]
    B --> E[Compute actor CPU budget]
    C --> E
    D --> F{Explicit minimum plan feasible?}
    E --> F
    F -- No --> G[Raise infeasible-plan error]
    F -- Yes --> H[Grow automatic actor pools within budget]
    H --> I[Apply final concurrency overrides]
    I --> J[Build and execute Ray datasets]

Loading

_{Reviews (5): Last reviewed commit: "Merge branch 'main' into jioffe502/graph..." | Re-trigger Greptile}