fix: route known text directly with safe Ray admission

[jioffe502]

Summary

Known .txt inputs are classified before graph construction, but the resolved text branch still used the generic MultiTypeExtractOperator, whose Ray actor pool defaulted to one worker. Batch admission also did not reserve CPU for schema-normalization tasks created after extraction branches were resolved.

This PR routes known text directly through TxtSplitActor and admits the complete resolved batch graph around its known actor and task requirements.

What changes

• Builds a direct TxtSplitActor graph for known text inputs; MultiTypeExtractOperator remains available for auto and other composite extraction paths.
• Sizes direct text extraction to at most 8 CPU workers.
• Reserves 1 CPU per extraction dataset when multiple datasets may require schema normalization.
• Reduces only automatically derived actor pools when readers, normalization, or other resolved stages need capacity.
• Keeps explicit BatchTuningParams and node_overrides authoritative; infeasible explicit plans fail before execution.
• Treats the maximum of (min, max) and (min, max, initial) Ray actor-pool tuples as requested capacity.
• Logs actor CPUs, source-read CPUs, other task CPUs, GPU allocation, and final pool sizes separately.

Filesystem-reader capacity remains owned by the existing source_cpu_reservation contract. Normalization capacity is added without counting readers twice.

Evidence

Check	Result
4-CPU direct text run on 100 real BRIGHT files	Admitted 3 TxtSplitActor workers plus 1 reader CPU; physical plan was ReadBinary -> TxtSplitCPUActor; 100/100 paths, 100 rows, 0 errors; Ray dataset completed in 3.12 s
BrowseComp-Plus 10k direct text run	Admitted 8 TxtSplitActor workers; 10,000/10,000 paths produced exactly 75,065 rows with 0 errors; Ray dataset completed in 33.18 s and the public call completed in 76.42 s including worker-environment setup
BrowseComp-Plus worker ablation motivating the bounded pool	1 to 8 workers reduced the same 10k extraction workload from approximately 377 s to 66 s while preserving all 75,065 rows
Explicit 4-worker text request on 4 CPUs	Rejected before execution because the file reader requires a fifth CPU
Mixed text/PDF workload on 32 CPUs	Shared admission completed 268 files in 80.914 s and stored 3,145 valid rows
Previous full acceptance on this admission branch	Full BRIGHT completed across all 12 domains; BO767, Earnings, FinanceBench, JP20, and 8 ViDoRe ingests completed

The text changes improve extraction throughput. Embedding and the final store remain separate downstream bottlenecks for full BrowseComp ingestion.

Scope

• Direct graph construction applies to known text inputs in library batch and in-process modes.
• Automatic actor sizing and resource admission apply only to library batch mode.
• Service-mode actor residency is unchanged.
• The deprecated NRL benchmark harness is unchanged; workload construction and telemetry remain NRB-owned.
• The LanceDB sink is unchanged. BrowseComp streaming writes remain tracked separately in Bound LanceDB ingestion memory with an Arrow-native transactional sink #2476.

Validation

• Full NRL non-integration suite: 3,260 passed, 146 skipped, 12 deselected, 26 subtests passed
• Focused graph, planner, manifest, ingest-interface, and inline-text suites: 254 passed, 3 skipped, 4 deselected
• Ray text integration: 2 passed (file/inline parity and combined file-plus-inline ingestion)
• Synthetic merge with current main: focused suites passed
• Pre-commit on all changed files: all hooks passed
• Runtime acceptance evidence and harness changes: NRB MR !13

Reviewer focus

• Does the direct text graph preserve TextChunkParams and split_config["text"] precedence?
• Is the direct TxtSplitActor pool reduced around reader/task capacity while explicit overrides remain authoritative?
• Is normalization capacity added to source-reader capacity exactly once?

[greptile-apps]

Greptile Summary

The PR routes known text inputs directly through TxtSplitActor and extends Ray batch admission to account for source reads and schema-normalization work.

• Adds bounded automatic concurrency for direct text extraction.
• Preserves explicit worker and node overrides while validating their resource feasibility.
• Reserves task capacity when multiple extraction datasets require normalization.
• Documents the revised resource-planning behavior and adds focused graph and admission tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/graph/executor.py	Extends actor-pool bounds handling and shared CPU admission to reserve capacity for source reads and other task work.
nemo_retriever/src/nemo_retriever/graph/ingestor_runtime.py	Builds a direct text extraction graph and derives a bounded automatic TxtSplitActor pool from available CPUs.
nemo_retriever/src/nemo_retriever/ingestor/branch_extraction.py	Passes extraction mode into tuning and includes multi-dataset schema-normalization capacity in shared preflight.
nemo_retriever/src/nemo_retriever/ingestor/graph_ingestor.py	Propagates the resolved extraction mode into batch node-override planning.
docs/docs/extraction/performance_guide.md	Documents direct text concurrency, normalization reservations, explicit-plan validation, and actor-pool tuple accounting.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    I[Known text inputs] --> G[Direct TxtSplitActor graph]
    C[Cluster resources] --> P[Shared resource preflight]
    R[Source-read reservation] --> P
    N[Normalization reservation] --> P
    O[Explicit and automatic pool settings] --> P
    P --> A[Admitted actor concurrency]
    A --> G
    G --> S[Common post-extraction stages]

Loading

_{Reviews (8): Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." | Re-trigger Greptile}

[jioffe502]

check to see if this is fundamentally a router problem or if this is the right fix.