`trace_filter` spawns unbounded concurrent block replays causing sync pipeline stall and MDBX long-reader warnings

[0x19dG87]

Summary

When trace_filter (or any trace API covering a block range) is called, reth spawns one trace_block_until_with_inspector blocking task per block in the range, all running concurrently. Each task holds an MDBX read transaction open for the full EVM replay duration.

Under concurrent trace load from multiple clients, this creates hundreds of simultaneous MDBX read transactions that:

1. Saturate NVMe I/O bandwidth to 100%
2. Starve the sync pipeline's MerkleExecute stage of I/O
3. Cause the node to fall progressively behind chain head
4. Trigger continuous WARN storage::db::mdbx: A database read transaction has been open for too long spam

The node becomes self-reinforcing: as it falls behind, it receives more forkchoice_updated payloads, which enqueue more sync work — while trace calls continue consuming all available I/O.

The concurrency ceiling is controlled by --rpc-cache.max-concurrent-db-requests (default/configured: 1024). With multiple concurrent clients each requesting 100-block ranges, the theoretical concurrent MDBX reader count is clients × blocks_per_call, up to the configured ceiling.

Ethereum mainnet is especially severe: each mainnet block trace replay takes 60–120 seconds (vs 5–15 seconds for less complex chains), generating thousands of random IOPS per concurrent task. Just 4–5 simultaneous mainnet trace replays are sufficient to saturate a high-end NVMe RAID array.

---

Environment

Field	Value
OS	Ubuntu 25.10, kernel 6.17.0-29-generic
CPU	AMD EPYC 7J13, 64 cores / 128 threads
RAM	500 GiB
Storage device	6× Micron 7450 7 TiB NVMe in RAID-0 (md0), XFS

---

Reth Configuration (relevant flags)

Docker Container:

node
  --chain=mainnet
  --storage.v2
  --db.max-size=4TB
  --db.max-readers=1024
  --http
  --http.api=all
  --rpc-cache.max-concurrent-db-requests=1024
  --rpc.max-trace-filter-blocks=2000
  --rpc.max-response-size=200
  --rpc.gascap=100000000
  --engine.cross-block-cache-size=4096
  --engine.memory-block-buffer-target=128
  --engine.persistence-threshold=4
  --engine.storage-worker-count=48
  --engine.account-worker-count=48
  --engine.prewarming-threads=32
  --rpc-cache.max-blocks=10000
  --rpc-cache.max-receipts=10000
  --rpc-cache.max-headers=5000
  --mem-limit 32g

---

Expected Behavior

1. trace_filter over a block range should not starve the sync/consensus pipeline of I/O
2. The node should maintain sync progress regardless of RPC trace load

---

Additional Notes

• The WARN storage::db::mdbx: A database read transaction has been open for too long message itself is not a bug — it is a correct diagnostic. The bug is the lack of back-pressure between RPC trace ops and the sync pipeline.
• The node does not crash or corrupt data. It eventually completes each MerkleExecute run (after 23 minutes in our case) and catches up between trace bursts. The risk is falling so far behind that lighthouse considers the execution client unresponsive.

---

Steps to reproduce

1. Run reth on Ethereum mainnet as an archive node with --http.api=all (trace APIs enabled) and --rpc-cache.max-concurrent-db-requests set to a large value (≥ 64).
2. Connect one or more RPC clients that issue trace_filter calls over block ranges (e.g., 50–100 blocks per call). The clients should be catching up from a point significantly behind chain head so requests are continuous rather than one-per-block.
3. Run multiple such clients concurrently (3–5 simultaneous connections, each with their own trace_filter loop).
4. Observe: WARN storage::db::mdbx: A database read transaction has been open for too long begins appearing repeatedly; MerkleExecute stage checkpoint stops advancing; NVMe utilization reaches 100%.

Minimum reproduction (single client):

1. Start reth mainnet archive node with --rpc-cache.max-concurrent-db-requests=1024.
2. Send a single trace_filter RPC call covering 50+ consecutive mainnet blocks from a historically complex range (e.g., blocks 18000000–18000050, high DeFi activity).
3. While the call is in flight, observe iostat -x 1 on the NVMe device and reth sync status — I/O will spike and pipeline stages will stall.

---

Node logs

Continuous WARN spam (135 occurrences in 10 minutes)

2026-05-23T08:56:45.061395Z WARN storage::db::mdbx: A database read transaction has been open for too long
  open_duration=60.000748474s id=9274075 backtrace=
   0: reth_db::implementation::mdbx::tx::MetricsHandler<K>::log_backtrace_on_long_read_transaction
             at ./crates/storage/db/src/implementation/mdbx/tx.rs:259:32
   1: reth_db::implementation::mdbx::tx::Tx<K>::execute_with_operation_metric
             at ./crates/storage/db/src/implementation/mdbx/tx.rs:165:29
   2: <reth_db::implementation::mdbx::tx::Tx<K> as reth_db_api::transaction::DbTx>::get_by_encoded_key
             at ./crates/storage/db/src/implementation/mdbx/tx.rs:297:14
   3: <reth_provider::providers::database::provider::DatabaseProvider<TX,N> as reth_storage_api::stage_checkpoint::StageCheckpointReader>::get_stage_checkpoint
             at ./crates/storage/provider/src/providers/database/provider.rs:2236:21
   4: <reth_provider::providers::database::provider::DatabaseProvider<TX,N> as reth_storage_api::block_id::BlockNumReader>::best_block_number
             at ./crates/storage/provider/src/providers/database/provider.rs:1798:14
   5: reth_provider::providers::state::historical::HistoricalStateProviderRef<Provider,N>::storage_history_lookup
             at ./crates/storage/provider/src/providers/state/historical.rs:221:41
   6: reth_provider::providers::state::historical::HistoricalStateProviderRef<Provider,N>::storage_by_lookup_key
             at ./crates/storage/provider/src/providers/state/historical.rs:247:20
   7: <reth_provider::providers::state::historical::HistoricalStateProviderRef<Provider,N> as reth_storage_api::state::StateProvider>::storage
             at ./crates/storage/provider/src/providers/state/historical.rs:671:14
  ...
  11: <T as reth_revm::database::EvmStateProvider>::storage
             at ./crates/revm/src/database.rs:58:9
  12: <reth_revm::database::StateProviderDatabase<DB> as revm_database_interface::DatabaseRef>::storage_ref
             at ./crates/revm/src/database.rs:161:19
  ...
  24: revm_interpreter::instructions::host::sload
             at /usr/local/cargo/registry/.../revm-interpreter-35.0.1/src/instructions/host.rs:205:32
  25: revm_interpreter::instructions::Instruction<W,H>::execute
  26: revm_interpreter::interpreter::Interpreter<IW>::step
  27: revm_inspector::handler::inspect_instructions
  28: revm_inspector::traits::InspectorEvmTr::inspect_frame_run
  ...
  33: revm_inspector::mainnet_inspect::<impl revm_inspector::inspect::InspectEvm for ...>::inspect_one_tx
  34: revm_inspector::inspect::InspectEvm::inspect_tx
  35: <alloy_evm::eth::EthEvm<DB,I,PRECOMPILE> as alloy_evm::evm::Evm>::transact_raw
  36: alloy_evm::evm::Evm::transact
  37: <alloy_evm::tracing::TracerIter<E,Txs,F> as core::iter::traits::iterator::Iterator>::next
  ...
  52: reth_rpc_eth_api::helpers::trace::Trace::trace_block_until_with_inspector::{{closure}}::{{closure}}
  53: reth_rpc_eth_api::helpers::call::Call::spawn_with_state_at_block::{{closure}}::{{closure}}
  54: reth_rpc_eth_api::helpers::blocking_task::SpawnBlocking::spawn_blocking_io_fut::{{closure}}
  ...
  70: reth_tasks::runtime::Runtime::spawn_on_rt::{{closure}}
  71: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  ...
  88: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}

All 135 WARNs trace to trace_block_until_with_inspector. Multiple simultaneous warnings with identical tx IDs confirm concurrent open transactions.

Sync pipeline stall — 23 minutes on 64 blocks

2026-05-23T08:51:49Z INFO reth_node_events::node: Executing stage pipeline_stages=8/13 stage=MerkleExecute checkpoint=25156717 target=25156781
2026-05-23T08:52:12Z INFO reth::cli: Status connected_peers=61 stage=MerkleExecute checkpoint=25156717 target=25156781
2026-05-23T08:53:52Z INFO reth::cli: Status connected_peers=68 stage=MerkleExecute checkpoint=25156717 target=25156781
2026-05-23T08:55:32Z INFO reth::cli: Status connected_peers=69 stage=MerkleExecute checkpoint=25156717 target=25156781
2026-05-23T08:57:12Z INFO reth::cli: Status connected_peers=80 stage=MerkleExecute checkpoint=25156717 target=25156781
2026-05-23T08:58:52Z INFO reth::cli: Status connected_peers=82 stage=MerkleExecute checkpoint=25156717 target=25156781
# ... checkpoint=25156717 unchanged for entire period ...
2026-05-23T09:15:07Z INFO reth::cli: Status connected_peers=130 stage=MerkleExecute checkpoint=25156781 target=25156877
# ^ Finally advanced after 23 minutes; chain was 226 blocks ahead

MerkleExecute for 64 blocks normally completes in under 1 second. It took 23 minutes because all I/O was consumed by concurrent trace replays.

System resource state during incident

System load average (128-thread machine):

465.47 415.66 307.65

reth-eth container stats:

CONTAINER   CPU %      MEM USAGE / LIMIT   PIDS
reth-eth    4152.78%   23.6GiB / 32GiB     1181

NVMe RAID-0 I/O — fully saturated:

Device   r/s       rkB/s      r_await  w/s       wkB/s     %util
md0      67083.90  1796190.84  0.22    13448.08  86983.82  91.39
md0      63756.00  2038088.00  0.72    25958.00  105352.00 100.10
md0      61955.45  2131053.47  0.74    6347.52   100479.21 99.01

---

Platform(s)

Linux (x86)

Container Type

Docker

What version/commit are you on?

Reth Version: 2.2.0
Commit SHA: 88505c7fcbfdebfd3b56d88c86b62e950043c6c4
Build Timestamp: 2026-04-29T19:53:57.473810535Z
Build Features: asm_keccak,jemalloc,keccak_cache_global,min_debug_logs,otlp,otlp_logs
Build Profile: maxperf-symbols

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

--chain mainnet (Ethereum mainnet, archive node, --storage.v2)

What type of node are you running?

Archive (default)

Code of Conduct

• I agree to follow the Code of Conduct

[HusseinAdeiza]

Investigation Update

Thanks for the detailed bug report! I've investigated this issue and identified the root cause.

Root Cause

Location: crates/rpc/rpc/src/trace.rs, lines 410-430

The trace_filter implementation spawns trace tasks for all blocks in the range concurrently, then waits for all of them with try_join_all():

// Lines 410-427: Spawns ALL tasks immediately
for block in &blocks {
    let traces = self.eth_api().trace_block_until(...);
    block_traces.push(traces);  // Accumulates futures without awaiting
}

// Line 430: Then waits for ALL simultaneously
let block_traces = futures::future::try_join_all(block_traces.drain(..)).await?;

With a 100-block range, this spawns 100 concurrent tasks, each holding an MDBX read transaction for 60-120 seconds (mainnet), exactly as you described.

Proposed Fix: Batch Processing

Process blocks in small batches instead of all at once:

const MAX_CONCURRENT_TRACES: usize = 4;  // Configurable

for chunk in blocks.chunks(MAX_CONCURRENT_TRACES) {
    let mut batch_traces = Vec::new();
    for block in chunk {
        let traces = self.eth_api().trace_block_until(...);
        batch_traces.push(traces);
    }
    let batch_results = futures::future::try_join_all(batch_traces).await?;
    all_traces.extend(batch_results.into_iter().flatten());
}

Benefits:

• Limits concurrent DB reads to 4-8 instead of 100+
• Still parallelizes work (faster than sequential)
• Provides back-pressure for sync pipeline
• Simple implementation

Questions

Before implementing:

1. Does this approach look reasonable?
2. What concurrency limit makes sense? (I'm thinking 4-8 for mainnet)
3. Should this reuse --rpc.max-tracing-requests or be a separate config?

Happy to submit a PR if this direction looks good!