fix(driver): fall back to another EOA when a submission account is unusable#4547
fix(driver): fall back to another EOA when a submission account is unusable#4547frdrckj wants to merge 11 commits into
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a fallback mechanism to retry transaction submissions using alternative accounts when encountering account-specific failures (such as insufficient funds, nonce issues, or underpriced transactions). Feedback focuses on improving the robustness of the error classification: first, by formatting the full anyhow::Error chain using format!("{err:#}") instead of err.to_string() to prevent underlying node errors from being hidden; and second, by loosening the substring matching logic to support non-Geth clients like Nethermind which format error messages differently.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
I have read the CLA Document and I hereby sign the CLA |
…usable EIP-7702 parallel submission picks an account from a FIFO pool but never falls back when the chosen account can't broadcast (no gas for the tx, stale nonce, or a pending tx that can't be replaced). A valid settlement then fails even when another funded account is available. Classify pre-broadcast node rejections as account-specific (`mempools::Error::SubmitterUnusable`) and, on such a failure, retry the settlement from another account while the submission deadline still holds. Spent accounts are held for the duration of the request so each retry picks a different one, and `try_acquire` keeps the fallback non-blocking to avoid deadlocking against concurrent settlements. Retrying is safe against double-submission: a settlement that actually lands always returns `Ok`, so only failures (where nothing was broadcast) are ever retried. `already known` is deliberately not treated as account-specific, since it means our exact tx is already in the mempool and may still be mined. Towards cowprotocol#4541.
Match keyword pairs (`insufficient`+`funds`, `nonce`+`low`) instead of exact
Geth phrases so non-Geth clients are also recognised (e.g. Nethermind's
`SenderInsufficientFunds` / `NonceTooLow`, which omit spaces), and format the
full error chain (`{:#}`) so a node message wrapped in extra context is still
matched.
frdrckj
force-pushed
the
fix/eip7702-submitter-fallback
branch
from
e2daf2d to
28dc8f5
Compare
Issue cowprotocol#4541 asks the driver to not only retry a settlement from another submission EOA on an account-specific failure, but to temporarily stop assigning settlements to a stuck lane until it recovers. - Quarantine: a delegated account that fails to broadcast for an account-specific reason (no gas, stuck nonce, underpriced) is now held out of the selection pool for UNHEALTHY_ACCOUNT_COOLDOWN before being returned, so concurrent and subsequent settlements skip it. The direct solver EOA is never benched. - Surface SubmitterUnusable out of the mempool race: select_ok returns the last error, which could mask an account-specific failure reported by another mempool and silently defeat the retry/quarantine. Prefer SubmitterUnusable unless a settlement-specific terminal error (revert/expired) is authoritative. Adds unit tests for the quarantine withholding and the race-error preference.
frdrckj
force-pushed
the
fix/eip7702-submitter-fallback
branch
from
28dc8f5 to
9950782
Compare
Don't retry/quarantine on a post-broadcast failure. In a multi-mempool race, select_ok can surface a pre-broadcast SubmitterUnusable from one lane while another lane already broadcast and then failed post-broadcast (e.g. the block stream ending), so retrying from a different EOA could double-submit the settlement. Track whether any lane broadcast via a per-execute flag set right after mempool.submit() succeeds, and have race_error keep last_error once a tx is on the wire, downgrading even a SubmitterUnusable it happened to return last. Only classify replacement underpricing as account-specific. A plain 'transaction underpriced' is a node/network fee-floor rejection that every account hits the same way, so retrying just re-fails and benches each submitter without fixing anything. Require both 'replacement' and 'underpriced'; global underpriced now falls through to a non-retryable error. Towards cowprotocol#4541.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod submitter_pool_tests { |
There was a problem hiding this comment.
These pool tests are useful, but we also need tests for the full retry flow.
Please cover direct-to-delegated fallback, delegated-to-direct fallback, no account available, deadline reached, unsafe errors, and tx already sent.
There was a problem hiding this comment.
I'll extract the retry loop into a small helper, submit_with_fallback(pool, guard, cooldown, deadline_reached, submit), that takes the submit step as a closure, so it can be unit-tested without the network. Tests will cover direct to delegated fallback, delegated to direct fallback, no account available, deadline reached (fails fast before submitting), an unsafe or ambiguous error (no retry), and success on the first attempt (no retry). These assert the helper's own contract; notify::executed stays in process_settle_request and is covered there
There was a problem hiding this comment.
I’ll leave this coverage question for the backend team to decide.
The any_broadcast flag only covers a confirmed send_transaction Ok. When several mempools race the same settlement, a sibling lane can fail before broadcast with an ambiguous error (timeout, connection reset, `already known`, `nonce too high`) whose tx might still be on the wire, while another lane cleanly reports an account failure. Surfacing SubmitterUnusable there and retrying from another EOA could double-submit the settlement. Track a saw_nonretryable flag in execute(), set for any lane error that is not SubmitterUnusable and not Disabled (Disabled is a clean skip that never touched the network). race_error now surfaces SubmitterUnusable only when no lane broadcast and no such error was seen; otherwise it keeps the real error and strips a SubmitterUnusable that select_ok happened to return last. Towards cowprotocol#4541.
…derpriced Only a replacement-underpriced error (a stuck nonce another account avoids) is account-specific and should trigger fallback; a plain or global underpriced is a node or network fee floor that every account hits the same way. The classifier already requires both "replacement" and "underpriced", so this is a naming change only. The metric and log label becomes replacement_underpriced to match. Per review feedback.
Before broadcasting, compare the signing account's balance to the gas the settlement needs (gas limit times the submission max_fee_per_gas). If it can't cover the cost, return SubmitterUnusable(InsufficientFunds) without sending, so the health-aware fallback picks another account and benches this one instead of wasting a submission. A failed balance lookup is not authoritative, so we log it and let the node decide. Per review feedback. Towards cowprotocol#4541.
…it better Extract the health-aware retry loop into submit_with_fallback, which takes the submission step as a closure so the fallback flow can be unit-tested without a live mempool. Adds tests for direct-to-delegated and delegated-to-direct fallback, no account available, deadline reached, an unsafe non-account error, and an immediate success. Also per review feedback: - check the submission deadline before each attempt, since acquiring a slot can wait past it - bench the failed account with its failure reason, and log which account was benched, why, and for how long - log the failed account, the next account, and the reason on each retry The solver notification still fires exactly once, after the loop, with the final result. Towards cowprotocol#4541.
|
Fixes look good so far. I'm going to have to ask @cowprotocol/backend squad to step in for a more detailed look. Thanks for your contribution! |
|
Thanks Igor, appreciate the review! Happy to work through any feedback from the backend squad. |
|
This pull request has been marked as stale because it has been inactive a while. Please update this pull request or it will be automatically closed. |
| // Proactively check the signer can cover the gas before broadcasting, so an | ||
| // underfunded account falls back to another one without a wasted submission | ||
| // (issue #4541). A failed balance lookup is not authoritative, so proceed | ||
| // and let the node decide. | ||
| let required_balance = settlement | ||
| .gas | ||
| .required_balance(eth::U256::from(final_gas_price.max_fee_per_gas)); | ||
| match self.ethereum.balance(signer).await { | ||
| Ok(balance) if balance < required_balance => { | ||
| tracing::warn!( | ||
| ?signer, | ||
| ?balance, | ||
| ?required_balance, | ||
| "submission account balance too low for gas, falling back" | ||
| ); | ||
| return Err(Error::SubmitterUnusable(AccountFailure::InsufficientFunds)); | ||
| } | ||
| Ok(_) => {} | ||
| Err(err) => tracing::warn!( | ||
| ?signer, | ||
| ?err, | ||
| "could not check submission account balance before submitting" | ||
| ), | ||
| } |
There was a problem hiding this comment.
IIUC the description says proactive balance pre-checks were left out on purpose, but this block adds exactly that check, one eth_getBalance per mempool before every broadcast, happy path included. The classifier already turns the node's own insufficient funds rejection into the same SubmitterUnusable, so the pre-check spends an RPC on every submission to save one doomed broadcast in the rare failure case. Probably makes sense to drop this block and let the node's rejection be the signal, which would also match the description, wdyt?
There was a problem hiding this comment.
yes, the code and the description is different. Igor asked for this pre-check earlier, which is why it's in. Functionally the reactive path already covers the case: the classifier turns the node's insufficient-funds rejection into the same SubmitterUnusable, so an underfunded account still gets skipped and benched without the pre-check. The only thing the pre-check buys is skipping the doomed broadcast, plus a bit of robustness if a sibling relay returns something ambiguous instead of a clean rejection. @igorroncevic, what do you think? should we drop it?
| fn race_error( | ||
| last_error: Error, | ||
| account_failure: Option<AccountFailure>, | ||
| broadcasted: bool, | ||
| saw_nonretryable: bool, | ||
| ) -> Error { |
There was a problem hiding this comment.
Did I get it correctly, that a terminal failure only wins the race when it happens to be the error select_ok returned last? If one lane reports SimulationRevert and another lane's SubmitterUnusable finishes last, the retry is still correctly suppressed (the revert sets saw_nonretryable), but the surfaced error becomes Other(...), so the solver is notified Fail instead of SimulationRevert and the metrics label shifts the same way. Did you consider tracking a terminal error from any lane, or is the label drift acceptable here?
There was a problem hiding this comment.
oh yes, i havent consider it yet. I can track a terminal error from any lane, the same way I capture the account failure, and return it ahead of the account-failure path so Revert/SimulationRevert/Expired always win regardless of race order. Adding a test for the two-lane case (one reverts, the other reports an account failure last).
Co-authored-by: ilya <squad.gazzz@gmail.com>
race_error only treated a terminal failure (revert/expired) as authoritative when it was the error select_ok returned last. If one lane reverted but another lane's SubmitterUnusable finished last, the retry was still suppressed, but the surfaced error dropped to Other, so the solver was notified Fail instead of the revert. Capture a terminal failure from any lane, like the existing account_failure, and return it first from race_error so it wins regardless of race order.
3 participants
Description
EIP-7702 parallel submission selects a submission account from a FIFO pool. When the selected account cannot broadcast the transaction for an account-specific reason (no balance for gas, a stale nonce, or a pending tx that cannot be replaced), the settlement currently fails even though another funded account may be available — and the stuck account keeps getting picked by later settlements. This is the gap described in #4541.
This PR makes EIP-7702 submission health-aware: it retries a failed settlement from another account and temporarily benches the unhealthy account so subsequent settlements stop assigning to a stuck lane until it recovers.
Changes
Reactive fallback + classification
mempools::Error::SubmitterUnusable(AccountFailure)and aclassify_submission_failurehelper that recognises pre-broadcast node rejections (insufficient funds,nonce too low,... underpriced) across Geth- and Nethermind-style wording.already knownandnonce too highare intentionally excluded.Mempool::submitreturns this variant (matching the full{:#}error chain) for broadcast-time rejections instead of the genericOther.Competition::process_settle_requestretries the settlement from another account onSubmitterUnusable. Spent accounts are held until the request finishes so each retry uses a different one, and a non-blockingSubmitterPool::try_acquireavoids deadlocking against concurrent settlements.Temporary quarantine — the "stop assigning to stuck lanes" half of #4541
UNHEALTHY_ACCOUNT_COOLDOWN(30s): it is withheld from the FIFO pool before being returned, so concurrent and subsequent settlements skip it until it recovers. It is retried automatically after the cooldown; a still-broken account is simply re-benched after one attempt. The direct solver EOA is never benched.Reliable propagation in multi-mempool setups
Mempools::executeraces mempools withselect_ok, which returns the last error when all fail — so a generic error (e.g. a timeout) from one mempool could mask aSubmitterUnusablefrom another and silently defeat the retry/quarantine. The race now surfacesSubmitterUnusablewhenever any mempool reported one, unless a settlement-specific terminal failure (revert/expired) is present, which stays authoritative.Metrics and solver notifications account for the new outcome.
Retrying is safe against double-submission: a settlement that actually lands always returns
Ok, so only failures (where nothing was broadcast) are ever retried. That is also whyalready knownis excluded.Scope
This covers the two behaviours #4541 asks for: skip-and-retry on account-specific failures, and temporarily marking unhealthy EOAs unavailable so stuck lanes stop receiving new submissions. Proactive balance/nonce RPC pre-checks before selecting an account are intentionally not added — they put RPC round-trips on the latency-sensitive settlement path and are inherently racy (balance/nonce can change between check and submit), while the node's actual rejection (which this PR classifies and benches on) is the source of truth. Quarantine already satisfies the "account is not already known to be stuck" signal by physically removing benched accounts from selection. Happy to add pre-checks in a follow-up if preferred.
How to test
Unit tests:
classifies_account_specific_submission_failures,classifies_non_geth_client_wording,does_not_classify_non_account_failures_as_account_specific— the classifier in both directions (guards against retrying reverts,already known, rate limits, transport errors,nonce too high).account_failure_is_surfaced_over_a_masking_race_error—SubmitterUnusablewins over a masking generic error but defers to settlement-specific failures.quarantined_account_is_withheld_from_the_pool,direct_slot_is_never_quarantined— benching withholds an unhealthy account while leaving healthy ones available, and never benches the solver EOA.Fixes #4541.