[bug]: sweeper batched 4 force-close outputs into invalid tx, then removed all inputs as `State=Fatal`

[openoms]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

0.20.0-beta

LND Configuration

restlisten=0.0.0.0:8080
rpclisten=0.0.0.0:10009
listen=0.0.0.0:9735
prometheus.listen=0.0.0.0:9092

bitcoin.active=1
bitcoin.mainnet=1
bitcoin.node=bitcoind
bitcoin.basefee=<configured>
bitcoin.feerate=<configured>
bitcoin.timelockdelta=60
bitcoind.rpchost=<internal-bitcoind-service>:8332
bitcoind.rpcuser=<redacted>
bitcoind.rpcpass=<redacted>
bitcoind.zmqpubrawblock=tcp://<internal-bitcoind-service>:28332
bitcoind.zmqpubrawtx=tcp://<internal-bitcoind-service>:28333

db.backend=postgres
db.use-native-sql=true
db.postgres.dsn=<redacted>
db.postgres.timeout=<configured>
db.postgres.maxconnections=<configured>

accept-keysend=1
allow-circular-route=1
stagger-initial-reconnect=1
protocol.wumbo-channels=1
maxchansize=500000000
max-commit-fee-rate-anchors=100
default-remote-max-htlcs=50
debuglevel=info
prometheus.enable=1
gc-canceled-invoices-on-the-fly=true
gc-canceled-invoices-on-startup=true
tor.active=true
tor.v3=true
tor.skip-proxy-for-clearnet-targets=true
minchansize=<configured>
alias=<redacted>

Backend Version

Bitcoin Core v28

Backend Configuration

debug=mempool
debug=rpc
shrinkdebugfile=1
server=1
txindex=1
blockfilterindex=1
printtoconsole=1
rpcuser=<redacted>
rpcpassword=<redacted>
zmqpubrawtx=tcp://0.0.0.0:28333
zmqpubrawblock=tcp://0.0.0.0:28332
bind=<redacted>
rpcbind=<redacted>
rpcallowip=<redacted>

OS/Distribution

kubernetes with https://github.com/blinkbitcoin/charts/tree/main/charts/lnd

Bug Details & Steps to Reproduce

On a mainnet LND node using Postgres for channel state, three force-closed
channels remained in limbo. After restart, LND re-registered the pending sweep
inputs. A later lncli wallet bumpfee --sat_per_vbyte 2 on one HTLC
second-level output caused the sweeper to build a single 4-input transaction
covering:

• one COMMITMENT_TIME_LOCK output
• two COMMITMENT_TO_REMOTE_CONFIRMED outputs
• one HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL output

Bitcoin Core rejected the transaction with:

mempool rejection: mandatory script verify flag failed

LND then removed all four inputs from the sweeper as State=Fatal. I am trying
to understand whether one invalid input poisoned the full batch, whether these
input types should have been isolated, and what the safest recovery path is for
a Postgres-backed node.

Environment

• LND: 0.20.0-beta commit=v0.20.0-beta
• LND commit hash: b9ea7070c20ad2ca8514a47d9b4d560a501f0487
• Network: Bitcoin mainnet
• Chain backend: Bitcoin Core
• Bitcoin Core: /Satoshi:28.0.0/
• Bitcoin Core relay fee: 0.00001000
• Bitcoin Core incremental fee: 0.00001000
• LND channel state backend: Postgres
• Node alias/pubkey: redacted

Force-close channels in limbo

From lncli pendingchannels, total limbo balance was about 0.676 BTC.

Channel	Closing tx	Limbo balance	Anchor	Notes
Channel A	redacted	about 0.072 BTC	LOST	no pending HTLCs
Channel B	redacted	about 0.551 BTC	RECOVERED	one incoming stage 2 HTLC for 8601 sat
Channel C	redacted	about 0.053 BTC	LOST	no pending HTLCs

Pending sweep inputs after restart

After restart, lncli wallet pendingsweeps showed four pending inputs:

Input	Witness type	Amount
Input A	COMMITMENT_TO_REMOTE_CONFIRMED	about 0.072 BTC
Input B	COMMITMENT_TO_REMOTE_CONFIRMED	about 0.053 BTC
Input C	COMMITMENT_TIME_LOCK	about 0.551 BTC
Input D	HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL	8601 sat

Triggering action

I bumped the fee for the HTLC second-level output:

lncli wallet bumpfee --sat_per_vbyte 2 <htlc-second-level-outpoint>

LND accepted it:

"Successfully registered rbf-tx with sweeper"

On the next block, the sweeper attempted a 4-input batch.

Relevant log excerpt

The full outpoints are redacted here, but the important sequence was:

[INF] WLKT: [BumpFee]: bumping fee for existing input=<htlc-second-level-outpoint>, new params=startingFeeRate={true 500}
[DBG] SWPR: Received new block: height=<height>, attempt sweeping 4 inputs:
<commitment-time-lock-outpoint> (CommitmentTimeLock)
<commitment-to-remote-confirmed-outpoint-a> (CommitmentToRemoteConfirmed)
<commitment-to-remote-confirmed-outpoint-b> (CommitmentToRemoteConfirmed)
<htlc-second-level-outpoint> (HtlcAcceptedSuccessSecondLevel)
[DBG] SWPR: Creating sweep tx for 4 inputs (...) using 500 sat/kw
[DBG] SWPR: Created sweep tx <sweep-txid> for inputs:
<commitment-time-lock-outpoint> (CommitmentTimeLock)
<commitment-to-remote-confirmed-outpoint-a> (CommitmentToRemoteConfirmed)
<commitment-to-remote-confirmed-outpoint-b> (CommitmentToRemoteConfirmed)
<htlc-second-level-outpoint> (HtlcAcceptedSuccessSecondLevel)
[DBG] SWPR: Failed to create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[ERR] SWPR: Initial broadcast failed: create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[DBG] SWPR: Sending result [Event=Fatal] for requestID=1
[ERR] SWPR: Failed to sweep input: <commitment-time-lock-outpoint> (CommitmentTimeLock), error: create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[ERR] SWPR: Failed to sweep input: <commitment-to-remote-confirmed-outpoint-a> (CommitmentToRemoteConfirmed), error: create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[ERR] SWPR: Failed to sweep input: <commitment-to-remote-confirmed-outpoint-b> (CommitmentToRemoteConfirmed), error: create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[ERR] SWPR: Failed to sweep input: <htlc-second-level-outpoint> (HtlcAcceptedSuccessSecondLevel), error: create RBF-compliant tx: tx=<sweep-txid> failed mempool check: mempool rejection: mandatory script verify flag failed
[DBG] SWPR: Removing input(State=Fatal) <commitment-to-remote-confirmed-outpoint-a> from sweeper
[DBG] SWPR: Removing input(State=Fatal) <commitment-to-remote-confirmed-outpoint-b> from sweeper
[DBG] SWPR: Removing input(State=Fatal) <commitment-time-lock-outpoint> from sweeper
[DBG] SWPR: Removing input(State=Fatal) <htlc-second-level-outpoint> from sweeper

After this, the next block log showed:

[DBG] SWPR: Received new block: height=<height>, attempt sweeping 0 inputs:
[DBG] SWPR: Sweeping 0 inputs

Actual behavior

The sweeper created a 4-input transaction. Bitcoin Core rejected it with:

mandatory script verify flag failed

All four inputs were then marked fatal and removed from the sweeper, including
the three large commitment outputs. The limbo channels remained listed by
lncli pendingchannels, but lncli wallet pendingsweeps became empty.

Expected Behavior

Expected behavior

I expected one of these outcomes:

• LND constructs a valid sweep transaction for the mature commitment outputs.
• If one input fails script verification, LND isolates that input or avoids
  marking unrelated inputs fatal.
• If HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL is unsafe to batch with commitment
  outputs, LND keeps it separate from COMMITMENT_TIME_LOCK and
  COMMITMENT_TO_REMOTE_CONFIRMED outputs.

Questions

1. Does this look like one bad input poisoning a sweep batch, or does the log
   indicate all four witnesses were invalid?
2. Should an HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL input be batched with
   COMMITMENT_TIME_LOCK and COMMITMENT_TO_REMOTE_CONFIRMED inputs?
3. Is it expected that all inputs in a batch are removed as State=Fatal after
   one batched transaction fails script verification?
4. For a Postgres-backed node where lncli wallet pendingsweeps is now empty
   but lncli pendingchannels still shows the limbo channels, is there a safe
   way to make LND retry these sweeps?
5. If LND cannot retry, which manual recovery path is recommended for each
   witness type? I am considering Chantools dry-runs only, without --publish,
   and then validating any raw tx with bitcoin-cli testmempoolaccept before
   any broadcast.

Debug Information

I can provide sanitized logs and exact outpoints privately on Slack or else suitable.

Environment

No response

[github-actions]

Found 1 possible duplicate issue:

1. [bug]: Sweeper Race Condition: Historical Rescan vs Input Grouping #10225

If this issue is a duplicate, please close it and 👍 the existing issue instead.

🤖 Generated with Claude Code

[yyforyongyu]

Can you inspect Bitcoin Core debug logs if debug=validation/script-level logging is available, or maybe just look at how bitcoind failed the verification? I think there is a deeper issue here as it's rare to see script verification fail.

[openoms]

Update: the exact raw LND-generated 4-input sweep tx is not recoverable from Bitcoin Core. It was rejected before mempool admission, and getrawtransaction <txid> false inside the bitcoind pod returns "No such mempool or blockchain transaction."

I enabled validation logging after the fact. For the next diagnostic step, I am going to use Chantools dry-runs without --publish to construct candidate recovery transactions per output type, then run bitcoin-cli decoderawtransaction and bitcoin-cli testmempoolaccept against each raw tx.

Plan:

• Test the COMMITMENT_TIME_LOCK output with chantools sweeptimelockmanual.
• Test each COMMITMENT_TO_REMOTE_CONFIRMED output separately with chantools sweepremoteclosed.
• Avoid manually constructing/publishing the HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL spend unless maintainers recommend a specific path.

I will share the testmempoolaccept results and sanitized decoded tx details.
Will not publish any Chantools transaction unless it is reviewed and locally accepted by testmempoolaccept.

[openoms]

Update: I was able to manually recover the largest COMMITMENT_TIME_LOCK
output that was removed as State=Fatal. Chantools generated a single-input
sweep transaction, and Bitcoin Core accepted it with testmempoolaccept.

This suggests at least one input from the failed batch was individually
spendable with correct witness data. So the linked PRs that isolate/retry failed
batch inputs seem directly relevant to this case.

For anyone attempting manual recovery: make sure --timelockaddr is the script
address for the exact stuck outpoint being recovered. In a force-close
transaction with multiple outputs, a valid address from a different output will
not derive the target script for the intended output.

[yyforyongyu]

make sure --timelockaddr is the script address for the exact stuck outpoint being recovered.

So there were some customized scripts used during the FC, which caused the script validation failure?

[openoms]

To clarify the previous question: no customized force-close scripts were used.
My --timelockaddr note was only about manual recovery with Chantools: in a
force-close transaction with multiple outputs, the address passed to Chantools
must be the script address of the exact output being swept.

Recovery/update:

• The large COMMITMENT_TIME_LOCK output was recovered with
  chantools sweeptimelockmanual as a single-input transaction. Bitcoin Core
  accepted the raw tx with testmempoolaccept, and the output was spent by the
  Chantools-generated sweep.
• Both significant COMMITMENT_TO_REMOTE_CONFIRMED outputs were also handled
  separately with chantools sweepremoteclosed. Each generated single-input
  transaction was accepted by Bitcoin Core with testmempoolaccept; the second
  has now also been broadcast.
  • first remote-closed sweep txid:
    a002000ae17a48872ec4595829604f57f6c2d9f9ef71b424d60a69b37a9876dc
    • input: 8fe215cccf973927943ed6a4889cb6cf41db5feaa4f60e436e7cd0208f66451a:3
    • vsize: 123
    • fee: 245 sats
    • effective feerate: about 1.99 sat/vB
  • second remote-closed sweep txid:
    a8862eef261e59279ad8bec1cfe3cefe1d5fe449be25e4396f3c64aa9d103568
    • input: a7a31ce5280d7ecf6a225fbded6ed524db5b218af6cc43a1b388d1dc2160b159:4
    • vsize: 123
    • fee: 245 sats
    • effective feerate: about 1.99 sat/vB

So the three significant-value inputs from the original failed mixed batch were
individually spendable when isolated and swept with the expected witness data:

• COMMITMENT_TIME_LOCK
• COMMITMENT_TO_REMOTE_CONFIRMED
• COMMITMENT_TO_REMOTE_CONFIRMED

The only remaining input from that original failed batch is the small
HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL output. Bitcoin Core currently sees it as
an unspent Taproot output worth 8451 sats. LND previously showed that HTLC as
8601 sats in pending channel state, but the chain UTXO value is 8451 sats.

What is the recommended recovery path for this witness type after the sweeper
has previously marked the mixed batch as fatal?

Specifically:

• Is there a supported lncli/LND way to re-register just that
  HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL input without batching it with the other
  force-close outputs?
• If manual recovery is required, is there a Chantools command or recommended
  witness construction path for this exact witness type?
• Which channel/HTLC fields are required for that manual spend, and where is the
  safest place to extract them from a Postgres-backed LND node?

I am avoiding any guessed manual spend for that HTLC until the supported path is
clear.

[ziggie1984]

Thanks for the update. One clarification: when you say Bitcoin Core sees the remaining HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL output as an unspent Taproot output, can you confirm whether the underlying channel was actually a simple/production taproot channel?

The issue text/log excerpts label the sweeper input as HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL, while LND normally has separate taproot witness types such as TAPROOT_HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL / TAPROOT_HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL_FINAL.

So I want to rule out a reporting/classification issue: was the input shown by lncli wallet pendingsweeps exactly as HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL, or was that label manually normalized/redacted? If possible, could you share the exact pendingsweeps witness type string for that HTLC output and the channel commitment type / taproot channel type?

[openoms]

Thanks for the clarification request @ziggie1984, good catch. Here is the exact
data from the node's recovery logs and chainstate:

1. Channel Commitment Type

The underlying channel was not a Taproot channel.

lncli pendingchannels for the affected channel
(ea321d1ec48d386efe4476171d7601d8982d39bea4a3762c06645c66d6692696:1)
returned:

"commitment_type": "ANCHORS"

The closing commitment transaction
(552390538f74091bba042c81da2253ba21902ec040811c5dc52166733b1938da)
has standard SegWit v0 P2WSH (v0_p2wsh) outputs on-chain, consistent with
ANCHORS.

2. Exact pendingsweeps Witness Type

The witness_type shown by lncli wallet pendingsweeps was exactly
HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL, not a TAPROOT_ variant. No manual
normalization or redaction was applied to this label. Raw JSON from
pendingsweeps after restart:

{
    "outpoint": "96b3c66b423f651b2448675298dc60a6492e59a4ed0b9d558b751a2fe86ccce1:0",
    "witness_type": "HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL",
    "amount_sat": 8601,
    "sat_per_vbyte": 0,
    "broadcast_attempts": 0,
    "requested_sat_per_vbyte": 0,
    "immediate": false,
    "budget": 4301,
    "deadline_height": 951917,
    "maturity_height": 950458
}

3. Why the Output Appears as Taproot On-Chain

The second-level HTLC success transaction
(96b3c66b423f651b2448675298dc60a6492e59a4ed0b9d558b751a2fe86ccce1) spent the
HTLC output from the commitment transaction (which was v0_p2wsh). Its single
output (vout 0) is however a Taproot (v1_p2tr) output:

scriptpubkey_type: v1_p2tr
address:           bc1p5lqjxxcwexke80kuu4rv7e68rsxpt3rkv6rse9gprlu0f7rekpqqahy02m
script:            5120a7c1231b0ec9ad93bedce546cf67471c0c15c47666870c95011ff8f4f879b040
value:             8451 sat

The 8451 sat value matches the 8601 sat HTLC minus the 150 sat
second-level tx fee. Bitcoin Core gettxout confirms this output is still
unspent.

4. Possible Mismatch

So the situation is:

• Channel commitment type: ANCHORS (SegWit v0)
• LND sweeper witness classification: HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL
  (legacy, not TAPROOT_ prefixed)
• Actual on-chain output being swept: v1_p2tr (Taproot)

If the sweeper constructed the witness using the legacy
HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL spending path (SegWit v0 ECDSA) against a
v1_p2tr output, that would explain the mandatory script verify flag failed
rejection from Bitcoin Core.

5. Manual Recovery Confirms the Other Inputs Were Valid

The three other inputs from the failed batch were individually recovered with
Chantools and confirmed on-chain:

• COMMITMENT_TIME_LOCK (552390...:4): swept with sweeptimelockmanual,
  testmempoolaccept returned allowed: true, confirmed at block 951141.
• COMMITMENT_TO_REMOTE_CONFIRMED (8fe215...:3): swept with
  sweepremoteclosed, confirmed.
• COMMITMENT_TO_REMOTE_CONFIRMED (a7a31c...:4): swept with
  sweepremoteclosed, confirmed.

Only the 8451 sat HTLC_ACCEPTED_SUCCESS_SECOND_LEVEL Taproot output remains
unspent.