Skip to content

Forward follower's append_entries_reply when it already has the snapshot#641

Merged
kjnilsson merged 1 commit into
mainfrom
snapshot-install-dead-lock-bug
Jul 6, 2026
Merged

Forward follower's append_entries_reply when it already has the snapshot#641
kjnilsson merged 1 commit into
mainfrom
snapshot-install-dead-lock-bug

Conversation

@kjnilsson

Copy link
Copy Markdown
Contributor

A follower whose last_applied is at or ahead of an incoming snapshot's index rejects the install_snapshot_rpc with an append_entries_reply rather than an install_snapshot_result. The snapshot sender treated this as fatal (exit(unexpected_install_snapshot_result)), so the reply never reached the leader and next_index was never corrected — the leader re-sent the same snapshot indefinitely, until the next election.

The sender now forwards that reply to the leader and exits normally. The leader's existing success = false handler advances next_index to the position the follower indicated, breaking the loop.

Adds ra_SUITE test snapshot_installation_target_already_has_snapshot.

A follower whose last_applied is at or ahead of an incoming snapshot's
index rejects the install_snapshot_rpc with an append_entries_reply
rather than an install_snapshot_result. The snapshot sender treated this
as fatal (exit(unexpected_install_snapshot_result)), so the reply never
reached the leader and next_index was never corrected — the leader
re-sent the same snapshot indefinitely, until the next election.

The sender now forwards that reply to the leader and exits normally. The
leader's existing success = false handler advances next_index to the
position the follower indicated, breaking the loop.

Adds ra_SUITE test snapshot_installation_target_already_has_snapshot.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a Raft snapshot retry loop where a follower that already has (or is ahead of) an incoming snapshot rejects install_snapshot_rpc with an append_entries_reply, and the snapshot-sender previously treated that as fatal—preventing the leader from correcting next_index.

Changes:

  • Forward #append_entries_reply{} responses from install_snapshot_rpc back to the leader and terminate the snapshot-sender normally.
  • Add regression test snapshot_installation_target_already_has_snapshot to ensure the forwarded reply reaches the leader and the cluster converges.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/ra_server_proc.erl Forward append_entries_reply to leader during snapshot send to avoid fatal exit and unblock next_index correction.
test/ra_SUITE.erl Adds regression coverage for the “target already has snapshot” scenario using meck to force the follower response and verify forwarding.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/ra_SUITE.erl
@kjnilsson kjnilsson marked this pull request as ready for review July 6, 2026 13:01
@kjnilsson kjnilsson merged commit 9ce468e into main Jul 6, 2026
9 checks passed
@kjnilsson kjnilsson deleted the snapshot-install-dead-lock-bug branch July 6, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants