Model Cosmos region failover states by tvaron3 · Pull Request #5134 · Azure/azure-sdk-for-rust

Tomas Varon (tvaron3) · 2026-08-26T15:43:18Z

Summary

model region offline/online behavior, including DNS-style transport failures and write-region promotion
model failover-priority changes and the NextWriteRegion / PreviousWriteRegion transition phases
model Strong-consistency multi-write gating, satellite write revocation, and hidden region buildout
preserve region identities and safely replay in-flight replication during region enrollment
document behavior derived from live-account observation and the Cosmos service implementation

Model region offline and online behavior, priority-driven failover phases, Strong multi-write gating, satellite write revocation, and hidden region buildout in the internal in-memory emulator. Add driver and public-client coverage for warm and cold topology changes, preference fallback, exclusions, session tokens, races, and replication safety. Document the behavior observed from live accounts and service code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9bda5b12-da90-43a1-afc4-5b6d3a9d2c14

azure-pipelines · 2026-08-26T15:43:47Z

Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Remove a public documentation link to a private configuration method so the Analyze job can build workspace documentation with warnings denied. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9bda5b12-da90-43a1-afc4-5b6d3a9d2c14

Allow the service-specific topology terms used by the in-memory emulator so CI cSpell validation accepts the documented offline and buildout scenarios. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9bda5b12-da90-43a1-afc4-5b6d3a9d2c14

azure-pipelines · 2026-08-26T17:48:31Z

Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Models Cosmos region lifecycle, failover transitions, write gating, and replication-safe enrollment in the in-memory emulator.

Changes:

Adds offline/online, failover-priority, and transition-state modeling.
Adds hidden region enrollment and pending replication replay.
Documents and tests routing and recovery behavior across driver and public SDK layers.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`sdk/cosmos/.cspell.json`	Adds topology terminology.
`sdk/cosmos/azure_data_cosmos/docs/in-memory-emulator-spec.md`	Documents new region semantics.
`sdk/cosmos/azure_data_cosmos/tests/in_memory_emulator_tests/mod.rs`	Registers public SDK tests.
`sdk/cosmos/azure_data_cosmos/tests/in_memory_emulator_tests/topology_parity.rs`	Tests public client topology behavior.
`sdk/cosmos/azure_data_cosmos_driver/src/in_memory_emulator/client.rs`	Models offline DNS failures.
`sdk/cosmos/azure_data_cosmos_driver/src/in_memory_emulator/config.rs`	Implements topology states and transitions.
`sdk/cosmos/azure_data_cosmos_driver/src/in_memory_emulator/store.rs`	Exposes lifecycle APIs and enrollment replay.
`sdk/cosmos/azure_data_cosmos_driver/src/in_memory_emulator/system_properties.rs`	Emits filtered account locations.
`sdk/cosmos/azure_data_cosmos_driver/tests/in_memory_emulator_tests/mod.rs`	Registers driver tests.
`sdk/cosmos/azure_data_cosmos_driver/tests/in_memory_emulator_tests/region_online_offline.rs`	Covers lifecycle, failover, and routing behavior.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Keep offline-triggered failover priorities aligned with service behavior, preserve retired region identity on rejected re-adds, and correct test docs. Make delayed region enrollment replay mutation-safe through catch-up, including deletes, concurrent registration, finalization, and region removal. Add regression coverage for each corrected edge case. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9bda5b12-da90-43a1-afc4-5b6d3a9d2c14

Simon Moreno (simorenoh) · 2026-09-02T18:32:53Z

Review summary

I found four correctness cases that look worth fixing before merge:

enrollment can replay an older mutation over newer seeded state;
delayed catch-up can overwrite a write committed locally;
changing write mode can make an old failover visible again; and
offline failover can leave an offline region at priority zero.

I added inline comments with the concrete sequences and possible fixes.

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

Simon Moreno (simorenoh) · 2026-09-02T18:33:12Z

+                // are atomic relative to each other: a mutation is either in
+                // this initial snapshot or appended to the new journal.
+                let in_flight = self.in_flight_replications.lock().unwrap();
+                let pending: Vec<_> = in_flight.values().cloned().collect();


🔴 Blocking · Correctness: Stale Mutation Replay

Could this replay mutations in the wrong order when a region is added?

For example, if a create is still pending but a newer delete has already completed, the delete is removed from this map while the create remains. The new region is seeded without the item, then the old create is replayed and brings it back.

I reproduced this by delaying a create, completing the delete, and removing/re-adding the target region. Could we track the latest mutation or tombstone per item instead of replaying every operation that happens to still be in flight? Should the delayed task also verify that it is still targeting the same region incarnation?

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

will fix this and the following in a follow up pr. Thanks for hte feedback

Simon Moreno (simorenoh) · 2026-09-02T18:33:12Z

+            );
+            tokio::spawn(async move {
+                tokio::time::sleep(delay).await;
+                target.catch_up_from(&seed_source);


🔴 Blocking · Correctness: Write Loss

Could delayed catch-up overwrite a write that has already committed locally?

A delayed region is already advertised and can accept writes. A write can be committed locally and pause before replicate() adds it to the catch-up journal. If this snapshot runs in between, it overwrites that local write. The journal does not contain the write yet, and when the request resumes, normal replication skips the source region, so the write is never restored there.

Could we make the local commit and journal registration atomic relative to catch-up finalization, or register the mutation before applying it locally?

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

Simon Moreno (simorenoh) · 2026-09-02T18:33:12Z

+            .filter(|r| {
+                r.name() == self.write_region
+                    || self.failing_over_from.as_deref() == Some(r.name())
+                    || self.next_write_region.as_deref() == Some(r.name())


🔴 Blocking · Correctness: Stale Failover State

Should changing write mode clear any active failover state?

If we announce a single-write failover, switch to multi-write, and then switch back to single-write, the old next_write_region becomes visible again. I reproduced this sequence and the final writable list contained both East and West instead of only East.

Could we either reject write-mode changes while a failover is active or clear next_write_region and failing_over_from when the mode changes?

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

Simon Moreno (simorenoh) · 2026-09-02T18:33:12Z

+            // failover: surviving regions move up and the offlined former hub
+            // moves to the lowest-priority position.
+            topology.priority_order.retain(|name| name != region_name);
+            topology.priority_order.push(region_name.to_string());


🔴 Blocking · Correctness: Priority Invariant

Could this leave priority_order inconsistent when an earlier candidate is already offline?

For example, starting with [East, West, Central], if West is offline and East is then offlined, Central is correctly promoted, but the stored order becomes [West, Central, East]. That leaves offline West at position zero even though Central is the write region. I reproduced this with a focused test.

Could we move the promoted region to position zero before moving the former write region to the end?

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

Tomas Varon (tvaron3) mentioned this pull request Aug 26, 2026

Model Cosmos region failover states tvaron3/azure-sdk-for-rust#18

Closed

Tomas Varon (tvaron3) and others added 2 commits August 26, 2026 12:32

Tomas Varon (tvaron3) marked this pull request as ready for review August 26, 2026 17:48

Tomas Varon (tvaron3) requested a review from a team as a code owner August 26, 2026 17:48

Copilot AI balanced review requested due to automatic review settings August 26, 2026 17:48

Copilot started reviewing on behalf of Tomas Varon (tvaron3) August 26, 2026 17:48 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Tomas Varon (tvaron3) added Cosmos The azure_cosmos crate Client This issue points to a problem in the data-plane of the library. labels Sep 1, 2026

Ashley Stanton-Nurse (analogrelay) approved these changes Sep 2, 2026

View reviewed changes

Tomas Varon (tvaron3) merged commit 7a2c639 into Azure:main Sep 2, 2026
25 checks passed

Simon Moreno (simorenoh) reviewed Sep 2, 2026

View reviewed changes

Tomas Varon (tvaron3) mentioned this pull request Sep 2, 2026

Harden emulator region transitions #5229

Open

Navigation Menu

Uh oh!

Model Cosmos region failover states - #5134

Model Cosmos region failover states#5134
Tomas Varon (tvaron3) merged 4 commits into
Azure:mainfrom
tvaron3:tvaron3-cosmos-emulator-region-failover

Tomas Varon (tvaron3) commented Aug 26, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Simon Moreno (simorenoh) commented Sep 2, 2026

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Uh oh!

Tomas Varon (tvaron3) Sep 2, 2026

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Tomas Varon (tvaron3) commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Simon Moreno (simorenoh) commented Sep 2, 2026

Review summary

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Tomas Varon (tvaron3) Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Simon Moreno (simorenoh) Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tomas Varon (tvaron3) commented Aug 26, 2026 •

edited

Loading