test(eventhubs): add live tests for the event processor

[Johnathan W (j7nw4r)]

Summary

This change adds 11 live tests for the Event Hubs event processor. The tests cover checkpoint resume across processor instances, graceful shutdown and restart, ownership records in the checkpoint store, the balanced load balancing strategy across two processors, invalid configuration errors, and an end to end run on BlobCheckpointStore against live Event Hubs and live Storage.

Motivation

Checkpoint resume is the core promise of the processor. A processor that restarts and reprocesses from the start, or skips events, breaks the checkpoint contract, and no existing test detects it. The processor tests before this change never call shutdown(), never share a checkpoint store between two processors, never read list_ownerships or list_checkpoints, and never assert an error. Load balancing and graceful shutdown are the other two behaviors that a caller depends on in production.

Changes

• Adds 9 live tests to sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_processor.rs for checkpoint resume, shutdown, restart, ownership records, the balanced strategy, an unknown event hub name, and an unknown consumer group.
• Adds sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_processor_blob.rs with 2 live tests that run the processor on BlobCheckpointStore, one for a missing storage container and one for a checkpoint round trip.
• Adds a file level serial mutex to eventhubs_processor.rs and applies it to the new tests and to the five existing tests. Every processor receiver attaches at owner level 0, so two processor tests that run at the same time displace each other.
• Adds the sync feature to the tokio dev dependency, which the serial mutex needs. This is the only change outside a test file, and no file under src/ changes.
• Keeps the blob tests on the defaultGroup consumer group. Cargo runs test binaries in parallel and a file mutex cannot reach across binaries, so the separate consumer group is what keeps the two binaries apart.

Three behaviors of the current code shape these tests, and each one is a way a weaker test would pass while proving nothing.

• PartitionClient::update_checkpoint returns Ok(()) and writes nothing when the received event carries no message annotations. Both checkpoint tests therefore assert that the checkpoint reached the store before they assert anything about resume. The resume test reads list_checkpoints and matches the sequence number. The blob test seeds a known prior value and asserts the stored value changed.
• EventProcessor::get_start_position clones the configured default start position and overrides only the location, so the inclusive flag is inherited from the default. Every new test sets inclusive: false. With inclusive: true a resumed reader re-reads the checkpointed event and the resume assertion breaks silently.
• EventProcessor::shutdown clears a flag and closes no receiver, so a partition client that the processor already handed out keeps streaming. The shutdown tests assert that run() resolves to Ok(()) and that the processor issues no new partition client. They do not assert that an already issued partition client stops, because that assertion would fail against correct source. This gap is tracked in [Event Hubs] EventProcessor::shutdown() does not stop event delivery or release ownership #5096.

The .NET tests detect every invalid configuration at start time. The Rust processor surfaces the same conditions at three different call sites, which the issue permits. An unknown event hub name fails at build(), and a missing storage container fails at run() on the first dispatch. An unknown consumer group reaches neither: a live run held run() pending past 60 seconds, because the receiver attaches on the first poll of the partition client's stream, so that test asserts there. The lazy attach is tracked in #5094. Each test pins the call that returns the error rather than the error text, because an unknown event hub and an unknown consumer group both map to amqp:not-found and the crate error type has no not found variant. The missing container test is the exception and asserts the structured 404 with the ContainerNotFound error code.

The bad namespace case adds no new test. consumer_new_with_error in sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_consumer.rs already opens a consumer against an invalid host and asserts the full error chain. A processor flavored copy would assert strictly less.

Test plan

Run the live tests with this exact command, after deploying the test resources with eng/common/TestResources/New-TestResources.ps1 -ServiceDirectory eventhubs and setting EVENTHUBS_HOST, EVENTHUB_NAME, AZURE_STORAGE_BLOB_ENDPOINT, and AZURE_STORAGE_BLOB_CONTAINER.

AZURE_TEST_MODE=live cargo test --package azure_messaging_eventhubs --test 'eventhubs_processor*' -- --test-threads=1

The offline gates below all pass on this branch.

• cargo fmt --all -- --check exits 0.
• RUSTFLAGS=-Dwarnings cargo test --no-run --package azure_messaging_eventhubs --all-features exits 0.
• RUSTFLAGS=-Dwarnings cargo clippy -p azure_messaging_eventhubs --all-features --all-targets --no-deps exits 0.
• RUSTDOCFLAGS=-Dwarnings cargo doc -p azure_messaging_eventhubs --all-features --no-deps exits 0.
• cargo test --package azure_messaging_eventhubs --all-features --no-fail-fast -- --test-threads=1 exits 0. The suite reports 144 unit tests passed, 6 checkpoint store tests passed, 45 doc tests passed, and no failures. The new tests report as ignored, 14 in eventhubs_processor and 2 in eventhubs_processor_blob.
• cspell passes on all three changed files with the repository configuration.

Two limits are known and recorded rather than fixed.

• processor_shutdown_stops_new_partition_clients asserts a negative, so it first asserts that the processor has partitions left to claim. A processor that already holds every partition also issues no new client, and the test would prove nothing. The precondition fails loudly instead of passing hollow.
• The blob container is shared and azure_storage_blob exposes no container create, so two concurrent runs of processor_checkpoints_to_blob_store would race on one checkpoint blob. CI runs the suite once, so this is a hazard for parallel developers, not for CI.

Closes #4893

Live validation

Every test here ran against a live Event Hubs namespace on 2026-08-20: 13 of 14 in eventhubs_processor.rs and 2 of 2 in eventhubs_processor_blob.rs.

Command: AZURE_TEST_MODE=live cargo test --package azure_messaging_eventhubs --test eventhubs_processor --test eventhubs_processor_blob -- --test-threads=1.

Three tests were corrected after that run, and the commit message records each one. The remaining failure, second_processor_displaces_first_with_consumer_disconnected, is not from this change: it fails the same way on a detached worktree of an unmodified origin/main.

Two further findings from the live run have their own issues: a lost ownership claim ending run() under the balanced strategy is #5095, and update_checkpoint succeeding without writing when an event carries no annotations is #5097.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.