Skip to content

Storage stress testing utilities and blob stress test executable - #5169

Open
Jocelyn (jaschrep-msft) wants to merge 23 commits into
Azure:mainfrom
jaschrep-msft:feature/storage/stress-testing-2
Open

Storage stress testing utilities and blob stress test executable#5169
Jocelyn (jaschrep-msft) wants to merge 23 commits into
Azure:mainfrom
jaschrep-msft:feature/storage/stress-testing-2

Conversation

@jaschrep-msft

Copy link
Copy Markdown
Member

Stress test framework for storage with a blobs roundtrip transfer test implementation.

Work is split into two new crates. azure_storage_stress is a library containing the runner and traits to be implemented for different tests and services. azure_storage_stress_blobs is an executable containing a blob roundtrip test definition and minimal setup code.

Sample outputs

Sample --help usage of the roundtrip test.

PS C:\Users\jaschrep\Repos\azure-sdk-for-rust> .\target\release\azure_storage_stress_blob.exe roundtrip --help
Azure Storage Blobs Stress Test
Continuously upload then download blobs

Usage: azure_storage_stress_blob.exe roundtrip [OPTIONS] --data-len <BYTES> <DATA_SOURCE>

Arguments:
  <DATA_SOURCE>
          Type of data source to upload from

          Possible values:
          - generated-stream: A stream which uploads in-memory data through a stream interface
          - direct-memory:    Direct contiguous memory access

Options:
      --concurrency <NUM WORKERS>
          Concurrency value for transfer options
          
          [default: 2]

      --parallel <PARALLEL>
          Parallel operations to run
          
          [default: 1]

      --block-len <BYTES>
          Block length for transfer options
          
          [default: 4194304]

      --duration <SECONDS>
          Duration of the stress test, excluding setup and cleanup
          
          [default: 10]

      --data-len <BYTES>
          Data length of blob(s) to transfer

<several args cut for sample brevity>

Sample run with info-level logging enabled:

PS C:\Users\jaschrep\Repos\azure-sdk-for-rust> .\target\release\azure_storage_stress_blob.exe roundtrip direct-memory --data-len 1024 --log-pretty
Azure Storage Blobs Stress Test
[2026-08-27T14:30:11Z INFO  azure_storage_stress_blob] Runner options: {
      "parallel": 1,
      "duration": {
        "secs": 10,
        "nanos": 0
      },
      "setup_timeout": null,
      "operation_timeout": null,
      "cleanup_timeout": null,
      "results_log_frequency": 100,
      "log_pretty": true,
      "fault_injection_file": null,
      "use_default_fault_injection": false,
      "fault_overrides": {
        "partial_response_hang": null,
        "partial_response_close": null,
        "partial_response_abort": null,
        "partial_response_normal": null,
        "no_response_hang": null,
        "no_response_close": null,
        "no_response_abort": null
      },
      "command": {
        "Roundtrip": {
          "concurrency": 2,
          "block_len": 4194304,
          "data_len": 1024,
          "data_source": "DirectMemory"
        }
      }
    }
    
[2026-08-27T14:30:11Z INFO  azure_storage_stress] Begin global setup
[2026-08-27T14:30:17Z INFO  azure_storage_stress_blob::roundtrip_test] Container created.
[2026-08-27T14:30:17Z INFO  azure_storage_stress] Begin stress test
[2026-08-27T14:30:18Z INFO  azure_storage_stress] {
      "total_loops": 100,
      "loops_success": 100,
      "loops_graceful_error": 0,
      "loops_timeout": 0,
      "loops_panic": 0,
      "loops_data_corruption": 0
    }
[2026-08-27T14:30:19Z INFO  azure_storage_stress] {
      "total_loops": 200,
      "loops_success": 200,
      "loops_graceful_error": 0,
      "loops_timeout": 0,
      "loops_panic": 0,
      "loops_data_corruption": 0
    }
<sample cut off for brevity>

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a reusable Storage stress-test framework and a Blob Storage roundtrip stress executable.

Changes:

  • Adds concurrent stress orchestration, metrics, timeouts, data generation, and fault injection.
  • Implements repeated blob upload/download integrity checks.
  • Registers both internal crates and their dependencies.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
Cargo.toml Registers the stress crates.
Cargo.lock Locks their dependencies.
sdk/storage/azure_storage_stress/Cargo.toml Defines the framework crate.
sdk/storage/azure_storage_stress/src/args.rs Defines runner and fault options.
sdk/storage/azure_storage_stress/src/data.rs Adds generated test data streams.
sdk/storage/azure_storage_stress/src/fault_injection.rs Implements probabilistic fault injection.
sdk/storage/azure_storage_stress/src/futures_ext.rs Adds optional future timeouts.
sdk/storage/azure_storage_stress/src/lib.rs Implements stress orchestration and metrics.
sdk/storage/azure_storage_stress/src/value_parsers.rs Adds CLI value parsers.
sdk/storage/azure_storage_stress_blob/Cargo.toml Defines the Blob stress executable.
sdk/storage/azure_storage_stress_blob/src/clients.rs Configures authentication, clients, and proxy transport.
sdk/storage/azure_storage_stress_blob/src/main.rs Adds the executable entry point.
sdk/storage/azure_storage_stress_blob/src/roundtrip_test.rs Implements upload/download integrity testing.
Suppressed comments (2)

sdk/storage/azure_storage_stress/src/data.rs:79

  • from_iter accepts an empty iterator with a positive length, but both read implementations call next().unwrap() and panic on first use. Reject empty generators during construction or represent exhaustion as an I/O error instead of exposing a stream that panics.
    #[allow(clippy::should_implement_trait)]
    pub fn from_iter(iter: I, len: u64, chunk: Option<usize>) -> Self {
        GeneratedStream {
            generator: iter.clone().cycle(),
            generator_reset_src: iter.cycle(),

sdk/storage/azure_storage_stress/Cargo.toml:18

  • These new third-party dependency versions are managed only in this crate, while repository dependencies are centralized under the root [workspace.dependencies] and storage manifests inherit them with workspace = true (for example, sdk/storage/azure_storage_blob/Cargo.toml:20-31). Add crc-fast and log at the workspace root and inherit them here.
crc-fast = "1.9.0"
futures.workspace = true
log = "0.4.33"

Comment thread sdk/storage/azure_storage_stress/src/lib.rs Outdated
Comment thread sdk/storage/azure_storage_stress/src/args.rs Outdated
Comment thread sdk/storage/azure_storage_stress/src/lib.rs
Comment thread sdk/storage/azure_storage_stress/src/lib.rs
Comment thread sdk/storage/azure_storage_stress/src/data.rs Outdated
Comment thread sdk/storage/azure_storage_stress_blob/Cargo.toml
Comment thread Cargo.toml Outdated
Comment thread sdk/storage/azure_storage_stress_blob/Cargo.toml Outdated
&self.options
}

pub async fn run(&self) -> Result<()> {
Comment thread sdk/storage/azure_storage_stress/src/data.rs Outdated
1.10.0 bumps msrv to 1.89. We need to support 1.88.
Technically, this is for a non-published crate, so this shouldn't be necessary.

@heaths Heath Stewart (heaths) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use trace or explain why these new, different logging facilities are needed when trace has been working fine for us. You can even default trace to a different listener to produce a different format if you need.

Comment thread .vscode/cspell.json
Comment thread Cargo.toml Outdated
Comment thread Cargo.toml Outdated
@heaths

Copy link
Copy Markdown
Member

A more descriptive PR title - which becomes the commit title - wouldn't hurt either. Think of git log or git blame.

@jaschrep-msft Jocelyn (jaschrep-msft) changed the title Feature/storage/stress testing 2 Storage stress testing utilities and blob stress test executable Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine to me though admittedly I did not look too closely at the implementation details. A README on usage would be helpful.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we get a README on how to run the stress tests and some of the common configuration points?

@heaths Heath Stewart (heaths) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants