Storage stress testing utilities and blob stress test executable - #5169
Storage stress testing utilities and blob stress test executable#5169Jocelyn (jaschrep-msft) wants to merge 23 commits into
Conversation
Standardize logging and formatting. Option for pretty logs. Option to alter how many loops before tally log.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Introduces a reusable Storage stress-test framework and a Blob Storage roundtrip stress executable.
Changes:
- Adds concurrent stress orchestration, metrics, timeouts, data generation, and fault injection.
- Implements repeated blob upload/download integrity checks.
- Registers both internal crates and their dependencies.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
Cargo.toml |
Registers the stress crates. |
Cargo.lock |
Locks their dependencies. |
sdk/storage/azure_storage_stress/Cargo.toml |
Defines the framework crate. |
sdk/storage/azure_storage_stress/src/args.rs |
Defines runner and fault options. |
sdk/storage/azure_storage_stress/src/data.rs |
Adds generated test data streams. |
sdk/storage/azure_storage_stress/src/fault_injection.rs |
Implements probabilistic fault injection. |
sdk/storage/azure_storage_stress/src/futures_ext.rs |
Adds optional future timeouts. |
sdk/storage/azure_storage_stress/src/lib.rs |
Implements stress orchestration and metrics. |
sdk/storage/azure_storage_stress/src/value_parsers.rs |
Adds CLI value parsers. |
sdk/storage/azure_storage_stress_blob/Cargo.toml |
Defines the Blob stress executable. |
sdk/storage/azure_storage_stress_blob/src/clients.rs |
Configures authentication, clients, and proxy transport. |
sdk/storage/azure_storage_stress_blob/src/main.rs |
Adds the executable entry point. |
sdk/storage/azure_storage_stress_blob/src/roundtrip_test.rs |
Implements upload/download integrity testing. |
Suppressed comments (2)
sdk/storage/azure_storage_stress/src/data.rs:79
from_iteraccepts an empty iterator with a positive length, but both read implementations callnext().unwrap()and panic on first use. Reject empty generators during construction or represent exhaustion as an I/O error instead of exposing a stream that panics.
#[allow(clippy::should_implement_trait)]
pub fn from_iter(iter: I, len: u64, chunk: Option<usize>) -> Self {
GeneratedStream {
generator: iter.clone().cycle(),
generator_reset_src: iter.cycle(),
sdk/storage/azure_storage_stress/Cargo.toml:18
- These new third-party dependency versions are managed only in this crate, while repository dependencies are centralized under the root
[workspace.dependencies]and storage manifests inherit them withworkspace = true(for example,sdk/storage/azure_storage_blob/Cargo.toml:20-31). Addcrc-fastandlogat the workspace root and inherit them here.
crc-fast = "1.9.0"
futures.workspace = true
log = "0.4.33"
| &self.options | ||
| } | ||
|
|
||
| pub async fn run(&self) -> Result<()> { |
1.10.0 bumps msrv to 1.89. We need to support 1.88. Technically, this is for a non-published crate, so this shouldn't be necessary.
Heath Stewart (heaths)
left a comment
There was a problem hiding this comment.
Please use trace or explain why these new, different logging facilities are needed when trace has been working fine for us. You can even default trace to a different listener to produce a different format if you need.
|
A more descriptive PR title - which becomes the commit title - wouldn't hurt either. Think of |
Jacob Lauzon (jalauzon-msft)
left a comment
There was a problem hiding this comment.
This looks fine to me though admittedly I did not look too closely at the implementation details. A README on usage would be helpful.
There was a problem hiding this comment.
Could we get a README on how to run the stress tests and some of the common configuration points?
Stress test framework for storage with a blobs roundtrip transfer test implementation.
Work is split into two new crates.
azure_storage_stressis a library containing the runner and traits to be implemented for different tests and services.azure_storage_stress_blobsis an executable containing a blob roundtrip test definition and minimal setup code.Sample outputs
Sample
--helpusage of the roundtrip test.Sample run with info-level logging enabled: