Skip to content

perf(aead-frame): add Criterion benches for the storage profile, so its cost is measured - #534

Merged
coderdan merged 5 commits into
mainfrom
feat/aead-frame-benches
Nov 1, 2026
Merged

coderdan merged 5 commits into
mainfrom
feat/aead-frame-benches

Conversation

@coderdan

@coderdan coderdan commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #562. Stacked on #532.

Summary

This adds Criterion benches for the storage profile. They write and read five values with storage::to_vec and storage::from_slice. The transport serde format, serde_json, MessagePack (rmp_serde) and Borsh run on the same values for comparison. The results are also in the crate README, under Performance.

 packages/aead-frame/
 ├── Cargo.toml        # + criterion 0.8 (no default features), borsh, rmp-serde dev-dependencies; [[bench]] storage, needs `serde`
+├── benches/
+│   └── storage.rs    # 5 values × {storage, transport, json, msgpack, borsh} × {write, read}
 └── README.md         # + Performance section with the results

Before it is timed, each value is checked to read back equal in all five formats. Each write includes dropping its result, so the wipe is measured too. MessagePack is written with field names (to_vec_named), as the storage profile, the transport format and JSON carry them. Borsh carries only the values in order, so it shows what a format costs that carries nothing else.

Neither MessagePack, Borsh nor JSON wipes anything; none of them depends on zeroize. They write into a plain Vec<u8>, so each time the buffer grows the old one is freed with its bytes still in it, and the result is freed unwiped when it is dropped. Wrapping their output in Zeroizing would cover only the drop. So on large values most of the gap below is the wipe, not the format.

Run them with:

cargo bench -p vitaminc-aead-frame --features serde --bench storage

Evidence

These ran on an Apple M4 (4 performance and 6 efficiency cores, 24 GB), macOS 26.5, rustc 1.98.1, Criterion 0.8.2 at its default settings, on mains power. Each time is the median of 100 samples. The background load was under one of the 10 cores (OrbStack, the window server, the terminal).

Size of the bytes written:

Value Storage Transport JSON MessagePack Borsh
One record of 7 fields 164 B 159 B 126 B 98 B 86 B
1,000 records 160 KB 159 KB 132 KB 102 KB 86 KB
Map of 1,000 text keys 22.0 KB 21.0 KB 14.9 KB 11.6 KB 20.0 KB
64 KiB of text 65.5 KB 65.5 KB 65.5 KB 65.5 KB 65.5 KB
64 KiB of bytes 65.5 KB 65.5 KB 234 KB 65.5 KB 65.5 KB

Time to write:

Value Storage Transport JSON MessagePack Borsh
One record of 7 fields 986 ns 1.09 µs 198 ns 337 ns 72 ns
1,000 records 937 µs 741 µs 217 µs 139 µs 30.3 µs
Map of 1,000 text keys 115 µs 80.0 µs 18.0 µs 13.6 µs 6.21 µs
64 KiB of text 35.5 µs 33.8 µs 33.8 µs 1.93 µs 2.03 µs
64 KiB of bytes 33.7 µs 33.7 µs 280 µs 1.87 µs 1.99 µs

Time to read:

Value Storage Transport JSON MessagePack Borsh
One record of 7 fields 1.35 µs 1.41 µs 440 ns 313 ns 196 ns
1,000 records 1.48 ms 1.47 ms 489 µs 467 µs 230 µs
Map of 1,000 text keys 268 µs 303 µs 216 µs 202 µs 85.4 µs
64 KiB of text 8.48 µs 8.49 µs 11.6 µs 4.14 µs 6.16 µs
64 KiB of bytes 1.91 µs 1.99 µs 881 µs 1.88 µs 2.85 µs

Two results were noisy: the MessagePack read of 1,000 records (95% interval 387 to 567 µs) and the transport write of 1,000 records (724 to 773 µs). Every other interval is within a few percent of its median. The storage, transport and JSON numbers match the first run of this PR, without MessagePack and Borsh, to within a few percent.

To explain the numbers, I ran some throwaway probes. They are not committed.

  • The wipe limits large writes. MessagePack and Borsh write 64 KiB in about 2 µs. Without the drop, a 64 KiB storage write takes 2.4 µs, the same. Wiping a 64 KiB Vec with zeroize alone takes 31.5 µs, about 2 GiB/s, because it writes one byte at a time with volatile stores. That is 93% of the 33.7 µs write. The transport writer pays the same cost.
  • Structured values pay for each part. The storage profile writes 1,000 records 6.7 times as slowly as MessagePack and 31 times as slowly as Borsh, and reads them 3.2 and 6.4 times as slowly. MessagePack carries the same field names, so the names are not the cost. In the probes, one u64 is written in 50 ns, an array of seven in 307 ns, and a struct of seven u64 fields in 703 ns (621 ns with the transport format): about 37 ns for each element and 90 ns for each field. The drop is only about 140 ns of the record's 986 ns. Reading the writer, I found no allocation for each child. I haven't found the cause: no profiler is installed here apart from xctrace. This is the clearest place to look for a speedup.
  • The storage profile is the largest format. 1,000 records take 57% more bytes than in MessagePack, and the map 89% more. Every length and count is a fixed four bytes and every value has a kind byte, where MessagePack packs small lengths and integers into one byte. Borsh is smallest for records but writes integers at full width, so its map is nearly as large as the storage profile's. The storage format is frozen at version 1, so this is a fact to note, not something this PR can change.
  • A storage write reads its blob back. It does this to find a repeated key, so it costs 26% more than a transport write for 1,000 records and 43% more for the map. Storage and transport reads are within about 12% of each other.

Checks: cargo clippy -p vitaminc-aead-frame --all-targets --all-features -D warnings, cargo +1.88.0 check --all-targets --features serde (the MSRV; criterion 0.8.2 needs 1.86), doctests, and cargo fmt --check.

Merge Danger

Door: two-way

This adds only a bench and dev-dependencies. Nothing in the library changes.

Blast Radius: none

The bench is built only by cargo bench, and by CI's --all-targets clippy and MSRV jobs. Criterion adds 19 crates to Cargo.lock, for dev builds only. borsh and rmp-serde were already in the lock file (rmp-serde is a dev-dependency of vitaminc-protected and vitaminc-traits). The README numbers depend on the machine they ran on, and will drift as the code changes.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T01:42:57.818382Z 631bf7f PR opened
🔒 Security Review ✅ Completed 2026-10-26T01:44:21.368254Z 631bf7f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

🧬 Mutation testing (cargo-mutants, --in-diff)

No mutants were generated for the changed lines.

@github-actions

Copy link
Copy Markdown
Contributor

✅ No CRAP threshold violations

1500 function(s) analyzed · threshold 30

@cipherstash-bot cipherstash-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: 🟢 merge as it is

Nothing must change before merging. This PR adds only a bench, dev-dependencies and README text. Three optional comments can wait: a doc comment, a variable name, and a CI step that runs the bench once.

How this review was made
Agent Model Review type Result
claude claude-opus-5-5 test-gap 1 found, 1 posted
claude claude-opus-5-5 rust 2 found, 2 posted
codex gpt-5.6-terra test-gap 0 found, 0 posted
codex gpt-5.6-terra rust 0 found, 0 posted

Synthesis: claude-sonnet-5-5 merged the findings, removed duplicates and dropped findings it could not confirm in the code. 0 posted finding(s) were raised by two or more models.

Plain language: claude-sonnet-5-5 read every comment as a new reader would. 2 comment(s) had a problem that stopped the reader acting; it rewrote 0.

Stack: position 3 of 3 (#532, *️⃣ #534, #557). *️⃣ marks this pull request.

Context loaded: the description, 0 linked issue(s) and 3 discussion entries.

Comment thread packages/aead-frame/benches/storage.rs Resolved
Comment thread packages/aead-frame/benches/storage.rs Resolved
Comment thread packages/aead-frame/benches/storage.rs Resolved

@auxesis auxesis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderdan thanks for taking the time to get this added — it's a really important quality signal.

@coderdan
coderdan force-pushed the feat/aead-frame-zeroize-audit branch from fb57376 to b28f087 Compare October 28, 2026 13:06
…ts cost is measured

Writes and reads five values with the storage profile, the transport serde format and serde_json, and records the results in the crate README.
…rsh, so its cost is set against compact binary formats
…sentence, and call Borsh's bytes borsh_bytes so they read apart from the crate
@coderdan
coderdan force-pushed the feat/aead-frame-benches branch from c2664d7 to 1937be3 Compare October 28, 2026 13:08
@coderdan
coderdan changed the base branch from feat/aead-frame-zeroize-audit to main November 1, 2026 09:24
@coderdan
coderdan merged commit 13367c7 into main Nov 1, 2026
11 checks passed
@coderdan
coderdan removed this pull request from stack #558 November 1, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aead-frame: measure what the storage profile costs against other formats

3 participants