Skip to content

QFw: remeasure the instrumentation cost with the transport spans and logs - #140

Merged
DougSO merged 1 commit into
mainfrom
overhead-budget-logs
Oct 11, 2026
Merged

DougSO merged 1 commit into
mainfrom
overhead-budget-logs

Conversation

@DougSO

@DougSO DougSO commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Docs only: the "Instrumentation Cost" section of docs/design/benchmarking.md.

Why

The budget was measured on 2026-10-08 with three telemetry states, before the transport spans (#120), the logs tier (#123 to #136) and Loki (QFw-SLURM-Cluster #48) existed. It called full tracing "a benchmark or a demonstration", which the cluster overlay no longer runs: the overlay ships with the transport spans on and the logs tier at debug. The logs tier is also the first signal whose cost follows what a process logs rather than what a job does, and nobody had measured it.

What was measured

The extended harness (QFw-SLURM-Cluster #52) on the reference cluster, against the image of QFw main b82d19f: the same 300-job fake-IQM stream in seven states, each adding one thing to the one before it, three repetitions in that order and three in reverse. Medians of the six:

State p50 p95 Client CPU QPM CPU Log lines per job
Off 10.91 ms 12.43 ms 7.51 ms 7.28 ms 0
Metrics (production default) +0.13 12.29 +0.27 +0.17 0
Metrics and traces +0.31 12.91 +0.47 +0.43 0
Plus the transport spans +0.39 12.95 +0.58 +0.53 0
Plus logs at error +0.49 13.16 +0.69 +0.68 0
Plus logs at debug (the overlay as shipped) +0.67 13.87 +0.81 +1.33 10
Plus logs at all +1.15 19.83 +0.87 +2.32 32

The two orders agree on every delta to within 0.1 ms of p50 (one state excepted) and 0.5 ms of CPU. The 2026-10-08 numbers for the first three states are within the spread of the repetitions.

What changed in the doc

  • The table has the seven states and a column for the lines a job leaves in Loki.
  • The prose explains the logs tier's cost: about 80 µs of QPM CPU per exported line, the p95 jump at all, and the plane's status polls that put about 16 lines a second into Loki at debug whatever the job rate (some 200 MB a day on an idle plane, which is what sets Loki's size over its retention).
  • The budget gains a figure for the overlay as shipped: up to 1 ms of p50 and 3 ms of CPU per job. all is for a diagnosis, not a budget.
  • The footprint counts Loki: the fifth container, 145 MB of image, the five containers' memory after two days of data, Tempo's size with the transport spans, and Loki's storage per line.

…logs

The budget section was measured before the transport spans, the logs
tier and Loki existed, and it called full tracing the demonstration
state, which the cluster overlay no longer runs. Measured again on the
reference cluster with seven states, each adding one thing to the one
before it: off, metrics, traces, the transport spans, and the logs tier
at error, debug and all; six repetitions, three in that order and three
in reverse, which bounds the order's own effect.

The table gains the four states and a column for the log lines a job
leaves; the prose says what the logs tier costs and why it is the first
signal whose cost follows what a process logs rather than what a job
does, including the plane's status polls between jobs; the budget
gains a figure for the overlay as shipped. The footprint counts Loki:
the fifth container, its image, its memory after two days of data, and
its storage per line and per day.

Signed-off-by: Doug Oucharek <dougso@me.com>
@DougSO
DougSO merged commit bdede12 into main Oct 11, 2026
6 checks passed
@DougSO
DougSO deleted the overhead-budget-logs branch October 11, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant