Repository navigation
QFw: remeasure the instrumentation cost with the transport spans and logs - #140
Merged
Merged
Conversation
…logs The budget section was measured before the transport spans, the logs tier and Loki existed, and it called full tracing the demonstration state, which the cluster overlay no longer runs. Measured again on the reference cluster with seven states, each adding one thing to the one before it: off, metrics, traces, the transport spans, and the logs tier at error, debug and all; six repetitions, three in that order and three in reverse, which bounds the order's own effect. The table gains the four states and a column for the log lines a job leaves; the prose says what the logs tier costs and why it is the first signal whose cost follows what a process logs rather than what a job does, including the plane's status polls between jobs; the budget gains a figure for the overlay as shipped. The footprint counts Loki: the fifth container, its image, its memory after two days of data, and its storage per line and per day. Signed-off-by: Doug Oucharek <dougso@me.com>
This was referenced Oct 11, 2026
QFw: bring main's registration-monitor fix and the remeasured cost section into defw2-prototype
#142
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only: the "Instrumentation Cost" section of
docs/design/benchmarking.md.Why
The budget was measured on 2026-10-08 with three telemetry states, before the transport spans (#120), the logs tier (#123 to #136) and Loki (QFw-SLURM-Cluster #48) existed. It called full tracing "a benchmark or a demonstration", which the cluster overlay no longer runs: the overlay ships with the transport spans on and the logs tier at
debug. The logs tier is also the first signal whose cost follows what a process logs rather than what a job does, and nobody had measured it.What was measured
The extended harness (QFw-SLURM-Cluster #52) on the reference cluster, against the image of QFw main
b82d19f: the same 300-job fake-IQM stream in seven states, each adding one thing to the one before it, three repetitions in that order and three in reverse. Medians of the six:errordebug(the overlay as shipped)allThe two orders agree on every delta to within 0.1 ms of p50 (one state excepted) and 0.5 ms of CPU. The 2026-10-08 numbers for the first three states are within the spread of the repetitions.
What changed in the doc
all, and the plane's status polls that put about 16 lines a second into Loki atdebugwhatever the job rate (some 200 MB a day on an idle plane, which is what sets Loki's size over its retention).allis for a diagnosis, not a budget.