Repository navigation
QFw: write a job's story in five log lines, stitched to its trace - #125
Conversation
|
Second commit, found by running this on the cluster: a DEFw service sets its Python log level after QFw's |
…tier QFw's own code writes a job's story at debug: a QPM's device query, the circuit's qubit cap, a driver's progress. On the reference cluster those are the lines a trace should show, and the warning tier never saw them, while the debug tier drowned them in DEFw's work-request and RPC chatter. The category filter no longer depends on the level. DEFw's transport internals (CORE, WORKER, RPC, STACKTRACE) leave only with the new "all" tier, whatever level they would otherwise pass, so "debug" is QFw's own lines plus DEFw's service and application lines, and "all" is everything. The seam takes a logs_internals flag for tests, which cover both. Signed-off-by: Doug Oucharek <dougso@me.com>
With the logs tier on, the instrumentation layer now logs the job's milestones itself: submitted and finished on the client, received, executing and done on the QPM. Each line is written while the job's span is current, so the tier stamps it with the trace, and the logs panel under a waterfall has something to show for every backend. The fake IQM logs nothing of its own inside a job's spans, so until now its traces had no lines at all. Debug, like the rest of what QFw logs about a job, and only when the tier is on: with it off these cost a boolean test and write nothing to DEFw's file. The spans that are not current when a milestone happens, the execute span opened by begin_backend_execution and the job span outside job_scope, are made current for the one call. Signed-off-by: Doug Oucharek <dougso@me.com>
A DEFw service sets its Python log level after QFw has configured telemetry, and DEFw's set_logging_level_helper removes every handler on the root logger when it does. The tier's handler went with them, so in every real service process the tier believed it was on while nothing it owned was listening; the unit tests, with no DEFw, could not see it. logs_enabled() now puts the handler back whenever it finds it gone. The story lines ask it before they write, so the first line of the next job restores the tier for everything the process logs from then on. A test strips the root logger the way DEFw does and checks the recovery. Signed-off-by: Doug Oucharek <dougso@me.com>
d4bfeb9 to
4384208
Compare
|
Rebased onto main at |
Follow-up to #123 from wiring Loki into the cluster: the fake IQM logs nothing of its own inside a job's spans (its two per-job lines belong to the device query the client makes before the job starts), so its traces had no lines under the waterfall.
With the logs tier on, the instrumentation layer now writes the job's milestones itself, at debug like the rest of what QFw logs about a job:
submitted job <id>: N circuit(s), S shotsandjob <id> completed after T ms;received circuit <cid> (qtask N): Q qubits, S shots,executing circuit <cid> on <device> via <api path>,circuit <cid> completed on <device> after T ms.Each line is written while the job's span is current, so the tier stamps it with the trace and span ids. Two of those spans are not current on their own, the execute span
begin_backend_executionopens and the job span outsidejob_scope, so the helper makes the span current for the one call. With the tier off the lines cost a boolean test and write nothing, to Loki or to DEFw's file.The mock suite now runs the tier at debug and asserts the five lines against the trace ids of the QPM run and the client job. Syntax clean, mock 587 passed, unit 124 passed.