Repository navigation
QFw: bring main's Loki logs and the dashboard's Grafana link into defw2-prototype - #51
Merged
Merged
Conversation
The telemetry stack is optional and runs beside the cluster, with Grafana publishing one host port. The dashboard's Docker records now carry each container's published ports, and the health widget shows a "Telemetry" link to Grafana, at the host the page was loaded from and the port Grafana publishes, whenever its container is running. Nothing to configure, and nothing shown when the stack is absent. Signed-off-by: Doug Oucharek <dougso@me.com>
The health widget is the last group on the canvas, about five screens down, and the canvas pans only with a middle-button drag, which a trackpad does not have. The same link now also sits in the view controls of the top bar, where it is always in view. Signed-off-by: Doug Oucharek <dougso@me.com>
The cluster side of QFw's logs tier (QFw #123, #124). A Loki container with its own volume and two weeks of retention joins the stack, the collector forwards OTLP logs to it through a pipeline that trims the resource the way the metrics one does, and Grafana gets a Loki data source. The Tempo source links a span to its trace's lines, the Loki source links a line's trace id back to its trace, the QFw Trace dashboard shows a job's lines under its waterfall, and QFw Jobs lists recent warnings and errors. The overlay sets QFW_TELEMETRY_LOGS=debug, the tier at which QFw writes a job's story, and qfw-site-services forwards it to the service hosts. Loki keeps service_name, qfw_component_role and qfw_device_name as stream labels; trace_id, span_id and severity_text ride as structured metadata, which is what the queries select on. The README describes it. Signed-off-by: Doug Oucharek <dougso@me.com>
QFw: keep QFw's log lines in Loki, beside the traces
QFw: link the dashboard's health widget to Grafana when the stack is up
The Logs section said a client out of the box exports nothing. With QFw's logs tier opening its own qfw.* loggers, such a client sends its story lines and the warnings and errors of everything else; say that instead. Signed-off-by: Doug Oucharek <dougso@me.com>
QFw: say what a Qiskit client's logs tier sends
The logs panels show a line's body and hide its labels, and a job's story has two writers, so the eight lines under a trace gave no hint which two came from the client. Each query now ends in a line_format that prefixes the line with its writer, "[client]" or "[qpm fake-iqm-20q]", on the QFw Trace panel, the warnings panel of QFw Jobs and the hop from a span to its logs in Explore. The labels stay in a line's details. Signed-off-by: Doug Oucharek <dougso@me.com>
QFw: say which component wrote each log line
…w2-prototype Signed-off-by: Doug Oucharek <dougso@me.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings cluster main
5de7bb7intodefw2-prototype(528e3c5), the same kind of catch-up as #45. Merge commit7596a2f.What comes over
How it merged
No conflicts. Two files changed on both sides since #45:
tools/qfw-site-servicesand its man page. Main's change there is one line from #48, which addsQFW_TELEMETRY_LOGSto the telemetry variables the tool forwards to site services over ssh. It merged into the prototype's version, which carries the DEFw v2 plane, without touching anything else.Checked
test_scripts/test_qfw_site_services.shpasses in a container (bash 5.2), on this merge and ondefw2-prototypebefore it. On macOS's bash 3.2 it stops atforwarded[@]: unbound variableon both, since bash 3.2 treats an empty array as unset underset -u.QFW_TELEMETRY_LOGS=debug. With the image's current tool it does not.The image bakes
tools/qfw-site-services, so the next image rebuild picks this up.