Repository navigation
feat(monitoring): implement Dashboard 2 (pipeline ops) and Dashboard 3 (data quality & drift) - #6175
Conversation
…3 (data quality & drift) - Add Dashboard 2: Ingestion & Worker Pipeline Operations (Pub/Sub backlog, drain speed, worker latencies, failed tasks, reconciler & deleter metrics). - Add Dashboard 3: Vulnerability Data Quality & Ecosystem Health (drift detection, linter findings, feed staleness, archive sizes, and Gitter ops). - Register dashboards in environments/oss-vdb-test/monitoring/kustomization.yaml configMapGenerator.
…g configs mktemp creates directories with 0700 permissions by default, preventing the non-root grafana user in the Docker container from reading the mounted dashboards.yaml and datasources.yaml provisioning files.
| c.collectDatasetAndExportMetrics(ctx) | ||
| } | ||
|
|
||
| func (c *Collector) collectSourceFreshnessAndPrefixes(ctx context.Context) map[string]string { |
There was a problem hiding this comment.
I don't really like this function doing two distinct things (recording the last updates metrics, and returning a prefix -> source map)
Could this maybe just return the prefixes and the last-updated times as maps, then have the caller iterate through and publish the metrics?
| metrics.RecordCheckerDrift.WithLabelValues("gcs_missing_or_stale").Set(float64(len(invalidIDs))) | ||
| countsBySource := make(map[string]int) | ||
| for _, id := range invalidIDs { | ||
| prefix := strings.Split(id, "-")[0] + "-" |
There was a problem hiding this comment.
nit: I like strings.Cut better for this
But also, a few of our android prefixes don't have the trailing - on their IDs. Not sure if we should just standardise the android ones, or swap this to a slices.ContainsFunc calling strings.HasPrefix(prefix, id)
| // hard-coded cluster not ideal, but this is the only cluster in the instance. | ||
| "project_id": project, | ||
| "location": "us-central1-f", |
There was a problem hiding this comment.
so... we moved the cluster to us-west2-b on test / us-west2-c on prod............
I guess we could query the gce metadata server, or just put it as an environment variable
Multi-hyphen prefixes: Android IDs (e.g., ASB-A-153358911 or PUB-A-12345) have multiple hyphens. Any cut or split on - (such as strings.Cut(id, "-")) cuts at the first hyphen, yielding "ASB" (or "ASB-"), which would never match "ASB-A" so even if source.yaml were updated to standardize Android prefixes with trailing hyphens (ASB-A-, PUB-A-), strings.Cut(id, "-") would still fail for the reason above.
2 participants
Adds Prometheus/GMP metrics and Grafana Dashboards 2 & 3 to monitor the ingestion pipeline, worker pools, Gitter, and vulnerability data quality.
How Metrics Are Collected
Long-running services (
worker,gitter,recoverer):/metricsendpoint on port9090(go/internal/metrics), scraped by GMPPodMonitoringevery 15s.default,reimport, andcvespools), Gitter requests/cache disk usage, and recovery tasks.Batch & dataset state metrics (
custommetrics):importer-reconcilerandimporter-deleternow save run summaries to Datastore (JobData).custommetrics(go/cmd/custommetrics) into a lightweight 1-replicaDeploymentthat polls Datastore (JobData,SourceRepository,ImportFindings) and GCS (all.zip,modified_id.csv) every 5 minutes and exposes Prometheus gauges on:9090/metrics(drift counts, linter findings, feed freshness, and dataset size).Grafana & GKE Config (
deployment/clouddeploy/gke-workers/):PodMonitoringscraping forworkers,gitter,recoverer, andcustommetrics.pipeline-ingestion-ops.json) and Dashboard 3 (data-quality-drift.json).It is somewhat difficult to know if this works without deploying it unfortunately.