feat(prd): move Loki to IONOS S3 and restore a highly-available distributed topology

[groundnuty]

Problem

Loki persists ~470 days of logs on prd, but only about 2 hours is queryable in Grafana.

Both things people were saying turned out to be true at once:

Claim	Verdict
"Application logs are persisted"	Correct — 127 GiB of chunks on the ingester volume
"We only see the running instance, and it's incomplete"	Correct — only ~2 h is readable

They reconcile because the data is persisted but unreachable.

Root cause

The filesystem object store requires one filesystem visible to every component. In the distributed layout the ingester and querier each hold a separate ReadWriteOnce volume:

ingester-0  PVC 200Gi -> 127.3 GB used (69%)   <- all history is here
querier-0   PVC 200Gi ->      148 KB used (0%) <- queries read from here

The querier searches an empty disk. Queries return status: success with totalChunksRef: 0, so Grafana renders "No data" rather than an error — which is why this looked like missing logs rather than a broken backend.

This cannot be fixed in place. Mounting the ingester's volume into a second pod — even readOnly, even pinned to the same node — fails:

MountVolume.SetUp failed ... mount: ... failed: Resource busy

ext4 will not mount one block device twice, and IONOS offers no RWX storage class. Distributed Loki with filesystem storage is unfixable on IONOS. Object storage removes the constraint entirely.

What changed

Storage

• object_store / shared_store: filesystem → s3 (bucket loki-prd, eu-central-3)
• schema: v11, store: boltdb-shipper and the 24h index period are unchanged — this is what makes migrated historical chunks resolve
• Credentials via SealedSecret + -config.expand-env=true; nothing secret in git

High availability

• replication_factor 1 → 2. Previously a single ingester restart stopped ingestion cluster-wide — observed twice (210 push-failures on 2026-07-13)
• ingester / distributor / querier / query-frontend / gateway → 2 replicas
• PDBs become meaningful now that replicas > 1 (relates to PRD: PodDisruptionBudgets require HA (2+ replicas) for stateful services #2497)

Fixes carried along

• query-frontend was capped at 64Mi and OOMKilled every ~8 min (23 restarts) — this was the "intermittent Loki errors" users reported. Introduced by Infrastructure: PRD — loki-distributed resource requests #2681; now given real resources.
• memcached (index-queries, chunks, frontend) enabled. Without caching every query refetches from S3 and browsing months of logs is unusably slow.
• serviceMonitor enabled. Loki previously exported no metrics to Prometheus, so failures of the logging stack itself were invisible.
• ingester PVC 200Gi → 10Gi (WAL only), querier PVC removed — chunks live in S3 now.

Complete log collection

• promtail ran on 4 of 5 nodes: the DaemonSet did not tolerate ingress-node=true:NoSchedule, so nothing on ingress-pool-* was collected — including the ingress-nginx controller's HTTP access logs, exactly what you need to debug a 4xx/5xx from a component. Now 5/5.
• tolerations replaces the chart defaults, so the master/control-plane entries are carried over deliberately.

Developer access — new "Application Logs" dashboard

• Namespace / Pod / Level / Search variables, so nobody has to write LogQL by hand
• Log volume by pod and errors over time (stacked), so a spike or a pod going silent is obvious
• Line count / error count / pods-reporting stats
• Time shortcuts out to 1y so the retained history is discoverable rather than hidden
• Deploys through the existing ionos_prd/grafana-dashboards source of kube-prometheus-stack — no Application change needed
• Datasource referenced via a datasource variable, because the Loki datasource has no fixed UID in this cluster

Data migration — already done and verified

1,892,200 objects / 122.28 GiB covering 2025-04-18 → 2026-07-31 were copied into loki-prd, from a snapshot clone so production was never touched. Zero failures. All 466 index tables present.

Verified by running a Loki with no volume at all, reading only from S3:

  1d  7d  30d  90d  180d  300d  400d   -> all resolve chunks from S3
  marketplace/desmos confirmed at 30 / 180 / 300 d
  sample retrieved: 2025-06-26T23:18:29Z  dome-wallet-frontend-5b6546dffd-j677w

Identical to the pre-migration filesystem results. (No data at 460d for any namespace — a real data boundary, not a migration gap.)

Verification done

• helm template against loki-distributed 0.79.0 renders clean; confirmed object_store: s3, shared_store: s3, replication_factor: 2, and that every Loki component receives -config.expand-env=true + envFrom: loki-s3-credentials
• helm template against promtail 6.16.0 confirms all three tolerations reach the DaemonSet
• SealedSecret applied and confirmed to unseal correctly
• S3 read / write / delete confirmed against loki-prd
• Every dashboard query and both label_values() variable queries executed against a Loki reading the migrated bucket — all HTTP 200 with data; 21 namespaces listed, pod lists correctly scoped per namespace
• Dashboard confirmed provisioned by Grafana as uid=dome-application-logs

Worth noting what the current backend does with those same dashboard queries. Run against production Loki today, the aggregations do not merely return "No data" — they fail outright:

HTTP 500  failed to load chunk 'ZmFrZS80...': open /var/loki/chunks/...: no such file or directory

That is this bug reproduced from the user's side. After cutover the identical queries return 200 with data.

⚠️ Deployment notes — please read before syncing

1. Two StatefulSets must be deleted before sync. volumeClaimTemplates are immutable, and this PR changes the ingester claim (200Gi → 10Gi) and removes the querier's. ArgoCD sync will otherwise fail with a Forbidden error:

kubectl -n loki-distributed delete sts loki-distributed-ingester loki-distributed-querier

Then sync. Old PVCs are retained, not deleted — they are the rollback path.

2. A delta copy is required after cutover. The migration snapshot froze at 2026-07-31 ~16:20 UTC. Anything written after that is only on the old volume. Once this is live, re-running the migration job copies just the delta (it skips objects already present, so it takes minutes). Without this, that window silently disappears from queries.

3. Rollback: revert this PR. The old 200Gi volume still holds every chunk.

[Copilot]

Pull request overview

Moves the production Loki deployment from local filesystem-backed storage to IONOS S3 to restore full log queryability, while re-establishing a highly-available distributed topology and improving operability (caching, metrics, dashboards).

Changes:

• Switch Loki object_store/shared_store to S3 (IONOS object storage), reduce local persistence to WAL-only, and scale core components to 2 replicas with meaningful disruption settings.
• Enable Loki memcached caches and ServiceMonitor scraping to improve query performance and observability of the logging stack.
• Ensure complete log collection (Promtail tolerations) and add a new Grafana “Application Logs” dashboard.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ionos_prd/promtail/values.yaml	Adds tolerations so Promtail schedules on ingress-tainted nodes and collects previously-missed logs.
ionos_prd/loki-distributed/values.yaml	Reconfigures Loki for S3-backed storage, HA replicas, caching, metrics, and updated persistence/resources.
ionos_prd/loki-distributed/loki-s3-sealed-secret.yaml	Adds SealedSecret providing S3 credentials via env expansion.
ionos_prd/grafana-dashboards/application-logs.yaml	Introduces an “Application Logs” dashboard with namespace/pod/level/search variables and overview panels.
applications_prd/monitoring/loki-distributed.yaml	Updates ArgoCD app to deploy the Loki S3 SealedSecret alongside the Helm release.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.