bug: `dataset rm` cannot delete staging files — ingestor (uid 65534) vs jobs-manager uid mismatch, no shared fsGroup

[saadqbal]

Summary

tracebloc dataset rm <name> drops the table but fails to delete the dataset's staging files on the shared PVC. The error is:

teardown incomplete — the table <schema>.<dataset> was dropped, but removing its files failed;
re-run `tracebloc dataset rm <dataset>` to remove the leftover files: removing PVC paths:
exec stream against tracebloc/<jobs-manager-pod>: command terminated with exit code 1
(rm: cannot remove '/data/shared/.tracebloc-staging/<dataset>/labels.csv': Permission denied)

The suggested "re-run" never succeeds — it fails on the same permission error every time. Orphaned staging files accumulate on the shared PVC while the dataset appears removed (the table is gone), masking the leak.

Reported during dataset ingestion/removal testing.

Root cause (verified)

A UID mismatch with no fsGroup bridge:

• The ingestor Job writes staging files as uid 65534 — tracebloc/data-ingestors Dockerfile:55 → USER 65534.
• The teardown rm -rf is exec'd inside the jobs-manager pod, which runs as its image UID (not 65534):
  • tracebloc/cli internal/push/teardown.go:91 builds the rm (append([]string{"rm","-rf"}, plan.PVCPaths...)), error wrap :93; exec stream internal/push/stream.go:100; user-facing wrap internal/cli/dataset_rm.go:190-191.
• The jobs-manager pod sets runAsNonRoot: true but no runAsUser and no fsGroup — client chart client/templates/jobs-manager-deployment.yaml:30-33.

A non-root UID that is not 65534 cannot delete uid-65534-owned files in a directory that is not group-writable → Permission denied. The cli comment at internal/cli/dataset_rm.go:187 ("idempotent, so re-running completes the cleanup") assumes a transient failure; that assumption does not hold for a permission error, so the retry advice is dead-end.

Caveat that makes this non-trivial

fsGroup is not applied to hostPath volumes (kubernetes/kubernetes#138411 — already noted in this chart for the bare-metal mysql init). On bare-metal / hostPath clusters, adding fsGroup alone will not fix it.

Options (design decision needed before coding)

1. Shared fsGroup on both pods + group-writable staging — clean on CSI / dynamic PVs, no-op on hostPath.
2. Ingestor creates staging dirs group-writable / setgid so any group member can clean up.
3. Ingestor owns cleanup of its own staging (delete from a uid-65534 context); cli only drops the table.
4. Run the teardown rm as uid 65534 (dedicated pod / initContainer).

Affected repos: client (chart securityContext — this issue's home), client-runtime (jobs-manager image / uid), data-ingestors (staging dir perms), cli (teardown path + the misleading retry message).

Secondary fix (cli)

tracebloc/cli internal/cli/dataset_rm.go:187-191: do not advise "re-run … to remove the leftover files" when the failure is a permission error — re-running cannot help. Detect EACCES and give accurate guidance (or an operator-side privileged cleanup path).

Acceptance criteria

• tracebloc dataset rm <name> removes both the table and all staging files on supported volume types; hostPath behavior documented explicitly.
• No "re-run" advice surfaced for a non-recoverable permission failure.

Refs

• Related (a failed ingest that reaches file-transfer leaks the same staging files this can't clean up): bug: validator-rejected ingest leaves an orphaned table; no rollback blocks the next ingest data-ingestors#260

[LukasWodka]

Refinement: corrected root cause + recommended fix

Did a code-level deep dive. Two refinements, one of which changes the fix.

Root cause is three UIDs, not two

The shared PVC is touched by three non-aligned identities, and the file that actually fails to delete is written by the CLI's ephemeral stage pod, not the ingestor:

Actor	UID	Writes	Code
CLI stage pod (ephemeral)	65532 (+ FSGroup 65532)	.tracebloc-staging/<table>/ (source files incl. labels.csv)	cli internal/push/pod.go:188,229
Ingestor Job	65534	/data/shared/<table>/ (sidecar files: images/masks/…)	client-runtime submit_ingestion_run.py (pod securityContext)
jobs-manager (runs the rm)	1001, no runAsUser/fsGroup	— just execs the rm	client templates/jobs-manager-deployment.yaml:30

dataset rm does a single rm -rf <FinalDest> <Staged> by exec-ing into the jobs-manager pod (1001) (internal/push/teardown.go:91). The failing path .tracebloc-staging/<table>/labels.csv is owned by 65532, so the precise blocker is uid 1001 cannot delete a directory owned by uid 65532. For a tabular dataset there are no sidecar files, so /data/shared/<table> doesn't exist and the staging dir is the only leftover. A fix framed around the ingestor's 65534 files misses the actual blocker.

Option 1 (fsGroup on jobs-manager) is a no-op on installer/bare-metal

Installer / bare-metal deployments use hostPath, and fsGroup is not applied to hostPath volumes (kubernetes/kubernetes#138411 — already noted in this chart for the mysql init). So an fsGroup-only fix repairs CSI clusters (AKS/EKS/OpenShift) but leaves every hostPath cluster broken. That rules out Option 1 as the primary.

Recommendation: sharpened Option 4 — ephemeral teardown pod as uid 65532

Run the teardown rm from an ephemeral pod that mirrors the stage pod's identity (uid 65532 + FSGroup 65532, PVC-mounted). The CLI already builds exactly this pod for staging (internal/push/pod.go), so we reuse the machinery.

• The staging dir was created by uid 65532, so a uid-65532 teardown pod owns it and deletes it by ownership — no group/fsGroup needed → works on hostPath and CSI alike. This fully fixes the tabular/common case on every volume type.
• Reuses the existing PSA-restricted, PVC-mounting pod pattern; adds no PVC-delete powers to the long-lived jobs-manager.

Complement, only for image/sidecar datasets (ingestor wrote /data/shared/<table> as 65534): align that path's group with the teardown identity — ingestor Job fsGroup/supplementalGroups: 65532 + create DEST_PATH group-writable + setgid in data-ingestors. Not needed for tabular datasets.

Secondary (cli): drop the "re-run … to remove the leftover files" advice on a permission failure (internal/cli/dataset_rm.go:187-191). The comment there assumes a transient error; that doesn't hold for EACCES, so the retry advice is a dead end. With the ephemeral-pod fix the rm now succeeds anyway.

Mapping to the listed options: Option 1 → out (hostPath no-op). The answer is Option 4, but as uid 65532 (the stage-pod uid that actually wrote the files), not 65534. Option 3 (ingestor owns cleanup) is the right fix for the sibling failed-ingest leak (data-ingestors#260), not for rm.

Where it lands

• cli — the real fix: ephemeral-65532 teardown pod + EACCES message. Ships independently and fixes hostPath. (implementing now)
• client-runtime + data-ingestors — only for image/sidecar datasets: ingestor fsGroup 65532 + group-writable DEST_PATH.
• client chart — Option 1 dropped as primary.

Acceptance criteria

• dataset rm <name> removes the table and all staging files on hostPath and CSI.
• No "re-run" advice surfaced for a non-recoverable permission failure.

Operator unblock on a hostPath box in the meantime: sudo rm -rf <host-data-dir>/.tracebloc-staging/<dataset> directly on the node.

[LukasWodka]

Opened the cli fix: tracebloc/cli#78 — runs the teardown rm in an ephemeral uid-65532 stage-identity pod (owns the staging files → deletes by ownership on hostPath and CSI), plus the EACCES message fix. Includes a regression test pinning "rm runs in a 65532 pod, not jobs-manager."

Fully resolves the tabular case (no sidecar files) on every volume type. Leaving this issue open for the image/sidecar complement (ingestor /data/shared/<table> written as uid 65534 → needs ingestor fsGroup 65532 + group-writable DEST_PATH in client-runtime/data-ingestors to be deletable on hostPath).