Keep kopia's cache on the data directory filesystem and limit its size - #2208
Keep kopia's cache on the data directory filesystem and limit its size#2208davotoula wants to merge 1 commit into
Conversation
kopia kept its cache at /kopia/cache which lives on the data partition. On
Raspberry Pi installs that partition is small (~19GB / 1.26M inodes) and the
cache grows unbounded: kopia's default soft limits are only enforced by sweeps
when a repository is opened, so nothing bounds growth between opens. Once the
cache exhausts the partition's inodes the repository can no longer open, the
sweep can never run, and backups silently deadlock.
- Point XDG_CACHE_HOME and KOPIA_LOG_DIR at ${dataDirectory}/kopia so the
cache and logs live on the (usually much larger) data directory filesystem.
XDG_CACHE_HOME rather than KOPIA_CACHE_DIRECTORY so kopia keeps its
per-repository cache subdirectories.
- Exclude the new kopia directory from snapshots via .kopiaignore so backups
don't back up their own cache.
- Pin hard cache size limits in connect(). connect() runs before every
repository operation and rewrites the local config, so the limits can never
drift. Hard limits are enforced continuously, not just at repository open.
- Remove any cache left behind at the legacy location on startup.
|
This would be great to have a look at as it fixes a real bug and the fix is proven to work. I am using the fix now but it will be wiped on next Umbrel update. First time umbrel contributor so happy to receive any pointers on how to improve the contribution. |
|
Thanks for the ping looking into it now 🫡 |
Soak-test update — 54 h on a live Raspberry Pi 4 (umbrelOS 1.7.4)Still running clean. Numbers below are all measured on the box today (2026-08-19 19:54 UTC), ~54 h and ~54 hourly backups after the patch went on. SD card (
Inode count is flat — actually 25 lower than at deploy. For comparison, the un-patched box was accruing roughly 70–100k inodes/day on this same partition, which is what took it from clean to fatal in 22 days. SSD ( Both halves of the patch verified in place, diffed against the stock file kept alongside as
The limit flags are the half that can't be applied out-of-band: Backups themselves are healthy throughout — this isn't a "cache is small because nothing ran" result:
Happy to keep this soaking and report again at the two-week mark, or to rebase if that's useful. |
Fixes #2207
Problem
kopia keeps its cache at
/kopia/cache, which is a bind mount of/data/umbrel-os/kopiaon the data partition. On Raspberry Pi installs that partition is small (~19 GB / 1.26M
inodes) and the cache grows without bound: kopia's default limits are soft limits, only
enforced by a sweep when the repository opens, and hard limits can't be pinned out-of-band
because umbreld re-runs
repository connectbefore every operation, rewriting the configwith defaults.
On my Pi the cache reached 13 GB / 1.26M inodes for ~640 MB of backed-up data,
exhausted the partition's inodes, and deadlocked: the repository could no longer open, so
the sweep could never run, and backups were silently dead —
df -hshowed free space,only
df -ishowed the cause. See #2207 for the full incident details.Changes (all in
backups.ts)XDG_CACHE_HOMEandKOPIA_LOG_DIRnow point at
${dataDirectory}/kopia/{cache,logs}, which lives on the largest drivein the system instead of the data partition.
kopiaadded tocreateIgnoreFile(),so backups don't back up their own cache (which would balloon snapshots and regrow the
cache while snapshotting it).
connect()— the root-cause fix. Becauseconnect()runs before every repository operation and rewrites the local config, thelimits are permanently enforced and can never drift. Hard limits are enforced
continuously by kopia, not just at open. Values: content 500 MiB soft / 1000 MiB hard,
metadata 1000 MiB soft / 2000 MiB hard (~3 GB worst case, ~1.5 GB typical). This also
protects SD-only installs, where the data directory shares a partition with
/kopiaand relocation alone wouldn't help.
/kopia/cache/kopia,non-blocking so a large stale cache doesn't delay startup; no-op when absent.
Why
XDG_CACHE_HOMEand notKOPIA_CACHE_DIRECTORYWhen kopia derives the cache path from
XDG_CACHE_HOMEit gives every repository its ownhashed subdirectory (
<cache>/kopia/<hash>—repo/caching.go,setupCachingOptionsWithDefaults). An explicitKOPIA_CACHE_DIRECTORYis used verbatimand shared by all repositories — umbrelOS supports multiple backup repositories, whose
caches would mix — and
kopia repository disconnectrunsos.RemoveAllon the configuredcache directory, which with a shared directory would delete every repository's cache.
XDG_CACHE_HOMEavoids all of that. (Verified against kopia v0.19.0, the versionumbrelOS ships.)
Config files intentionally stay at
/kopia/config: the three call sites pass an explicit--config-file=/kopia/config/...which overridesXDG_CONFIG_HOME, and config files aretiny — only the cache was the problem.
Testing
Integration tests (umbrel-dev Docker environment, same setup as CI):
backups.integration.test.ts, written first and watched fail againstunmodified
masterbefore implementing:${dataDirectory}/kopia, thekopiadirectory is excluded from the snapshot, and all four cache limit values arepinned byte-exact in the repository config (connect flags multiply MB by 2^20),
/kopia/cache/kopiais removed on startup.masterbaseline in a freshly restarted container: identical results apart from the two new
tests — no regressions.
tsc --noEmitandprettier --checkclean.Field test (live Raspberry Pi on umbrelOS 1.7.4, NAS backup destination):
backups.tsapplied to/opt/umbreldon a device whose data partition was at52% inode use from the old cache (~650k inodes for ~640 MB of backed-up data).
successfully with the cache now on the data drive; the data partition has stayed at 1%.
directory and all four limit values present exactly as set