Skip to content

Keep kopia's cache on the data directory filesystem and limit its size - #2208

Open
davotoula wants to merge 1 commit into
getumbrel:masterfrom
davotoula:fix/kopia-cache-relocation-and-limits
Open

Keep kopia's cache on the data directory filesystem and limit its size#2208
davotoula wants to merge 1 commit into
getumbrel:masterfrom
davotoula:fix/kopia-cache-relocation-and-limits

Conversation

@davotoula

Copy link
Copy Markdown

Fixes #2207

Problem

kopia keeps its cache at /kopia/cache, which is a bind mount of /data/umbrel-os/kopia
on the data partition. On Raspberry Pi installs that partition is small (~19 GB / 1.26M
inodes) and the cache grows without bound: kopia's default limits are soft limits, only
enforced by a sweep when the repository opens, and hard limits can't be pinned out-of-band
because umbreld re-runs repository connect before every operation, rewriting the config
with defaults.

On my Pi the cache reached 13 GB / 1.26M inodes for ~640 MB of backed-up data,
exhausted the partition's inodes, and deadlocked: the repository could no longer open, so
the sweep could never run, and backups were silently dead — df -h showed free space,
only df -i showed the cause. See #2207 for the full incident details.

Changes (all in backups.ts)

  1. Move cache and logs to the data directoryXDG_CACHE_HOME and KOPIA_LOG_DIR
    now point at ${dataDirectory}/kopia/{cache,logs}, which lives on the largest drive
    in the system instead of the data partition.
  2. Exclude the new location from snapshotskopia added to createIgnoreFile(),
    so backups don't back up their own cache (which would balloon snapshots and regrow the
    cache while snapshotting it).
  3. Pin hard cache size limits in connect() — the root-cause fix. Because
    connect() runs before every repository operation and rewrites the local config, the
    limits are permanently enforced and can never drift. Hard limits are enforced
    continuously by kopia, not just at open. Values: content 500 MiB soft / 1000 MiB hard,
    metadata 1000 MiB soft / 2000 MiB hard (~3 GB worst case, ~1.5 GB typical). This also
    protects SD-only installs, where the data directory shares a partition with /kopia
    and relocation alone wouldn't help.
  4. Clean up the legacy cache on startup — rename-then-delete of /kopia/cache/kopia,
    non-blocking so a large stale cache doesn't delay startup; no-op when absent.

Why XDG_CACHE_HOME and not KOPIA_CACHE_DIRECTORY

When kopia derives the cache path from XDG_CACHE_HOME it gives every repository its own
hashed subdirectory (<cache>/kopia/<hash>repo/caching.go,
setupCachingOptionsWithDefaults). An explicit KOPIA_CACHE_DIRECTORY is used verbatim
and shared by all repositories — umbrelOS supports multiple backup repositories, whose
caches would mix — and kopia repository disconnect runs os.RemoveAll on the configured
cache directory, which with a shared directory would delete every repository's cache.
XDG_CACHE_HOME avoids all of that. (Verified against kopia v0.19.0, the version
umbrelOS ships.)

Config files intentionally stay at /kopia/config: the three call sites pass an explicit
--config-file=/kopia/config/... which overrides XDG_CONFIG_HOME, and config files are
tiny — only the cache was the problem.

Testing

Integration tests (umbrel-dev Docker environment, same setup as CI):

  • Two new tests in backups.integration.test.ts, written first and watched fail against
    unmodified master before implementing:
    • after a backup, kopia's cache and logs live under ${dataDirectory}/kopia, the
      kopia directory is excluded from the snapshot, and all four cache limit values are
      pinned byte-exact in the repository config (connect flags multiply MB by 2^20),
    • a stale legacy cache at /kopia/cache/kopia is removed on startup.
  • Full backups integration suite run with the change and again on a stashed master
    baseline in a freshly restarted container: identical results apart from the two new
    tests — no regressions.
  • tsc --noEmit and prettier --check clean.

Field test (live Raspberry Pi on umbrelOS 1.7.4, NAS backup destination):

  • Patched backups.ts applied to /opt/umbreld on a device whose data partition was at
    52% inode use from the old cache (~650k inodes for ~640 MB of backed-up data).
  • On restart the legacy cleanup dropped the partition to 1% (383 inodes) within minutes.
  • A manually triggered backup and subsequent scheduled hourly backups completed
    successfully with the cache now on the data drive; the data partition has stayed at 1%.
  • Repository config inspected directly: per-repository cache subdirectory under the data
    directory and all four limit values present exactly as set

kopia kept its cache at /kopia/cache which lives on the data partition. On
Raspberry Pi installs that partition is small (~19GB / 1.26M inodes) and the
cache grows unbounded: kopia's default soft limits are only enforced by sweeps
when a repository is opened, so nothing bounds growth between opens. Once the
cache exhausts the partition's inodes the repository can no longer open, the
sweep can never run, and backups silently deadlock.

- Point XDG_CACHE_HOME and KOPIA_LOG_DIR at ${dataDirectory}/kopia so the
  cache and logs live on the (usually much larger) data directory filesystem.
  XDG_CACHE_HOME rather than KOPIA_CACHE_DIRECTORY so kopia keeps its
  per-repository cache subdirectories.
- Exclude the new kopia directory from snapshots via .kopiaignore so backups
  don't back up their own cache.
- Pin hard cache size limits in connect(). connect() runs before every
  repository operation and rewrites the local config, so the limits can never
  drift. Hard limits are enforced continuously, not just at repository open.
- Remove any cache left behind at the legacy location on startup.
@davotoula

Copy link
Copy Markdown
Author

@lukechilds

This would be great to have a look at as it fixes a real bug and the fix is proven to work. I am using the fix now but it will be wiped on next Umbrel update.

First time umbrel contributor so happy to receive any pointers on how to improve the contribution.

@lukechilds

Copy link
Copy Markdown
Member

Thanks for the ping looking into it now 🫡

@davotoula

Copy link
Copy Markdown
Author

@lukechilds

Soak-test update — 54 h on a live Raspberry Pi 4 (umbrelOS 1.7.4)

Still running clean. Numbers below are all measured on the box today (2026-08-19 19:54 UTC), ~54 h and ~54 hourly backups after the patch went on.

SD card (/dev/mmcblk0p6, the partition that originally hit ENOSPC at 100% inodes):

Before the fix At deploy (2026-08-17) Now (+54 h)
Inodes used 1,261,567 / 1 free (100%) 383 (1%) 358 (1%)
Space used 15 G (76%) 85 M (1%) 100 M (1%)

Inode count is flat — actually 25 lower than at deploy. For comparison, the un-patched box was accruing roughly 70–100k inodes/day on this same partition, which is what took it from clean to fatal in 22 days.

SSD (/dev/sda1, where the cache now lives): 773 K of 117 M inodes used — 1%. The relocation has somewhere on the order of a thousand times the inode headroom of the SD partition, and it is no longer unbounded anyway because of the hard limits below.

Both halves of the patch verified in place, diffed against the stock file kept alongside as backups.ts.orig:

stock patched
XDG_CACHE_HOME /kopia/cache (SD card) ${this.#umbreld.dataDirectory}/kopia/cache (SSD)
--content-cache-size-mb / -limit-mb (absent) 500 / 1000
--metadata-cache-size-mb / -limit-mb (absent) 1000 / 2000

The limit flags are the half that can't be applied out-of-band: connect() runs on ordinary read operations and rewrites <repo-id>.config, so any manual kopia cache set is discarded on the next backups.* RPC. Re-passing them on every connect() is what makes them stick.

Backups themselves are healthy throughout — this isn't a "cache is small because nothing ran" result:

  • 32 snapshots retained, hourly, most recent 23 min before this check
  • snapshot size steady at ~640–660 MB
  • backups.getRepositories.query and backups.listBackups.query both return cleanly (the former was the RPC that failed outright with unable to write config file: ... no space left on device during the incident)

Happy to keep this soaking and report again at the two-week mark, or to rebase if that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backups die silently when kopia's unbounded cache exhausts the data partition's inodes

2 participants