Skip to content

fix: stop recursiveRemoveDir from deleting sibling cache files - #101

Merged
k1LoW merged 11 commits into
mainfrom
fix-disk-cache-cleanup
Jun 30, 2026
Merged

k1LoW merged 11 commits into
mainfrom
fix-disk-cache-cleanup

Conversation

@k1LoW

@k1LoW k1LoW commented Jun 30, 2026

Copy link
Copy Markdown
Member

TestDiskCacheMaxKeys was failing on CI (Linux) because evicting one cache entry could wipe out a sibling entry's files.

The old recursiveRemoveDir counted only directories in the parent, not files. With two keys sharing the same path stem (e.g. te/st/0 and te/st/1), evicting test0 walked all the way up to cacheRoot and RemoveAll'd the entire te/ subtree, deleting test1's files too. Because ttlcache.OnEviction runs in its own goroutine, the race surfaced only on slower schedulers.

This change replaces the heuristic with os.Remove walking upward, which only succeeds on empty directories and stops naturally when a sibling file or directory remains.

Failing run: https://github.com/2manymws/rcutil/actions/runs/28413839254

The previous implementation counted only directories when walking
the parent of an evicted entry, ignoring files. When two keys
shared the same parent dir (e.g. "test0" and "test1" mapping to
"te/st/0" and "te/st/1"), evicting one walked all the way up to
cacheRoot and then RemoveAll'd the entire "te/" subtree, taking
the surviving key's files with it.

ttlcache.OnEviction runs in its own goroutine, so the race
surfaced non-deterministically. Linux CI failed TestDiskCacheMaxKeys
with "no such file or directory", while macOS happened to finish
Load before the cleanup goroutine reached recursiveRemoveDir.

Replace the dir-count heuristic with os.Remove walking upward.
os.Remove only succeeds on empty directories, so it naturally
stops as soon as any sibling file or directory remains, and the
function never touches cacheRoot itself.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@k1LoW k1LoW added the bug Something isn't working label Jun 30, 2026
@k1LoW k1LoW self-assigned this Jun 30, 2026
@k1LoW
k1LoW requested a review from Copilot June 30, 2026 01:49

This comment was marked as outdated.

The previous loop swallowed every os.Remove error except IsNotExist,
hiding permission/IO failures behind the same code path as the
expected "directory not empty" stop condition.

Check emptiness explicitly with os.ReadDir before calling os.Remove.
Real errors from ReadDir or Remove now propagate to the caller, while
IsNotExist (path stem doesn't exist, or raced with another remover)
keeps the walk going as before.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

This comment was marked as outdated.

ReadDir + Remove has a TOCTOU window where another goroutine can
store a sibling key under the same prefix between the emptiness
check and the remove call. The OS then returns ENOTEMPTY, which
the previous version would surface as a real cleanup failure.

Match ENOTEMPTY explicitly with errors.Is and stop walking instead
of propagating it. Permission and IO errors still propagate as
before.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

This comment was marked as outdated.

Two robustness fixes to recursiveRemoveDir:

1. NewDiskCache now stores filepath.Clean(cacheRoot). Paths derived
   from filepath.Join in Store are already cleaned, but cacheRoot
   was kept as user input. A trailing slash on the caller's value
   would make the "dir != cacheRoot" stop check miss, letting the
   walk reach and Remove the cache root itself once it became empty.

2. Extract isDirNotEmpty so the TOCTOU stop condition stays portable.
   POSIX returns syscall.ENOTEMPTY; Windows returns ERROR_DIR_NOT_EMPTY
   (145) instead, which syscall.ENOTEMPTY does not match.
@github-actions

This comment has been minimized.

This comment was marked as outdated.

POSIX (SUSv3) allows rmdir to return either ENOTEMPTY or EEXIST for
a non-empty directory. Linux and macOS use ENOTEMPTY, but legacy
POSIX-compatible filesystems may surface EEXIST. Match both in
isDirNotEmpty so the TOCTOU stop path stays correct everywhere.
@github-actions

This comment has been minimized.

This comment was marked as outdated.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

This comment was marked as outdated.

The existing TestRecursiveRemoveDir cases only exercise sibling
directories. The regression that motivated this PR is the sibling
*file* case, which until now was covered only indirectly by
TestDiskCacheMaxKeys, whose async eviction goroutine made the
failure race-dependent and flaky on CI.

Add a deterministic test that lays out 0.request/0.response and
1.request/1.response under a shared parent, removes the "0" files,
and asserts that recursiveRemoveDir leaves the "1" files and the
parent directory intact.
@github-actions

This comment has been minimized.

This comment was marked as outdated.

Avoid duplicating the suffix literals from diskcache.go. The test
lives in the same package, so it can reference the unexported
constants directly and stay aligned if they ever change.
@github-actions

This comment has been minimized.

This comment was marked as outdated.

gosec flags 0644 on os.WriteFile. The test only needs to create
readable files for its own process; drop to 0600.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

This comment was marked as outdated.

filepath.Clean("") returns ".", so an empty cacheRoot would silently
be treated as the current working directory and cache files would
land somewhere the caller never intended. Reject the empty input
explicitly so the caller gets a clear error instead.
@github-actions

Copy link
Copy Markdown
Contributor

Code Metrics Report

main (6e90955) #101 (1ad9980) +/-
Coverage 77.4% 74.1% -3.4%
Code to Test Ratio 1:1.9 1:1.9 +0.0
Details
  |                    | main (6e90955) | #101 (1ad9980) |  +/-  |
  |--------------------|----------------|----------------|-------|
- | Coverage           |          77.4% |          74.1% | -3.4% |
  |   Files            |              2 |              2 |     0 |
  |   Lines            |            324 |            336 |   +12 |
- |   Covered          |            251 |            249 |    -2 |
+ | Code to Test Ratio |          1:1.9 |          1:1.9 |  +0.0 |
  |   Code             |            658 |            678 |   +20 |
+ |   Test             |           1275 |           1315 |   +40 |

Code coverage of files in pull request scope (77.2% → 73.1%)

Files Coverage +/- Status
diskcache.go 73.1% -4.1% modified

Reported by octocov

@github-actions

Copy link
Copy Markdown
Contributor

BenchmarkDiscCache1MBBody-4

main (-) #101 (1ad9980) +/-
Number of iterations 1000 1000 0
Nanoseconds per iteration 227224 ns/op 219506 ns/op -7718 ns/op
Bytes allocated per iteration 70774 B/op 70887 B/op 113 B/op
Allocs per iteration 265 allocs/op 264 allocs/op -1 allocs/op
Metadata
main (-) #101 (1ad9980)
goos linux linux
goarch amd64 amd64
pkg github.com/2manymws/rcutil github.com/2manymws/rcutil
cpu AMD EPYC 7763 64-Core Processor AMD EPYC 7763 64-Core Processor

BenchmarkEncodeDecode1MBBody-4

main (-) #101 (1ad9980) +/-
Number of iterations 1000 1000 0
Nanoseconds per iteration 2321874 ns/op 2318046 ns/op -3828 ns/op
Bytes allocated per iteration 54492 B/op 55196 B/op 704 B/op
Allocs per iteration 68 allocs/op 68 allocs/op 0 allocs/op
Metadata
main (-) #101 (1ad9980)
goos linux linux
goarch amd64 amd64
pkg github.com/2manymws/rcutil github.com/2manymws/rcutil
cpu AMD EPYC 7763 64-Core Processor AMD EPYC 7763 64-Core Processor

BenchmarkNGINXCache1MBBody-4

main (-) #101 (1ad9980) +/-
Number of iterations 1000 1000 0
Nanoseconds per iteration 373987 ns/op 380043 ns/op 6056 ns/op
Bytes allocated per iteration 15230 B/op 15227 B/op -3 B/op
Allocs per iteration 98 allocs/op 98 allocs/op 0 allocs/op
Metadata
main (-) #101 (1ad9980)
goos linux linux
goarch amd64 amd64
pkg github.com/2manymws/rcutil github.com/2manymws/rcutil
cpu AMD EPYC 7763 64-Core Processor AMD EPYC 7763 64-Core Processor

Reported by octocov

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@k1LoW
k1LoW marked this pull request as ready for review June 30, 2026 04:55
@k1LoW
k1LoW merged commit 86fb2bb into main Jun 30, 2026
3 checks passed
@k1LoW
k1LoW deleted the fix-disk-cache-cleanup branch June 30, 2026 04:56
@github-actions github-actions Bot mentioned this pull request Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants