Skip to content

Cache CGCs, F- and R-symbols and direct products with MemoizationKit - #59

Draft
lkdvos wants to merge 2 commits into
mainfrom
ld/memoizationkit
Draft

lkdvos wants to merge 2 commits into
mainfrom
ld/memoizationkit

Conversation

@lkdvos

@lkdvos lkdvos commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

The caches of SUNRepresentations, now with MemoizationKit.jl:

  • F- and R-symbols were cached in one LRU per rank, which takes a lock on every lookup, hits included, so that TensorKit reading them from many threads serializes on it. They are now @cached functions SUN_Fsymbol and SUN_Rsymbol, in MemoizationKit's default ClockCache, which is read without a lock (since MemoizationKit 0.1.1).
  • Direct products: the distinct irreps of s1 ⊗ s2 (productirreps) are cached in the same way. Their result is shared and must not be mutated. Nsymbol stays uncached: for small products it costs about as much as a cache lookup.
  • Clebsch-Gordan coefficients (SUN_CGC) are cached in RAM and in MemoizationKit's disk cache, an SQLite database per machine, which replaces the JLD2 files with pidfile locks. Processes on one machine share it, and writes are transactions. The coefficients are written in a compact format of their own (CGCSerializer: the size, and the linear indices and values of the nonzero entries), which does not depend on the Julia version, and is a third smaller and 7× faster to read than that of Serialization.
  • The CGC, F- and R-symbol caches keep their previous size of 10⁵ entries in RAM (MemoizationKit.default_cache_settings), which can be changed with MemoizationKit's preferences; the direct products use MemoizationKit's default of 10⁴.
  • The README example of ⊗ now shows the sorted order of Sort ⊗ output and speed up Nsymbol and ⊗ #58.

Breaking (for 0.5.0):

  • use_disk_cache, cgc_cache_dir, clear_disk_cache!, ram_cache_info and disk_cache_info are removed, in favour of MemoizationKit's preferences and functions: set_cache_preferences!(SUNRepresentations; disk_path = ...) or disk = false, empty_disk_caches!(SUNRepresentations.SUN_CGC) and empty_caches!(SUNRepresentations). precompute_disk_cache(N, a_max, T) (without force) and cache_info() remain; the latter lists all four caches and the disk cache of the CGCs. The README describes the new setup.
  • Previously cached CGCs are not read anymore, and are computed again.
  • Dependencies: SQLite and Serialization replace JLD2, Pidfile, Scratch and LRUCache. SQLite loads in 0.18 s, JLD2 in 0.78 s.

Tests: the CGC disk cache is filled by precompute_disk_cache, and read back once the RAM cache is emptied (counted as disk hits, and equal to the coefficients computed without any cache); CGCSerializer round-trips CGCs with Float64 and Float32 entries.

Benchmarks

The reproducer of #56 (cold construction of the recoupling data of one permutation in a CTMRG step of an SU(3)-symmetric PEPS), minimum over two interleaved rounds:

main this PR
cold permute, 1 thread 0.96 s 0.60 s
cold permute, 8 threads 0.41 s 0.15 s

Cache hits (@belapsed), and when many threads hit at once (8 tasks, each looking up the same 2404 warm SU(3) F-symbols, time per lookup):

main this PR
Fsymbol hit 126 ns 38 ns
Rsymbol hit 95 ns 22 ns
iterate (1, 1) ⊗ (2, 1) 157 ns 19 ns
Fsymbol hits, 8 tasks 968 ns 14 ns

Reading CGCs from the disk cache, with the RAM cache emptied, per CGC, with the cache in its default location (a scratch space in ~/.julia, here on a network file system) and on a local disk:

default: main default: this PR local: main local: this PR computing
SU(3), labels ≤ 2 (309 CGCs) 7.0 ms 0.71 ms 0.33 ms 0.03 ms 0.5 ms
SU(3), labels 3–4 ⊗ ≤ 2 (300 CGCs) 6.9 ms 0.75 ms 0.39 ms 0.04 ms 0.6–1.2 ms
SU(4), labels ≤ 1 (256 CGCs) 7.3 ms 0.90 ms 0.39 ms 0.13 ms 2.5–4.2 ms
SU(5), labels ≤ 1 (348 CGCs) 12.4 ms 6.9 ms 4.9 ms 5.6 ms 140–170 ms

On the network file system, the file locks of main made reading a small CGC from disk over 10× slower than computing it; with SQLite, reading the smallest CGCs takes about as long as computing them, and the disk cache pays off for everything larger. On a local disk, reading is faster than computing for all of them. For large CGCs (SU(5)), reading the entries dominates, and the two are within 20% on a local disk.

Caveats
  • Computations run outside the locks, so threads that miss the same key at the same time compute it more than once (1901 misses for 1501 F-symbols in the reproducer), as with LRUCache before.
  • Disk lookups take SQLite's file locks, which are slower on network file systems; a local disk_path is faster still.

MemoizationKit v0.1.1, TensorKit v0.17.2, TensorKitSectors v0.3.11, Julia 1.13.2, shared 32-core workstation (load average 1–9 during these runs; the tables use the minimum).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/SUNRepresentations.jl 100.00% <ø> (ø)
src/caching.jl 94.44% <100.00%> (+4.97%) ⬆️
src/clebschgordan.jl 97.46% <100.00%> (+0.14%) ⬆️
src/gtpatterns.jl 95.83% <100.00%> (ø)
src/sector.jl 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lkdvos
lkdvos force-pushed the ld/sorted-directproduct branch from 8beb0d7 to cba7abe Compare October 15, 2026 15:35
Base automatically changed from ld/sorted-directproduct to main October 16, 2026 12:23
@lkdvos
lkdvos force-pushed the ld/memoizationkit branch from 54c98dc to c613a5d Compare October 23, 2026 13:37
@lkdvos lkdvos changed the title Cache F- and R-symbols and direct products with MemoizationKit Cache CGCs, F- and R-symbols and direct products with MemoizationKit Oct 23, 2026
lkdvos and others added 2 commits October 23, 2026 10:51
The F- and R-symbols were cached in one `LRU` per rank, which takes a lock on every lookup,
hits included, so that TensorKit reading them from many threads serializes on it. They are now
`@cached` with MemoizationKit (`SUN_Fsymbol` and `SUN_Rsymbol`), whose default `ClockCache` is
read without a lock (since MemoizationKit 0.1.1). The distinct irreps of a direct product, which
`⊗` iterates over (`productirreps`), are cached in the same way. `Nsymbol` stays uncached: for
small products it costs about as much as a cache lookup.

The Clebsch-Gordan coefficients (`SUN_CGC`) are cached in RAM and in MemoizationKit's disk
cache, an SQLite database per machine, which replaces the JLD2 files with pidfile locks. They are
written in a compact format of their own (`CGCSerializer`), which does not depend on the Julia
version, and is a third smaller and 7x faster to read than that of `Serialization`. The
location of the disk cache and whether it is used are MemoizationKit preferences, which replace
`use_disk_cache`, `cgc_cache_dir` and `clear_disk_cache!`; `precompute_disk_cache` and
`cache_info` remain. SQLite replaces JLD2, Pidfile, Scratch and LRUCache as dependencies. The
CGC, F- and R-symbol caches keep their previous size of 10^5 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lkdvos
lkdvos force-pushed the ld/memoizationkit branch from c613a5d to 0942f3f Compare October 24, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant