perf(core): bucket and index timeline projections - #1722
Open
hmans wants to merge 5 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Room Timeline retained decoded protobuf payloads and string-heavy metadata for
every room and every point in history. Threads independently duplicated many of
the same stable IDs in maps and summaries. Their RAM cost therefore grew with
the complete retained message history.
Closes #1721.
What changed
preserving its one ordered EVT consumer and lightweight global indexes.
(
core.room_timeline_hot_window/CHATTO_CORE_ROOM_TIMELINE_HOT_WINDOW, default30d).direct reads, atomic installation, shared concurrent success/failure results,
and context-aware errors.
long-lived thread does not load its complete history.
timeline rows, posting lists, body state, echo/attachment indexes, thread
mappings, summaries, participants, and follows with 32-bit handles, dense
slices, and packed flags.
readiness, snapshots, and failure remain independently owned.
process-local handles, with canonical stable-ID ordering and byte-identical
round-trip coverage.
tests, ADR-056, architecture/FDR updates, and operator configuration docs.
Results
to 276 retained heap bytes/message across the full PR (77.9%).
Timeline from 432.2 to 276.0 bytes/message (36.1%), Threads from 311.3 to
118.1 bytes/message (62.1%), and their combined retained metadata from 743.5
to 394.1 bytes/message (47.0%).
53,763-event history replayed Room Timeline in 1.14 seconds and Threads in
1.14 seconds.
already-bucketed branch to 34.66 MB with compact indexes (23.2%). The
projection-specific diagnostics were also corrected to account for compact
slice capacity instead of charging map overhead per element.
Access timestamps and resident-byte accounting prepare a later bounded LRU.
Compatibility and rollout
v2contract. The indexed representation preserves that stable protobuf shape;
handles never cross the snapshot boundary.
v1snapshot contract and likewise rebuildsits independent handles during restore.
and safely cold-replay EVT once. Rolling back uses the old binary's
independent contract namespace.
retraction or user-key shred cannot reintroduce or return deleted content.
Test plan
mise codegen-protohandle-order restore
mise test-climise lintmise license-checkproduction-derived EVT messages
events