ci: Update index cache for host use - #5662
Merged
Merged
Conversation
The index cache records each file's mtime, and a consumer's boot index decides what to revisit by comparing that against the mtime on disk — skipping the files where the two agree. `git clone` stamps every file with the checkout time, so the mtimes in the dump exist nowhere but the runner that built it, no comparison ever matches, and a consumer re-indexes every realm it imported. Derive each file's mtime from a hash of its own content instead, so two checkouts of the same bytes agree. Same content gives the same mtime and the file is skipped; different content gives a different one and the file is revisited along with whatever the invalidation fan-out reaches from it. Content rather than commit time because CI clones are shallow and the skills realm is a clone of a separate repo — neither carries the history a commit-time scheme needs. Scoped to base, skills and openrouter: the realms a test stack boots without a SKIP_* flag switching it off. The remaining realms still land in the dump carrying checkout mtimes. No consumer changes behavior yet — the dev import flow sets REALM_SERVER_FULL_INDEX_ON_STARTUP=false and never compares mtimes, so this only changes what a future consumer can rely on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A consumer pays the download and the replay of everything in the snapshot it imports, and the full one covers every realm this job indexes. A host test shard serves only base, skills and openrouter — the realms it leaves out are the instance-heavy ones, so on a representative database they are four fifths of the dump (292 MB of 1487 MB uncompressed). Their rows are pure cost to a shard, and they put cards from realms it never mounts into its index. Publish a second artifact holding just those three realms. pg_dump takes no row filter, so the COPY blocks are assembled from `COPY (SELECT …) TO STDOUT` with a generated header per table; header and body share one column list so their order cannot drift, and `-q` keeps psql from writing a command tag into the data. Verified locally by replaying a generated dump into an empty database: same row counts and an identical checksum over every row. The realm filter reads the realm_urls the index actually holds and keeps the ones whose path names a wanted realm, because they are not uniformly the serving URL — the base realm registers under its canonical https://cardstack.com/base/ while the others carry the serving origin, so building either form from REALM_BASE_URL gets base wrong. Assert every scoped realm contributed a realm_url. A filter that matches nothing still produces a structurally valid dump whose COPY blocks are simply empty, which the verify step accepts, and every consumer would quietly go back to indexing from scratch with nothing to indicate why. Additive: the full artifact is built and uploaded exactly as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
FadhlanR
approved these changes
Aug 3, 2026
FadhlanR
left a comment
FadhlanR
left a comment
Contributor
There was a problem hiding this comment.
[Claude Code 🤖] Approving — well-engineered CI change. Verified the load-bearing assumptions: all four cached tables carry realm_url (host filter is valid), the import TRUNCATE list matches CACHE_TABLES, and the host dump's COPY format/column order matches the full pg_dump so either replays unchanged. The content-hash mtime scheme handles shallow/separately-cloned realms and content drift cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As preparation for #5655, which updates host tests in CI to use cached index tables, this tweaks the cache-building job on
mainto: