Run the rollup backfill scan from the worker instead of the scheduler - #28
Merged
Conversation
The scheduler app has no S3 credentials, so listing archived days from it failed every tick with ECONNREFUSED against the localhost fallback and no backfill job was ever enqueued. The scheduler now enqueues a single parameterless scan job, and the worker lists the archive, picks the missing days, and backfills them oldest first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The rollup backfill never processed old archived snapshots in production: the scheduler listed archived days from S3, but the scheduler app has no S3 credentials, so every tick failed with ECONNREFUSED against the localhost fallback and no backfill job was ever enqueued. The scheduler now enqueues a single parameterless scan job, and the worker — which has S3 access — lists the archive, filters days fully past the snapshot retention window, diffs them against already rolled-up days, and backfills up to 4 missing days per scan, oldest first. This also removes the retention and skip checks that were duplicated between the two apps, and no new Fly secrets are needed to deploy.
🤖 Generated with Claude Code