Cap bills during daily scraper refreshes

[ThatXliner]

What changed

• make each scheduled Congress and Open States refresh pass --recent 100 --retain 100
• run retention once after successful ingestion for only the refreshed jurisdiction
• rank and delete overflow entirely inside PostgreSQL, returning aggregate counts instead of candidate rows or deleted UUIDs
• delete evicted bills and their polymorphic feed images, briefs, lenses, saved references, and cascading brief-change images atomically
• keep prune-bills as a manual, read-only-by-default inventory and repair command
• remove the separate weekly retention job

Why

Fetching the newest 100 bills bounds source API work, but it does not bound storage: the membership of that window changes every day, so the database retains the union of every historical window. The first implementation fixed storage growth with a weekly job, but selected victims in JavaScript and therefore transferred bill metadata from Supabase solely to make a database-local decision.

Retention now runs at the natural end of each daily bill refresh. A single data-modifying CTE selects overflow, deletes dependent rows, and returns only counts. Each jurisdiction remains independent, so federal activity cannot displace state measures.

Impact and safety

The current production read-only preview selects 1,104 bills: 27 California, 20 Missouri, 18 Texas, and 1,039 federal bills. North Carolina is already below the cap. No production rows were deleted while preparing this PR.

Routine deletion is opt-in through the supervisor's explicit --retain 100 argument and is only accepted alongside --recent for the Congress or Open States scraper. Targeted fetches, bulk imports, cursor walks, and non-bill scrapers cannot invoke it. Retention uses a transaction plus lock and statement timeouts; a busy database fails the job for a later retry rather than waiting indefinitely. PostgreSQL autovacuum makes deleted space reusable, although reported physical disk size may not fall immediately.

Checks

• pnpm --filter @acme/scraper test (130 passed)
• pnpm --filter @acme/scraper build
• pnpm --filter @acme/supervisor test (37 passed)
• pnpm --filter @acme/supervisor build
• Prettier check on all changed files
• git diff --check
• standalone local CLI dry run with writes disabled
• production inventory query (read-only)
• local transactional database fixture covering ranking, every polymorphic dependent, brief-image cascade, and rollback

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
billion-nextjs	Ready	Preview	Aug 16, 2026 3:38pm