perf: mitigate slow follow/tag/rank queries (2026-07-12 outage)

[ety001]

Summary

Short-term mitigation for the three slow-query classes that caused the 2026-07-12 beta-hivemind outage (ELB MaxResponseTime 30s, jussi 502s, ~2h duration). Targets the 21.9s / 12.9s / 12.7s queries identified in the RDS slow log. No architecture changes — maximally reuses existing patterns (the @cacher decorator, pids_by_blog caching, the CONCURRENTLY migration style from #367/#371).

Investigation report: agent-share/notes/hivemind/2026-07-12-beta-hivemind-slow-query-investigation.md

Changes

1. Follow queries — hardcode state + cache (5f566a6)

get_followers, get_followers_by_page, get_following_by_page were uncached (only get_following had a 30s cache). The parameterized state IN :state tuple binding also prevented the planner from matching the v23 partial index idx_follows_follower_state_created_desc (WHERE state IN (1,3)), forcing a fallback to the full-ASC ix5a with a cross-value sort — the 21.9s query.

• Hardcode state IN (1,3) / state IN (2,3) literals so the planner can use the partial index
• Add 30s cache to the three uncached functions (aligns with get_following)
• Complete the INNER JOIN migration for get_following_by_page (missed in 3274329 and 67c6d5e where its three siblings were converted)

2. Tag filtering — cache (eed7f5e)

pids_by_category (bridge_api) and pids_by_query (condenser_api) ran uncached. Hot tags match tens of thousands of post_ids via IN (subquery) — the 12.9s query. Apply the same query_col + cache_key pattern already used by pids_by_blog in the same file, with a 60s TTL.

3. Symmetric follows index + drop duplicates (d35e4a4, v28→v29)

get_followers (WHERE following = :id ORDER BY created_at DESC) had no following-led partial index — the only one (idx_follows_follower_state_created_desc) is follower-led and serves get_following. Hardcoding state (change #1) is not enough without a symmetric index; this is the real fix for the 21.9s query.

• Add idx_follows_following_state_created_desc (following, state, created_at DESC, follower) WHERE state IN (1,3)
• Drop hive_follows_5a/5b: legacy v9 duplicates of ix5a/ix5b (identical columns), pure write-amplification; never managed by _disableable_indexes
• Bumps DB_VERSION 28→29

4. Reduce fetch_ranks frequency (56a60e7)

Accounts.fetch_ranks runs SELECT id FROM hive_accounts ORDER BY vote_weight DESC (full-table sort of million-scale rows, ~12s) hourly on the indexer's sync connection — the 12.7s query. Separate from the API pool but competes for RDS IOPS (which hit the 3000 cap during the outage). Reduced to every 6h; rank is an approximate notification-bucket score, a 6h stale window is acceptable.

Verification

• All 5 files byte-compile (py_compile)
• Diff reviewed line-by-line; clean (no formatting noise)
• Static checks: state= bindings removed from all 4 follow funcs; cache_key+cache_ttl on all target functions; DB_VERSION=29; migration block complete
• make test-all not run — local env has no PostgreSQL and the aiopg git-tarball dependency times out to install. Test suite requires a live DB; deferred to CI.

Out of scope (mid/long-term, per investigation report)

• Redis sorted-set caching for follow graph / tags
• Read replica for API queries; raise RDS IOPS
• Table partitioning

Deployment notes

• The v28→v29 migration runs on hive sync startup via CREATE INDEX CONCURRENTLY (no table lock), but consumes IOPS while building on the large hive_follows table — recommend deploying during a low-traffic window. Migration logs: [HIVE] Creating idx_follows_following_state_created_desc index...
• Caching requires REDIS_URL to be configured (the @cacher decorator skips cache when redis_cache is None)