python-math #33: docs(delphi): overnight pre-cutover audit (s6) — CUTOVER_RUNBOOK + journal

delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md

@@ -3966,3 +3966,82 @@ airtight version). Push + CI follow the verdicts.
    are intentionally inert, Q1); improved mode's ban path has no live
    equiv coverage. Follow-up candidate for the cutover phase (improved
    mode is the post-cutover option per MATH_POLLER_DESIGN.md).
+
+## Session 6 (2026-07-26, overnight): pre-cutover audit + fixes
+
+Stack audit for the clojure-off/python-on decision (Julien overnight
+mandate). State found + actions:
+
+- **Stack**: 32 Draft PRs, base-pointer chain verified correct end-to-end
+  (spr's 4th-column ❌ on the upper 13 is metadata cosmetics; every head
+  OID checked == local). NEW top commit since s5: #2658 zid-sharding
+  (another session, 2026-07-25 — scheduling scaffolding, opt-in,
+  process-per-shard; measured thread-pool serial fraction 0.988 → 15.7x
+  at 16 processes).
+- **CI**: (a) #2637's own run failed on ONE test —
+  test_py_poller_runner_cmd_cwd_env asserted cwd.name == "delphi", which
+  is "app" in the CI container; layout-fragile assertion dropped, fix
+  squashed into #2657's commit, stack pushed (retriggers checks).
+  (b) python-ci dispatched on the sharding head (was never run there).
+  (c) #2648 mid-stack red = two certify tests failing at that STACK
+  POSITION only (they pass from #2656's position up — the fix rode a
+  later commit); old run re-run; if still red it is a stack-position
+  artifact, not a tip defect. (d) observed in #2637's run logs: a
+  postgres "null zid" constraint ERROR from the integration flow with no
+  failing test — noted, unexplained, non-blocking.
+- **Copilot**: 14 older PRs reviewed (all threads resolved but two);
+  the two unresolved threads (#2618 engine_mode coupling, #2622 stale
+  group_clusterings) were both ALREADY FIXED by later stack PRs (#2641
+  resolver move; #2642 Q4 overwrite) — replied with citations, resolved.
+  18 newer PRs (#2641-#2658) had NO Copilot review → requested on all 18
+  (goal-doc authorization, one per PR at review-ready; Julien explicitly
+  asked). Triage the incoming reviews at next orientation.
+- **Our review agent**: coverage verified from journal records across the
+  stack (per-Draft agents early; #2648/49, #2651-55 batches; #2656,
+  #2657 individually); #2658 reviewed tonight (report in this entry's
+  follow-up).
+- **CUTOVER_RUNBOOK.md written** (docs commit): evidence base, risk
+  register (no prod shadow soak yet; Q10 large-conv intentional
+  divergence WILL flag in shadow compare; throughput/sharding note;
+  Q19 moot post-cutover), step 0 merge → step 1 same-day shadow →
+  step 2 evening flip (env-var, instantly reversible) → step 3
+  decommission.
+
+Follow-up: #2658 sharding review (our agent, tonight) — verdict SHIP.
+No-behavior-change claim HOLDS (shard branch gated on shard_count>1;
+defaults never enter it; watermark/write paths untouched; 74/74 tests).
+Assignment is zid % shard_count (no hash() — PYTHONHASHSEED-immune,
+process-stable); config validation rejects all malformed index/count
+shapes; shard filter correctly precedes allowlist. Two non-defects
+flagged: negative-zid partition untested-but-true (moot, DB serials);
+and NO code guard against two fleet processes sharing a shard_index —
+deployment-layer responsibility, now noted in CUTOVER_RUNBOOK.md.
+
+## Session 6 (cont., 2026-07-27): Copilot triage fan-out — 25 threads closed, 16 fixes applied
+
+Four parallel triage agents on the 18 landed Copilot reviews (25 comments):
+2 QUIRK rejections with ledger citations (a group_votes flattening that
+would break the Q2 restart restore; a proj_probe load-file nit), 3
+DECLINEs (replied+resolved), 1 deliberate deferral (#2644's
+behavior-identical legacy reindex — not worth a re-certification cycle in
+the cutover window), and 16 APPLYs applied serially by one agent (TDD on
+the substantive one: improved-mode early returns leaked stale
+group_clusterings/group_k_smoother across engine-mode switches — RED
+observed, reset now gated `if not legacy_mode:`; plus certify manifest
+now hashes comments CSVs — STRICT, forces a one-time clj re-record of
+the mod entries; lru_cache on tree hashes; dataset-mismatch guard;
+slug-glob sanitization; NaN-propagating Q11 clamp; Q16 always-2-wide
+projection; shard-bench fd/kill cleanup; docstring/test hardening).
+Combined gate: 647 passed / 5 pre-existing skips. Battery ×2 relaunched
+on the new tree (s6 logs).
+
+Julien decisions recorded (POST_CUTOVER_IMPROVEMENTS.md): py-round
+WONTFIX; equiv = release-gate script, not CI; bans confirmed negligible
+by fresh prodclone SQL (201/67/735,226 — and never honored by Clojure);
+sharding NOT needed at current traffic (fresh prodclone analysis: p95=3,
+p99=5, max 14 distinct active convs/min in 2024+ vs ~100 ticks/min
+serial capacity; all-time peak 116/min would want 2-4 shards).
+Category-1 nondeterminism issues opened: #2660 (Q10), #2661 (Q12),
+#2662 (Q13/Q18) — all "fixed by the Python push", determinism pinned by
+test_driver.py::test_determinism_bit_identical_except_wall_clock + the
+battery's pass-pair bit-comparisons.

delphi/docs/CUTOVER_RUNBOOK.md

@@ -0,0 +1,98 @@
+# Clojure→Python math cutover runbook
+
+Written 2026-07-26 (post R1-parity DONE, GOAL_STATE.md). Companion:
+MATH_POLLER_DESIGN.md §4 (phases), MATH_POLLER_EQUIV_SPEC.md (the live
+equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).
+
+## Evidence base (what is PROVEN as of 2026-07-26)
+
+- R1 battery: 20/20 MATCH on four consecutive clean pass-pairs (2026-07-24),
+  ledger zero open. Warm chains, moderation, meta, bans, revotes, degenerate
+  ticks, restart seams.
+- Live poller equivalence vs the REAL Clojure container: vw 8/8 batches +
+  pc-meta-02 6/6 batches MATCH (moderation stream live, kill+restart seam,
+  tick/watermark semantics, bidToPid/ptptstats row-identical; floats within
+  the measured clj self-jitter envelope).
+- Full delphi suite green (1134 passed at goal close; CI green at stack tip).
+
+## What is NOT yet proven (risk register)
+
+1. **No prod-shaped shadow soak yet.** The equivalence harness ran replay
+   datasets (small/mid convs). Prod adds: conversation churn, concurrent
+   zids at scale, very large convs.
+2. **Large convs (>10k ptpts or >5k comments) intentionally DIVERGE from
+   Clojure**: clj dispatches to unseeded-random mini-batch PCA (Q10 — not
+   even self-consistent); python runs deterministic full PCA at every size
+   (documented improvement, same blob shape, server-compatible). A shadow
+   comparer WILL flag these convs — expected, not a defect. Decide the
+   acceptance for them up front (structural-only, or exclude from compare).
+3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2
+   (r8g.4xlarge, cost-model study) on biodiversity-sized replays.
+   **PRE-FLIP MEASUREMENT REQUIRED: one full-PCA tick of the largest
+   prodclone conv (33k ptpts) on the target EC2 instance** — 7 historical
+   convs sit above the old Clojure large-conv cutoffs and python runs
+   full PCA at every size (est. 0.5-2 min/tick at the extreme; fine for
+   7 rarely-active convs, but measure, don't estimate).
+   Sharding (#2658) is the scale-out path, opt-in via POLL_SHARD_INDEX/
+   POLL_SHARD_COUNT — one shard = one process. Start UNSHARDED (defaults
+   are a verified no-op); shard only if the shadow soak shows lag.
+   If sharding: the deployment layer MUST guarantee each shard_index
+   appears exactly once per MATH_ENV — no code guard exists against two
+   processes owning the same slice (per-zid serialization is per-process
+   only; #2658 review, 2026-07-26).
+4. Q19 (clj actor race losing votes on new-zid discovery) is a CLOJURE bug;
+   python's per-zid FIFO+lock design does not have it (equivalence runs
+   verified py carries the full vote stream).
+
+## Step 0 — land the stack (morning)
+
+1. Triage the 18 Copilot reviews requested overnight (2026-07-26) on
+   #2641-#2658; apply/reply per the standing triage rules.
+2. Confirm CI green: stack-tip python-ci dispatch + PR checks (see
+   spr status; #2648's mid-stack red is a stack-position artifact — the
+   same tests pass from #2656 upward — cosmetic for deploy, which builds
+   the tip).
+3. Merge bottom-up: `jj spr merge --count <N>` (spr handles squash order).
+   NEVER the GitHub UI. Then a normal edge deploy.
+
+## Step 1 — shadow in prod (same day)
+
+Infrastructure already in compose (#2625): service `delphi-math-poller`,
+profile `delphi-math`, `MATH_ENV=${DELPHI_MATH_ENV:-delphi}` — distinct
+from Clojure's env, rows invisible to the server (UNIQUE(zid, math_env)).
+
+```
+docker compose --profile delphi-math up -d delphi-math-poller
+# env: POLISMATH_ENGINE_MODE=clojure-legacy  DELPHI_MATH_ENV=delphi
+```
+
+Verify within minutes:
+- math_main rows appearing under math_env='delphi' with advancing
+  caching_tick;
+- no errorconv dumps / parked zids in the poller log;
+- spot-compare a few active zids' blobs vs the clojure rows (the certify
+  StepComparer acceptance; scripts/poller_equiv.py compare machinery is
+  reusable for row pairs).
+
+Soak: hours-to-a-day of prod traffic. Exit = no structural divergence on
+small/mid convs; large-conv divergence understood per risk #2.
+
+## Step 2 — flip (evening, if soak clean)
+
+One env change, instantly reversible:
+- Set the python poller's MATH_ENV to the server's Config.mathEnv ('prod');
+  stop the clojure `math` service. (Or flip the server's MATH_ENV to
+  'delphi' — pick ONE mechanism and write it down.)
+- Watch: TS prefetch (pca.ts caching_tick > last, ~2.5s poll) keeps
+  serving; nextComment routing gets comment-priorities; participants
+  bidToPid present.
+
+Rollback = revert the env var + restart clojure math. Rows for both envs
+coexist; nothing is destroyed by the flip in either direction.
+
+## Step 3 — decommission (later)
+
+Remove the `math` service from compose/deploy; archive the Clojure tree
+(it remains the R1 oracle). Follow-ups parked in the journal: equiv-in-CI
+decision, improved-mode ban coverage, fraction-cut py-round fix, quirk
+un-replication in improved mode (the post-cutover engine option).

delphi/docs/GOAL_CUTOVER_READY.md

@@ -0,0 +1,91 @@
+# GOAL: Cutover-ready Python math — mode collapse, clarity, re-certification
+
+Standing autonomous goal (set with Julien, 2026-07-27). Successor to
+GOAL_R1_PARITY.md (achieved 2026-07-24 — its DONE evidence stands; this goal
+prepares the actual Clojure-off/Python-on switch per Julien's rulings in
+POST_CUTOVER_IMPROVEMENTS.md and CUTOVER_RUNBOOK.md). Work session after
+session; decide, document, keep going. Propose-then-wait is suspended for
+this goal; compensating controls are per-change journal notes and the
+walkthrough section in GOAL_STATE at each milestone. Stop only for hard
+blockers (broken environment, usage limit, AWS denial — see Constraints).
+
+## DONE means (ALL must hold, evidenced in-repo)
+
+1. **Mode collapse**: the engine has ONE code path — exact legacy semantics.
+   Gate: `grep -rn "ENGINE_MODE\|engine_mode\|resolve_engine_mode" delphi/polismath/`
+   returns ZERO hits (the flag machinery itself is deleted); improved-only
+   branches extracted to parked jj commits (side bookmark `improvements/*`,
+   one commit per POST_CUTOVER_IMPROVEMENTS.md queue item) BEFORE deletion.
+   Participant-ban filtering DELETED outright (not parked — dropped feature).
+   run_delphi.py/job_poller (the API-called pipeline) exercises the same
+   single path — no pipeline-only math branches.
+2. **Clarity refactor landed PRE-cutover** (Julien ruling 2026-07-27):
+   PR 14b/14c — `compute_group_comment_stats_df` reads like the deleted
+   scalar recipe; vectorized blob-injection tests green. Bit-identity guarded
+   by the battery.
+3. **Golden snapshots re-recorded** at the collapse commit (legacy values,
+   verified against the certified battery recordings before recording —
+   never blind), full regression suite green vs the new baseline.
+4. **Battery**: TWO consecutive fully-clean passes (20/20 MATCH, zero open
+   ledger entries) on the exact final tree.
+5. **Equivalence release gate**: scripts/poller_equiv.py full-run verdict
+   PASS (non-vacuous) on vw AND pc-meta-02 on the final tree.
+6. **Large-conv EC2 measurement recorded**: one full-PCA tick of the largest
+   prodclone conv shape (33k ptpts × 783 cmts; synthesize the shape if
+   extraction is impractical) timed on the target EC2 class via the `bench`
+   AWS profile; number + verdict (serial OK / needs deterministic
+   large-conv path) written into CUTOVER_RUNBOOK.md risk register.
+7. GOAL_STATE.md first line flips to `STATUS: DONE` only after 1-6 hold.
+
+A session ending with open ledger entries, an incomplete collapse, or an
+unrecorded measurement has made PROGRESS, not achieved the goal.
+
+## Constraints
+
+- Extract-then-delete: every improved-mode branch worth re-landing is first
+  moved VERBATIM to a parked commit on the `improvements` side chain (per
+  queue item), so post-cutover PRs are rebases, not rewrites. Ban filtering
+  is deleted WITHOUT parking (dropped feature).
+- TDD calibrated per GOAL_R1_PARITY.md (RED mandatory for behavior pins;
+  full-suite gate per push, delegated per the gate protocol).
+- Golden snapshots: re-record ONLY after cross-checking against the battery's
+  certified clj recordings; never blind.
+- NEVER merge PRs; everything ships as Draft on the spr stack. python-ci on
+  spr branches needs manual workflow_dispatch (dispatch at wind-down, check
+  at next orientation).
+- Privacy: real_data/.local stays gitignored; slugs OK, zids/report-ids/
+  content never committed.
+- AWS: `bench` profile ONLY (tagged EC2, us-east-1, budget+forecast alarms
+  exist). NEVER widen a policy or switch profiles on AccessDenied — stop and
+  report. Terminate instances when done; verify termination.
+- The Clojure tree stays untouched (it remains the oracle until decommission).
+
+## Method
+
+- **Phase 0 — battery tooling speedup FIRST** (Julien 2026-07-27: the ~36-min
+  first-pass py re-replay blocks every code change; data in journal s6):
+  (a) scope the py cache tree-hash to the ENGINE surface — exclude pure
+  harness files (certify.py, poller_equiv.py, prodclone.py, shard_bench.py,
+  polismath/poller/**) whose changes cannot alter replay outputs (driver.py/
+  schedule.py/real_data.py DO shape replays — keep them in);
+  (b) parallelize battery entries (independent by construction) with
+  ~6 workers → target <8 min first pass;
+  (c) prove both with an A/B run before relying on them.
+- **Phase 1 — inventory**: enumerate every engine_mode branch site (grep) and
+  classify: DELETE (ban filtering, dead), PARK (queue items 2-8), KEEP-AS-ONLY
+  (legacy behavior). Write the inventory to the journal before cutting.
+- **Phase 2 — collapse**, bottom-up, re-running the (fast) battery per chunk.
+- **Phase 3 — clarity refactor** (14c then 14b), battery-guarded.
+- **Phase 4 — goldens re-record + full gates + equiv release gate.**
+- **Phase 5 — EC2 large-conv measurement** (bench profile; reuse the
+  cost-model harness patterns; runbook update).
+- Reviews per push: Claude review subagent; Copilot ONCE per PR at
+  review-ready. Triage against the quirks ledger (a "fix" undoing legacy
+  semantics is now a POST-cutover queue item, not a code change).
+
+## Session protocol
+
+Orientation = GOAL doc + GOAL_STATE.md only. Wind-down: finish the cycle,
+gate, commit, rewrite GOAL_STATE (numbered next actions, file:line), push,
+dispatch CI. Durable state on disk, never in chat. Token floor per
+GOAL_R1_PARITY.md (unchanged).