diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 6793eb886..151861d3d 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -3966,3 +3966,82 @@ airtight version). Push + CI follow the verdicts. are intentionally inert, Q1); improved mode's ban path has no live equiv coverage. Follow-up candidate for the cutover phase (improved mode is the post-cutover option per MATH_POLLER_DESIGN.md). + +## Session 6 (2026-07-26, overnight): pre-cutover audit + fixes + +Stack audit for the clojure-off/python-on decision (Julien overnight +mandate). State found + actions: + +- **Stack**: 32 Draft PRs, base-pointer chain verified correct end-to-end + (spr's 4th-column ❌ on the upper 13 is metadata cosmetics; every head + OID checked == local). NEW top commit since s5: #2658 zid-sharding + (another session, 2026-07-25 — scheduling scaffolding, opt-in, + process-per-shard; measured thread-pool serial fraction 0.988 → 15.7x + at 16 processes). +- **CI**: (a) #2637's own run failed on ONE test — + test_py_poller_runner_cmd_cwd_env asserted cwd.name == "delphi", which + is "app" in the CI container; layout-fragile assertion dropped, fix + squashed into #2657's commit, stack pushed (retriggers checks). + (b) python-ci dispatched on the sharding head (was never run there). + (c) #2648 mid-stack red = two certify tests failing at that STACK + POSITION only (they pass from #2656's position up — the fix rode a + later commit); old run re-run; if still red it is a stack-position + artifact, not a tip defect. (d) observed in #2637's run logs: a + postgres "null zid" constraint ERROR from the integration flow with no + failing test — noted, unexplained, non-blocking. +- **Copilot**: 14 older PRs reviewed (all threads resolved but two); + the two unresolved threads (#2618 engine_mode coupling, #2622 stale + group_clusterings) were both ALREADY FIXED by later stack PRs (#2641 + resolver move; #2642 Q4 overwrite) — replied with citations, resolved. + 18 newer PRs (#2641-#2658) had NO Copilot review → requested on all 18 + (goal-doc authorization, one per PR at review-ready; Julien explicitly + asked). Triage the incoming reviews at next orientation. +- **Our review agent**: coverage verified from journal records across the + stack (per-Draft agents early; #2648/49, #2651-55 batches; #2656, + #2657 individually); #2658 reviewed tonight (report in this entry's + follow-up). +- **CUTOVER_RUNBOOK.md written** (docs commit): evidence base, risk + register (no prod shadow soak yet; Q10 large-conv intentional + divergence WILL flag in shadow compare; throughput/sharding note; + Q19 moot post-cutover), step 0 merge → step 1 same-day shadow → + step 2 evening flip (env-var, instantly reversible) → step 3 + decommission. + +Follow-up: #2658 sharding review (our agent, tonight) — verdict SHIP. +No-behavior-change claim HOLDS (shard branch gated on shard_count>1; +defaults never enter it; watermark/write paths untouched; 74/74 tests). +Assignment is zid % shard_count (no hash() — PYTHONHASHSEED-immune, +process-stable); config validation rejects all malformed index/count +shapes; shard filter correctly precedes allowlist. Two non-defects +flagged: negative-zid partition untested-but-true (moot, DB serials); +and NO code guard against two fleet processes sharing a shard_index — +deployment-layer responsibility, now noted in CUTOVER_RUNBOOK.md. + +## Session 6 (cont., 2026-07-27): Copilot triage fan-out — 25 threads closed, 16 fixes applied + +Four parallel triage agents on the 18 landed Copilot reviews (25 comments): +2 QUIRK rejections with ledger citations (a group_votes flattening that +would break the Q2 restart restore; a proj_probe load-file nit), 3 +DECLINEs (replied+resolved), 1 deliberate deferral (#2644's +behavior-identical legacy reindex — not worth a re-certification cycle in +the cutover window), and 16 APPLYs applied serially by one agent (TDD on +the substantive one: improved-mode early returns leaked stale +group_clusterings/group_k_smoother across engine-mode switches — RED +observed, reset now gated `if not legacy_mode:`; plus certify manifest +now hashes comments CSVs — STRICT, forces a one-time clj re-record of +the mod entries; lru_cache on tree hashes; dataset-mismatch guard; +slug-glob sanitization; NaN-propagating Q11 clamp; Q16 always-2-wide +projection; shard-bench fd/kill cleanup; docstring/test hardening). +Combined gate: 647 passed / 5 pre-existing skips. Battery ×2 relaunched +on the new tree (s6 logs). + +Julien decisions recorded (POST_CUTOVER_IMPROVEMENTS.md): py-round +WONTFIX; equiv = release-gate script, not CI; bans confirmed negligible +by fresh prodclone SQL (201/67/735,226 — and never honored by Clojure); +sharding NOT needed at current traffic (fresh prodclone analysis: p95=3, +p99=5, max 14 distinct active convs/min in 2024+ vs ~100 ticks/min +serial capacity; all-time peak 116/min would want 2-4 shards). +Category-1 nondeterminism issues opened: #2660 (Q10), #2661 (Q12), +#2662 (Q13/Q18) — all "fixed by the Python push", determinism pinned by +test_driver.py::test_determinism_bit_identical_except_wall_clock + the +battery's pass-pair bit-comparisons. diff --git a/delphi/docs/CUTOVER_RUNBOOK.md b/delphi/docs/CUTOVER_RUNBOOK.md new file mode 100644 index 000000000..cd78ab6a0 --- /dev/null +++ b/delphi/docs/CUTOVER_RUNBOOK.md @@ -0,0 +1,98 @@ +# Clojure→Python math cutover runbook + +Written 2026-07-26 (post R1-parity DONE, GOAL_STATE.md). Companion: +MATH_POLLER_DESIGN.md §4 (phases), MATH_POLLER_EQUIV_SPEC.md (the live +equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). + +## Evidence base (what is PROVEN as of 2026-07-26) + +- R1 battery: 20/20 MATCH on four consecutive clean pass-pairs (2026-07-24), + ledger zero open. Warm chains, moderation, meta, bans, revotes, degenerate + ticks, restart seams. +- Live poller equivalence vs the REAL Clojure container: vw 8/8 batches + + pc-meta-02 6/6 batches MATCH (moderation stream live, kill+restart seam, + tick/watermark semantics, bidToPid/ptptstats row-identical; floats within + the measured clj self-jitter envelope). +- Full delphi suite green (1134 passed at goal close; CI green at stack tip). + +## What is NOT yet proven (risk register) + +1. **No prod-shaped shadow soak yet.** The equivalence harness ran replay + datasets (small/mid convs). Prod adds: conversation churn, concurrent + zids at scale, very large convs. +2. **Large convs (>10k ptpts or >5k comments) intentionally DIVERGE from + Clojure**: clj dispatches to unseeded-random mini-batch PCA (Q10 — not + even self-consistent); python runs deterministic full PCA at every size + (documented improvement, same blob shape, server-compatible). A shadow + comparer WILL flag these convs — expected, not a defect. Decide the + acceptance for them up front (structural-only, or exclude from compare). +3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2 + (r8g.4xlarge, cost-model study) on biodiversity-sized replays. + **PRE-FLIP MEASUREMENT REQUIRED: one full-PCA tick of the largest + prodclone conv (33k ptpts) on the target EC2 instance** — 7 historical + convs sit above the old Clojure large-conv cutoffs and python runs + full PCA at every size (est. 0.5-2 min/tick at the extreme; fine for + 7 rarely-active convs, but measure, don't estimate). + Sharding (#2658) is the scale-out path, opt-in via POLL_SHARD_INDEX/ + POLL_SHARD_COUNT — one shard = one process. Start UNSHARDED (defaults + are a verified no-op); shard only if the shadow soak shows lag. + If sharding: the deployment layer MUST guarantee each shard_index + appears exactly once per MATH_ENV — no code guard exists against two + processes owning the same slice (per-zid serialization is per-process + only; #2658 review, 2026-07-26). +4. Q19 (clj actor race losing votes on new-zid discovery) is a CLOJURE bug; + python's per-zid FIFO+lock design does not have it (equivalence runs + verified py carries the full vote stream). + +## Step 0 — land the stack (morning) + +1. Triage the 18 Copilot reviews requested overnight (2026-07-26) on + #2641-#2658; apply/reply per the standing triage rules. +2. Confirm CI green: stack-tip python-ci dispatch + PR checks (see + spr status; #2648's mid-stack red is a stack-position artifact — the + same tests pass from #2656 upward — cosmetic for deploy, which builds + the tip). +3. Merge bottom-up: `jj spr merge --count ` (spr handles squash order). + NEVER the GitHub UI. Then a normal edge deploy. + +## Step 1 — shadow in prod (same day) + +Infrastructure already in compose (#2625): service `delphi-math-poller`, +profile `delphi-math`, `MATH_ENV=${DELPHI_MATH_ENV:-delphi}` — distinct +from Clojure's env, rows invisible to the server (UNIQUE(zid, math_env)). + +``` +docker compose --profile delphi-math up -d delphi-math-poller +# env: POLISMATH_ENGINE_MODE=clojure-legacy DELPHI_MATH_ENV=delphi +``` + +Verify within minutes: +- math_main rows appearing under math_env='delphi' with advancing + caching_tick; +- no errorconv dumps / parked zids in the poller log; +- spot-compare a few active zids' blobs vs the clojure rows (the certify + StepComparer acceptance; scripts/poller_equiv.py compare machinery is + reusable for row pairs). + +Soak: hours-to-a-day of prod traffic. Exit = no structural divergence on +small/mid convs; large-conv divergence understood per risk #2. + +## Step 2 — flip (evening, if soak clean) + +One env change, instantly reversible: +- Set the python poller's MATH_ENV to the server's Config.mathEnv ('prod'); + stop the clojure `math` service. (Or flip the server's MATH_ENV to + 'delphi' — pick ONE mechanism and write it down.) +- Watch: TS prefetch (pca.ts caching_tick > last, ~2.5s poll) keeps + serving; nextComment routing gets comment-priorities; participants + bidToPid present. + +Rollback = revert the env var + restart clojure math. Rows for both envs +coexist; nothing is destroyed by the flip in either direction. + +## Step 3 — decommission (later) + +Remove the `math` service from compose/deploy; archive the Clojure tree +(it remains the R1 oracle). Follow-ups parked in the journal: equiv-in-CI +decision, improved-mode ban coverage, fraction-cut py-round fix, quirk +un-replication in improved mode (the post-cutover engine option). diff --git a/delphi/docs/GOAL_CUTOVER_READY.md b/delphi/docs/GOAL_CUTOVER_READY.md new file mode 100644 index 000000000..6622d9e75 --- /dev/null +++ b/delphi/docs/GOAL_CUTOVER_READY.md @@ -0,0 +1,91 @@ +# GOAL: Cutover-ready Python math — mode collapse, clarity, re-certification + +Standing autonomous goal (set with Julien, 2026-07-27). Successor to +GOAL_R1_PARITY.md (achieved 2026-07-24 — its DONE evidence stands; this goal +prepares the actual Clojure-off/Python-on switch per Julien's rulings in +POST_CUTOVER_IMPROVEMENTS.md and CUTOVER_RUNBOOK.md). Work session after +session; decide, document, keep going. Propose-then-wait is suspended for +this goal; compensating controls are per-change journal notes and the +walkthrough section in GOAL_STATE at each milestone. Stop only for hard +blockers (broken environment, usage limit, AWS denial — see Constraints). + +## DONE means (ALL must hold, evidenced in-repo) + +1. **Mode collapse**: the engine has ONE code path — exact legacy semantics. + Gate: `grep -rn "ENGINE_MODE\|engine_mode\|resolve_engine_mode" delphi/polismath/` + returns ZERO hits (the flag machinery itself is deleted); improved-only + branches extracted to parked jj commits (side bookmark `improvements/*`, + one commit per POST_CUTOVER_IMPROVEMENTS.md queue item) BEFORE deletion. + Participant-ban filtering DELETED outright (not parked — dropped feature). + run_delphi.py/job_poller (the API-called pipeline) exercises the same + single path — no pipeline-only math branches. +2. **Clarity refactor landed PRE-cutover** (Julien ruling 2026-07-27): + PR 14b/14c — `compute_group_comment_stats_df` reads like the deleted + scalar recipe; vectorized blob-injection tests green. Bit-identity guarded + by the battery. +3. **Golden snapshots re-recorded** at the collapse commit (legacy values, + verified against the certified battery recordings before recording — + never blind), full regression suite green vs the new baseline. +4. **Battery**: TWO consecutive fully-clean passes (20/20 MATCH, zero open + ledger entries) on the exact final tree. +5. **Equivalence release gate**: scripts/poller_equiv.py full-run verdict + PASS (non-vacuous) on vw AND pc-meta-02 on the final tree. +6. **Large-conv EC2 measurement recorded**: one full-PCA tick of the largest + prodclone conv shape (33k ptpts × 783 cmts; synthesize the shape if + extraction is impractical) timed on the target EC2 class via the `bench` + AWS profile; number + verdict (serial OK / needs deterministic + large-conv path) written into CUTOVER_RUNBOOK.md risk register. +7. GOAL_STATE.md first line flips to `STATUS: DONE` only after 1-6 hold. + +A session ending with open ledger entries, an incomplete collapse, or an +unrecorded measurement has made PROGRESS, not achieved the goal. + +## Constraints + +- Extract-then-delete: every improved-mode branch worth re-landing is first + moved VERBATIM to a parked commit on the `improvements` side chain (per + queue item), so post-cutover PRs are rebases, not rewrites. Ban filtering + is deleted WITHOUT parking (dropped feature). +- TDD calibrated per GOAL_R1_PARITY.md (RED mandatory for behavior pins; + full-suite gate per push, delegated per the gate protocol). +- Golden snapshots: re-record ONLY after cross-checking against the battery's + certified clj recordings; never blind. +- NEVER merge PRs; everything ships as Draft on the spr stack. python-ci on + spr branches needs manual workflow_dispatch (dispatch at wind-down, check + at next orientation). +- Privacy: real_data/.local stays gitignored; slugs OK, zids/report-ids/ + content never committed. +- AWS: `bench` profile ONLY (tagged EC2, us-east-1, budget+forecast alarms + exist). NEVER widen a policy or switch profiles on AccessDenied — stop and + report. Terminate instances when done; verify termination. +- The Clojure tree stays untouched (it remains the oracle until decommission). + +## Method + +- **Phase 0 — battery tooling speedup FIRST** (Julien 2026-07-27: the ~36-min + first-pass py re-replay blocks every code change; data in journal s6): + (a) scope the py cache tree-hash to the ENGINE surface — exclude pure + harness files (certify.py, poller_equiv.py, prodclone.py, shard_bench.py, + polismath/poller/**) whose changes cannot alter replay outputs (driver.py/ + schedule.py/real_data.py DO shape replays — keep them in); + (b) parallelize battery entries (independent by construction) with + ~6 workers → target <8 min first pass; + (c) prove both with an A/B run before relying on them. +- **Phase 1 — inventory**: enumerate every engine_mode branch site (grep) and + classify: DELETE (ban filtering, dead), PARK (queue items 2-8), KEEP-AS-ONLY + (legacy behavior). Write the inventory to the journal before cutting. +- **Phase 2 — collapse**, bottom-up, re-running the (fast) battery per chunk. +- **Phase 3 — clarity refactor** (14c then 14b), battery-guarded. +- **Phase 4 — goldens re-record + full gates + equiv release gate.** +- **Phase 5 — EC2 large-conv measurement** (bench profile; reuse the + cost-model harness patterns; runbook update). +- Reviews per push: Claude review subagent; Copilot ONCE per PR at + review-ready. Triage against the quirks ledger (a "fix" undoing legacy + semantics is now a POST-cutover queue item, not a code change). + +## Session protocol + +Orientation = GOAL doc + GOAL_STATE.md only. Wind-down: finish the cycle, +gate, commit, rewrite GOAL_STATE (numbered next actions, file:line), push, +dispatch CI. Durable state on disk, never in chat. Token floor per +GOAL_R1_PARITY.md (unchanged). diff --git a/delphi/docs/GOAL_STATE.md b/delphi/docs/GOAL_STATE.md index 4bf71203c..ec04dafde 100644 --- a/delphi/docs/GOAL_STATE.md +++ b/delphi/docs/GOAL_STATE.md @@ -1,42 +1,51 @@ -STATUS: DONE - -# GOAL_STATE — R1 parity goal ACHIEVED (2026-07-24, session 5) - -All three GOAL_R1_PARITY.md "DONE means" conditions hold, evidenced in-repo: - -1. **Battery: TWO consecutive fully-clean passes on the final tree** — - 20/20 MATCH, DIVERGENCE=0, ERROR=0, exit 0 on both - (real_data/.local/replays/battery_s5_pass{7,8}.log — the FINAL tree, - post #2657-review hardening; same-day clean pairs pass{1,2}, {3,4}, - {5,6} on predecessor trees: FOUR consecutive clean pairs total). Battery: 20 - entries = all real_data datasets + prodclone extractions + the goal - doc's edge cases (mod-heavy, revote-heavy, banned, meta, degenerate, - zero-votes, restart seams). divergences.json: ZERO open (70 resolved + - 11 carved-out, every carve diagnosed + quirk-ledgered Q1-Q19). - Subgroup-* carve (Q7) logged on every run, never silent. -2. **Poller equivalence PASSES** (MATH_POLLER_EQUIV_SPEC.md harness, live - vs the REAL Clojure container): vw verdict MATCH 8/8 batches + - pc-meta-02 verdict MATCH 6/6 batches with the moderation stream - (146/146 mod events), both including a kill+restart seam mid-schedule; - identical math_main/bidToPid/ptptstats rows (structural identity; - floats within the measured clj self-jitter envelope — the declared, - per-run-reported tolerance); caching_tick/math_ticks/watermark - semantics verified. Evidence: real_data/.local/replays/poller_equiv/ - {vw,pc-meta-02}/ (verdict JSONs, manifests, runner logs) + 6 kept DBs. -3. **This line**: STATUS: DONE (written only after 1+2 held). - -## For Julien's walkthrough (morning review) - -- Journal sessions 4-5 (CLJ-PARITY-FIXES-JOURNAL.md) narrate every - per-change note. Highlights: restart-seam from_dict restore; Q17 hash - tie-break PORT; Q18 uniqify ulp knife-edge CARVE (+pc-meta-02 swap); - **Q19 = REAL Clojure production bug** (conv-actor race losing votes — - wants an upstream fix); FOUR real py-poller production bugs fixed - (pid/tid/zid ints; derive_ptptstats wrote the WRONG STATISTIC — now - the verbatim repness.clj geometric port). -- Stack (all Draft, NEVER merged per constraints): docs PR #2626, - feature PR #2656, NEW harness PR (created this push), ci commit. - python-ci dispatched at final push — check the run. -- Parked: fraction-cut py-round clj-driver fix (needs ~2h cache - re-record); Q19 upstream fix decision; poller cutover phases - (MATH_POLLER_DESIGN.md §4) are the natural NEXT goal. +STATUS: IN PROGRESS + +# GOAL_STATE — checkpoint for GOAL_CUTOVER_READY.md (cap ~50 lines) + +Predecessor GOAL_R1_PARITY.md: ACHIEVED 2026-07-24 (evidence pointers in its +final journal entries; battery ×4 clean pairs, live equiv PASS ×2 datasets). + +## Where we are (2026-07-27, session 6 wind-down) + +- Copilot triage COMPLETE: 25 threads — 16 fixes applied (TDD; 647-test gate + green), 2 quirk-rejections cited, 3 declines, 1 deferral (#2644). Battery + ×2 re-ran on the triaged tree (s6 logs — check verdicts at orientation if + this session ended before they landed). Issues #2660/#2661/#2662 opened + (Q10/Q12/Q13+Q18 — "fixed by the Python push"). +- Julien rulings captured in POST_CUTOVER_IMPROVEMENTS.md + + CUTOVER_RUNBOOK.md: bans DROPPED entirely; equiv = release gate not CI; + sharding parked (data: p99=5 active convs/min vs ~100 ticks/min EC2- + measured serial capacity); clarity refactor moved PRE-cutover; large-conv + EC2 tick measurement is a flip precondition; run_delphi.py is + PRODUCTION-called (POST /api/v3/delphi/jobs → job_poller FULL_PIPELINE). +- Battery timing DATA (journal s6): first pass after an engine change ≈36 + min (py re-replay, clj cached); second consecutive pass ≈21 SECONDS; + full clj re-record ≈91 min. Phase 0 tooling attacks the 36-min pass. + +## Next actions + +1. s6 battery CONFIRMED clean: 20/20 MATCH ×2 on the triaged tree + (battery_s6_pass{1,2}.log; only 3 clj re-records fired — the restart + entries carry no comments CSV). Commit split DONE: triage fixes = PR + #2663 (spr/edge/efb914d7), docs = #2659. python-ci dispatched on + #2663's head (run 30276844539) — CHECK at orientation. Thread + replies/resolves: CONFIRMED complete — all 20 open threads across 11 + PRs replied (citing #2663) and resolved; 25/25 Copilot comments closed. +2. Phase 0 of GOAL_CUTOVER_READY.md: battery tooling speedup (hash scoping + + parallel entries; A/B-prove — timing data in journal s6: 36-min + first pass / 21-s cached pass). THEN Phase 1 inventory (grep all + engine_mode branch sites; classify DELETE/PARK/KEEP; journal it). +3. Review the new PRs per protocol when review-ready: #2659 (docs), #2663 + (triage batch — request Copilot once; our agent already covered the + content via the triage itself). +4. Parked questions needing input: none — all rulings recorded. + +## Pointers + +- Contract: GOAL_CUTOVER_READY.md. Roadmap: POST_CUTOVER_IMPROVEMENTS.md. + Runbook: CUTOVER_RUNBOOK.md. Quirks: CLOJURE_QUIRKS.md Q1-Q19. +- Battery: scripts/certify.py run (20 entries). Equiv release gate: + scripts/poller_equiv.py full-run (needs pgproxy 127.0.0.1:15432 up: + `docker start polis-dev-postgres-1 pgproxy`). +- Suite baseline: 1134 passed / 22 skipped / 46 xfailed (2026-07-24) + s6 + additions; gate delegation protocol in GOAL_R1_PARITY.md. diff --git a/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md new file mode 100644 index 000000000..2d16ba1a3 --- /dev/null +++ b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md @@ -0,0 +1,100 @@ +# Post-cutover improvement roadmap (the ONE central list) + +Established 2026-07-27 with Julien's decisions on the deferred items. +Companions: CUTOVER_RUNBOOK.md (the switch itself), CLOJURE_QUIRKS.md +(Q1-Q19 — each row's "Later fix" column feeds this list), +MATH_POLLER_DESIGN.md §4 (cutover phases). + +## Standing decisions (Julien, 2026-07-27) + +- Equivalence protocol stays a RELEASE-GATE SCRIPT, not CI: its purpose is + the one-time conviction that Python is equivalent; Clojure is removed + once convinced. (scripts/poller_equiv.py full-run.) +- fraction-cut py-round clj-driver bug: WONTFIX (test tooling for the + component being deleted). +- Participant bans: CONFIRMED negligible on prodclone — 201 banned + participants / 67 conversations / 735,226 total (0.027%), and Clojure + never honored them, so bans have never affected prod math. DROPPED + ENTIRELY as a feature (not part of Polis) — see queue item 1. +- Sharding: NOT needed at current traffic — prodclone vote history shows + p95 = 3, p99 = 5 distinct active conversations per minute (2024+), + max 14; all-time historical peak 116/min. One serial Python process + sustains ~100 recompute-ticks/min — MEASURED ON EC2 (r8g.4xlarge, + cost-model study, #2658's bench) on biodiversity-sized replays, NOT + the dev laptop. Caveat: that tick rate is for small/mid convs; the + large-conv (Q10-class) tick cost is unmeasured on EC2 — see queue + item 9. The scaffolding (#2658) stays opt-in and parked unless a + scale event approaches the historical peak (which would want 2-4 + shards). + +## THE CUTOVER PRECONDITION — mode collapse (Julien directive 2026-07-27) + +The committed, switched-to code must be the EXACT legacy match with NO +improvement code paths. Improvements re-land AFTERWARD as separate, +sequential PRs so the git history documents each one. Mechanically: +extract today's improved-mode branches into parked jj commits, collapse +the engine to unconditional legacy behavior, re-certify, switch; then +rebase the improvement commits on top one at a time. + +Implication to decide consciously: the "improved" branches are what the +delphi DynamoDB pipeline (run_delphi.py) executes today, and what the +golden snapshots pin. run_delphi.py is PRODUCTION-CALLED, not a dev +script: POST /api/v3/delphi/jobs (server/src/routes/delphi/jobs.ts) +enqueues DynamoDB jobs and delphi's job_poller.py FULL_PIPELINE branch +executes run_delphi.py. After the collapse there is only ONE engine +path, so the API pipeline automatically follows the poller's exact +legacy semantics (Julien requirement 2026-07-27). Collapsing to legacy-only changes THAT service's +math outputs too (to the Clojure-equivalent values prod consumers have +always seen from the math worker) and requires re-recording the golden +snapshots at the collapse commit. + +## The improvement queue (one PR each, in rough order) + +Each PR: change + tests + re-certified outputs + a CHANGELOG-quality +description. Sources: CLOJURE_QUIRKS "Later fix" column, journal parked +items. + +1. ~~Honor participant bans~~ — DROPPED ENTIRELY (Julien 2026-07-27): + bans are not a Polis feature (201 rows ever, never honored by any + engine). The mode collapse DELETES the improved-mode ban-filtering + branch outright; mod_out_ptpts stays ingested-but-inert exactly as + prod has always behaved. +2. Degenerate-tick guards — un-replicate Q4/Q5 (skip clustering below + 2 participants/base-clusters instead of running k=2 on one point). +3. Group-level k-means gets its intended 100 iterations — un-replicate + Q3 (Clojure silently ran 20). +4. Persistent moderation watermark — un-replicate Q15 (Clojure drops it + every votes tick). +5. Comment priorities read CURRENT-tick group-votes — un-replicate Q2 + (Clojure reads the previous tick's). +6. Distance formula: true euclidean instead of the cancellation-lossy + vectorz form — un-replicate Q11 (removes the 0.0-tie merges) and with + it the Q17 hash-order tie-break scaffolding. +7. Projection rank-1 fix — un-replicate Q16 (pad pc2 instead of zeroing + everything). +8. Modern solver paths (sklearn PCA/k-means) where they beat the ports — + the original "improved" aspiration, now landing with certification + discipline. +9. **Deterministic large-conversation handling** (Julien 2026-07-27, from + the Q10 discussion): 7 of 15,575 prodclone convs ever crossed the + Clojure cutoffs (largest: 33,422 ptpts x 783 cmts / 2.0M votes; + 18,082 x 9,030). Python currently runs full PCA at every size — + deterministic but O(ptpts x cmts x iters) per tick; large-conv tick + cost on EC2 is UNMEASURED (see CUTOVER_RUNBOOK risk register). If + measurement says it is too slow, add a DETERMINISTIC large-conv path + (seeded mini-batch or randomized-SVD with fixed seed) as its own PR — + keeping determinism, unlike Clojure's unseeded sampling (#2660). +10. MOVED PRE-CUTOVER (Julien 2026-07-27: "we need to land clean code"): + vectorized-code readability + blob-injection tests — PR 14b/14c from + HANDOFF_PR14_VECTORIZED_REFACTOR.md (14a shipped as #2564). Now Phase 3 + of GOAL_CUTOVER_READY.md, battery-guarded; listed here for lineage only. +11. Cleanup pass: the deferred cosmetic simplifications (e.g. the #2644 + legacy reindex no-op), dead update_moderation seams, the Q7 subgroup + computation deletion upstream if the Clojure tree is still around. + +## Explicitly NOT planned + +- Clojure-side fixes for #2660/#2661/#2662 (Q10/Q12/Q13+Q18 + nondeterminism) — resolved by replacement, not repair. +- Q19 (Clojure conv-actor race): moot at decommission; the Python + design (per-zid FIFO + lock) is the fix.