Fix/audit cycle 33987355439 - #29
Merged
Merged
Conversation
…ith a URL sidecar; one URL grammar; a build gate on failed embeds Squashed from the audit branch by file set (3 original commits contributed; their messages follow). --- f918165 fix(images): retry + durable cache for textimages, one URL grammar, and a build gate on failed embeds Builds on the owner's in-progress work (downloadWithRetry with 3 attempts and backoff, HTTP 418 treated as final, durable cache via -PimageCacheDir) and closes what the v27 build logs showed: - collectUrls stopped at whitespace while IMG_TAG_REGEX did not, so `…/ShaarHahakdamot/Screenshot 2023-… .png` was fetched truncated and could never be substituted even after a successful download. One shared URL_BODY now drives both; the literal URL stays the map/cache key and only the HTTP request is percent-encoded (existing `%20` is left alone). - newlyDownloaded/failed were plain Ints incremented from 16 coroutines (`cached=410` vs `downloaded=409`); now AtomicInteger + ConcurrentHashMap. - A failed download silently left the remote URL in the row (10/420 images shipped as network URLs in v27). The embedder now writes image-embed-report.json (counts + one entry per failed URL with reason: timeout / http <status> / content-filter 418 (NetFree on the self-hosted runner) / malformed-url / oversized), and the build step reads it: ::warning:: per failed URL, ::error:: and exit 1 above IMAGE_EMBED_MAX_FAILURES (default 20) or when the report is missing. - Log volume: one summary line (greppable `cached=` prefix) and one stack-trace-free line per failure instead of a stack trace per attempt (81% of the embedder's log lines were repeated frames); cache hits are a count. Tests: 5 new (spaced-URL round trip, request-only encoding, failure report contents, exact counters under 120 concurrent URLs, report-path property); :sefariasqlite:jvmTest --tests '*SefariaImageEmbedder*' 14/14. --- 5888ffd fix(images): content-addressed cache keys with a URL sidecar; migrate the legacy cache without a download cacheFileName replaced `/`, `:` and `?` with `_`, so two distinct image URLs could map to one durable cache file and the next build served the wrong image from the cache and reported success. The key is now sha256(url); each entry carries a `<sha>.url` sidecar written before the bytes (temp + atomic move), and a hit requires the sidecar to name the requested URL, otherwise the entry is evicted and fetched again. The legacy layout is migrated in place: on a miss the old name is looked up, claimed by writing the sidecar and renamed, so a warm cache costs zero downloads (verified on the 417 URLs of the 2026-09-01 export: 417 served from disk, 0 fetched, 0 legacy collisions) and the IMAGE_EMBED_MAX_FAILURES gate is never exposed. Legacy names that two URLs of the same export would share are refused outright so a historical collision is repaired rather than frozen. URLs whose legacy name is not a valid path on the host are treated as a miss instead of throwing. --- 35c8f42 fix(images): a cache hit requires a byte-exact URL sidecar; drop the provenance-free legacy migration Round 3 of the audit of cycle 33987355439 (item S15). The legacy cache name was the URL suffix with `/ : ?` folded to `_`, which is not injective (…/Tikkunei_Zohar/40.png and …/Tikkunei_Zohar_40.png collide), and the migration adopted such a blob under the new sha256(url) key with a "verified" sidecar without any proof of which URL had produced it. The durable cache directory was introduced by this very branch and no release run has executed on a branch head (the single run on 15ef349 never started a job), so there are no legacy entries to migrate: the migration, the dual read and the two-URLs-one-name refusal are removed. A hit now requires `<sha>.url` whose bytes equal the URL exactly; anything else is a miss and a fresh download, with the sidecar written before the bytes as before. Tests: no hit without a matching sidecar, a byte-inexact sidecar is a miss, a legacy-named blob (alone or as a collision pair) is never adopted, miss → download → sidecar + blob. SefariaImageEmbedderTest 21/21. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…hase summaries; dropped data summarised and named; fail closed on seed, snapshot and buildstate; atomic candidate publish Squashed from the audit branch by file set (4 original commits contributed; their messages follow). --- ace5f88 fix(generator): one DEBUG line per book, a throttled progress line, and a phase summary instead of 17,800 INFO lines Audit of cycle 33987355439 (item S4a, report 06-db-generator-step). The "Generate Seforim Database" step logged 24,207 lines / 2.8 MB for ~1,500 books, 94.8% of it boilerplate: a 6-line INFO block per book, the book header printed twice from two call sites, a line tick at lineIndex == 0 for every book (`0/N (0%)`), and `Books progress: n/N` after every single book. Generator.kt (logging only; the serial book-insert loop, its order, id allocation and every insert call are untouched — verified by diffing the control-flow skeleton with all logger calls stripped): - the per-book INFO block is one DEBUG line (id, title, file, category, lines, toc, acronyms, ms); processBookContent/processLinesWithTocEntries return BookContentStats; a logger.e + rethrow keeps the "which book blew up" breadcrumb for content failures. - the three `Books progress` sites fold into one throttled INFO line every 100 books or 60 s: `books 1234/1501 (82.2%) · lines 5.2M · toc 301.0K · elapsed 27:08 · eta ~05:52` (15 lines on the real run). - line ticks need lineIndex > 0, a book above 5,000 lines and ≥10% or 60 s since the last tick (3,239 -> 507 on the real book sizes; the old rule replayed over them reproduces exactly the 3,239 observed). - the duplicated `📚 Processing book` site in processDirectory is gone; the createAndProcessBook site (which also serves the priority-list caller and carries the categoryId) stays. - acronym inserts, `===` directory lines, `✅ Category` and the `📝` merge lines are DEBUG; their counts land in one summary per phase: `Otzaria book import (phase 1): books=…/… lines=… tocEntries=… directories=… skipped=… acronymBooks=… hearotMerged=… elapsed=mm:ss` (counters reset per phase — QA fix). GeneratorProgress.kt: clock-injected cadence gates and locale-free formatters; GeneratorProgressTest.kt: 18 tests on a fake clock. DEBUG is off in CI (Logger.setMinSeverity(Info) on every CLI entry; the audited log has 0 Debug lines), so the demotion is real. No consumer parses any removed marker (repo, pipeline-monitor, otzaria-library, LinkerToOtzaria checked); `build_state.db snapshot written`, `Persisting in-memory DB`, SefariaDirect `Processed N/N books` and `BUILD SUCCESSFUL` are byte-identical. Measured on build-job-34024655297.log: -17,838 lines (-73.7%) and -1.98 MB on the step. Left for S4b: the 1,286 `Target book not found` / `Original path` lines (their fix is a WARN summary for the 643 dropped manual links) and the insert-path failure breadcrumb. Tests: :otzariasqlite:jvmTest --rerun 36/36. --- f2079c5 fix(generator): summarise dropped data instead of spamming it; make the summaries agree; name what was silently skipped Audit of cycle 33987355439 (item S4b, report 06-db-generator-step). Logging, counting and report-file changes only: the serial book-insert loop, id allocation, SQL and every row written are unchanged (verified by a skeleton diff with all logger calls stripped, 19 hunks in Generator.kt, each a counter / set insertion / report call / currentBook assignment / log-and-rethrow / the _headings filter). - 37 Talmud `<masechet>_headings` JSON files (heading -> line maps) were fed to the link parser: 74 parse attempts (each tried twice), a 40-line JSON dump per failure, then `Source book not found for links` x37. They are excluded from link-file discovery with one INFO line. Proven no-op for the data: parseLinksFromJson returned emptyList() for them, so zero rows and zero ids ever came from them. - Priority list 430/431 missing (Generator.kt list never re-pinned after the directory restructure): 430 WARNs -> one WARN with counts, the first paths and a pointer; full list in the report file. The list is NOT re-pinned or deleted (changes insert order -> ids; operator decision). - 642 x `Target book not found: עולת ראיה` + `Original path` -> one WARN per missing target with count and path (bounded), plus a total. Links are still dropped exactly as before (otzaria-library O7 fixes the data). - Contradictory summaries: `type=REFERENCE resolvedPairs=4421 written=106912` was a labelling problem — the read counters are keyed by the CSV's `Conection Type`, `written` by the stored type after blank inference. Labels are now csvRowsRead/csvDropped/csvResolvedPairs/ storedWritten with a reconciling totals line (3,899,477 resolved vs 3,886,967 written; gap 12,510 = heading/self-link filters + INSERT OR IGNORE). `GenerationApplyResult(…, 0)` hard-coded unmatched=0 next to "13 unmatched" — fixed. All-metadata's 1,116 unmatched titles are named (bounded WARN + report file) for otzaria-library's O6. The JSON metrics consumed by the QA scripts are byte-identical. - build_state.db written 5x: all five writers traced to consumers (four feed the next stage's InMemoryIdAllocator.load, the fifth is the published seforim.db.buildstate.zst); none dropped; each write now prints its seconds. - 407 books without a source hash: Sefaria's hash computer walks merged.json before the blacklist filter, so 395 blacklisted books contribute hashes no id claims; Otzaria's 16 are classified. Named per class; recording behaviour unchanged (operator decision). - Havrouta 38 found / 37 processed: the unmatched tractate is named with the reason. line_ref's 90 ambiguous keys are named (bounded) and written to the report file. - The insert-path failure breadcrumb removed by S4a is restored via a currentBook field reported by the outermost catch (no re-indent, no control-flow change). GenerateLinks.kt logged the literal `${tables.size}` — fixed. - New common/reports/GeneratorReport.kt writes the bounded lists to <root>/build/generator-reports (pinned in root build.gradle.kts because each generator stage forks with its own subproject cwd). Nothing collects that directory yet: the workflow contract forbids Actions artifact storage (every handoff is a content-addressed pre-release), so wiring it is an operator decision. Measured on build-job-34024655297.log: about -4,750 lines / -0.40 MB (the 40-line exception blocks were the audit's "blank stack-frame" lines), +~19 summary lines. Tests 495 -> 527 with the same 9 pre-existing Windows path failures; compiler warnings 8 -> 8; workflow contract tests 42 -> 44 (the workflow itself is untouched). --- f4d5c01 fix(generator): fail closed when the seed copy or the buildstate snapshot fails; verify the written state GenerateLinks and GenerateLines wrapped the seed (ATTACH the base DB and copy every table) in runCatching and continued on failure; in appendOtzariaLinks the base DB and the persist target are the same file, so a failed seed let the later VACUUM INTO replace the 7 GiB DB with an empty one and the build reported success. Both seeds now log the base DB and the table the copy died on and rethrow, before any VACUUM INTO; the pre-existing target is left byte-identical. The five snapshotTo sites (Sefaria, metadata seed, lines, links, havrouta) swallowed a failed buildstate write with a warning, so a DB could ship with an allocator state the next build would reuse to hand out ids twice. All five now rethrow, and GenerateLinks snapshots after its persist so a failed persist never advances the state. After every snapshot, BuildStateVerifier reopens only the meta and id_counters tables, checks the stamp is this run's, and asserts next_id > MAX(id) for each allocator-issued table (one b-tree edge per table, ~20 ms on a 5 GB DB); alt_toc_entry is excluded with the evidence that its ids are implicit rowids by design, which real generator output confirmed. --- 691b8a9 fix(generator): a missing required seed fails before the target is touched; publish through an atomic candidate Round 3 of the audit of cycle 33987355439 (item S16). GenerateLinks and GenerateLines warned and continued with an empty database when the seed was absent while appendExistingDb was set, and the release tasks pass one path as both base and target, so the reachable outcome was an empty DB shipped as a success. Now: appendExistingDb with no seed throws before any write to the target, naming the path and the opt-in that permits an empty base (-PallowEmptyBase, forwarded by the four seed-using JavaExec tasks because a Gradle -P is not a system property of the forked JVM). Both generators publish through DbPublish: the result is written to <target>.candidate in the same directory, verified, and moved over the target with ATOMIC_MOVE + REPLACE_EXISTING; stale -journal/-wal/-shm beside the candidate and the target are removed around the move. The publish → snapshot → BuildStateVerifier order from f4d5c01 is unchanged. Tests: OtzariaBuildFailClosedTest 13/13 (throw before the target is modified, no candidate left behind, stale WAL does not survive the publish, the release tasks forward the opt-in and pass one path as base and target). otzariasqlite 57/57. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ot silently skipped; QA drift is gated and observable Squashed from the audit branch by file set (1 original commits contributed; their messages follow). --- 9b586e0 fix(qa): a Sefaria schema that parses but is not a book is counted, not silently skipped load_schema_books failed on unreadable JSON but silently `continue`d over valid JSON whose top level is not an object or that has no object-valued `schema` key, so such a book vanished from both the expected set and the DB and could slip under the 2%/10-book drift gate (--expect-snapshot runs in no workflow). Both shapes now land in the same `unreadable` accounting: counted, named with the reason, allowlistable through KNOWN_UNREADABLE_SCHEMAS like Sheet.json, and otherwise a failure. Measured against the 2026-09-01 export (6,601 schema files): zero files take either branch, so this is a no-op today and fail-closed for the next export that ships a different shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…locked monotonic use stamp; progress and named failures in the patch fan Squashed from the audit branch by file set (4 original commits contributed; their messages follow). --- 06b5d0b fix(ci): durable, digest-verified patch-fan anchor cache; progress and named failures in the patch fan Audit of cycle 33987355439 (item S8, reports 07/08). Prefetched patch-fan anchors (~1.3 GB seforim.db.zst each, ~5.2 GB) were downloaded in the background, deleted by the cleanup step's `rm -rf … prefetch` when attempt 34021998271 failed 45 min later, and re-downloaded in full by the successful retry (237+241+232+243 = 953 s, ~249 s wall at parallel=5, plus downlink contention with the generator). Anchors are immutable content-addressed release assets. - prefetch_patch_anchors.sh: durable cache at ${XDG_CACHE_HOME:-$HOME/.cache}/seforimlibrary/patch-anchors (the same per-runner root the image embedder uses; never the tmpfs — path AND `stat -f` filesystem-type guard), keyed by release tag, sha256-verified against the release's published digest before every reuse (mismatch, republished tag, or missing digest with no recorded sha -> ::warning:: + fresh download), hardlinked in and out (copy fallback across filesystems; a partial copy is removed so gh's no-clobber download can proceed), bounded at 8 anchors / 30 days by cache_prune under a lock that goes stale after 600 s (a pruner killed mid-run no longer disables the bound), applied on all-hit runs too. Tag names are validated as single path components (`..` refused). One line per anchor: `reused <tag> from cache (sha256 ok)` / `downloaded <tag> in Ns`; unwritable cache -> warn and download to the run dir. The cleanup step only reports on the cache; S2's temp sweep is disjoint by root and by name; `rm -rf prefetch` drops only hardlinks. - The five bare `ANCHOR` lines are `anchor <tag> (offset N): <asset> <size>`; the abort paths of prefetch_patch_anchors.sh and upload_early_release_assets.sh say what they aborted and why (still never failing the job); a download failure names the missing asset. - "Produce + verify patch fan" was dark for 22 minutes: numbered per-anchor start lines, a per-anchor end line (elapsed, patch size, verify result, the folded full-snapshot columns) and a 5-minute heartbeat while producing. The launcher command line and anchor selection are unchanged. - QA fixes: the new cache-report call in the if: always() cleanup step is guarded (a job dying before the pipeline-control checkout would have gone red on exit 127); cache-store failures are ::warning::. Tests: test_prefetch_patch_anchors.py (cold/warm/corrupted/bound/stale lock/abort + 8 edge cases: republished tag, absent digest, no-hardlink fs, partial copy, unwritable root, hardlink survival, age bound, set -u without HOME), workflow-contract tests for the cache path, cleanup exclusion, sweep exclusion, progress lines and guarded helper calls; 123 pytest; 45/45 step bodies bash -n; 0 CR in the diff. Known limit: the heartbeat lists anchors in batch order, so an anchor that finishes out of order stays listed until its predecessor lands (over-inclusive, never a false all-clear). --- 2350a6c fix(ci): rank the patch-anchor cache by a monotonic use stamp, not by whole-second mtime prefetch_patch_anchors.sh pruned by `stat %Y` + `sort -rn`: entries stored or refreshed within the same second tied, sort fell back to comparing the rest of the line, and the entry just downloaded (v23-x) could rank below older ones and be evicted. In the reconcile job the eviction test flaked 0/6 green; in production the cache could discard the anchor it had spent ~16 minutes fetching. Each entry now records used_ns = max(now_ns, highest used_ns in the cache + 1) in .meta on store and on every hit (a Lamport stamp: later writes always rank higher, whatever the clock or filesystem resolution), and the pruner sorts by that key with the .meta mtime as the tie-break for pre-fix entries. The refresh rewrites .meta via temp+rename under $BASHPID (the five parallel fetches are subshells sharing $$). Keep count, age cutoff, eviction report line and lock protocol are unchanged; the new tests fail on the previous script with the defect itself. --- 1ba52c6 test(ci): gate the anchor-cache sandbox suites on a capability probe instead of failing on BSD tools The three sandbox classes in test_prefetch_patch_anchors.py drive prefetch_patch_anchors.sh and the fan step through a real bash and need mapfile (bash 4), GNU stat --format, GNU sed -i and sha256sum; on macOS they failed instead of skipping. One cached bash probe at import now decides HAS_GNU_SANDBOX and the classes skip with the missing capability named. The probe is by feature, not platform: Windows Git Bash still runs all 22 tests, and on Linux a probe that comes back false raises instead of skipping, so a broken probe cannot silently retire the S8/S12 coverage in the reconcile and contracts jobs. --- e4956d8 fix(ci): serialise the anchor-cache use stamp under flock; the prune never falls back to tag order Round 3 of the audit of cycle 33987355439 (item S18). The five parallel prefetchers allocated `used_ns` as read-max → write with no lock, so two could get one stamp (reachable on the highest+1 branch or with a %N-less date), and a full tie made `sort` decide by whole line, i.e. by tag — the freshest anchor could still be evicted. Allocation now runs under `flock -w 30` on `<cache>/.stamp.lock`, the floor is persisted in `<cache>/.stamp` so a `.meta` not yet written cannot be overtaken, the fallback key is `<key>.<BASHPID>` when flock is unavailable, and the prune sorts stably on three numeric keys. Tests: STAMP_ALLOC_RACE phase (20 concurrent marks → distinct, strictly increasing keys; the last-marked anchor survives a keep-1 prune under forced future stamps) and a rank-contract test on the flock/sort text. Prefetch suite 26/26 on a CR-normalised copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…light and no-snapshot recovery, named failures in publish and validators, every run: script under GitHub's 21,000-character ceiling Squashed from the audit branch by file set (7 original commits contributed; their messages follow). --- 81ab27e ci(build): make the relink wait observable — heartbeat, a named cause on cancel/failure, and no swallowed poll errors In run 33991433362 the wait step printed `relink run: 33994031370` and then nothing for 28,875 s (~480 polls), followed by the generic `##[error]relink run 33994031370 concluded cancelled`. The child had in fact hit its own timeout-minutes. Poll stderr went to a file that was only grepped for HTTP 401 and overwritten each minute; every other API failure was "transient", fell to `*) queued_for=0` and thereby reset the 30-minute pre-start stuck cap, so a sustained outage would poll silently to 1440 min. - One line per 10 minutes: elapsed, child status, current job/step (one jobs-API call, tolerated), run URL; the URL is printed once up front. - completed:cancelled compares the longest job's duration with the child's timeout budget (kaggle 90 / local 1440 / server 480, mirrored from relink.yml) and says whether the cancel matches it or was external; completed:failure lists the failed jobs and steps; every terminal branch ends with the recovery hint (`relink-recovery request=… parent=…`, then a rebuild with relink_recovery_run_id) and the unchanged ##[error] line. - Transient poll failures print the first stderr line with a consecutive counter, fail the step after 30 in a row with the last stderr, and no longer touch queued_for. The 401 fast-fail and the 1800 s cap are byte-identical. - The step's prose no longer describes the split 90m+480m+30m budget for the default target=local single job. Driven under a stubbed gh across success, cancel-at-budget, cancel-early, failure, timed_out, 31×503, 401 and queued interleaved with 502 (the cap now fires; it did not before). test_manual_release_workflow.py: 36 tests. --- d2fb699 fix(ci): recovery mode publishes no snapshot; delete unconsumed snapshot on failure; idempotent gradle.properties; safe temp sweep Audit of cycle 33987355439 (item S2, report 08-seforimlibrary-build). - Publish-snapshot step is skipped when relink_recovery_run_id is set. A recovery build never dispatches a relink, so nothing ever downloads its own rebuilt snapshot; runs 34021998271 and 34024655297 each spent 464 s uploading a 987 MiB pre-release that was orphaned immediately. The dump still runs (Phase-2 semantic compare needs the local bytes). - build_provenance.json snapshot_sha256 now names the snapshot the link set was produced from (EXPECTED_LINKER_SNAPSHOT_ZST_SHA256, which the relink step defaults to this build's own digest on every normal path). In recovery mode the rebuilt snapshot is unpublished, and both standalone-relink consumers (fetch_relink_inputs.sh, relink.yml manual mode) hard-fail on a 404 tag, so naming it would break them. - New failure/cancelled hook deletes this attempt's own lines-snapshot-sha256-<sha> pre-release, only when this attempt created it (SNAPSHOT_RELEASE_CREATED_HERE, set solely on a successful gh release create) AND the relink was never dispatched (RELINK_DISPATCH_STARTED unset) AND the relink step was not cancelled (GITHUB_ENV application for a cancelled step is undocumented). Tag shape is regex-checked; every branch exits 0. - ~/.gradle/gradle.properties heaps are written as one marked block (awk drops any previous block and bare duplicates); the file no longer grows every week. Values unchanged; the grep -Fxq gate still matches. - Cleanup step removes this attempt's lines-snapshot-<run>-<attempt>.db and manual-links-inputs-<run>-<attempt>/ with sizes, then a bounded -maxdepth 1 -mtime +2 sweep of the same two name shapes. Everything is best-effort (|| true): the step is if: always() under bash -e and must not turn a published build red. - Teardown: recovery mode says it dispatches no relink of its own (the old text claimed it never reached the dispatch boundary); the closing line lists the run ids it actually cancelled. QA: 51 tests pass, YAML loads, bash -n on all 44 step bodies, deletion guard exercised in a sandbox with a stub gh (7 branches), gradle block idempotent over 3 runs. Known accepted gap: a publish step that creates the release and then fails during upload exports no marker, so that release leaks rather than risking a wrong delete. --- c8fb855 fix(ci): preflight the recovery relink inputs at dispatch time; contract test on the Linker's real meta.json layout Audit of cycle 33987355439 (item S3, report 08-build-failed-34021998271). Recovery build 34021998271 ran 48 minutes (DB generation, snapshot dump, a 987 MiB upload) before Phase-2 rejected the Linker's schema-3 meta.json ("recovery snapshot comparison requires schema 2"). The file that failed was downloadable at second 0: the recovery relink had finished 37 s before the build started, and meta.json is published as a 759-byte standalone asset on every linker-release-sha256-* release. - manual-generate-release.yml: new step "Preflight the recovery relink inputs", last step of the reconcile job (ubuntu-latest, before the host lease, DB generation, dump and publish), gated on relink_recovery_run_id != '' and reuse != 'true'. It walks the same chain Phase-2 walks later (run -> display_title -> linker-output-<req> -<attempt> -> pointer/manifest -> linker-release-sha256-* -> meta.json -> lines-snapshot-sha256-<sha> in this repo) and runs the SAME verifier code (--preflight) on the real meta.json. Every refusal is one ::error:: naming file, field, value seen and expected; success prints exactly one line. Nothing in the build job's guard or Phase-2 is weakened or moved. - verify_relink_recovery_snapshot.py: check_payload_meta() / check_baseline_manifest() shared by --preflight and verify(); cheap input checks run before the 5 GiB hash; snapshot.sha256 must be 64 hex; --preflight combined with the comparison flags is refused (it silently skipped the comparison and exited 0 — fail-open, found by QA); PAYLOAD_META_SCHEMAS documented as the single source of truth citing the Linker writer (src/incremental.py write_meta, schema 3 since a9ae2d4). - test_verify_relink_recovery_snapshot.py: fixture is the Linker's real writer layout (key-for-key); positive test per accepted schema in both modes, negative at max+1 naming file and value, both entry points refuse identically, tripwire if the accepted set outruns the observed Linker schema; sys.executable instead of python3 (13 tests failed on Windows before). - test_manual_release_workflow.py: pins the new step (in the reconcile job, its gate incl. the reuse clause, ordered before tmpfs/lease/DB generation/dump/Phase-2, one positive line, Phase-2 still runs the full verifier without --preflight). 11/11 mutations caught. QA fixes: `IFS=$'\t' read` collapsed a null conclusion (tab is IFS whitespace) -> jq `// "none"`; a jq failure in a process substitution died under set -e without an annotation -> `||` arm. Sandbox: 36 cases with a stub gh (1 happy path = one line, 35 refusals = one ::error:: each); live read-only replay of run 34021656701 passes every check. 72 tests pass, 45/45 step bodies bash -n clean. Known limit: the line-baseline manifest (schema 2) lives only inside the 128 MB payload and is still checked at Phase-2. --- 807b36b fix(ci): make publish, validators, manifest, dispatch and QA drift observable; gate reference-snapshot drift Audit of cycle 33987355439 (item S6, report 07-build-success-nongenerate). - release_draft.sh: the 27-minute publish printed two lines for 3.4 GiB across 10 assets. upload_asset now prints one line per asset (`uploaded <name> <size> sha256=<12> in <s>s (<MB/s>)`, or `reused …` when the content-addressed dedupe skipped a re-send) and a closing `release <tag>: N assets, <bytes> bytes, <s>s`. Every failure message, retry budget and both gh release upload invocations are byte-identical. - validate_build_provenance.py, validate_otzaria_provenance.py and validate-sefaria-release-metadata.py print exactly one `ok: …` line on success (failure contract unchanged: stderr, non-zero, empty stdout); the reconcile reuse scan runs the first with --quiet and prints one `ok: build_provenance checked on N releases` instead of ~27 lines. - Release manifest: `gh release view … 2>/dev/null` wrote `latest: null` on ANY failure and the step stayed green. Only gh's `release not found` / a 404 on a …/releases… URL yields `latest: null` (with an explicit line); anything else is ::error:: + exit 1. Same fix in update-release-manifest.yml (same bug, same file, same branch); both now print the manifest summary, the commit or "no changes". - Dispatch: one line before/after the otzaria-library repository dispatch (a 204 has no body) and one at the relink `gh workflow run` (workflow, target, non-secret inputs); the run URL stays with the wait loop (pinned to one occurrence). - QA: reference-snapshot drift was printed and never gated (this cycle: baseProvenance=1 12970 vs 13056 = -0.659%, PASS). gate_snapshot_drift in scripts/qa/common.py, wired into check1 (total + per type), check2 and check7: a shrink is ::error:: + exit 1 only when it exceeds BOTH QA_DRIFT_MAX_SHRINK_PCT (workflow: 2, a 3x margin over this cycle's worst) AND the absolute floor QA_DRIFT_MIN_SHRINK_ABS (workflow: 10; without it a -1 on guides=2 / midrash=5 / inferred pairs=20 / targum=45 would fail the weekly release over one book leaving the export); everything else is ::warning:: with the numbers. The gate is off when the env is unset, and run_all.py --require-all (release mode, the workflow's QA step) fails if it is unset, so it cannot vanish silently. - Sheet.json: a 0-byte file in the pinned export (Sefaria's source-sheet pseudo-index, the only unparseable one of 6,601 schemas; the Kotlin importer drops it identically). Now an explicit KNOWN_UNREADABLE_SCHEMAS entry with one INFO line; any OTHER unparseable schema is a hard failure instead of a silent coverage cut that still reports PASS. - catalog.pb: never produced by design on this branch (the workflow rm -f's stale copies, BRANCH-STACK.md records its removal, the manifest omits catalogBlobName), so the 5x WARN is one INFO; an explicitly requested missing catalog keeps its WARN. QA: stub-gh sandboxes for the upload path (incl. 0-byte asset), 9 manifest scenarios on both step bodies, the drift gate matrix (run as a script), Sheet.json confirmed 0 bytes on a real export; 103 pytest + QA synthetic matrix pass; 47/47 step bodies bash -n; no S1-S5 hunk reworked. --- d5fc42e fix(ci): keep every run: script under GitHub's 21,000-character expression ceiling GitHub refused to load manual-generate-release.yml at 15ef349: "(Line: 1974, Col: 14): Exceeded max expression length 21000" (run 34195296928, zero jobs). S1 and S8 had grown the relink-wait step to 21,230 characters and the patch-fan step to 22,592; the workflow could not run at all. Two cohesive blocks move out unchanged: the fan's read_schema / produce_anchor / heartbeat / drain_batch into patch_fan_lib.sh, sourced by the step (produce_anchor is backgrounded and drain_batch mutates the driver's arrays, so they stay in the step's shell), and the relink poll loop into wait_for_relink_run.sh, run as a child with RUN, RUN_URL and RELINK_REQUEST_ID passed as env (nothing after the loop reads its variables; it re-states set -euo pipefail, which the step already used). Largest run: script is now 16,210 characters. A test asserts every run: script in every workflow stays under 19,000, and ci.yml gains a hosted `contracts` job running the same Python suite loop the reconcile job runs, so pull requests get that check even when the release workflow itself cannot load. test_prefetch_patch_anchors' eviction sandbox now ages its older entries explicitly instead of relying on sub-second ordering (the production tie-break is addressed separately). --- c0ebaf6 fix(ci): give the release parent a budget that outlives its relink child; cap the wait explicitly build-and-release ran under timeout-minutes: 1440 while the relink child it dispatches (target=local) has its own 1440-minute ceiling; the comment already admitted the contradiction. A child legally running to its cap outlived the parent, GitHub killed the parent, the always() step cancelled the child, and the week shipped nothing. The parent is now 2880 minutes (self-hosted jobs may run five days), leaving ~1385 minutes after a maximal child for Phase-2, the patch fan and the uploads (~130-160 measured); both host-lease TTLs move to 176400 s with the same margin over the new ceiling. wait_for_relink_run.sh caps the wait at the tighter of RELINK_WAIT_CAP_MIN and child_timeout_min + 60 and fails with a named error carrying the recovery recipe, so a child stuck past its own ceiling surfaces a day before the parent would have died; garbage or missing cap values still cap, a healthy fast child is never truncated. The Kotlin workflow contract test pins the same literals and moves with them; it also still pinned upload-artifact@v4 after S9 moved the workflow to @v6, which had left the CI test job red on this branch. --- 1a78e92 fix(ci): the relink wait cap covers the child's whole job chain per path, not one job Round 3 of the audit of cycle 33987355439 (item S17). wait_for_relink_run.sh capped the wait at the child's single relink job timeout plus an hour, but relink.yml is a chain: on target=kaggle relink (90) → resolve (480) → publish (30) = 600 minutes, on server 480 + 30, on local 1440 + 30. A healthy kaggle run was declared out of contract at minute 150 and cancelled by the parent's cleanup. One table now derives two numbers per path: child_job_max_min (the longest single job, used only to tell a self-timeout from an external cancel) and child_path_budget_min (the sum of the path's jobs, the wall clock waited on); the cap is budget + 60 → 660 / 570 / 1530, and RELINK_WAIT_CAP_MIN moves 1500 → 1530 so the local path keeps a real margin. The parent's 2880-minute ceiling leaves ≈ 1295 minutes for Phase 2 after the largest cap; the host leases (176400 s) still outlive the parent by an hour. Tests: per-path caps replace the assertion that pinned 150; a kaggle child still running at minute 300 is waited on; the overdue cancel fires at the new cap; the table's numbers are pinned against relink.yml's timeout-minutes (cross-checked against a sibling Linker checkout when present). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…shes only, test counts in every gate, workflow contract tests incl. the run: length guard, __pycache__ untracked and ignored Squashed from the audit branch by file set (3 original commits contributed; their messages follow). --- 15ef349 ci: setup-gradle v5 with real cache reuse; contract.yml on otzaria pushes only; test counts in every gate; native-access flag ends 60 JVM warnings Audit of cycle 33987355439 (items S9 + S7, reports 12 and 07). S9 - ci.yml: gradle/gradle-build-action@v3 (deprecated; no inputs; the audited run executed 67/67 tasks after a ~400 s cache step) -> gradle/actions/setup-gradle@v5 with cache-read-only on non-otzaria refs (write on trunk, read on PRs). contract.yml's setup-gradle v4 -> v5 so the repo runs one generation (test pins them equal). v6 is not taken: it moves caching into a proprietary component under commercial terms and drops configuration-cache support, which this repo enables (operator decision). - contract.yml: `push: branches: [otzaria]` (+ unfiltered pull_request); the pipeline's own release/handoff refs no longer re-trigger it. No tag trigger existed, nothing in the four repos looks up a contract run, and the trunk has no branch protection, so no required check is lost. - Every Python-suite gate prints `tests: <suite> N passed` per suite and a total, and fails on a failing suite AND on a suite collecting zero tests; ci.yml and contract.yml count the JUnit XML (`tests: allTests …`) and fail if no XML exists. The reconcile job's loop now also runs test_ci_workflows.py, test_patch_anchor_schema.py and test_prefetch_patch_anchors.py (hermetic: stub gh, no network) — S8's prefetch work was gated by nothing before; a contract test asserts the set of test_*.py files on disk equals the wired set. - Node 20 deprecations: every ubuntu-latest job moves to the lowest node24 major (checkout v5, setup-java v5, upload-artifact v6, setup-gradle v5), each verified from the action's action.yml; self- hosted jobs untouched. The "Gradle-8" line is KGP's own inert wording (0 .java sources) and is explained in ci.yml rather than silenced. S7 - The 60 `WARNING: A restricted method in java.lang.System has been called … System::load` lines were 15 forked-JVM blocks from org.sqlite.SQLiteJDBCLoader (Generate x9, Dump lines, Apply LINKER, patch fan x4). `--enable-native-access=ALL-UNNAMED` is applied centrally via a CommandLineArgumentProvider on every JavaExec/Test task in the root build (not jvmArgs, which ~25 tasks assign and two lines of which the recovery overlay greps verbatim) and via patchPipelineJvmArgs for the fan's direct java. Proven: 4 warnings -> 0 on the same class; a real JavaExec reaching NativeDB -> 0; the launcher spec publishes the flag; configuration cache stored and reused. - `--warning-mode all` on ci.yml's single Gradle invocation: exactly two deprecations, both from the Android Gradle plugin (lint-gradle, aapt2), none in this repo's build scripts. - The remaining gh stderr leak (prefetch's unexpected seforim.db.zst failure) folds gh's reason into its named line. - Not done, by decision: moving the 45 `run:` bodies (2,403 lines, ~127 KB of echoed script per log) to files before Thursday. QA: action majors and `with:` keys verified per action.yml; loops executed (failing suite, zero-collection, empty XML dir); 137 pytest; 108 step bodies across 9 workflows bash -n; 0 CR in the diff. --- f0c95a1 chore: untrack .github/scripts/__pycache__ (added by mistake in 807b36b) and ignore it --- 3772bac Merge origin/otzaria (PR #27 per-table content hash) into fix/audit-cycle-33987355439 contract.yml was the only conflict: base added the logical-hash fixture pair (env, test -f, cmp, --tests '*LogicalHashContractTest*'); this branch added the otzaria push filter, the @v5 action majors and the JUnit-XML test count. Resolved by keeping both: 6 fixtures present, 3 byte-identical pairs, and the count line labels both test classes. test_ci_workflows.py pins those literals, so its three assertions move in lockstep (4→6, 2→3, combined label). PatchPipelineCli.kt auto-merged (non-overlapping hunks); no script in .github/ parses the delta manifest's new fromTableContentHashes/toTableContentHashes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
palmoni5
force-pushed
the
fix/audit-cycle-33987355439
branch
from
September 9, 2026 09:14
5888ffd to
14b5413
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.