From e74a7bf5dcb37c55b8dbe14161d31b2a524c9a0f Mon Sep 17 00:00:00 2001 From: ypl <7353755@gmail.com> Date: Wed, 9 Sep 2026 12:13:33 +0300 Subject: [PATCH 1/7] fix(images): retry and a durable content-addressed textimages cache with a URL sidecar; one URL grammar; a build gate on failed embeds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed from the audit branch by file set (3 original commits contributed; their messages follow). --- f918165 fix(images): retry + durable cache for textimages, one URL grammar, and a build gate on failed embeds Builds on the owner's in-progress work (downloadWithRetry with 3 attempts and backoff, HTTP 418 treated as final, durable cache via -PimageCacheDir) and closes what the v27 build logs showed: - collectUrls stopped at whitespace while IMG_TAG_REGEX did not, so `…/ShaarHahakdamot/Screenshot 2023-… .png` was fetched truncated and could never be substituted even after a successful download. One shared URL_BODY now drives both; the literal URL stays the map/cache key and only the HTTP request is percent-encoded (existing `%20` is left alone). - newlyDownloaded/failed were plain Ints incremented from 16 coroutines (`cached=410` vs `downloaded=409`); now AtomicInteger + ConcurrentHashMap. - A failed download silently left the remote URL in the row (10/420 images shipped as network URLs in v27). The embedder now writes image-embed-report.json (counts + one entry per failed URL with reason: timeout / http / content-filter 418 (NetFree on the self-hosted runner) / malformed-url / oversized), and the build step reads it: ::warning:: per failed URL, ::error:: and exit 1 above IMAGE_EMBED_MAX_FAILURES (default 20) or when the report is missing. - Log volume: one summary line (greppable `cached=` prefix) and one stack-trace-free line per failure instead of a stack trace per attempt (81% of the embedder's log lines were repeated frames); cache hits are a count. Tests: 5 new (spaced-URL round trip, request-only encoding, failure report contents, exact counters under 120 concurrent URLs, report-path property); :sefariasqlite:jvmTest --tests '*SefariaImageEmbedder*' 14/14. --- 5888ffd fix(images): content-addressed cache keys with a URL sidecar; migrate the legacy cache without a download cacheFileName replaced `/`, `:` and `?` with `_`, so two distinct image URLs could map to one durable cache file and the next build served the wrong image from the cache and reported success. The key is now sha256(url); each entry carries a `.url` sidecar written before the bytes (temp + atomic move), and a hit requires the sidecar to name the requested URL, otherwise the entry is evicted and fetched again. The legacy layout is migrated in place: on a miss the old name is looked up, claimed by writing the sidecar and renamed, so a warm cache costs zero downloads (verified on the 417 URLs of the 2026-09-01 export: 417 served from disk, 0 fetched, 0 legacy collisions) and the IMAGE_EMBED_MAX_FAILURES gate is never exposed. Legacy names that two URLs of the same export would share are refused outright so a historical collision is repaired rather than frozen. URLs whose legacy name is not a valid path on the host are treated as a miss instead of throwing. --- 35c8f42 fix(images): a cache hit requires a byte-exact URL sidecar; drop the provenance-free legacy migration Round 3 of the audit of cycle 33987355439 (item S15). The legacy cache name was the URL suffix with `/ : ?` folded to `_`, which is not injective (…/Tikkunei_Zohar/40.png and …/Tikkunei_Zohar_40.png collide), and the migration adopted such a blob under the new sha256(url) key with a "verified" sidecar without any proof of which URL had produced it. The durable cache directory was introduced by this very branch and no release run has executed on a branch head (the single run on 15ef349 never started a job), so there are no legacy entries to migrate: the migration, the dual read and the two-URLs-one-name refusal are removed. A hit now requires `.url` whose bytes equal the URL exactly; anything else is a miss and a fresh download, with the sidecar written before the bytes as before. Tests: no hit without a matching sidecar, a byte-inexact sidecar is a miss, a legacy-named blob (alone or as a collision pair) is never adopted, miss → download → sidecar + blob. SefariaImageEmbedderTest 21/21. Co-Authored-By: Claude Fable 5.1 --- generator/sefariasqlite/build.gradle.kts | 10 + .../sefariasqlite/SefariaImageEmbedder.kt | 406 +++++++++++++++-- .../sefariasqlite/SefariaImageEmbedderTest.kt | 410 +++++++++++++++++- 3 files changed, 782 insertions(+), 44 deletions(-) diff --git a/generator/sefariasqlite/build.gradle.kts b/generator/sefariasqlite/build.gradle.kts index 83fa916c..55fde879 100644 --- a/generator/sefariasqlite/build.gradle.kts +++ b/generator/sefariasqlite/build.gradle.kts @@ -129,6 +129,16 @@ tasks.register("generateSefariaSqlite") { if (project.hasProperty("linkerSidecar")) { systemProperty("linkerSidecarPath", project.property("linkerSidecar") as String) } + // Durable textimages.sefaria.org cache (see SefariaImageEmbedder.defaultCacheDir). + if (project.hasProperty("imageCacheDir")) { + systemProperty("imageCacheDir", project.property("imageCacheDir") as String) + } + // Where the embedder reports what it could NOT inline; the workflow gates + // on it (see SefariaImageEmbedder.defaultReportPath). Absolute path please — + // this JavaExec runs with the project dir as its working directory. + if (project.hasProperty("imageEmbedReport")) { + systemProperty("imageEmbedReport", project.property("imageEmbedReport") as String) + } // Optional JVM tuning (similar to generator) jvmArgs = listOf( diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedder.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedder.kt index e59bec55..9e3c8b3c 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedder.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedder.kt @@ -7,18 +7,23 @@ import kotlinx.coroutines.awaitAll import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.sync.Semaphore import kotlinx.coroutines.sync.withPermit +import kotlinx.serialization.json.Json import java.net.URI +import java.net.URISyntaxException import java.net.http.HttpClient import java.net.http.HttpRequest import java.net.http.HttpResponse +import java.net.http.HttpTimeoutException +import java.io.IOException import java.nio.file.Files import java.nio.file.Path import java.nio.file.Paths import java.nio.file.StandardCopyOption +import java.security.MessageDigest import java.time.Duration import java.util.Base64 import java.util.concurrent.ConcurrentHashMap -import kotlin.io.path.exists +import java.util.concurrent.atomic.AtomicInteger import kotlin.io.path.isRegularFile import kotlin.io.path.readBytes @@ -43,6 +48,36 @@ object SefariaImageEmbedder { private const val USER_AGENT = "SeforimLibrary-SefariaImageEmbedder/1.0" private const val DOWNLOAD_PARALLELISM = 16 private const val MAX_IMAGE_BYTES = 5 * 1024 * 1024 // 5 MiB ceiling per image + private const val DOWNLOAD_ATTEMPTS = 3 + + /** `-PimageCacheDir=` / `-DimageCacheDir=`: where downloaded images persist. */ + internal const val CACHE_DIR_PROPERTY = "imageCacheDir" + internal const val CACHE_DIR_ENV = "SEFORIM_IMAGE_CACHE_DIR" + + /** `-PimageEmbedReport=` / `-DimageEmbedReport=`: where [ImageEmbedReport] lands. */ + internal const val REPORT_PROPERTY = "imageEmbedReport" + internal const val REPORT_ENV = "SEFORIM_IMAGE_EMBED_REPORT" + + /** + * Sidecar written beside every cached image: `.url`, holding the URL + * that produced `` verbatim (UTF-8, no trailing newline). It is what + * makes a cache entry self-describing — see [cacheFileName]. + */ + internal const val SIDECAR_SUFFIX = ".url" + + private const val HEX = "0123456789abcdef" + + /** + * Pause before download attempt 2 and 3. A transient timeout on + * textimages.sefaria.org (10 of 420 images timed out at 30 s in build + * 34021998271 while every one of them downloaded a day earlier) turns + * into a remote-URL line, which then differs from the previous build's + * inline data URI and breaks relink recovery. Tests zero this. + */ + internal var retryBackoffMillis: LongArray = longArrayOf(2_000, 6_000) + + /** Injectable for tests; production downloads over HTTP. */ + internal var downloader: (String) -> ByteArray = ::downloadBytes private val client: HttpClient by lazy { HttpClient.newBuilder() @@ -52,11 +87,20 @@ object SefariaImageEmbedder { .build() } + // How far a Sefaria image URL runs — ONE definition, shared by the up-front + // scan ([collectUrls]) and the rewrite ([IMG_TAG_REGEX]). The two must agree: + // a path may contain spaces (…/Screenshot 2023-05-04 at 3.09.20 PM.png), and + // a scan that stopped at whitespace keyed the cache on a truncated URL the + // rewrite could never look up — so that image stayed remote even when it + // downloaded. Stops at the attribute quote, at merged.json's `\"` escape, + // and at any line break. + private const val URL_BODY = """[^"'<>\\\n\r\t]+""" + // Regex used both to detect URLs up-front and to rewrite them inline. // Captures: (1) the full URL, (2) nothing — we replace the URL inside the // matched `` tag so surrounding attributes are preserved. private val IMG_TAG_REGEX = Regex( - "]*src=[\"'](https://textimages\\.sefaria\\.org/[^\"']+)[\"'][^>]*/?>", + "]*src=[\"'](https://textimages\\.sefaria\\.org/$URL_BODY)[\"'][^>]*/?>", RegexOption.IGNORE_CASE ) @@ -74,43 +118,62 @@ object SefariaImageEmbedder { * Scan [mergedJsonPaths], extract unique Sefaria image URLs, download any * that aren't already in [cacheDir], and populate the in-memory data-URI map. * Safe to call multiple times — already-cached entries are reused. + * Writes [reportPath] on every run: whatever did not download ships as a + * live network URL, so the build has to be able to see it. */ suspend fun prefetch( mergedJsonPaths: Collection, - cacheDir: Path = Paths.get("build", "sefaria", "image-cache"), + cacheDir: Path = defaultCacheDir(), + reportPath: Path = defaultReportPath(), logger: Logger = Logger.withTag("SefariaImageEmbedder") ) = coroutineScope { Files.createDirectories(cacheDir) + logger.i { "Embedder: image cache at ${cacheDir.toAbsolutePath()}" } val urls = collectUrls(mergedJsonPaths) - if (urls.isEmpty()) { - enabled = true - return@coroutineScope - } logger.i { "Embedder: ${urls.size} unique Sefaria image URLs to process" } val semaphore = Semaphore(DOWNLOAD_PARALLELISM) - var newlyDownloaded = 0 - var failed = 0 + // Counted from DOWNLOAD_PARALLELISM coroutines: plain `Int++` loses + // increments (34024655297 printed cached=410 with downloaded=409 on a + // cold cache), and these numbers now gate the build. + val newlyDownloaded = AtomicInteger() + val fromDiskCache = AtomicInteger() + val failures = ConcurrentHashMap() urls.map { url -> async(Dispatchers.IO) { semaphore.withPermit { - val cached = ensureCachedBytes(url, cacheDir, logger) - if (cached != null) { - if (cached.fromNetwork) newlyDownloaded++ - dataUriByUrl[url] = toDataUri(url, cached.bytes) - } else { - failed++ + when (val fetched = ensureCachedBytes(url, cacheDir, logger)) { + is Fetched.Bytes -> { + if (fetched.fromNetwork) newlyDownloaded.incrementAndGet() + else fromDiskCache.incrementAndGet() + dataUriByUrl[url] = toDataUri(url, fetched.bytes) + } + is Fetched.Failure -> failures[url] = fetched.reason } } } }.awaitAll() enabled = true + val report = ImageEmbedReport( + attempted = urls.size, + cached = dataUriByUrl.size, + downloaded = newlyDownloaded.get(), + fromDiskCache = fromDiskCache.get(), + failed = failures.size, + // Sorted so two runs over the same export produce identical bytes. + failures = failures.entries.sortedBy { it.key }.map { ImageEmbedFailure(it.key, it.value) }, + ) + writeReport(report, reportPath, logger) logger.i { - "Embedder: cached=${dataUriByUrl.size} (downloaded=$newlyDownloaded, failed=$failed)" + "Embedder: cached=${report.cached} (downloaded=${report.downloaded}, " + + "fromDiskCache=${report.fromDiskCache}, failed=${report.failed}) of ${report.attempted}" } + // One line per failure, no stack trace: the frames are always the same + // HttpClient chain, and every one of these ships as a remote . + report.failures.forEach { logger.w { "Embedder: not embedded (${it.reason}): ${it.url}" } } } /** @@ -127,12 +190,57 @@ object SefariaImageEmbedder { } } + /** + * The on-disk image cache. Configured through the `imageCacheDir` system + * property (gradle `-PimageCacheDir=`) or `SEFORIM_IMAGE_CACHE_DIR`; the + * fallback lives under the build dir, so on the CI runner (tmpfs build dir) + * it is lost with every build. A durable directory makes the embedded + * images identical across builds of the same export, which the relink + * recovery comparison depends on. + */ + internal fun defaultCacheDir(): Path { + val configured = System.getProperty(CACHE_DIR_PROPERTY)?.takeIf { it.isNotBlank() } + ?: System.getenv(CACHE_DIR_ENV)?.takeIf { it.isNotBlank() } + return configured?.let { Paths.get(it) } ?: Paths.get("build", "sefaria", "image-cache") + } + + /** + * Where the [ImageEmbedReport] is written. Configured through the + * `imageEmbedReport` system property (gradle `-PimageEmbedReport=`) or + * `SEFORIM_IMAGE_EMBED_REPORT`; CI points it beside the DB output and gates + * the build on it, a local run gets it under the build dir. + */ + internal fun defaultReportPath(): Path { + val configured = System.getProperty(REPORT_PROPERTY)?.takeIf { it.isNotBlank() } + ?: System.getenv(REPORT_ENV)?.takeIf { it.isNotBlank() } + return configured?.let { Paths.get(it) } + ?: Paths.get("build", "sefaria", "image-embed-report.json") + } + + /** + * Written atomically (temp file + ATOMIC_MOVE) so the gate never reads a + * half-written report. A write failure is not worth aborting a 40-minute + * import over — the gate fails the build on the missing file instead. + */ + private fun writeReport(report: ImageEmbedReport, reportPath: Path, logger: Logger) { + runCatching { + val dir = reportPath.toAbsolutePath().parent + Files.createDirectories(dir) + val tmp = Files.createTempFile(dir, "image-embed-report", ".json.tmp") + Files.writeString(tmp, report.toJsonReport()) + Files.move(tmp, reportPath, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING) + logger.i { "Embedder: report written to ${reportPath.toAbsolutePath()}" } + }.onFailure { logger.w(it) { "Failed to write image-embed report to $reportPath" } } + } + /** * Drop all in-memory cache state (does not touch disk cache). Mostly for tests. */ internal fun resetForTest() { dataUriByUrl.clear() enabled = false + downloader = ::downloadBytes + retryBackoffMillis = longArrayOf(2_000, 6_000) } /** @@ -146,7 +254,7 @@ object SefariaImageEmbedder { private fun collectUrls(mergedJsonPaths: Collection): Set { val out = HashSet() val bytePattern = URL_PREFIX.toByteArray(Charsets.UTF_8) - val textRegex = Regex("https://textimages\\.sefaria\\.org/[^\"'<>\\s\\\\]+") + val textRegex = Regex("https://textimages\\.sefaria\\.org/$URL_BODY") for (p in mergedJsonPaths) { val bytes = runCatching { Files.readAllBytes(p) }.getOrNull() ?: continue if (indexOfSubSequence(bytes, bytePattern) < 0) continue @@ -161,8 +269,9 @@ object SefariaImageEmbedder { private fun stripTrailingJsonArtifacts(url: String): String { // Sefaria JSON sometimes embeds URLs ending with a stray backslash or // punctuation from the surrounding content. Keep only characters that - // are legal in a URL path. - return url.trimEnd('\\', '\'', '"', ',', '.', ';', ')', ']', '}') + // are legal in a URL path — including the trailing space [URL_BODY] now + // admits so that spaced paths survive the scan intact. + return url.trimEnd(' ', '\\', '\'', '"', ',', '.', ';', ')', ']', '}') } private fun indexOfSubSequence(haystack: ByteArray, needle: ByteArray): Int { @@ -176,51 +285,228 @@ object SefariaImageEmbedder { return -1 } - private data class CachedBytes(val bytes: ByteArray, val fromNetwork: Boolean) + /** What one URL produced: bytes to inline, or why it stays a remote URL. */ + private sealed interface Fetched { + data class Bytes( + val bytes: ByteArray, + val fromNetwork: Boolean, + ) : Fetched + data class Failure(val reason: String) : Fetched + } - private fun ensureCachedBytes(url: String, cacheDir: Path, logger: Logger): CachedBytes? { + /** + * A hit requires an entry at `sha256(url)` whose `.url` sidecar holds this + * exact URL — there is no other way into the cache. In particular a file + * left over from the pre-sha256 naming scheme (path with `/ : ?` folded to + * `_`) is NOT adopted: that scheme was not injective (`…/Tikkunei_Zohar/40.png` + * and `…/Tikkunei_Zohar_40.png` both wrote `Tikkunei_Zohar_40.png`) and the + * file records no URL, so adopting it would inline one URL's image under + * another URL's key, durably and reported as a success. Such a file is + * simply ignored and left where it is — nothing reads it, and deleting + * files this build did not write in a cache dir that may be shared with a + * second checkout is the more dangerous option. + */ + private fun ensureCachedBytes( + url: String, + cacheDir: Path, + logger: Logger, + ): Fetched { val cachePath = cacheDir.resolve(cacheFileName(url)) - if (cachePath.exists() && cachePath.isRegularFile()) { - return runCatching { CachedBytes(cachePath.readBytes(), fromNetwork = false) } - .getOrElse { - logger.w(it) { "Corrupted cached image at $cachePath; redownloading" } - runCatching { Files.delete(cachePath) } - null - } - ?: return ensureCachedBytes(url, cacheDir, logger) + val sidecarPath = sidecarOf(cachePath) + + readVerifiedEntry(url, cachePath, sidecarPath, logger)?.let { + return Fetched.Bytes(it, fromNetwork = false) } - val bytes = runCatching { downloadBytes(url) }.getOrElse { - logger.w(it) { "Failed to download image: $url" } - return null + val bytes = runCatching { downloadWithRetry(url, logger) }.getOrElse { + // Reported once, by the caller, with the URL — see [failureReason]. + return Fetched.Failure(failureReason(it)) } if (bytes.size > MAX_IMAGE_BYTES) { - logger.w { "Skipping oversized image (${bytes.size} bytes): $url" } + return Fetched.Failure("oversized (${bytes.size} bytes)") + } + writeEntry(cacheDir, url, cachePath, sidecarPath, bytes) + return Fetched.Bytes(bytes, fromNetwork = true) + } + + /** `.url` for `` — see [SIDECAR_SUFFIX]. */ + private fun sidecarOf(path: Path): Path = + path.resolveSibling(path.fileName.toString() + SIDECAR_SUFFIX) + + private fun readSidecar(sidecarPath: Path): String? = + runCatching { Files.readString(sidecarPath) }.getOrNull() + + private fun evict(vararg paths: Path) { + for (path in paths) runCatching { Files.deleteIfExists(path) } + } + + /** + * A usable hit at the current name, or null (with the unusable entry + * evicted). The name is already sha256(url), so the sidecar is not what + * separates two URLs — it is what makes an entry that did NOT come from this + * URL (a hand-copied cache dir, a blob dropped in by another tool, a torn + * write) a miss instead of the wrong image reported as a success. It is also + * the ONLY way in: no name-shaped heuristic ever adopts bytes. + */ + private fun readVerifiedEntry(url: String, cachePath: Path, sidecarPath: Path, logger: Logger): ByteArray? { + if (!cachePath.isRegularFile()) return null + val owner = readSidecar(sidecarPath) + if (owner != url) { + logger.w { + if (owner == null) "Cached image at $cachePath has no $SIDECAR_SUFFIX sidecar; redownloading: $url" + else "Cached image at $cachePath was written for another URL; redownloading: $url" + } + evict(cachePath, sidecarPath) + return null + } + val bytes = runCatching { cachePath.readBytes() }.getOrElse { + logger.w(it) { "Corrupted cached image at $cachePath; redownloading" } + evict(cachePath, sidecarPath) + return null + } + if (bytes.isEmpty()) { + logger.w { "Empty cached image at $cachePath; redownloading" } + evict(cachePath, sidecarPath) return null } - val tmp = cachePath.resolveSibling(cachePath.fileName.toString() + ".part") - Files.createDirectories(cachePath.parent) - Files.write(tmp, bytes) - Files.move(tmp, cachePath, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE) - return CachedBytes(bytes, fromNetwork = true) + return bytes + } + + /** + * Sidecar first, bytes second: a crash in between leaves an orphan `.url` + * (harmless — the next run finds no bytes and downloads) instead of bytes no + * reader is allowed to trust. Both land through [atomicWrite]. + */ + private fun writeEntry(cacheDir: Path, url: String, cachePath: Path, sidecarPath: Path, bytes: ByteArray) { + atomicWrite(cacheDir, sidecarPath, url.toByteArray(Charsets.UTF_8)) + atomicWrite(cacheDir, cachePath, bytes) + } + + /** + * Unique temp file in the cache dir + ATOMIC_MOVE. The temp name is unique + * (not `.part`) because two builds can share the runner's durable + * cache, and two writers on one `.part` interleave into a corrupt image that + * the move then publishes. + */ + private fun atomicWrite(cacheDir: Path, target: Path, bytes: ByteArray) { + Files.createDirectories(cacheDir) + val tmp = Files.createTempFile(cacheDir, target.fileName.toString(), ".part") + try { + Files.write(tmp, bytes) + Files.move(tmp, target, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE) + } catch (error: IOException) { + // A unique temp name cannot be reused by the next attempt, so a + // failed write has to take its own file with it (the cache dir has + // no pruning of any kind). + evict(tmp) + throw error + } + } + + /** A status the server may answer differently in a moment (408, 429, 5xx). */ + internal class RetriableHttpStatus(val status: Int, url: String) : IOException("HTTP $status for $url") + + /** A definitive answer (404, the content filter's 418, ...): retrying only delays the build. */ + internal class FinalHttpStatus(val status: Int, url: String) : IllegalStateException("HTTP $status for $url") + + internal fun httpFailure(status: Int, url: String): Throwable = + if (status == 408 || status == 429 || status >= 500) RetriableHttpStatus(status, url) + else FinalHttpStatus(status, url) + + /** Timeouts (HttpTimeoutException is an IOException), resets, and [RetriableHttpStatus]. */ + internal fun isRetriable(error: Throwable): Boolean = error is IOException + + /** + * The short reason string [ImageEmbedReport] carries and the workflow turns + * into one `::warning::` per URL. Kept stable and greppable — a 418 is the + * self-hosted runner's content filter answering, never Sefaria (see the + * NetFree pattern), and that reads very differently from a real 404. + */ + internal fun failureReason(error: Throwable): String = when { + error is FinalHttpStatus && error.status == 418 -> + "content-filter (HTTP 418 — NetFree on the self-hosted runner)" + error is FinalHttpStatus -> "http ${error.status}" + error is RetriableHttpStatus -> "http ${error.status}" + error is HttpTimeoutException -> "timeout" + error is IllegalArgumentException || error is URISyntaxException -> "malformed-url" + else -> "${error.javaClass.simpleName}: ${error.message}" + } + + private fun downloadWithRetry(url: String, logger: Logger): ByteArray { + for (attempt in 1..DOWNLOAD_ATTEMPTS) { + try { + return downloader(url) + } catch (error: Throwable) { + if (error is InterruptedException) throw error + if (!isRetriable(error) || attempt == DOWNLOAD_ATTEMPTS) throw error + val backoff = retryBackoffMillis.getOrElse(attempt - 1) { retryBackoffMillis.last() } + logger.w { + "Image download attempt $attempt/$DOWNLOAD_ATTEMPTS failed " + + "(${error.javaClass.simpleName}: ${error.message}); retrying in $backoff ms: $url" + } + if (backoff > 0) Thread.sleep(backoff) + } + } + throw IllegalStateException("unreachable: $url") + } + + /** + * Characters a [URI] refuses verbatim: a raw space (Sefaria has + * `…/Screenshot 2023-05-04 at 3.09.20 PM.png`), the RFC 3986 "unwise" set, + * and anything non-ASCII. `%` is deliberately absent — most Sefaria paths + * arrive already encoded (`image%20-%200019.png`) and must not be encoded + * twice. + */ + private const val REQUEST_UNSAFE_CHARS = " \"<>\\^`{|}" + + /** + * The wire form of [url]. Encoding happens HERE and nowhere else: the + * cache map stays keyed on the URL exactly as it appears in the content, + * which is what [substituteImages] looks up. + */ + internal fun encodeForRequest(url: String): String { + val out = StringBuilder(url.length) + for (byte in url.toByteArray(Charsets.UTF_8)) { + val code = byte.toInt() and 0xFF + if (code in 0x21..0x7E && code.toChar() !in REQUEST_UNSAFE_CHARS) out.append(code.toChar()) + else out.append(String.format("%%%02X", code)) + } + return out.toString() } private fun downloadBytes(url: String): ByteArray { - val request = HttpRequest.newBuilder(URI.create(url)) + val request = HttpRequest.newBuilder(URI.create(encodeForRequest(url))) .timeout(Duration.ofSeconds(30)) .header("User-Agent", USER_AGENT) .header("Accept", "image/*") .build() val response = client.send(request, HttpResponse.BodyHandlers.ofByteArray()) if (response.statusCode() !in 200..299) { - throw IllegalStateException("HTTP ${response.statusCode()} for $url") + throw httpFailure(response.statusCode(), url) } return response.body() } - private fun cacheFileName(url: String): String { - val keyPart = url.removePrefix(URL_PREFIX) - return keyPart.replace('/', '_').replace(':', '_').replace('?', '_') + /** + * `sha256(url)`, lowercase hex. The name is a digest of the WHOLE URL, so + * two URLs can no longer share one file: the pre-sha256 name (the URL suffix + * with `/ : ?` folded to `_`) mapped both `a/b.png` and `a_b.png` onto + * `a_b.png`, and a hit is served without a request, so the wrong image was + * embedded, kept across builds (the cache is durable, has no TTL and stores + * no provenance) and counted as a success. + * The fixed 64 chars also retire the >255-byte name a deep path threw on and + * the `* " < > |` a Windows run choked on. No extension: the data-URI MIME + * comes from the requested URL ([toDataUri]), never from the cache file, and + * [SIDECAR_SUFFIX] is what keeps the directory greppable. + */ + internal fun cacheFileName(url: String): String { + val digest = MessageDigest.getInstance("SHA-256").digest(url.toByteArray(Charsets.UTF_8)) + val out = StringBuilder(digest.size * 2) + for (byte in digest) { + val code = byte.toInt() and 0xFF + out.append(HEX[code ushr 4]).append(HEX[code and 0x0F]) + } + return out.toString() } private fun toDataUri(url: String, bytes: ByteArray): String { @@ -236,3 +522,37 @@ object SefariaImageEmbedder { return "data:$mime;base64,$b64" } } + +/** One URL that could not be inlined, and why — see [SefariaImageEmbedder.failureReason]. */ +@kotlinx.serialization.Serializable +internal data class ImageEmbedFailure(val url: String, val reason: String) + +/** + * Machine-readable outcome of one [SefariaImageEmbedder.prefetch], written + * beside the DB output so the build can gate on it (see the "Generate Seforim + * Database" step in manual-generate-release.yml). Every entry in [failures] + * ships as a live `https://textimages.sefaria.org/…` `` — a broken + * image for an offline client — and an INFO count gated nothing: 10 of 420 + * went out that way in v27. + * + * - [attempted]: unique URLs found in the export. + * - [cached]: URLs with a data URI in hand, i.e. actually embeddable. + * - [downloaded]/[fromDiskCache]: how those were obtained. + * + * Shape is deterministic (declared key order; [failures] sorted by URL). + */ +@kotlinx.serialization.Serializable +internal data class ImageEmbedReport( + val attempted: Int, + val cached: Int, + val downloaded: Int, + val fromDiskCache: Int, + val failed: Int, + val failures: List, +) + +private val imageEmbedReportJson = Json { prettyPrint = true } + +/** JSON form written beside seforim.db for the workflow gate. */ +internal fun ImageEmbedReport.toJsonReport(): String = + imageEmbedReportJson.encodeToString(ImageEmbedReport.serializer(), this) diff --git a/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedderTest.kt b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedderTest.kt index 233f03b2..f7e6075b 100644 --- a/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedderTest.kt +++ b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaImageEmbedderTest.kt @@ -1,10 +1,19 @@ package io.github.kdroidfilter.seforimlibrary.sefariasqlite +import kotlinx.coroutines.runBlocking +import kotlinx.serialization.json.Json +import java.net.http.HttpTimeoutException +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths +import java.util.concurrent.atomic.AtomicInteger +import kotlin.io.path.writeText import kotlin.test.AfterTest import kotlin.test.BeforeTest import kotlin.test.Test import kotlin.test.assertEquals import kotlin.test.assertFalse +import kotlin.test.assertNotEquals import kotlin.test.assertTrue /** @@ -14,7 +23,406 @@ import kotlin.test.assertTrue */ class SefariaImageEmbedderTest { @BeforeTest fun reset() = SefariaImageEmbedder.resetForTest() - @AfterTest fun clean() = SefariaImageEmbedder.resetForTest() + @AfterTest fun clean() { + SefariaImageEmbedder.resetForTest() + System.clearProperty(SefariaImageEmbedder.CACHE_DIR_PROPERTY) + System.clearProperty(SefariaImageEmbedder.REPORT_PROPERTY) + } + + private fun mergedJsonWith(vararg urls: String): Pair { + val dir = Files.createTempDirectory("image-cache-test") + return dir to mergedJsonIn(dir, "merged.json", *urls) + } + + private fun mergedJsonIn(dir: Path, name: String, vararg urls: String): Path { + val json = dir.resolve(name) + val tags = urls.joinToString(" ") { """ text""" } + json.writeText("""{"text": ["$tags"]}""") + return json + } + + /** The cache file for [url] under the current (sha256) key. */ + private fun cacheFile(dir: Path, url: String): Path = + dir.resolve(SefariaImageEmbedder.cacheFileName(url)) + + /** Its `.url` sidecar. */ + private fun sidecarFile(dir: Path, url: String): Path = + dir.resolve(SefariaImageEmbedder.cacheFileName(url) + SefariaImageEmbedder.SIDECAR_SUFFIX) + + /** Writes a complete cache entry (bytes + sidecar) the way the embedder does. */ + private fun writeCacheEntry(dir: Path, url: String, bytes: ByteArray, sidecarUrl: String = url) { + Files.write(cacheFile(dir, url), bytes) + Files.writeString(sidecarFile(dir, url), sidecarUrl) + } + + /** Keep every test's report inside its own temp dir. */ + private fun reportPath(dir: Path): Path = dir.resolve("image-embed-report.json") + + private fun reportIn(dir: Path): ImageEmbedReport = + Json.decodeFromString(ImageEmbedReport.serializer(), Files.readString(reportPath(dir))) + + @Test + fun transientTimeoutIsRetriedThenCachedOnDisk() = runBlocking { + val url = "https://textimages.sefaria.org/retry/a.png" + val (dir, json) = mergedJsonWith(url) + var calls = 0 + SefariaImageEmbedder.retryBackoffMillis = longArrayOf(0, 0) + SefariaImageEmbedder.downloader = { + calls++ + if (calls < 3) throw HttpTimeoutException("request timed out") else byteArrayOf(1, 2, 3) + } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(3, calls, "two timeouts, then success") + assertTrue(Files.isRegularFile(cacheFile(dir, url)), "cached on disk") + assertEquals(url, Files.readString(sidecarFile(dir, url)), "the sidecar names the URL that wrote it") + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + } + + @Test + fun finalHttpStatusIsNotRetried() = runBlocking { + val url = "https://textimages.sefaria.org/blocked/b.png" + val (dir, json) = mergedJsonWith(url) + var calls = 0 + SefariaImageEmbedder.retryBackoffMillis = longArrayOf(0, 0) + SefariaImageEmbedder.downloader = { calls++; throw SefariaImageEmbedder.httpFailure(418, it) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls, "a content-filter 418 is final") + assertFalse(Files.exists(cacheFile(dir, url))) + assertFalse(Files.exists(sidecarFile(dir, url)), "no orphan sidecar when nothing downloaded") + val input = """""" + assertEquals(input, cleanSefariaLine(input), "remote URL kept when the download failed") + } + + @Test + fun exhaustedRetriesLeaveTheRemoteUrl() = runBlocking { + val url = "https://textimages.sefaria.org/flaky/c.png" + val (dir, json) = mergedJsonWith(url) + var calls = 0 + SefariaImageEmbedder.retryBackoffMillis = longArrayOf(0, 0) + SefariaImageEmbedder.downloader = { calls++; throw SefariaImageEmbedder.httpFailure(503, it) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(3, calls, "5xx is retried up to the attempt budget") + assertFalse(Files.exists(cacheFile(dir, url))) + assertEquals("http 503", reportIn(dir).failures.single().reason) + } + + @Test + fun cachedFileIsReusedWithoutAnyDownload() = runBlocking { + val url = "https://textimages.sefaria.org/warm/d.png" + val (dir, json) = mergedJsonWith(url) + writeCacheEntry(dir, url, byteArrayOf(1, 2, 3)) + SefariaImageEmbedder.downloader = { error("network must not be touched for a cached image") } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + // Disk hits are a counter, never a log line per file. + val report = reportIn(dir) + assertEquals(1, report.fromDiskCache) + assertEquals(0, report.downloaded) + } + + // --- cache key: two URLs, two files (audit claim 5) ------------------- + + /** `a/b.png` and `a_b.png` — one file under the pre-sha256 key. */ + private val collidingA = "https://textimages.sefaria.org/a/b.png" + private val collidingB = "https://textimages.sefaria.org/a_b.png" + private val legacyNameOfBoth = "a_b.png" + + /** The pre-sha256 name, reproduced here because production no longer has it. */ + private fun legacyName(url: String): String = + url.removePrefix("https://textimages.sefaria.org/") + .replace('/', '_').replace(':', '_').replace('?', '_') + + @Test + fun urlsThatCollidedUnderTheLegacyKeyGetTheirOwnFileAndBytes() = runBlocking { + assertEquals( + legacyName(collidingA), + legacyName(collidingB), + "premise: these two URLs shared one cache file before the sha256 key" + ) + val (dir, json) = mergedJsonWith(collidingA, collidingB) + SefariaImageEmbedder.downloader = { url -> + if (url == collidingA) byteArrayOf(1, 2, 3) else byteArrayOf(4, 5, 6) + } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(2, reportIn(dir).downloaded, "no hit may be served across the two URLs") + assertNotEquals(cacheFile(dir, collidingA), cacheFile(dir, collidingB), "distinct cache files") + assertEquals(collidingA, Files.readString(sidecarFile(dir, collidingA))) + assertEquals(collidingB, Files.readString(sidecarFile(dir, collidingB))) + // Each line gets ITS image, not whichever of the two was fetched first. + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,BAUG")) + } + + // --- the sidecar is the ONLY way into the cache ----------------------- + + @Test + fun aLoneLegacyNamedBlobIsNeverAdopted() = runBlocking { + // THE HAZARD, exactly: a past export wrote `a_b.png` — under the + // pre-sha256 name, which folded `/ : ?` to `_` and so was not injective + // — from `a_b.png` (U2). This export contains only `a/b.png` (U1), which + // maps to the SAME legacy name. Nothing on disk says whose bytes those + // are, so U1 must not inherit them: it downloads its own. + val (dir, json) = mergedJsonWith(collidingA) + assertEquals(legacyNameOfBoth, legacyName(collidingA), "premise: A's legacy name") + assertEquals(legacyNameOfBoth, legacyName(collidingB), "premise: B's legacy name, the same") + Files.write(dir.resolve(legacyNameOfBoth), byteArrayOf(4, 5, 6)) // U2's image + var calls = 0 + SefariaImageEmbedder.downloader = { calls++; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls, "a blob with no sidecar naming this URL is a miss") + val report = reportIn(dir) + assertEquals(1, report.downloaded) + assertEquals(0, report.fromDiskCache) + assertTrue( + cleanSefariaLine("""""").contains("data:image/png;base64,AQID"), + "U1 gets its OWN image, never U2's" + ) + assertEquals(collidingA, Files.readString(sidecarFile(dir, collidingA))) + assertEquals( + byteArrayOf(4, 5, 6).toList(), + Files.readAllBytes(dir.resolve(legacyNameOfBoth)).toList(), + "a file this build did not write is ignored where it lies, not moved and not deleted" + ) + } + + @Test + fun bothUrlsOfALegacyCollisionDownloadTheirOwnBytes() = runBlocking { + // Both members of the colliding pair in one export, the legacy file + // present: neither may claim it, and the two end up with distinct files. + val (dir, json) = mergedJsonWith(collidingA, collidingB) + Files.write(dir.resolve(legacyNameOfBoth), byteArrayOf(9, 9, 9)) + val calls = AtomicInteger() + SefariaImageEmbedder.downloader = { url -> + calls.incrementAndGet() + if (url == collidingA) byteArrayOf(1, 2, 3) else byteArrayOf(4, 5, 6) + } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(2, calls.get()) + assertEquals(0, reportIn(dir).fromDiskCache, "a legacy file is served to nobody") + assertEquals( + byteArrayOf(9, 9, 9).toList(), + Files.readAllBytes(dir.resolve(legacyNameOfBoth)).toList(), + "left untouched" + ) + assertNotEquals(cacheFile(dir, collidingA), cacheFile(dir, collidingB)) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,BAUG")) + } + + @Test + fun blobAtTheRightNameWithoutASidecarIsNotAHit() = runBlocking { + // Bytes sitting at sha256(url) with no `.url` beside them have no + // provenance either (a hand-copied cache dir, a crash between the two + // writes): the entry is evicted and refetched. + val url = "https://textimages.sefaria.org/warm/f.png" + val (dir, json) = mergedJsonWith(url) + Files.write(cacheFile(dir, url), byteArrayOf(9, 9, 9)) + var calls = 0 + SefariaImageEmbedder.downloader = { calls++; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls, "no sidecar, no hit") + assertEquals(0, reportIn(dir).fromDiskCache) + assertEquals(url, Files.readString(sidecarFile(dir, url)), "the refetch writes the sidecar") + assertEquals(byteArrayOf(1, 2, 3).toList(), Files.readAllBytes(cacheFile(dir, url)).toList()) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + } + + @Test + fun aMissDownloadsAndWritesBothSidecarAndBlob() = runBlocking { + val url = "https://textimages.sefaria.org/cold/g.png" + val (dir, json) = mergedJsonWith(url) + var calls = 0 + SefariaImageEmbedder.downloader = { calls++; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls) + assertTrue(Files.isRegularFile(cacheFile(dir, url)), "blob written under the digest") + assertEquals(url, Files.readString(sidecarFile(dir, url)), "sidecar names the URL byte-for-byte") + assertEquals(byteArrayOf(1, 2, 3).toList(), Files.readAllBytes(cacheFile(dir, url)).toList()) + // No `.part` temp survives a successful write. + assertTrue( + dir.toFile().listFiles().orEmpty().none { it.name.endsWith(".part") }, + "the atomic write leaves no temp behind" + ) + + // Second run over the same durable cache: the sidecar makes it a hit. + SefariaImageEmbedder.resetForTest() + SefariaImageEmbedder.downloader = { error("a verified entry must not re-download") } + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + assertEquals(1, reportIn(dir).fromDiskCache) + assertEquals(0, reportIn(dir).downloaded) + } + + @Test + fun entryWhoseSidecarNamesAnotherUrlIsEvictedAndRefetched() = runBlocking { + val url = "https://textimages.sefaria.org/warm/e.png" + val (dir, json) = mergedJsonWith(url) + writeCacheEntry(dir, url, byteArrayOf(9, 9, 9), sidecarUrl = "https://textimages.sefaria.org/other.png") + var calls = 0 + SefariaImageEmbedder.downloader = { calls++; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls, "a sidecar naming another URL is a miss") + val report = reportIn(dir) + assertEquals(1, report.downloaded) + assertEquals(0, report.fromDiskCache) + assertEquals(url, Files.readString(sidecarFile(dir, url)), "the entry now names its real owner") + assertEquals(byteArrayOf(1, 2, 3).toList(), Files.readAllBytes(cacheFile(dir, url)).toList()) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + } + + @Test + fun aSidecarThatIsNotTheUrlByteForByteIsNotAHit() = runBlocking { + // The hit rule is byte equality, not "looks like the URL": a sidecar + // written by anything but [writeEntry] — a shell redirect that appends + // a newline, an editor, a partially flushed write — has not proved the + // bytes came from this URL, so the entry is evicted and refetched. This + // also pins the writer: it must never append a terminator of its own. + val url = "https://textimages.sefaria.org/warm/h.png" + val (dir, json) = mergedJsonWith(url) + writeCacheEntry(dir, url, byteArrayOf(9, 9, 9), sidecarUrl = "$url\n") + var calls = 0 + SefariaImageEmbedder.downloader = { calls++; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(1, calls, "a sidecar that differs by one byte is a miss") + assertEquals(0, reportIn(dir).fromDiskCache) + assertEquals(url, Files.readString(sidecarFile(dir, url)), "rewritten with no trailing newline") + assertEquals(byteArrayOf(1, 2, 3).toList(), Files.readAllBytes(cacheFile(dir, url)).toList()) + } + + @Test + fun cacheDirComesFromTheSystemPropertyWhenSet() { + System.setProperty(SefariaImageEmbedder.CACHE_DIR_PROPERTY, "durable/textimages") + assertEquals(Paths.get("durable", "textimages"), SefariaImageEmbedder.defaultCacheDir()) + System.setProperty(SefariaImageEmbedder.CACHE_DIR_PROPERTY, " ") + assertEquals( + Paths.get("build", "sefaria", "image-cache"), + SefariaImageEmbedder.defaultCacheDir(), + "blank property falls back to the build dir" + ) + } + + @Test + fun reportPathComesFromTheSystemPropertyWhenSet() { + System.setProperty(SefariaImageEmbedder.REPORT_PROPERTY, "out/image-embed-report.json") + assertEquals(Paths.get("out", "image-embed-report.json"), SefariaImageEmbedder.defaultReportPath()) + System.clearProperty(SefariaImageEmbedder.REPORT_PROPERTY) + assertEquals( + Paths.get("build", "sefaria", "image-embed-report.json"), + SefariaImageEmbedder.defaultReportPath(), + "unset property falls back to the build dir" + ) + } + + @Test + fun spacedUrlIsScannedAndSubstitutedWithTheSameKey() = runBlocking { + // The scan used to stop at whitespace and cache ".../Screenshot", a key + // substituteImages could never look up — so this image stayed remote in + // every build even when the bytes arrived. + val url = "https://textimages.sefaria.org/ShaarHahakdamot/Screenshot 2023-05-04 at 3.09.20 PM.png" + val (dir, json) = mergedJsonWith(url) + val requested = mutableListOf() + SefariaImageEmbedder.downloader = { requested += it; byteArrayOf(1, 2, 3) } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + assertEquals(listOf(url), requested, "the whole spaced URL is the cache key") + assertEquals(0, reportIn(dir).failed) + assertTrue(cleanSefariaLine("""""").contains("data:image/png;base64,AQID")) + } + + @Test + fun onlyTheRequestIsPercentEncoded() { + assertEquals( + "https://textimages.sefaria.org/a%20b/c.png", + SefariaImageEmbedder.encodeForRequest("https://textimages.sefaria.org/a b/c.png") + ) + // Sefaria's own encoded paths must not be encoded a second time. + val encoded = "https://textimages.sefaria.org/shalach_teshalach/image%20-%200019.png" + assertEquals(encoded, SefariaImageEmbedder.encodeForRequest(encoded)) + assertEquals( + "https://textimages.sefaria.org/%D7%90.png", + SefariaImageEmbedder.encodeForRequest("https://textimages.sefaria.org/א.png") + ) + } + + @Test + fun everyFailureIsNamedInTheReport() = runBlocking { + val blocked = "https://textimages.sefaria.org/ShaarHahakdamot/blocked.png" + val gone = "https://textimages.sefaria.org/gone/x.png" + val slow = "https://textimages.sefaria.org/slow/z.png" + val ok = "https://textimages.sefaria.org/ok/y.png" + val (dir, json) = mergedJsonWith(blocked, gone, slow, ok) + SefariaImageEmbedder.retryBackoffMillis = longArrayOf(0, 0) + SefariaImageEmbedder.downloader = { url -> + when (url) { + blocked -> throw SefariaImageEmbedder.httpFailure(418, url) + gone -> throw SefariaImageEmbedder.httpFailure(404, url) + slow -> throw HttpTimeoutException("request timed out") + else -> byteArrayOf(1, 2, 3) + } + } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + val report = reportIn(dir) + assertEquals(4, report.attempted) + assertEquals(1, report.cached) + assertEquals(1, report.downloaded) + assertEquals(3, report.failed) + assertEquals(listOf(blocked, gone, slow), report.failures.map { it.url }, "sorted by URL") + assertEquals( + "content-filter (HTTP 418 — NetFree on the self-hosted runner)", + report.failures[0].reason + ) + assertEquals("http 404", report.failures[1].reason) + assertEquals("timeout", report.failures[2].reason) + } + + @Test + fun countersAreExactUnderConcurrentDownloads() = runBlocking { + // The counters are written from DOWNLOAD_PARALLELISM coroutines and now + // gate the build, so a lost `Int++` is a wrong gate, not a cosmetic log. + val urls = (1..120).map { "https://textimages.sefaria.org/concurrent/$it.png" } + val (dir, json) = mergedJsonWith(*urls.toTypedArray()) + val calls = AtomicInteger() + SefariaImageEmbedder.downloader = { url -> + calls.incrementAndGet() + val n = url.substringAfterLast('/').removeSuffix(".png").toInt() + if (n % 2 == 0) byteArrayOf(1, 2, 3) else throw SefariaImageEmbedder.httpFailure(404, url) + } + + SefariaImageEmbedder.prefetch(listOf(json), cacheDir = dir, reportPath = reportPath(dir)) + + val report = reportIn(dir) + assertEquals(120, calls.get(), "a 404 is final, so one call per URL") + assertEquals(120, report.attempted) + assertEquals(60, report.downloaded) + assertEquals(0, report.fromDiskCache) + assertEquals(60, report.cached) + assertEquals(60, report.failed) + assertEquals(60, report.failures.size) + } @Test fun substituteImagesReplacesKnownUrlsInImgTags() { From 1e35d5767be9fbe4a32271c84d04ef9a37ecf3bd Mon Sep 17 00:00:00 2001 From: ypl <7353755@gmail.com> Date: Wed, 9 Sep 2026 12:13:33 +0300 Subject: [PATCH 2/7] fix(generator): one DEBUG line per book with throttled progress and phase summaries; dropped data summarised and named; fail closed on seed, snapshot and buildstate; atomic candidate publish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed from the audit branch by file set (4 original commits contributed; their messages follow). --- ace5f88 fix(generator): one DEBUG line per book, a throttled progress line, and a phase summary instead of 17,800 INFO lines Audit of cycle 33987355439 (item S4a, report 06-db-generator-step). The "Generate Seforim Database" step logged 24,207 lines / 2.8 MB for ~1,500 books, 94.8% of it boilerplate: a 6-line INFO block per book, the book header printed twice from two call sites, a line tick at lineIndex == 0 for every book (`0/N (0%)`), and `Books progress: n/N` after every single book. Generator.kt (logging only; the serial book-insert loop, its order, id allocation and every insert call are untouched — verified by diffing the control-flow skeleton with all logger calls stripped): - the per-book INFO block is one DEBUG line (id, title, file, category, lines, toc, acronyms, ms); processBookContent/processLinesWithTocEntries return BookContentStats; a logger.e + rethrow keeps the "which book blew up" breadcrumb for content failures. - the three `Books progress` sites fold into one throttled INFO line every 100 books or 60 s: `books 1234/1501 (82.2%) · lines 5.2M · toc 301.0K · elapsed 27:08 · eta ~05:52` (15 lines on the real run). - line ticks need lineIndex > 0, a book above 5,000 lines and ≥10% or 60 s since the last tick (3,239 -> 507 on the real book sizes; the old rule replayed over them reproduces exactly the 3,239 observed). - the duplicated `📚 Processing book` site in processDirectory is gone; the createAndProcessBook site (which also serves the priority-list caller and carries the categoryId) stays. - acronym inserts, `===` directory lines, `✅ Category` and the `📝` merge lines are DEBUG; their counts land in one summary per phase: `Otzaria book import (phase 1): books=…/… lines=… tocEntries=… directories=… skipped=… acronymBooks=… hearotMerged=… elapsed=mm:ss` (counters reset per phase — QA fix). GeneratorProgress.kt: clock-injected cadence gates and locale-free formatters; GeneratorProgressTest.kt: 18 tests on a fake clock. DEBUG is off in CI (Logger.setMinSeverity(Info) on every CLI entry; the audited log has 0 Debug lines), so the demotion is real. No consumer parses any removed marker (repo, pipeline-monitor, otzaria-library, LinkerToOtzaria checked); `build_state.db snapshot written`, `Persisting in-memory DB`, SefariaDirect `Processed N/N books` and `BUILD SUCCESSFUL` are byte-identical. Measured on build-job-34024655297.log: -17,838 lines (-73.7%) and -1.98 MB on the step. Left for S4b: the 1,286 `Target book not found` / `Original path` lines (their fix is a WARN summary for the 643 dropped manual links) and the insert-path failure breadcrumb. Tests: :otzariasqlite:jvmTest --rerun 36/36. --- f2079c5 fix(generator): summarise dropped data instead of spamming it; make the summaries agree; name what was silently skipped Audit of cycle 33987355439 (item S4b, report 06-db-generator-step). Logging, counting and report-file changes only: the serial book-insert loop, id allocation, SQL and every row written are unchanged (verified by a skeleton diff with all logger calls stripped, 19 hunks in Generator.kt, each a counter / set insertion / report call / currentBook assignment / log-and-rethrow / the _headings filter). - 37 Talmud `_headings` JSON files (heading -> line maps) were fed to the link parser: 74 parse attempts (each tried twice), a 40-line JSON dump per failure, then `Source book not found for links` x37. They are excluded from link-file discovery with one INFO line. Proven no-op for the data: parseLinksFromJson returned emptyList() for them, so zero rows and zero ids ever came from them. - Priority list 430/431 missing (Generator.kt list never re-pinned after the directory restructure): 430 WARNs -> one WARN with counts, the first paths and a pointer; full list in the report file. The list is NOT re-pinned or deleted (changes insert order -> ids; operator decision). - 642 x `Target book not found: עולת ראיה` + `Original path` -> one WARN per missing target with count and path (bounded), plus a total. Links are still dropped exactly as before (otzaria-library O7 fixes the data). - Contradictory summaries: `type=REFERENCE resolvedPairs=4421 written=106912` was a labelling problem — the read counters are keyed by the CSV's `Conection Type`, `written` by the stored type after blank inference. Labels are now csvRowsRead/csvDropped/csvResolvedPairs/ storedWritten with a reconciling totals line (3,899,477 resolved vs 3,886,967 written; gap 12,510 = heading/self-link filters + INSERT OR IGNORE). `GenerationApplyResult(…, 0)` hard-coded unmatched=0 next to "13 unmatched" — fixed. All-metadata's 1,116 unmatched titles are named (bounded WARN + report file) for otzaria-library's O6. The JSON metrics consumed by the QA scripts are byte-identical. - build_state.db written 5x: all five writers traced to consumers (four feed the next stage's InMemoryIdAllocator.load, the fifth is the published seforim.db.buildstate.zst); none dropped; each write now prints its seconds. - 407 books without a source hash: Sefaria's hash computer walks merged.json before the blacklist filter, so 395 blacklisted books contribute hashes no id claims; Otzaria's 16 are classified. Named per class; recording behaviour unchanged (operator decision). - Havrouta 38 found / 37 processed: the unmatched tractate is named with the reason. line_ref's 90 ambiguous keys are named (bounded) and written to the report file. - The insert-path failure breadcrumb removed by S4a is restored via a currentBook field reported by the outermost catch (no re-indent, no control-flow change). GenerateLinks.kt logged the literal `${tables.size}` — fixed. - New common/reports/GeneratorReport.kt writes the bounded lists to /build/generator-reports (pinned in root build.gradle.kts because each generator stage forks with its own subproject cwd). Nothing collects that directory yet: the workflow contract forbids Actions artifact storage (every handoff is a content-addressed pre-release), so wiring it is an operator decision. Measured on build-job-34024655297.log: about -4,750 lines / -0.40 MB (the 40-line exception blocks were the audit's "blank stack-frame" lines), +~19 summary lines. Tests 495 -> 527 with the same 9 pre-existing Windows path failures; compiler warnings 8 -> 8; workflow contract tests 42 -> 44 (the workflow itself is untouched). --- f4d5c01 fix(generator): fail closed when the seed copy or the buildstate snapshot fails; verify the written state GenerateLinks and GenerateLines wrapped the seed (ATTACH the base DB and copy every table) in runCatching and continued on failure; in appendOtzariaLinks the base DB and the persist target are the same file, so a failed seed let the later VACUUM INTO replace the 7 GiB DB with an empty one and the build reported success. Both seeds now log the base DB and the table the copy died on and rethrow, before any VACUUM INTO; the pre-existing target is left byte-identical. The five snapshotTo sites (Sefaria, metadata seed, lines, links, havrouta) swallowed a failed buildstate write with a warning, so a DB could ship with an allocator state the next build would reuse to hand out ids twice. All five now rethrow, and GenerateLinks snapshots after its persist so a failed persist never advances the state. After every snapshot, BuildStateVerifier reopens only the meta and id_counters tables, checks the stamp is this run's, and asserts next_id > MAX(id) for each allocator-issued table (one b-tree edge per table, ~20 ms on a 5 GB DB); alt_toc_entry is excluded with the evidence that its ids are implicit rowids by design, which real generator output confirmed. --- 691b8a9 fix(generator): a missing required seed fails before the target is touched; publish through an atomic candidate Round 3 of the audit of cycle 33987355439 (item S16). GenerateLinks and GenerateLines warned and continued with an empty database when the seed was absent while appendExistingDb was set, and the release tasks pass one path as both base and target, so the reachable outcome was an empty DB shipped as a success. Now: appendExistingDb with no seed throws before any write to the target, naming the path and the opt-in that permits an empty base (-PallowEmptyBase, forwarded by the four seed-using JavaExec tasks because a Gradle -P is not a system property of the forked JVM). Both generators publish through DbPublish: the result is written to .candidate in the same directory, verified, and moved over the target with ATOMIC_MOVE + REPLACE_EXISTING; stale -journal/-wal/-shm beside the candidate and the target are removed around the move. The publish → snapshot → BuildStateVerifier order from f4d5c01 is unchanged. Tests: OtzariaBuildFailClosedTest 13/13 (throw before the target is modified, no candidate left behind, stale WAL does not survive the publish, the release tasks forward the opt-in and pass one path as base and target). otzariasqlite 57/57. Co-Authored-By: Claude Fable 5.1 --- build.gradle.kts | 72 +++ generator/common/build.gradle.kts | 8 +- .../common/buildstate/BuildStateVerifier.kt | 196 ++++++++ .../common/buildstate/BuildStateWriter.kt | 18 +- .../common/patch/PatchPipelineCli.kt | 25 +- .../common/refs/BuildLineRefIndexCli.kt | 65 ++- .../common/reports/GeneratorReport.kt | 143 ++++++ .../buildstate/BuildStateVerifierTest.kt | 226 +++++++++ .../buildstate/BuildStateWriteTimingTest.kt | 79 +++ .../common/refs/BuildLineRefIndexCliTest.kt | 17 + .../common/reports/GeneratorReportTest.kt | 192 ++++++++ generator/otzariasqlite/build.gradle.kts | 19 + .../seforimlibrary/otzariasqlite/Generator.kt | 419 ++++++++++++++-- .../otzariasqlite/GeneratorProgress.kt | 223 +++++++++ .../seforimlibrary/otzariasqlite/DbPublish.kt | 187 +++++++ .../otzariasqlite/GenerateHavroutaLinks.kt | 61 ++- .../otzariasqlite/GenerateLines.kt | 105 +++- .../otzariasqlite/GenerateLinks.kt | 81 ++- .../otzariasqlite/GeneratorProgressTest.kt | 289 +++++++++++ .../HavroutaUnmatchedTractateTest.kt | 96 ++++ .../OtzariaBuildFailClosedTest.kt | 466 ++++++++++++++++++ .../OtzariaGeneratorDiagnosticsTest.kt | 349 +++++++++++++ .../sefariasqlite/GenerateSefariaSqlite.kt | 32 +- .../SeedAllMetadataPostProcess.kt | 78 ++- .../SeedGenerationsPostProcess.kt | 7 +- .../sefariasqlite/SefariaBlacklists.kt | 16 +- .../sefariasqlite/SefariaDirectImporter.kt | 83 ++++ .../sefariasqlite/SefariaLinksImporter.kt | 59 ++- .../SeedAllMetadataSourceTest.kt | 30 ++ .../SefariaGeneratorDiagnosticsTest.kt | 294 +++++++++++ 30 files changed, 3787 insertions(+), 148 deletions(-) create mode 100644 generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifier.kt create mode 100644 generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReport.kt create mode 100644 generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifierTest.kt create mode 100644 generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriteTimingTest.kt create mode 100644 generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReportTest.kt create mode 100644 generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgress.kt create mode 100644 generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/DbPublish.kt create mode 100644 generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgressTest.kt create mode 100644 generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/HavroutaUnmatchedTractateTest.kt create mode 100644 generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaBuildFailClosedTest.kt create mode 100644 generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaGeneratorDiagnosticsTest.kt create mode 100644 generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaGeneratorDiagnosticsTest.kt diff --git a/build.gradle.kts b/build.gradle.kts index 7511e7fa..b3d81775 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -7,6 +7,42 @@ plugins { alias(libs.plugins.android.application).apply(false) } +// ─── JDK 25: pre-approve the native access sqlite-jdbc already performs ──── +// Every JVM that opens a seforim.db makes sqlite-jdbc call System::load, and +// JDK 25 answers on stderr with +// WARNING: A restricted method in java.lang.System has been called +// WARNING: java.lang.System::load has been called by org.sqlite.SQLiteJDBCLoader … +// WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning … +// WARNING: Restricted methods will be blocked in a future release … +// Run 34024655297 printed that four-line block 15 times — 60 log lines — from +// "Generate Seforim Database" (×9), "Dump lines snapshot for the linker", +// "Apply LINKER links (Phase-2)" and "Produce + verify patch fan" (×4). The +// flag grants exactly the access the code already takes: no bytecode, no +// behaviour and no artifact change. The last line is also the reason not to +// leave it: the same call becomes an error in a future JDK. +// +// Added as an argument PROVIDER rather than by appending to `jvmArgs`: ~25 +// JavaExec tasks assign `jvmArgs = listOf(…)` in their own configuration +// blocks (manual-generate-release.yml greps two of those lines verbatim), and +// an assignment discards anything appended here. jvmArgumentProviders is a +// separate list that no assignment can clear. +// +// The patch fan does not fork through Gradle at all — it runs PatchPipelineCli +// with `java` from generator/common's published launcher spec — so its copy of +// the flag lives in `patchPipelineJvmArgs` there. +class EnableNativeAccess : org.gradle.process.CommandLineArgumentProvider { + override fun asArguments(): Iterable = listOf("--enable-native-access=ALL-UNNAMED") +} + +allprojects { + tasks.withType().configureEach { + jvmArgumentProviders.add(EnableNativeAccess()) + } + tasks.withType().configureEach { + jvmArgumentProviders.add(EnableNativeAccess()) + } +} + tasks.register("generateSeforimDb") { group = "application" description = "Generate build/seforim.db from Sefaria, append Otzaria, and release info." @@ -41,6 +77,42 @@ project(":generator-common").tasks.matching { it.name == "stampSchemaVersion" }. mustRunAfter(":generator-common:buildLineDhIndex") } +// Generator diagnostics side-channel (see GeneratorReport). Findings that are +// too long for the build log — the missing priority entries, the metadata +// records that matched no book, the books with no source hash, the ambiguous +// line_ref keys — log one bounded summary line and write the full list here. +// +// Every generator stage is a forked JavaExec whose working directory is its OWN +// subproject, so GeneratorReport's relative default would scatter the files +// across generator/*/build/generator-reports and none of them would be under +// the root build/ that the workflow's tmpfs and its release staging address. +// Pin all writers to one absolute directory in the ROOT build dir. +// -PgeneratorReportDir overrides it. +// +// NOTE: nothing collects this directory off the runner today — build/ is a +// tmpfs the job unmounts, and manual-generate-release.yml deliberately has no +// Actions artifact upload to hook into (see +// test_weekly_workflow_has_no_actions_artifact_handoffs and +// test_generator_reports_are_not_collected_off_the_runner_yet). Every finding's +// counts and its first names are in the build log regardless; only the tail of +// each list dies with the run. Publishing them is an operator decision. +val generatorReportDir: String = + (findProperty("generatorReportDir") as String?) + ?: layout.buildDirectory.dir("generator-reports").get().asFile.absolutePath +listOf( + ":sefariasqlite" to "generateSefariaSqlite", + ":sefariasqlite" to "seedAllMetadata", + ":otzariasqlite" to "generateLines", + ":otzariasqlite" to "generateLinks", + ":otzariasqlite" to "appendOtzariaLines", + ":otzariasqlite" to "appendOtzariaLinks", + ":generator-common" to "buildLineRefIndex", +).forEach { (projectPath, taskName) -> + project(projectPath).tasks.matching { it.name == taskName }.configureEach { + (this as JavaExec).systemProperty("generatorReportDir", generatorReportDir) + } +} + // line_ref is derived from line.heRef + book.title, so it must be rebuilt // after every stage that writes or renames books and lines. project(":generator-common").tasks.matching { it.name == "buildLineRefIndex" }.configureEach { diff --git a/generator/common/build.gradle.kts b/generator/common/build.gradle.kts index 3b7a3ca6..ab645890 100644 --- a/generator/common/build.gradle.kts +++ b/generator/common/build.gradle.kts @@ -50,7 +50,13 @@ kotlin { // publishes them verbatim, so the direct invocation cannot drift from the task // it stands in for. val patchPipelineMainClass = "io.github.kdroidfilter.seforimlibrary.common.patch.PatchPipelineCliKt" -val patchPipelineJvmArgs = listOf("-Xmx$generatorHeap", "-XX:+UseG1GC") +// --enable-native-access: the fan runs this CLI through `java` directly (see +// the launcher below), so it never passes through the root build's JavaExec +// argument provider; without the flag every fan fork re-prints JDK 25's +// four-line sqlite-jdbc restricted-method warning (16 log lines across the 4 +// anchors of run 34024655297). It changes no bytecode and no output. +val patchPipelineJvmArgs = + listOf("-Xmx$generatorHeap", "-XX:+UseG1GC", "--enable-native-access=ALL-UNNAMED") // A function, not a val: like every other JavaExec here it must resolve inside // a task's configuration block, never at script-evaluation time. fun patchPipelineClasspath() = files(tasks.named("jvmJar")) + configurations.getByName("jvmRuntimeClasspath") diff --git a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifier.kt b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifier.kt new file mode 100644 index 00000000..fb71b9ef --- /dev/null +++ b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifier.kt @@ -0,0 +1,196 @@ +package io.github.kdroidfilter.seforimlibrary.common.buildstate + +import co.touchlab.kermit.Logger +import java.nio.file.Files +import java.nio.file.Path +import java.sql.Connection +import java.sql.DriverManager + +/** + * Post-write self-check of `build_state.db`, run by every generator stage right + * after its `snapshotTo`. + * + * It answers two questions the stages could not answer before: + * + * 1. **Is the file on disk the one this stage just wrote?** A failed snapshot + * used to leave the PREVIOUS release's state in place, and nothing + * distinguishes it from a fresh one — same name, plausible size, valid + * SQLite. [verifyFreshSnapshot] matches the `meta` rows this stage passed as + * `extraMeta` (`generator` + `generated_at`, unique per run) against what the + * file actually holds. + * 2. **Are its counters ahead of the DB it describes?** `id_counters.next_id` is + * the id the NEXT build hands to a freshly-discovered natural key. If the DB + * already holds that id, the next build re-issues a published id to a + * different row — the delta/patch fan then changes the meaning of rows rather + * than their content. + * + * Cost: deliberately NOT [BuildStateReader], which materialises the full + * `id_line` / `id_link` maps (millions of rows, ~990 MB file). This reads only + * `meta` and `id_counters` from the snapshot, plus one `MAX(id)` per allocator + * table from the DB — every one of those `id` columns is an INTEGER PRIMARY KEY, + * i.e. the rowid, so SQLite answers from the b-tree edge without scanning. + * Milliseconds even on the 7 GiB DB. + */ +object BuildStateVerifier { + + /** + * Allocator tables whose row ids the generator deliberately does NOT take + * from the allocator, so `next_id > MAX(id)` does not hold for them and must + * not be asserted. + * + * `alt_toc_entry` is the only one: both builders insert its rows with an + * implicit rowid — the Sefaria one via `repository.insertAltTocEntry(...)` + * (`SefariaAltTocBuilder`, six call sites, all with `id = 0`) and the Otzaria + * one at `Generator.kt` ("Entry ids stay auto-allocated, matching the Sefaria + * builder's Phase 1.5 deferral"). `IdAllocatorBindings.insertAltTocEntryStable` + * exists but has no call site, so `IdTable.ALT_TOC_ENTRY`'s counter never + * leaves 1 while the shipped DB holds tens of thousands of rows: measured on + * a real `generateSefariaSqlite` output, `next_id=1` against `MAX(id)=71602`. + * Asserting here would fail every stage of every release. + * + * Delete the entry the moment those inserts start going through the allocator. + */ + private val NOT_ALLOCATOR_ISSUED: Set = setOf(IdTable.ALT_TOC_ENTRY) + + /** The cheap half of [BuildStateSnapshot]: `meta` + `id_counters`, nothing else. */ + data class Header(val meta: Map, val counters: Map) + + /** Reads [Header] from a build_state.db without touching the id maps. */ + fun readHeader(path: Path): Header { + Class.forName("org.sqlite.JDBC") + DriverManager.getConnection("jdbc:sqlite:${path.toAbsolutePath()}").use { conn -> + conn.autoCommit = true + val meta = HashMap() + if (tableExists(conn, "meta")) { + conn.createStatement().use { st -> + st.executeQuery("SELECT key, value FROM meta").use { rs -> + while (rs.next()) meta[rs.getString(1)] = rs.getString(2) + } + } + } + val counters = HashMap() + if (tableExists(conn, "id_counters")) { + conn.createStatement().use { st -> + st.executeQuery("SELECT table_name, next_id FROM id_counters").use { rs -> + while (rs.next()) { + val table = IdTable.fromTableName(rs.getString(1)) ?: continue + counters[table] = rs.getLong(2) + } + } + } + } + return Header(meta, counters) + } + } + + /** + * Throws [IllegalStateException] unless the build_state at [buildStatePath] is + * the one this stage just wrote ([expectedMeta], typically the very map passed + * to `snapshotTo`) and its counters are ahead of every id in [dbPath]. + * + * Tables absent from [dbPath] or without an `id` column are skipped and named + * in the log; in practice `SeforimDb.Schema.create` gives every stage all 15, + * so this only fires for hand-built fixtures. Tables in [NOT_ALLOCATOR_ISSUED] + * are skipped for a different reason and reported separately. + */ + fun verifyFreshSnapshot( + buildStatePath: Path, + dbPath: Path, + expectedMeta: Map, + logger: Logger = Logger.withTag("BuildStateVerifier"), + ) { + check(Files.exists(buildStatePath)) { + "build_state was reported written but $buildStatePath does not exist" + } + val header = readHeader(buildStatePath) + + val schemaVersion = header.meta["schema_version"]?.toIntOrNull() + checkNotNull(schemaVersion) { + "build_state at $buildStatePath has no usable meta.schema_version " + + "(got '${header.meta["schema_version"]}') — it is not a snapshot this build wrote" + } + check(schemaVersion <= BuildStateSchema.CURRENT_VERSION) { + "build_state at $buildStatePath has schema_version=$schemaVersion, " + + "newer than supported ${BuildStateSchema.CURRENT_VERSION}" + } + for ((key, expected) in expectedMeta) { + val actual = header.meta[key] + check(actual == expected) { + "build_state at $buildStatePath is not the snapshot this stage wrote: " + + "meta.$key='$actual', expected '$expected'. A previous build's state was " + + "left in place — publishing it would re-issue ids already handed out." + } + } + check(header.counters.isNotEmpty()) { + "build_state at $buildStatePath carries no id_counters rows" + } + + check(Files.exists(dbPath)) { + "build_state at $buildStatePath cannot be verified: the DB it describes, $dbPath, does not exist" + } + + val violations = ArrayList() + val skipped = ArrayList() + val notIssued = ArrayList() + var checked = 0 + Class.forName("org.sqlite.JDBC") + DriverManager.getConnection("jdbc:sqlite:${dbPath.toAbsolutePath()}").use { conn -> + conn.autoCommit = true + for (table in IdTable.values()) { + if (table in NOT_ALLOCATOR_ISSUED) { + notIssued += table.tableName + continue + } + if (!tableExists(conn, table.tableName) || !hasIdColumn(conn, table.tableName)) { + skipped += table.tableName + continue + } + val maxId = maxId(conn, table.tableName) + val nextId = header.counters[table] + if (nextId == null) { + violations += "${table.tableName}: no id_counters row, but the DB holds MAX(id)=$maxId" + continue + } + // next_id is handed to the next fresh key, so it must be strictly + // above everything the DB already holds. + if (nextId <= maxId) { + violations += "${table.tableName}: next_id=$nextId <= MAX(id)=$maxId" + } + checked++ + } + } + check(violations.isEmpty()) { + "build_state at $buildStatePath is behind the DB at $dbPath — the next build would " + + "re-issue ids this build already published: ${violations.joinToString("; ")}" + } + logger.i { + "build_state verified against $dbPath: $checked counters ahead of the DB" + + (if (skipped.isEmpty()) "" else " (absent here: ${skipped.joinToString()})") + + (if (notIssued.isEmpty()) "" else " (ids not allocator-issued: ${notIssued.joinToString()})") + } + } + + private fun maxId(conn: Connection, table: String): Long = + conn.createStatement().use { st -> + st.executeQuery("SELECT COALESCE(MAX(id), 0) FROM \"$table\"").use { rs -> + if (rs.next()) rs.getLong(1) else 0L + } + } + + private fun hasIdColumn(conn: Connection, table: String): Boolean = + conn.createStatement().use { st -> + st.executeQuery("PRAGMA table_info(\"$table\")").use { rs -> + while (rs.next()) if (rs.getString("name") == "id") return true + } + false + } + + private fun tableExists(conn: Connection, name: String): Boolean { + conn.prepareStatement( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", + ).use { ps -> + ps.setString(1, name) + ps.executeQuery().use { rs -> return rs.next() } + } + } +} diff --git a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriter.kt b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriter.kt index 9954ed01..1269a0c7 100644 --- a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriter.kt +++ b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriter.kt @@ -15,6 +15,7 @@ import java.sql.DriverManager class BuildStateWriter(private val logger: Logger = Logger.withTag("BuildStateWriter")) { fun write(snapshot: BuildStateSnapshot, target: Path) { + val startedAtNanos = System.nanoTime() Files.createDirectories(target.toAbsolutePath().parent) val tmp = target.resolveSibling("${target.fileName}.tmp") if (Files.exists(tmp)) Files.delete(tmp) @@ -36,13 +37,28 @@ class BuildStateWriter(private val logger: Logger = Logger.withTag("BuildStateWr java.nio.file.StandardCopyOption.REPLACE_EXISTING, java.nio.file.StandardCopyOption.ATOMIC_MOVE, ) + // The elapsed time goes BEFORE the counter list on purpose: the pipeline + // monitor's phase splitter anchors this line on `links=)` at + // end-of-line (pipeline-monitor/generate_phases.sh, marker + // `buildstate_write_2`), so nothing may be appended after it. + // Five of these run per build (one per generator stage, each in its own + // JVM, each seeding the next stage's IdAllocator) and together they were + // 546 s / 26% of the step — a cost that was previously invisible because + // the write reported no duration at all. + val elapsedSeconds = elapsedSecondsSince(startedAtNanos) logger.i { - "build_state.db snapshot written to $target (" + + "build_state.db snapshot written to $target in ${elapsedSeconds}s (" + "books=${snapshot.books.size}, lines=${snapshot.lines.size}, " + "tocEntries=${snapshot.tocEntries.size}, links=${snapshot.links.size})" } } + /** `113.2` — one decimal, integer arithmetic so the runner's locale cannot change it. */ + private fun elapsedSecondsSince(startedAtNanos: Long): String { + val tenths = (System.nanoTime() - startedAtNanos) / 100_000_000L + return "${tenths / 10}.${tenths % 10}" + } + private fun applyDdl(conn: Connection) { conn.createStatement().use { st -> BuildStateSchema.statements.forEach { st.executeUpdate(it) } diff --git a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/patch/PatchPipelineCli.kt b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/patch/PatchPipelineCli.kt index 24f13a4a..73d25fd5 100644 --- a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/patch/PatchPipelineCli.kt +++ b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/patch/PatchPipelineCli.kt @@ -138,8 +138,19 @@ fun main(args: Array) { // it out client-side. Without this the manifest claims a catalogBlobName // but the patch.db ships with an empty blobs table — caught by the real // e2e on Zayit (catalog.pb timestamp stayed at v1). - val catalogPath = System.getProperty("catalogPb") - ?: System.getenv("CATALOG_PB_PATH") + // + // On THIS branch catalog.pb is no longer produced at all (see BRANCH-STACK + // "ביטול catalog.pb"; the release workflow even `rm -f build/catalog.pb`s any + // stale copy a dirty self-hosted workspace might smuggle in), and the + // manifest below correctly omits catalogBlobName when nothing was embedded — + // so a consumer never looks for a blob that is not there. The default, + // nobody-asked-for-a-catalog case is therefore expected and is reported as + // INFO; it was a WARN, which meant every cycle shipped 5 standing warnings + // (one per anchor) that no one could ever act on. Only an EXPLICIT request + // (-DcatalogPb / CATALOG_PB_PATH) pointing at a missing file is a real gap + // and keeps its warning. + val requestedCatalogPath = System.getProperty("catalogPb") ?: System.getenv("CATALOG_PB_PATH") + val catalogPath = requestedCatalogPath ?: outPath.resolveSibling("catalog.pb").toAbsolutePath().toString() val catalogFile = Paths.get(catalogPath) val catalogEmbedded = Files.isRegularFile(catalogFile) @@ -152,8 +163,16 @@ fun main(args: Array) { } } logger.i { "Embedded catalog.pb (${Files.size(catalogFile)} bytes) into patch.blobs" } + } else if (requestedCatalogPath != null) { + logger.w { + "No catalog.pb at the explicitly requested $catalogPath — patch ships without a " + + "catalog blob and its manifest omits catalogBlobName" + } } else { - logger.w { "No catalog.pb at $catalogPath — patch ships without a catalog blob" } + logger.i { + "no catalog.pb produced by this pipeline — patch ships without a catalog blob and " + + "its manifest omits catalogBlobName, which is the supported shape" + } } // Compress the patch with zstd. The .db file remains around so diff --git a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCli.kt b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCli.kt index f49a46c8..ee7803c1 100644 --- a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCli.kt +++ b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCli.kt @@ -2,6 +2,7 @@ package io.github.kdroidfilter.seforimlibrary.common.refs import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport import io.github.kdroidfilter.seforimlibrary.core.refs.RefKey import java.nio.file.Files import java.nio.file.Paths @@ -41,16 +42,54 @@ fun main() { } } if (report.ambiguousKeys > 0) { - logger.w { "line_ref: ${report.ambiguousKeys} keys resolving to more than one line" } + // "90 keys" alone is unactionable: nothing said WHICH refs collide, + // so nobody could look at the data. The colliding refs are named + // here (bounded) and listed in full in the report file. + logger.w { + "line_ref: ${report.ambiguousKeys} keys resolving to more than one line, e.g. " + + report.ambiguous.take(MAX_REPORTED_AMBIGUOUS).joinToString { + "'${it.bookTitle}' ${it.heRef} (lines ${it.firstLineIndex}, ${it.secondLineIndex})" + } + } + GeneratorReport.write("line-ref-ambiguous-keys", logger) { + put("ambiguousKeys", report.ambiguousKeys.toLong()) + put("indexedKeys", report.indexed.toLong()) + put("books", report.books.toLong()) + putRows( + "ambiguous", + report.ambiguous.map { + mapOf( + "bookId" to it.bookId, + "bookTitle" to it.bookTitle, + "heRef" to it.heRef, + "firstLineIndex" to it.firstLineIndex, + "secondLineIndex" to it.secondLineIndex, + ) + }, + ) + } } } } +/** How many colliding refs the single WARN line names before deferring to the report file. */ +private const val MAX_REPORTED_AMBIGUOUS = 20 + +/** One ref key that two or more lines of the same book resolve to. */ +internal data class AmbiguousLineRef( + val bookId: Long, + val bookTitle: String, + val heRef: String, + val firstLineIndex: Long, + val secondLineIndex: Long, +) + internal data class LineRefIndexReport( val books: Int, val indexed: Int, val ambiguousKeys: Int, val titleMismatchBooks: List, + val ambiguous: List = emptyList(), ) internal fun rebuildLineRefIndex(conn: Connection, logger: Logger): LineRefIndexReport { @@ -84,6 +123,9 @@ internal fun indexAllBooks(conn: Connection, logger: Logger): LineRefIndexReport var indexed = 0 var ambiguous = 0 val mismatched = ArrayList() + // Named collisions, in discovery order: the first line that claimed a hash + // and the second one that collided with it. + val ambiguousRefs = ArrayList() val bookRows = ArrayList>() conn.prepareStatement("SELECT id, title, heRef FROM book ORDER BY id").use { ps -> @@ -102,6 +144,9 @@ internal fun indexAllBooks(conn: Connection, logger: Logger): LineRefIndexReport val aliases = listOfNotNull(bookHeRef, title).filter { it.isNotBlank() } val seen = HashSet() val seenAmbiguous = HashSet() + // hash -> (heRef, lineIndex) of the line that claimed it first, + // so a collision can be reported with both sides named. + val firstByHash = HashMap>() var hasTitleMismatch = false selectLines.setLong(1, bookId) @@ -114,7 +159,21 @@ internal fun indexAllBooks(conn: Connection, logger: Logger): LineRefIndexReport // for titles that themselves contain a hyphen or Hebrew maqaf. val key = RefKey.ofLine(heRef, aliases) ?: continue val hash = RefKey.hash(key) - if (!seen.add(hash) && seenAmbiguous.add(hash)) ambiguous++ + if (!seen.add(hash)) { + if (seenAmbiguous.add(hash)) { + ambiguous++ + val (firstRef, firstIndex) = firstByHash[hash] ?: (heRef to lineIndex) + ambiguousRefs += AmbiguousLineRef( + bookId = bookId, + bookTitle = title, + heRef = firstRef, + firstLineIndex = firstIndex, + secondLineIndex = lineIndex, + ) + } + } else { + firstByHash[hash] = heRef to lineIndex + } insert.setLong(1, bookId) insert.setLong(2, hash) insert.setLong(3, lineIndex) @@ -130,5 +189,5 @@ internal fun indexAllBooks(conn: Connection, logger: Logger): LineRefIndexReport } } - return LineRefIndexReport(books, indexed, ambiguous, mismatched) + return LineRefIndexReport(books, indexed, ambiguous, mismatched, ambiguousRefs) } diff --git a/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReport.kt b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReport.kt new file mode 100644 index 00000000..6d6a4497 --- /dev/null +++ b/generator/common/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReport.kt @@ -0,0 +1,143 @@ +package io.github.kdroidfilter.seforimlibrary.common.reports + +import co.touchlab.kermit.Logger +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths +import java.nio.file.StandardCopyOption + +/** + * Side-channel for generator diagnostics that are too long for the build log. + * + * The logging rule for this pipeline is "every noise cut becomes one summary + * line with counts" — but a summary line is only actionable if the full list it + * summarises is recoverable somewhere. Findings such as "430 priority entries + * are missing" or "1,116 metadata records matched no book" therefore log one + * bounded line and write the complete list here, as a small deterministic JSON + * file next to the build output. + * + * Location: `-DgeneratorReportDir=` / `GENERATOR_REPORT_DIR`, else + * `build/generator-reports`. Files are written atomically (temp + ATOMIC_MOVE) + * like the other generator reports, so a crash never leaves a half-written one. + * + * A failure to write a report is never fatal: these files are diagnostics, and + * losing one must not fail a build that otherwise succeeded. + * + * JSON is emitted by hand rather than through kotlinx.serialization because + * `:generator-common` does not depend on it, and because a fixed insertion + * order keeps the files byte-comparable between builds. + */ +object GeneratorReport { + + const val DIR_PROPERTY: String = "generatorReportDir" + const val DIR_ENV: String = "GENERATOR_REPORT_DIR" + const val DEFAULT_DIR: String = "build/generator-reports" + + /** Directory the reports are written to. Not created until something writes. */ + fun directory(): Path { + val explicit = System.getProperty(DIR_PROPERTY) ?: System.getenv(DIR_ENV) + return Paths.get(explicit ?: DEFAULT_DIR) + } + + /** + * Writes `/.json`, returning the path, or `null` when the + * write failed (already logged as a warning). + * + * [name] is the bare report name — no extension, no directory. + */ + fun write(name: String, logger: Logger, build: ReportBuilder.() -> Unit): Path? { + val body = ReportBuilder().apply(build).build() + return runCatching { + val dir = directory().toAbsolutePath() + Files.createDirectories(dir) + val target = dir.resolve("$name.json") + val tmp = Files.createTempFile(dir, name, ".json.tmp") + Files.writeString(tmp, body) + Files.move(tmp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING) + logger.i { "report written to $target" } + target + }.onFailure { logger.w(it) { "Failed to write $name report to ${directory()}" } }.getOrNull() + } +} + +/** + * Builds one report object. Keys keep insertion order; only the shapes the + * generator reports actually need are supported (scalars, string lists and + * lists of flat records). + */ +class ReportBuilder internal constructor() { + + private val entries = ArrayList>() + + fun put(key: String, value: String) { + entries += key to quote(value) + } + + fun put(key: String, value: Long) { + entries += key to value.toString() + } + + fun put(key: String, value: Int) = put(key, value.toLong()) + + fun putStrings(key: String, values: Collection) { + if (values.isEmpty()) { + entries += key to "[]" + return + } + val body = values.joinToString(separator = ",\n") { " " + quote(it) } + entries += key to "[\n$body\n ]" + } + + /** + * A list of flat records. Every value is rendered as a JSON string or, for + * a [Number], as a JSON number; `null` becomes `null`. Field order follows + * each map's iteration order, so pass a `LinkedHashMap` (`mapOf(...)` is). + */ + fun putRows(key: String, rows: Collection>) { + if (rows.isEmpty()) { + entries += key to "[]" + return + } + val body = rows.joinToString(separator = ",\n") { row -> + val fields = row.entries.joinToString(separator = ", ") { (k, v) -> + "${quote(k)}: ${scalar(v)}" + } + " { $fields }" + } + entries += key to "[\n$body\n ]" + } + + internal fun build(): String = buildString { + append("{\n") + entries.forEachIndexed { index, (key, rendered) -> + append(" ").append(quote(key)).append(": ").append(rendered) + if (index != entries.lastIndex) append(',') + append('\n') + } + append("}\n") + } + + private fun scalar(value: Any?): String = when (value) { + null -> "null" + is Number -> value.toString() + is Boolean -> value.toString() + else -> quote(value.toString()) + } + + private fun quote(value: String): String = buildString(value.length + 2) { + append('"') + for (ch in value) { + when (ch) { + '"' -> append("\\\"") + '\\' -> append("\\\\") + '\n' -> append("\\n") + '\r' -> append("\\r") + '\t' -> append("\\t") + // Locale-free: String.format would depend on the runner's locale. + else -> if (ch < ' ') append("\\u").append(ch.code.toString(16).padStart(4, '0')) + else append(ch) + } + } + append('"') + } +} diff --git a/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifierTest.kt b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifierTest.kt new file mode 100644 index 00000000..83ab5998 --- /dev/null +++ b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateVerifierTest.kt @@ -0,0 +1,226 @@ +package io.github.kdroidfilter.seforimlibrary.common.buildstate + +import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder +import java.nio.file.Files +import java.nio.file.Path +import java.security.MessageDigest +import java.sql.DriverManager +import kotlin.test.assertContains +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * The five generator stages used to publish whatever build_state.db happened to + * be on disk. These cover the post-write self-check that now stands between a + * snapshot and "stage completed": the file must be the one this stage wrote, and + * its counters must be ahead of every id the produced DB holds. + */ +class BuildStateVerifierTest { + + @JvmField + @Rule + val tmp = TemporaryFolder() + + private fun lineHash(seed: Int): ByteArray = + MessageDigest.getInstance("SHA-1").digest(byteArrayOf(seed.toByte())) + + /** A DB holding only the `id` columns the verifier reads. */ + private fun miniDb(name: String, tables: Map>): Path { + val db = tmp.newFolder().toPath().resolve(name) + Class.forName("org.sqlite.JDBC") + DriverManager.getConnection("jdbc:sqlite:${db.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + for ((table, ids) in tables) { + st.executeUpdate("CREATE TABLE \"$table\" (id INTEGER PRIMARY KEY NOT NULL, v TEXT)") + for (id in ids) st.executeUpdate("INSERT INTO \"$table\"(id, v) VALUES ($id, 'x')") + } + } + } + return db + } + + /** Two books, three lines in the first — counters land at book=3, line=4. */ + private fun snapshot(target: Path, meta: Map) { + val allocator = InMemoryIdAllocator.load(null) + val bookA = allocator.bookId("Otzaria", "A") + allocator.bookId("Otzaria", "B") + repeat(3) { i -> allocator.lineId(bookA, lineHash(i), 0) } + allocator.snapshotTo(target, meta) + } + + private fun meta(generatedAt: String) = mapOf( + "generator" to "test", + "generated_at" to generatedAt, + ) + + @Test + fun `a fresh snapshot whose counters lead the db passes`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + val db = miniDb("seforim.db", mapOf("book" to listOf(1L, 2L), "line" to listOf(1L, 2L, 3L))) + + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } + + @Test + fun `counters behind the db are rejected per table`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + // A stale/foreign buildstate: the DB holds ids this allocator never issued, + // so the next build would hand id 9 to some other book. + val db = miniDb("seforim.db", mapOf("book" to listOf(1L, 2L, 9L), "line" to listOf(1L, 2L, 3L))) + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } + assertContains(failure.message!!, "book: next_id=3 <= MAX(id)=9") + assertContains(failure.message!!, "re-issue ids this build already published") + } + + /** + * The boundary the gate exists for: `next_id` is handed to the NEXT fresh key, + * so `next_id == MAX(id)` is already a collision, not a near miss. Without this + * case `<=` could be weakened to `<` and the whole suite would still pass. + */ + @Test + fun `a counter equal to the db maximum is already a collision`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + // Counters are book=3, line=4; a book already sitting on id 3 means the next + // build would hand 3 to a different book. + val db = miniDb("seforim.db", mapOf("book" to listOf(1L, 2L, 3L), "line" to listOf(1L))) + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } + assertContains(failure.message!!, "book: next_id=3 <= MAX(id)=3") + } + + @Test + fun `a snapshot left over from another run is rejected`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + // Written by the PREVIOUS build; this build's snapshotTo failed, so the + // file simply stayed in place — same name, plausible size, valid SQLite. + snapshot(state, meta("2026-09-01T10:00:00Z")) + val db = miniDb("seforim.db", mapOf("book" to listOf(1L, 2L), "line" to listOf(1L, 2L, 3L))) + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, meta("2026-09-08T10:00:00Z")) + } + assertContains(failure.message!!, "not the snapshot this stage wrote") + assertContains(failure.message!!, "meta.generated_at='2026-09-01T10:00:00Z'") + } + + @Test + fun `a missing snapshot is rejected`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val db = miniDb("seforim.db", mapOf("book" to listOf(1L))) + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, meta("now")) + } + assertContains(failure.message!!, "does not exist") + } + + @Test + fun `a missing db is rejected`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + val missing = tmp.newFolder().toPath().resolve("seforim.db") + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, missing, expected) + } + assertContains(failure.message!!, "does not exist") + } + + @Test + fun `a snapshot that is not a build_state is rejected`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + Files.write(state, "certainly not a database".repeat(20).toByteArray()) + val db = miniDb("seforim.db", mapOf("book" to listOf(1L))) + + // Either the driver refuses the file or the meta/counters are absent — + // both must abort the stage rather than let it publish this. + assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, meta("now")) + } + } + + @Test + fun `readHeader returns counters and meta without the id maps`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + + val header = BuildStateVerifier.readHeader(state) + assertEquals(3L, header.counters[IdTable.BOOK]) + assertEquals(4L, header.counters[IdTable.LINE]) + assertEquals("2026-09-08T10:00:00Z", header.meta["generated_at"]) + assertTrue(header.meta.containsKey("schema_version")) + } + + /** + * `alt_toc_entry` rows are inserted with an implicit rowid by both alt-TOC + * builders and its counter therefore never leaves 1. Measured on a real + * `generateSefariaSqlite` output: next_id=1 against MAX(id)=71602. Asserting + * the counter there would have failed every stage of every release. + */ + @Test + fun `alt_toc_entry ids are not allocator-issued and do not fail the check`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + val db = miniDb( + "seforim.db", + mapOf( + "book" to listOf(1L, 2L), + "line" to listOf(1L, 2L, 3L), + "alt_toc_entry" to listOf(1L, 2L, 71602L), + ), + ) + + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } + + @Test + fun `an unrelated table is still checked when alt_toc_entry carries rows`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + val db = miniDb( + "seforim.db", + mapOf( + "book" to listOf(1L, 2L, 9L), + "alt_toc_entry" to listOf(71602L), + ), + ) + + val failure = assertFailsWith { + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } + assertContains(failure.message!!, "book: next_id=3 <= MAX(id)=9") + assertFalse(failure.message!!.contains("alt_toc_entry")) + } + + @Test + fun `tables the db does not carry are skipped, not failed`() { + val state = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + val expected = meta("2026-09-08T10:00:00Z") + snapshot(state, expected) + // Only `book` exists here. Real stage DBs carry all 15 (the schema is + // created wholesale at repository init); this pins the fixture path. + val db = miniDb("seforim.db", mapOf("book" to listOf(1L, 2L))) + + BuildStateVerifier.verifyFreshSnapshot(state, db, expected) + } +} diff --git a/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriteTimingTest.kt b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriteTimingTest.kt new file mode 100644 index 00000000..d6c6f588 --- /dev/null +++ b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/buildstate/BuildStateWriteTimingTest.kt @@ -0,0 +1,79 @@ +package io.github.kdroidfilter.seforimlibrary.common.buildstate + +import co.touchlab.kermit.LogWriter +import co.touchlab.kermit.Logger +import co.touchlab.kermit.Severity +import co.touchlab.kermit.StaticConfig +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder +import kotlin.test.assertContains +import kotlin.test.assertTrue + +/** + * The five build_state snapshots cost 546 s (26%) of the audited generation step + * and reported no duration at all, so the cost was invisible in the log and only + * showed up as an unexplained silent gap in the phase splitter. + * + * Two things are pinned here: the line now carries its own elapsed seconds, and + * it still ENDS with `links=)` — pipeline-monitor's `buildstate_write_2` + * marker anchors on that with a `$`-terminated regex, so appending anything + * after the counters would silently unhook the phase split. + */ +class BuildStateWriteTimingTest { + + @JvmField + @Rule + val tmp = TemporaryFolder() + + private class Capture : LogWriter() { + val lines = mutableListOf() + override fun log(severity: Severity, message: String, tag: String, throwable: Throwable?) { + lines += message + } + } + + private fun writeSnapshotAndCaptureLog(): String { + val capture = Capture() + val logger = Logger(StaticConfig(Severity.Verbose, listOf(capture)), "BuildStateWriter") + val target = tmp.newFolder().toPath().resolve("seforim.db.buildstate") + BuildStateWriter(logger).write(BuildStateSnapshot.empty(), target) + return capture.lines.single { it.startsWith("build_state.db snapshot written to ") } + } + + @Test + fun `the snapshot line reports its own elapsed seconds`() { + val line = writeSnapshotAndCaptureLog() + assertTrue( + Regex(""" in \d+\.\ds \(""").containsMatchIn(line), + "expected an ` in s (` duration, got: $line", + ) + } + + @Test + fun `the snapshot line still ends with the counters the phase splitter anchors on`() { + val line = writeSnapshotAndCaptureLog() + assertContains(line, "(books=0, lines=0, tocEntries=0, links=0)") + assertTrue( + Regex("""lines=\d+, tocEntries=\d+, links=\d+\)$""").containsMatchIn(line), + "pipeline-monitor's buildstate_write_2 marker requires this line to END " + + "with `links=)`; got: $line", + ) + } + + @Test + fun `the duration is locale-independent`() { + val previous = java.util.Locale.getDefault() + try { + // A locale whose decimal separator is a comma would turn "113.2" into + // "113,2" under String.format; the writer uses integer arithmetic. + java.util.Locale.setDefault(java.util.Locale.of("de", "DE")) + assertTrue( + Regex(""" in \d+\.\ds \(""").containsMatchIn(writeSnapshotAndCaptureLog()), + "the elapsed seconds must not follow the runner's locale", + ) + } finally { + java.util.Locale.setDefault(previous) + } + } +} diff --git a/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCliTest.kt b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCliTest.kt index f4918979..53f60bbe 100644 --- a/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCliTest.kt +++ b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/refs/BuildLineRefIndexCliTest.kt @@ -44,6 +44,23 @@ class BuildLineRefIndexCliTest { assertEquals(listOf("לא תואם"), report.titleMismatchBooks) assertEquals(8, report.indexed) assertEquals(1, report.ambiguousKeys) + // `90 keys resolving to more than one line` named none of them, so + // nobody could look at the data behind the warning. Each collision + // now carries the book and BOTH line indexes; the trailing `!`/`?` + // variants at 12 and 13 collapse onto the ref first claimed at 10, + // and a key is reported once no matter how many lines pile onto it. + assertEquals( + listOf( + AmbiguousLineRef( + bookId = 6, + bookTitle = "פת לחם", + heRef = "פת לחם, שער רביעי - שער הביטחון, א, א", + firstLineIndex = 10, + secondLineIndex = 12, + ), + ), + report.ambiguous, + ) conn.createStatement().use { st -> st.executeQuery("SELECT COUNT(DISTINCT refKeyHash) FROM line_ref WHERE bookId = 6").use { rs -> assertEquals(2, rs.getInt(1)) diff --git a/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReportTest.kt b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReportTest.kt new file mode 100644 index 00000000..7f7738a7 --- /dev/null +++ b/generator/common/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/common/reports/GeneratorReportTest.kt @@ -0,0 +1,192 @@ +package io.github.kdroidfilter.seforimlibrary.common.reports + +import co.touchlab.kermit.LogWriter +import co.touchlab.kermit.Logger +import co.touchlab.kermit.Severity +import co.touchlab.kermit.StaticConfig +import java.nio.file.Files +import java.nio.file.Path +import kotlin.io.path.readText +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertEquals +import kotlin.test.assertNull +import kotlin.test.assertTrue + +/** + * The report side-channel that lets a noisy finding collapse to one bounded log + * line without losing the full list behind it. + * + * Two properties matter and are pinned here: the emitted JSON is byte-stable + * (so two builds' reports can be diffed), and a failure to write one can never + * take a build down — these are diagnostics, and the audited step succeeds or + * fails on the database, not on its own paperwork. + */ +class GeneratorReportTest { + + private val previousDir: String? = System.getProperty(GeneratorReport.DIR_PROPERTY) + + @AfterTest + fun restoreDirProperty() { + if (previousDir == null) System.clearProperty(GeneratorReport.DIR_PROPERTY) + else System.setProperty(GeneratorReport.DIR_PROPERTY, previousDir) + } + + /** Collects everything logged, so a test can assert on severity + message. */ + private class Capture : LogWriter() { + val lines = mutableListOf>() + override fun log(severity: Severity, message: String, tag: String, throwable: Throwable?) { + lines += severity to message + } + } + + private fun capturingLogger(capture: Capture) = + Logger(StaticConfig(minSeverity = Severity.Verbose, logWriterList = listOf(capture)), "test") + + private fun withReportDir(dir: Path): Path { + System.setProperty(GeneratorReport.DIR_PROPERTY, dir.toAbsolutePath().toString()) + return dir + } + + // ─── builder output ──────────────────────────────────────────────────── + + @Test + fun `builder emits keys in insertion order with stable formatting`() { + val body = ReportBuilder().apply { + put("computed", 6216L) + put("notRecorded", 391) + putStrings("titles", listOf("אבן הראשה", "קול התור")) + }.build() + + assertEquals( + """ + { + "computed": 6216, + "notRecorded": 391, + "titles": [ + "אבן הראשה", + "קול התור" + ] + } + """.trimIndent() + "\n", + body, + ) + } + + @Test + fun `rows render numbers booleans and nulls unquoted`() { + val body = ReportBuilder().apply { + putRows( + "missingTargets", + listOf( + mapOf("title" to "עולת ראיה", "links" to 642, "kept" to false), + mapOf("title" to "x", "links" to null, "kept" to true), + ), + ) + }.build() + + assertEquals( + """ + { + "missingTargets": [ + { "title": "עולת ראיה", "links": 642, "kept": false }, + { "title": "x", "links": null, "kept": true } + ] + } + """.trimIndent() + "\n", + body, + ) + } + + @Test + fun `empty collections render as empty arrays rather than being dropped`() { + val body = ReportBuilder().apply { + putStrings("titles", emptyList()) + putRows("rows", emptyList()) + }.build() + assertEquals("{\n \"titles\": [],\n \"rows\": []\n}\n", body) + } + + @Test + fun `quoting escapes the characters that would otherwise break the JSON`() { + val body = ReportBuilder().apply { + put("path", "אוצריא\\מחשבת ישראל\\\"כתבי\"\tהרב\nקוק.txt") + put("control", "a" + 1.toChar() + "b") + }.build() + + assertEquals( + "{\n" + + " \"path\": \"אוצריא\\\\מחשבת ישראל\\\\\\\"כתבי\\\"\\tהרב\\nקוק.txt\",\n" + + " \"control\": \"a\\u0001b\"\n" + + "}\n", + body, + ) + } + + // ─── writing ─────────────────────────────────────────────────────────── + + @Test + fun `write puts the named report under the configured directory`() { + val dir = withReportDir(Files.createTempDirectory("generator-reports")) + val capture = Capture() + + val written = GeneratorReport.write("otzaria-priority-list-missing", capturingLogger(capture)) { + put("entries", 431L) + put("found", 1L) + } + + assertEquals(dir.resolve("otzaria-priority-list-missing.json"), written) + assertEquals("{\n \"entries\": 431,\n \"found\": 1\n}\n", written!!.readText()) + assertTrue( + capture.lines.any { it.first == Severity.Info && it.second.startsWith("report written to ") }, + "the log must still say where the full list went: ${capture.lines}", + ) + } + + @Test + fun `write leaves no temp file behind`() { + val dir = withReportDir(Files.createTempDirectory("generator-reports")) + GeneratorReport.write("a", capturingLogger(Capture())) { put("k", 1L) } + GeneratorReport.write("a", capturingLogger(Capture())) { put("k", 2L) } + + val names = Files.list(dir).use { s -> s.map { it.fileName.toString() }.sorted().toList() } + assertEquals(listOf("a.json"), names, "a rewrite must replace, not accumulate temp files") + assertEquals("{\n \"k\": 2\n}\n", dir.resolve("a.json").readText()) + } + + @Test + fun `a report that cannot be written warns and returns null instead of throwing`() { + // A regular file where the directory should be: createDirectories fails. + val blocker = Files.createTempFile("generator-reports", ".not-a-dir") + withReportDir(blocker.resolve("sub")) + val capture = Capture() + + val written = GeneratorReport.write("doomed", capturingLogger(capture)) { put("k", 1L) } + + assertNull(written, "a failed diagnostics write must not surface as a value") + val warning = capture.lines.single { it.first == Severity.Warn }.second + assertContains(warning, "Failed to write doomed report") + } + + // ─── directory resolution ────────────────────────────────────────────── + + @Test + fun `directory falls back to the build-relative default when nothing is set`() { + System.clearProperty(GeneratorReport.DIR_PROPERTY) + // The env var is not set in the test JVM; if a runner ever sets it, the + // property-wins case above still pins the precedence that matters. + if (System.getenv(GeneratorReport.DIR_ENV) == null) { + // Compared as a Path: the default is written with `/` but renders + // with the platform separator on Windows runners. + assertEquals(java.nio.file.Paths.get(GeneratorReport.DEFAULT_DIR), GeneratorReport.directory()) + } + } + + @Test + fun `the system property wins over everything else`() { + val dir = Files.createTempDirectory("explicit-reports") + System.setProperty(GeneratorReport.DIR_PROPERTY, dir.toString()) + assertEquals(dir.toString(), GeneratorReport.directory().toString()) + } +} diff --git a/generator/otzariasqlite/build.gradle.kts b/generator/otzariasqlite/build.gradle.kts index e027c921..7587b5d4 100644 --- a/generator/otzariasqlite/build.gradle.kts +++ b/generator/otzariasqlite/build.gradle.kts @@ -112,6 +112,10 @@ tasks.register("generateLines") { if (project.hasProperty("baseDb")) { systemProperty("baseDb", project.property("baseDb") as String) } + // The ONLY way past a missing base DB — see DbPublish.allowEmptyBase(). + if (project.hasProperty("allowEmptyBase")) { + systemProperty("allowEmptyBase", project.property("allowEmptyBase") as String) + } // If in-memory DB is used, persist destination (default to build/seforim.db) if (inMemory) { @@ -169,6 +173,10 @@ tasks.register("generateLinks") { } ) } + // The ONLY way past a missing base DB — see DbPublish.allowEmptyBase(). + if (project.hasProperty("allowEmptyBase")) { + systemProperty("allowEmptyBase", project.property("allowEmptyBase") as String) + } if (project.hasProperty("sourceDir")) { systemProperty("sourceDir", project.property("sourceDir") as String) @@ -207,6 +215,10 @@ tasks.register("appendOtzariaLines") { systemProperty("appendExistingDb", "true") systemProperty("baseDb", baseDb) systemProperty("persistDb", persistDb) + // The ONLY way past a missing base DB — see DbPublish.allowEmptyBase(). + if (project.hasProperty("allowEmptyBase")) { + systemProperty("allowEmptyBase", project.property("allowEmptyBase") as String) + } val defaultAcronymDb = layout.buildDirectory.file("acronymizer/acronymizer.db").get().asFile.absolutePath if (project.hasProperty("acronymDb")) { @@ -246,8 +258,15 @@ tasks.register("appendOtzariaLinks") { val persistDb = if (project.hasProperty("persistDb")) project.property("persistDb") as String else baseDb args(":memory:") + // baseDb == persistDb on purpose: phase 2 appends links to the DB phase 1 + // produced and publishes it back over the same path (atomically — see + // DbPublish). GenerateLinks refuses to run when that file is missing. systemProperty("baseDb", persistDb) systemProperty("persistDb", persistDb) + // The ONLY way past a missing base DB — see DbPublish.allowEmptyBase(). + if (project.hasProperty("allowEmptyBase")) { + systemProperty("allowEmptyBase", project.property("allowEmptyBase") as String) + } if (project.hasProperty("sourceDir")) { systemProperty("sourceDir", project.property("sourceDir") as String) diff --git a/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/Generator.kt b/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/Generator.kt index 89789595..651f6723 100644 --- a/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/Generator.kt +++ b/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/Generator.kt @@ -9,6 +9,7 @@ import io.github.kdroidfilter.seforimlibrary.common.countVisibleChars import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocator import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocatorBindings import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport import io.github.kdroidfilter.seforimlibrary.core.models.* import io.github.kdroidfilter.seforimlibrary.core.text.HebrewTextUtils import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository @@ -38,6 +39,32 @@ import kotlin.io.path.readText */ private const val LINE_FLUSH_THRESHOLD = 1000 +/** + * otzaria-library ships `_headings.json` inside the very same `links/` + * directory as the real `_links.json` files, but those are heading → line + * number maps (`{"בבא בתרא": 1, "דף ב.": 2, …}`), not link files. Nothing in this + * repository reads them: the link parser failed on all 37 of them twice per + * build (once from [DatabaseGenerator.buildHearotMergePlans] in phase 1, once + * from [DatabaseGenerator.processLinks] in phase 2), each failure dragging a + * JSON dump into the log, and each file then produced a "Source book not found + * for links" warning — 481 lines that buried any genuine parse failure. + * + * They are excluded at discovery instead. This is a strict no-op on database + * content: a `_headings` file parsed to an empty link list, and an empty list + * inserts nothing and allocates no id. + */ +private const val HEADINGS_FILE_SUFFIX = "_headings" + +/** True for the `links/` entries that really are link files. See [HEADINGS_FILE_SUFFIX]. */ +private fun isLinkJsonFile(file: Path): Boolean = + file.extension == "json" && !file.nameWithoutExtension.endsWith(HEADINGS_FILE_SUFFIX) + +/** Longest bounded name list any single summary line prints. */ +private const val MAX_NAMES_PER_SUMMARY_LINE = 20 + +/** Collapses newlines/tabs so a quoted excerpt can never break a log line in two. */ +private fun String.oneLine(): String = replace(Regex("\\s+"), " ").trim() + class DatabaseGenerator( private val sourceDirectory: Path, private val repository: SeforimRepository, @@ -117,9 +144,75 @@ class DatabaseGenerator( // Tracks books processed from the priority list to avoid double insertion private val processedPriorityBookKeys = mutableSetOf() - // Overall progress across books + // Overall progress across books. + // The operator-facing signal is `progress` (throttled, INFO); everything + // per-book is DEBUG and its counts are folded into the progress/summary + // lines. See GeneratorProgress.kt. private var totalBooksToProcess: Int = 0 - private var processedBooksCount: Int = 0 + private val progress = BookProgressReporter() + + // Counters folded into the end-of-phase summary line, replacing the + // per-book/per-directory INFO lines they used to be derived from. + private var directoriesVisited: Int = 0 + private var skippedBooksCount: Int = 0 + private var acronymBooksCount: Int = 0 + private var hearotMergedBooksCount: Int = 0 + + // Breadcrumb for the serial insert path. The per-book INFO block used to be + // the only thing naming the book when allocator.bookId / insertBook / + // getBook / updateBookCategoryId threw; those failures are otherwise + // anonymous now that the block is a single DEBUG line emitted only AFTER + // the insert succeeded. Assigned (never read) inside the loop, so the loop's + // control flow is unchanged; read by the phase-level catch. + private var currentBook: String? = null + + // Titles the importer deliberately did NOT insert, kept so the end-of-import + // source-hash accounting can say WHY a computed hash has no book id instead + // of just reporting "1545 / 1561". + private val skippedBlacklistedSourceTitles = mutableSetOf() + private val skippedSefariaShadowedTitles = mutableSetOf() + private val skippedBlacklistedFileTitles = mutableSetOf() + private val skippedUncategorizedTitles = mutableSetOf() + + // Manual-link drop accounting (finding: 643 of 2,932 links silently dropped, + // 642 of them for one missing book, logged as two INFO lines per link). + private class DroppedLinkTarget(var links: Int, val firstPath: String) + + private val droppedLinkTargets = LinkedHashMap() + private var manualLinkRowsSeen: Int = 0 + + /** + * Starts (or restarts) the book-import phase clock and zeroes every counter + * that feeds the end-of-phase summary, so a second phase on the same + * instance reports its own numbers rather than cumulative ones. + */ + private fun startBookImport(total: Int) { + directoriesVisited = 0 + skippedBooksCount = 0 + acronymBooksCount = 0 + hearotMergedBooksCount = 0 + currentBook = null + skippedBlacklistedSourceTitles.clear() + skippedSefariaShadowedTitles.clear() + skippedBlacklistedFileTitles.clear() + skippedUncategorizedTitles.clear() + progress.start(total) + } + + /** Counts one finished book (content or skip) and emits the throttled INFO line. */ + private fun noteBookFinished(lines: Int = 0, tocEntries: Int = 0) { + progress.onBookFinished(lines, tocEntries)?.let { line -> logger.i { line } } + } + + private fun bookImportSummary(label: String): String = progress.summaryLine( + label, + mapOf( + "directories" to directoriesVisited.toLong(), + "skipped" to skippedBooksCount.toLong(), + "acronymBooks" to acronymBooksCount.toLong(), + "hearotMerged" to hearotMergedBooksCount.toLong(), + ), + ) // Normalization helpers for categories/titles private fun normalizeHebrewLabel(raw: String): String { @@ -257,13 +350,72 @@ class DatabaseGenerator( internal fun recordSourceHashesAtEndOfImport() { if (otzariaSourceHashes.isEmpty()) return var recorded = 0 + // A hash with no allocated book id is a book that will be fully + // reprocessed on every future cycle, so the gap is worth naming rather + // than leaving as a bare "N / M". + val unrecorded = LinkedHashMap>() for ((key, hash) in otzariaSourceHashes) { if (allocator.peekBookId(key.sourceName, key.canonicalHeTitle) != null) { allocator.recordSourceHash(key, hash) recorded++ + } else { + unrecorded.getOrPut(classifyUnimportedBook(key.canonicalHeTitle)) { mutableListOf() } + .add(key.canonicalHeTitle) } } logger.i { "Recorded source hashes for $recorded / ${otzariaSourceHashes.size} Otzaria books" } + reportUnrecordedSourceHashes(unrecorded) + } + + /** + * Why a book whose source hash was computed never reached the allocator. + * Every branch mirrors one deliberate `continue`/early-return in the import + * path; anything left over is a genuinely unexplained gap and says so. + */ + private fun classifyUnimportedBook(title: String): String = when (title) { + in mergedHearotTitles -> "merged into a base book as inline notes" + in skippedSefariaShadowedTitles -> "already exists from Sefaria (priority source)" + in skippedBlacklistedSourceTitles -> "blacklisted source" + in skippedBlacklistedFileTitles -> "blacklisted file name" + in skippedUncategorizedTitles -> "file has no category (library root)" + else -> "not imported (reason not tracked)" + } + + /** One WARN per class with counts + bounded names; the full list goes to a report file. */ + private fun reportUnrecordedSourceHashes(byClass: Map>) { + if (byClass.isEmpty()) return + val total = byClass.values.sumOf { it.size } + logger.w { + "source hashes: $total of ${otzariaSourceHashes.size} Otzaria books have no source hash — " + + "they are fully reprocessed every cycle" + } + val ordered = byClass.entries.sortedWith( + compareByDescending>> { it.value.size }.thenBy { it.key }, + ) + for ((reason, titles) in ordered) { + val names = titles.sorted() + logger.w { + "source hashes: ${titles.size} not recorded — $reason " + + "(${names.take(MAX_NAMES_PER_SUMMARY_LINE).joinToString()}" + + (if (names.size > MAX_NAMES_PER_SUMMARY_LINE) ", … and ${names.size - MAX_NAMES_PER_SUMMARY_LINE} more)" else ")") + } + } + GeneratorReport.write("otzaria-source-hashes-not-recorded", logger) { + put("computed", otzariaSourceHashes.size.toLong()) + put("notRecorded", total.toLong()) + putRows( + "byReason", + ordered.map { (reason, titles) -> + mapOf("reason" to reason, "books" to titles.size) + }, + ) + putRows( + "books", + ordered.flatMap { (reason, titles) -> + titles.sorted().map { mapOf("title" to it, "reason" to reason) } + }, + ) + } } @@ -319,6 +471,7 @@ class DatabaseGenerator( } } catch (_: Exception) { 0 } logger.i { "Planned to process approximately $totalBooksToProcess books" } + startBookImport(totalBooksToProcess) // Process priority books first (if any), then process the full library runCatching { @@ -332,6 +485,7 @@ class DatabaseGenerator( preloadAllBookContents(libraryPath) processDirectory(libraryPath, null, 0, metadata) assertAllHearotMergesApplied() + logger.i { bookImportSummary("Otzaria book import") } // Process links processLinks() @@ -366,11 +520,20 @@ class DatabaseGenerator( repository.setJournalModeWal() } catch (_: Exception) {} - logger.e(e) { "Error during generation" } + logger.e(e) { "Error during generation${bookBreadcrumb()}" } throw e } } + /** + * ` (last book: 5848 'X' file=X.txt categoryId=612)` — the breadcrumb the + * deleted per-book INFO block used to provide for failures on the insert + * path (`allocator.bookId`, `insertBook`, `getBook`, + * `updateBookCategoryId`), which throw before the per-book DEBUG line is + * reached. Empty when nothing has been inserted yet. + */ + private fun bookBreadcrumb(): String = currentBook?.let { " (last book: $it)" } ?: "" + /** * Phase 1: Generate categories, books, TOCs and lines only (no links). */ @@ -436,6 +599,7 @@ class DatabaseGenerator( } } catch (_: Exception) { 0 } logger.i { "Planned to process approximately $totalBooksToProcess books (phase 1)" } + startBookImport(totalBooksToProcess) runCatching { processPriorityBooks(loadMetadata = { metadata }) } .onFailure { e -> logger.w(e) { "Failed processing priority list; continuing with full generation (phase 1)" } } @@ -443,12 +607,18 @@ class DatabaseGenerator( preloadAllBookContents(libraryPath) processDirectory(libraryPath, null, 0, metadata) assertAllHearotMergesApplied() + logger.i { bookImportSummary("Otzaria book import (phase 1)") } // Build category closure after categories insertion logger.i { "Building category_closure table (phase 1)..." } repository.rebuildCategoryClosure() } recordSourceHashesAtEndOfImport() + } catch (e: Throwable) { + // Rethrown unchanged — this exists only to name the book the serial + // insert loop was on. See [bookBreadcrumb]. + logger.e(e) { "Error during phase 1${bookBreadcrumb()}" } + throw e } finally { runCatching { enableForeignKeys() } runCatching { repository.setSynchronousNormal() } @@ -639,8 +809,9 @@ class DatabaseGenerator( var mergedPairs = 0 var totalNotes = 0 - val linkFiles = Files.list(linksDir).use { s -> s.filter { it.extension == "json" }.toList() } - .sortedBy { it.fileName.toString() } + val jsonFiles = Files.list(linksDir).use { s -> s.filter { it.extension == "json" }.toList() } + val linkFiles = jsonFiles.filter(::isLinkJsonFile).sortedBy { it.fileName.toString() } + noteSkippedHeadingFiles(jsonFiles.size - linkFiles.size) for (jsonFile in linkFiles) { val baseTitle = normalizeBookTitle(jsonFile.nameWithoutExtension.removeSuffix("_links")) val jsonBytes = runCatching { Files.readAllBytes(jsonFile) }.getOrNull() ?: continue @@ -737,6 +908,11 @@ class DatabaseGenerator( } } + /** One INFO line for the `_headings` files skipped at link discovery. See [HEADINGS_FILE_SUFFIX]. */ + private fun noteSkippedHeadingFiles(count: Int) { + if (count > 0) logger.i { "skipped $count *$HEADINGS_FILE_SUFFIX files (heading maps, not link files)" } + } + /** * Loud invariant: every planned merge was applied. An unapplied plan means a * base book was skipped after its companion's import was already suppressed — @@ -894,7 +1070,8 @@ class DatabaseGenerator( metadata: Map, parentPath: String = "" ) { - logger.i { "=== Processing directory: ${directory.fileName} with parentCategoryId: $parentCategoryId (level: $level) ===" } + directoriesVisited += 1 + logger.d { "=== Processing directory: ${directory.fileName} with parentCategoryId: $parentCategoryId (level: $level) ===" } Files.list(directory).use { stream -> val entries = stream.sorted { a, b -> @@ -909,7 +1086,7 @@ class DatabaseGenerator( logger.d { "Processing subdirectory: ${entry.fileName} with parentId: $parentCategoryId" } val placement = ensureCategoryHierarchy(entry.fileName.toString(), parentCategoryId, level, parentPath) val normalizedPath = placement.normalizedPath.joinToString(" / ") - logger.i { "✅ Category '${entry.fileName}' normalized to '$normalizedPath' with ID: ${placement.id} (parent: $parentCategoryId)" } + logger.d { "✅ Category '${entry.fileName}' normalized to '$normalizedPath' with ID: ${placement.id} (parent: $parentCategoryId)" } processDirectory(entry, placement.id, placement.leafLevel + 1, metadata, placement.canonicalPath) } @@ -923,18 +1100,25 @@ class DatabaseGenerator( val fname = entry.fileName.toString() // Skip files explicitly blacklisted by name if (fileNameBlacklist.contains(fname)) { + skippedBlacklistedFileTitles += normalizeBookTitle(fname.substringBeforeLast('.')) logger.i { "⛔ Skipping blacklisted file '$fname' by name" } continue } if (normalizeBookTitle(fname.substringBeforeLast('.')) in mergedHearotTitles) { - logger.i { "📝 Skipping '$fname' — merged into its base book as inline notes" } + hearotMergedBooksCount += 1 + logger.d { "📝 Skipping '$fname' — merged into its base book as inline notes" } continue } if (parentCategoryId == null) { + skippedUncategorizedTitles += normalizeBookTitle(fname.substringBeforeLast('.')) logger.w { "❌ Book found without category: $entry" } continue } - logger.i { "📚 Processing book ${entry.fileName} with categoryId: $parentCategoryId" } + // No per-book INFO header here: this site and the one inside + // createAndProcessBook each printed one per book (1,500 + 1,501 + // lines / 0.39 MB in one build). Both are gone; createAndProcessBook + // now emits a single per-book DEBUG line once the book is done, + // and that site also covers the priority-list caller. createAndProcessBook(entry, parentCategoryId, metadata) } @@ -944,7 +1128,7 @@ class DatabaseGenerator( } } } - logger.i { "=== Finished processing directory: ${directory.fileName} ===" } + logger.d { "=== Finished processing directory: ${directory.fileName} ===" } } @@ -965,16 +1149,16 @@ class DatabaseGenerator( val rawTitle = filename.substringBeforeLast('.') val title = normalizeBookTitle(rawTitle) val meta = metadata[rawTitle] ?: metadata[title] ?: metadata[stripQuotesForLookup(rawTitle)] - - logger.i { "Processing book: $title with categoryId: $categoryId" } + // Breadcrumb only — never read by this function. See [bookBreadcrumb]. + currentBook = "'$title' file=$filename categoryId=$categoryId" // Apply source blacklist val srcName = getSourceNameFor(path) if (sourceBlacklist.contains(srcName)) { + skippedBlacklistedSourceTitles += title logger.i { "⛔ Skipping '$title' from blacklisted source '$srcName'" } - processedBooksCount += 1 - val pct = if (totalBooksToProcess > 0) (processedBooksCount * 100 / totalBooksToProcess) else 0 - logger.i { "Books progress: $processedBooksCount/$totalBooksToProcess (${pct}%)" } + skippedBooksCount += 1 + noteBookFinished() return } @@ -983,16 +1167,17 @@ class DatabaseGenerator( if (existingBook != null) { val existingSource = repository.getSourceById(existingBook.sourceId) if (existingSource?.name == "Sefaria") { + skippedSefariaShadowedTitles += title logger.i { "⏭️ Skipping '$title' - already exists from Sefaria (priority source)" } - processedBooksCount += 1 - val pct = if (totalBooksToProcess > 0) (processedBooksCount * 100 / totalBooksToProcess) else 0 - logger.i { "Books progress: $processedBooksCount/$totalBooksToProcess (${pct}%)" } + skippedBooksCount += 1 + noteBookFinished() return } } // Assign a stable ID via IdAllocator so cross-build reproducibility holds. val currentBookId = allocator.bookId(srcName, title) + currentBook = "$currentBookId '$title' file=$filename categoryId=$categoryId" logger.d { "Assigning ID $currentBookId to book '$title' (source=$srcName) with categoryId: $categoryId" } // Pre-resolve author / pubPlace / pubDate IDs through the IdAllocator @@ -1038,23 +1223,37 @@ class DatabaseGenerator( logger.d { "Book '${book.title}' inserted with ID: $insertedBookId and categoryId: $categoryId" } // Insert acronyms for this book if an Acronymizer DB is available + var acronymCount = 0 try { val terms = fetchAcronymsForTitle(title) if (terms.isNotEmpty()) { repository.bulkInsertBookAcronyms(insertedBookId, terms) - logger.i { "Inserted ${terms.size} acronyms for '${title}'" } + acronymCount = terms.size + acronymBooksCount += 1 + logger.d { "Inserted ${terms.size} acronyms for '${title}'" } } } catch (e: Exception) { logger.w(e) { "Failed to insert acronyms for '$title'" } } - // Process content of the book - processBookContent(path, insertedBookId, title, categoryId) + // Process content of the book. One DEBUG line per book replaces the + // six INFO lines this used to print; on failure the book is named at + // ERROR level so the breadcrumb survives without the INFO chatter. + val startedAtMs = monotonicMillis() + val stats = try { + processBookContent(path, insertedBookId, title, categoryId) + } catch (e: Throwable) { + logger.e(e) { "Failed processing book $insertedBookId '$title' (file=$filename, categoryId=$categoryId)" } + throw e + } + logger.d { + "book $insertedBookId '$title' file=$filename categoryId=$categoryId " + + "lines=${stats.lines} toc=${stats.tocEntries} tocWithChildren=${stats.tocEntriesWithChildren} " + + "acronyms=$acronymCount ${monotonicMillis() - startedAtMs}ms" + } - // Book-level progress - processedBooksCount += 1 - val pct = if (totalBooksToProcess > 0) (processedBooksCount * 100 / totalBooksToProcess) else 0 - logger.i { "Books progress: $processedBooksCount/$totalBooksToProcess (${pct}%)" } + // Book-level progress (throttled: see BookProgressReporter) + noteBookFinished(lines = stats.lines, tocEntries = stats.tocEntries) } /** @@ -1063,9 +1262,13 @@ class DatabaseGenerator( * @param path The path to the book file * @param bookId The ID of the book in the database */ - private suspend fun processBookContent(path: Path, bookId: Long, bookTitle: String, categoryId: Long) = coroutineScope { + private suspend fun processBookContent( + path: Path, + bookId: Long, + bookTitle: String, + categoryId: Long, + ): BookContentStats = coroutineScope { logger.d { "Processing content for book ID: $bookId" } - logger.i { "Processing content of book ID: $bookId (ID generated by the database)" } // Prefer preloaded content from RAM if available val key = toLibraryRelativeKey(path) @@ -1079,20 +1282,30 @@ class DatabaseGenerator( val stats = HearotCompanionMerge.MergeStats() HearotCompanionMerge.mergeLines(bookTitle, rawLines, plan, stats).also { pendingHearotMergeBases.remove(bookTitle) - logger.i { "📝 '$bookTitle': merged companion notes — ${stats.inPlace} anchored in place, ${stats.appended} appended" } + logger.d { "📝 '$bookTitle': merged companion notes — ${stats.inPlace} anchored in place, ${stats.appended} appended" } } } - logger.i { "Number of lines: ${lines.size}" } // Process each line one by one, handling TOC entries as we go - processLinesWithTocEntries(bookId, bookTitle, categoryId, lines) + val tocStats = processLinesWithTocEntries(bookId, bookTitle, categoryId, lines) // Update the total number of lines repository.updateBookTotalLines(bookId, lines.size) - logger.i { "Content processed successfully for book ID: $bookId (ID generated by the database)" } + BookContentStats( + lines = lines.size, + tocEntries = tocStats.tocEntries, + tocEntriesWithChildren = tocStats.tocEntriesWithChildren, + ) } + /** Per-book counters folded into the per-book DEBUG line and the phase summary. */ + private data class BookContentStats( + val lines: Int, + val tocEntries: Int, + val tocEntriesWithChildren: Int, + ) + /** * Processes lines of a book, identifying and creating TOC entries. @@ -1100,7 +1313,12 @@ class DatabaseGenerator( * @param bookId The ID of the book in the database * @param lines The lines of the book content */ - private suspend fun processLinesWithTocEntries(bookId: Long, bookTitle: String, categoryId: Long, lines: List) { + private suspend fun processLinesWithTocEntries( + bookId: Long, + bookTitle: String, + categoryId: Long, + lines: List, + ): BookContentStats { logger.d { "Processing lines and TOC entries together for book ID: $bookId" } // Structure pour stocker toutes les entrées TOC créées @@ -1124,6 +1342,8 @@ class DatabaseGenerator( val lineBuffer = ArrayList(LINE_FLUSH_THRESHOLD) val tocEntryLineIdUpdates = ArrayList>() val lineTocEntryUpdates = ArrayList>() + // Line ticks: silent for small books, ≥10% (or 60 s) apart for large ones. + val lineTicks = LineTickGate(totalLines = lines.size) suspend fun flushLineBatch() { if (lineBuffer.isEmpty()) return @@ -1228,9 +1448,8 @@ class DatabaseGenerator( lineTocBuffer.clear() } - if (lineIndex % 1000 == 0) { - val pct = if (lines.isNotEmpty()) (lineIndex * 100 / lines.size) else 0 - logger.i { "Book $bookId '$bookTitle': $lineIndex/${lines.size} lines (${pct}%)" } + if (lineTicks.shouldTick(lineIndex)) { + logger.i { lineTicks.tickLine(bookId, bookTitle, lineIndex) } } } @@ -1252,9 +1471,11 @@ class DatabaseGenerator( val lastChildIds = entriesByParent.values.mapNotNull { it.lastOrNull() } repository.bulkUpdateTocEntryFlags(hasChildrenIds, lastChildIds) - logger.i { "✅ Finished processing lines and TOC entries for book ID: $bookId" } - logger.i { " Total TOC entries: ${allTocEntries.size}" } - logger.i { " Entries with children: ${parentIds.size}" } + return BookContentStats( + lines = lines.size, + tocEntries = allTocEntries.size, + tocEntriesWithChildren = parentIds.size, + ) } private fun cleanHtml(html: String): String { @@ -1331,6 +1552,14 @@ class DatabaseGenerator( logger.i { "Processing ${entries.size} priority entries first" } val metadata = loadMetadata() + // The list has drifted almost entirely out of the library (430 of 431 + // entries pointed at files that no longer exist in the audited build). + // One WARN summary replaces one WARN per entry; the full list goes to a + // report file so the operator can decide re-pin vs delete. Re-pinning or + // deleting the list here would change the book insertion ORDER, hence + // the allocated ids — never a fix this code may make on its own. + var priorityFilesFound = 0 + val priorityMissing = mutableListOf() outer@ for ((idx, relative) in entries.withIndex()) { // Build the absolute path under the library root @@ -1342,6 +1571,7 @@ class DatabaseGenerator( val bookFileName = parts.last() // Skip files explicitly blacklisted by name if (fileNameBlacklist.contains(bookFileName)) { + skippedBlacklistedFileTitles += normalizeBookTitle(bookFileName.substringBeforeLast('.')) logger.i { "⛔ Skipping blacklisted file in priority list: $bookFileName" } continue@outer } @@ -1356,9 +1586,11 @@ class DatabaseGenerator( val bookPath = currentPath.resolve(bookFileName) if (!Files.isRegularFile(bookPath)) { - logger.w { "Priority entry ${idx + 1}/${entries.size}: file not found: $bookPath" } + priorityMissing += relative + logger.d { "Priority entry ${idx + 1}/${entries.size}: file not found: $bookPath" } continue@outer } + priorityFilesFound += 1 // Avoid processing duplicates listed multiple times val key = toLibraryRelativeKey(bookPath) @@ -1389,6 +1621,36 @@ class DatabaseGenerator( // Mark as processed to avoid double insertion during full traversal processedPriorityBookKeys.add(key) } + + reportPriorityListDrift(entries.size, priorityFilesFound, priorityMissing) + } + + /** + * One WARN for the whole priority pass instead of one per missing entry + * (430 WARNs / 0.11 MB in the audited build). The list itself is left + * exactly as it is: re-pinning or deleting entries reorders the book + * inserts and therefore the allocated ids, which is an operator decision, + * not a logging fix. + */ + private fun reportPriorityListDrift(total: Int, found: Int, missing: List) { + if (missing.isEmpty()) { + logger.i { "priority list: $found/$total entries found" } + return + } + logger.w { + "priority list: $found/$total entries found, ${missing.size} missing " + + "(first: ${missing.take(3).joinToString()}) — list is " + + "otzariasqlite/src/commonMain/resources/priority.txt (loaded by " + + "DatabaseGenerator.loadPriorityList in Generator.kt); needs re-pinning or " + + "removal (operator decision)" + } + GeneratorReport.write("otzaria-priority-list-missing", logger) { + put("entries", total.toLong()) + put("found", found.toLong()) + put("missing", missing.size.toLong()) + put("listResource", "otzariasqlite/src/commonMain/resources/priority.txt") + putStrings("missingEntries", missing) + } } /** @@ -1509,8 +1771,12 @@ class DatabaseGenerator( logger.d { "Links in database before processing: $linksBefore" } logger.i { "Loading all link JSON files into RAM..." } + droppedLinkTargets.clear() + manualLinkRowsSeen = 0 // Preload all links JSON into memory to minimize IO - val linkFiles = Files.list(linksDir).use { s -> s.filter { it.extension == "json" }.toList() } + val jsonFiles = Files.list(linksDir).use { s -> s.filter { it.extension == "json" }.toList() } + val linkFiles = jsonFiles.filter(::isLinkJsonFile) + noteSkippedHeadingFiles(jsonFiles.size - linkFiles.size) val linksByBook = coroutineScope { linkFiles.map { file -> async { @@ -1529,10 +1795,12 @@ class DatabaseGenerator( logger.i { "Processing links from RAM..." } var totalLinks = 0 for ((bookTitle, links) in linksByBook) { + manualLinkRowsSeen += links.size val processedLinks = processLinksForBook(bookTitle, links) totalLinks += processedLinks logger.d { "Processed $processedLinks links for $bookTitle, total so far: $totalLinks" } } + reportDroppedManualLinks() // Count links after processing val linksAfter = repository.countLinks() @@ -1545,6 +1813,55 @@ class DatabaseGenerator( updateBookHasLinksTable() } + /** + * Manual links whose target book does not exist in the DB are dropped — that + * is upstream data drift (otzaria-library), not something this importer can + * repair, and the drop behaviour is deliberately unchanged. What changes is + * that it is now visible: one WARN per missing target book with its link + * count and original path, bounded, plus a total. Previously this was two + * INFO lines per dropped link and no summary at all, so a fifth of the + * manual-link corpus vanished without a single warning. + */ + private fun reportDroppedManualLinks() { + if (droppedLinkTargets.isEmpty()) return + val ordered = droppedLinkTargets.entries.sortedWith( + compareByDescending> { it.value.links }.thenBy { it.key }, + ) + val droppedLinks = ordered.sumOf { it.value.links } + logger.w { + "manual links: $droppedLinks of $manualLinkRowsSeen links dropped — " + + "${ordered.size} target book(s) not found" + } + for ((title, target) in ordered.take(MAX_NAMES_PER_SUMMARY_LINE)) { + logger.w { + "manual links: ${target.links} links dropped — target book not found: " + + "'$title' (${target.firstPath})" + } + } + val rest = ordered.drop(MAX_NAMES_PER_SUMMARY_LINE) + if (rest.isNotEmpty()) { + logger.w { + "manual links: … and ${rest.size} more missing target books " + + "(${rest.sumOf { it.value.links }} links)" + } + } + GeneratorReport.write("otzaria-manual-links-dropped", logger) { + put("linkRowsSeen", manualLinkRowsSeen.toLong()) + put("droppedLinks", droppedLinks.toLong()) + put("missingTargetBooks", ordered.size.toLong()) + putRows( + "missingTargets", + ordered.map { (title, target) -> + mapOf( + "title" to title, + "links" to target.links, + "originalPath" to target.firstPath, + ) + }, + ) + } + } + /** * Processes links that were preloaded in memory for a given source book. */ @@ -1566,7 +1883,9 @@ class DatabaseGenerator( val rangeBatch = mutableListOf() val coverageBatch = mutableListOf() val touchedLinkIds = mutableSetOf() - for ((index, linkData) in links.withIndex()) { + // `withIndex()` dropped with the per-link "Link i/N" log line it existed + // for: same list, same order, same iterations. + for (linkData in links) { try { val path = linkData.path_2 val targetTitle = if (path.contains('\\')) { @@ -1583,8 +1902,12 @@ class DatabaseGenerator( if (HearotCompanionMerge.isMergeableCompanionTitle(normalizeBookTitle(targetTitle))) { logger.d { "Skipping link into merged companion notes: $targetTitle" } } else { - logger.i { "Link ${index + 1}/${links.size} - Target book not found: $targetTitle" } - logger.i { "Original path: ${linkData.path_2}" } + // Counted, not printed: two INFO lines per dropped link + // was 1,286 lines for 643 drops. The drop itself is + // unchanged — see [reportDroppedManualLinks]. + droppedLinkTargets + .getOrPut(targetTitle) { DroppedLinkTarget(0, linkData.path_2) } + .links += 1 } continue } @@ -2154,7 +2477,15 @@ class DatabaseGenerator( logger.w { "DictaToOtzaria format conversion not yet implemented for $bookTitle" } emptyList() } catch (e2: Exception) { - logger.w(e2) { "Failed to parse links from file for $bookTitle in any known format" } + // One line, no throwable: kermit renders the JsonDecodingException + // together with the whole offending document, which was 146 log + // lines of JSON body per build. The head of the content says just + // as much about what the file actually is. + logger.w { + "Failed to parse links for '$bookTitle' in any known format " + + "(${e2::class.simpleName}: ${e2.message?.take(120)?.oneLine()}); " + + "content starts: ${content.take(80).oneLine()}" + } emptyList() } } diff --git a/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgress.kt b/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgress.kt new file mode 100644 index 00000000..a75bc82d --- /dev/null +++ b/generator/otzariasqlite/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgress.kt @@ -0,0 +1,223 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import kotlin.time.TimeSource + +/** + * Cadence gates and formatters for the generator's progress output. + * + * The book import used to emit ~15 INFO lines per book (a 6-line lifecycle + * block, a duplicated header, a `Books progress:` line and a line tick that + * always fired at index 0). For ~1,500 books that is ~18k lines of the + * "Generate Seforim Database" step. Everything per-book now goes to DEBUG + * (the CLI entry points pin kermit at [co.touchlab.kermit.Severity.Info], so + * DEBUG is genuinely off in CI) and the operator-facing INFO signal is a + * throttled progress line produced here. + * + * Both gates are pure apart from the injected clock, so their cadence is unit + * tested against a fake clock instead of against a 35-minute build. + */ + +/** Emit a book progress line at most every N finished books… */ +internal const val BOOK_PROGRESS_EVERY_N_BOOKS: Int = 100 + +/** …or every this many milliseconds, whichever comes first. */ +internal const val BOOK_PROGRESS_EVERY_MS: Long = 60_000L + +/** Books at or below this many lines never emit a line tick. */ +internal const val LINE_TICK_MIN_BOOK_LINES: Int = 5_000 + +/** Line ticks are only ever considered on multiples of this index. */ +internal const val LINE_TICK_STEP: Int = 1_000 + +/** A tick needs at least this much of the book to have passed since the last one… */ +internal const val LINE_TICK_MIN_PERCENT: Int = 10 + +/** …unless this long has passed, which keeps a slow huge book observable. */ +internal const val LINE_TICK_EVERY_MS: Long = 60_000L + +private val processStart = TimeSource.Monotonic.markNow() + +/** Monotonic milliseconds since class load; injected so tests can fake it. */ +internal val monotonicMillis: () -> Long = { processStart.elapsedNow().inWholeMilliseconds } + +private fun oneDecimal(value: Long, unit: Long, suffix: String): String { + val whole = value / unit + val tenths = (value % unit) * 10 / unit + return "$whole.$tenths$suffix" +} + +/** `934`, `372.9K`, `5.2M` — locale independent, integer arithmetic only. */ +internal fun formatCount(n: Long): String = when { + n >= 1_000_000L -> oneDecimal(n, 1_000_000L, "M") + n >= 10_000L -> oneDecimal(n, 1_000L, "K") + else -> n.toString() +} + +private fun pad2(v: Long): String = if (v < 10) "0$v" else "$v" + +/** `27:10`, `1:05:03` — mm:ss below an hour, h:mm:ss above it. */ +internal fun formatDuration(ms: Long): String { + val totalSeconds = (if (ms < 0) 0 else ms) / 1000 + val hours = totalSeconds / 3600 + val minutes = (totalSeconds % 3600) / 60 + val seconds = totalSeconds % 60 + return if (hours > 0) "$hours:${pad2(minutes)}:${pad2(seconds)}" else "${pad2(minutes)}:${pad2(seconds)}" +} + +/** `82.2` for 1234/1501; `?` when the total is unknown. */ +internal fun formatPercent(done: Int, total: Int): String { + if (total <= 0) return "?" + val tenths = done.toLong() * 1000L / total + return "${tenths / 10}.${tenths % 10}" +} + +/** + * Throttles and formats the book-loop progress line: + * + * books 1234/1501 (82.2%) · lines 5.2M · toc 301.4K · elapsed 27:10 · eta ~6:00 + * + * A line is produced every [everyNBooks] finished books or every [everyMs], + * whichever comes first. This replaces the per-book `Books progress: n/N (p%)` + * line, which fired 1,501 times in a single build. + */ +internal class BookProgressReporter( + private val nowMs: () -> Long = monotonicMillis, + private val everyNBooks: Int = BOOK_PROGRESS_EVERY_N_BOOKS, + private val everyMs: Long = BOOK_PROGRESS_EVERY_MS, +) { + /** Planned book count; set once the source tree has been walked. */ + var totalBooks: Int = 0 + + var booksDone: Int = 0 + private set + var linesInserted: Long = 0L + private set + var tocEntries: Long = 0L + private set + + private var startedAtMs: Long = 0L + private var lastEmitMs: Long = 0L + private var lastEmitBooks: Int = 0 + private var started: Boolean = false + + /** Starts (or restarts) the elapsed/ETA clock. Idempotent per phase. */ + fun start(totalBooks: Int = this.totalBooks) { + this.totalBooks = totalBooks + val now = nowMs() + startedAtMs = now + lastEmitMs = now + lastEmitBooks = 0 + booksDone = 0 + linesInserted = 0L + tocEntries = 0L + started = true + } + + fun elapsedMs(): Long = if (started) nowMs() - startedAtMs else 0L + + /** + * Records one finished book (skipped books count too, with zero content) + * and returns the INFO line to log, or `null` when this book is not due. + */ + fun onBookFinished(lines: Int = 0, tocEntries: Int = 0): String? { + if (!started) start() + booksDone += 1 + linesInserted += lines.toLong() + this.tocEntries += tocEntries.toLong() + + val now = nowMs() + val dueByCount = everyNBooks > 0 && booksDone - lastEmitBooks >= everyNBooks + val dueByTime = everyMs > 0 && now - lastEmitMs >= everyMs + if (!dueByCount && !dueByTime) return null + + lastEmitBooks = booksDone + lastEmitMs = now + return progressLine(now) + } + + /** The same line, unconditionally — used to close out a phase. */ + fun progressLine(now: Long = nowMs()): String { + val elapsed = now - startedAtMs + return buildString { + append("books ").append(booksDone).append('/').append(totalBooks) + append(" (").append(formatPercent(booksDone, totalBooks)).append("%)") + append(" · lines ").append(formatCount(linesInserted)) + append(" · toc ").append(formatCount(tocEntries)) + append(" · elapsed ").append(formatDuration(elapsed)) + val remaining = totalBooks - booksDone + if (totalBooks > 0 && booksDone > 0 && remaining > 0) { + val eta = elapsed / booksDone * remaining + append(" · eta ~").append(formatDuration(eta)) + } + } + } + + /** + * End-of-phase summary. [extras] are appended as `key=value` in iteration + * order, so pass a [LinkedHashMap] (`mapOf(...)` already is one). + */ + fun summaryLine(label: String, extras: Map = emptyMap()): String = buildString { + append(label) + append(": books=").append(booksDone).append('/').append(totalBooks) + append(" lines=").append(linesInserted) + append(" tocEntries=").append(tocEntries) + for ((key, value) in extras) { + append(' ').append(key).append('=').append(value) + } + append(" elapsed=").append(formatDuration(elapsedMs())) + } +} + +/** + * Per-book line-tick cadence. Construct one per book, then ask it about every + * line index. + * + * Old behaviour: `if (lineIndex % 1000 == 0)` — which fires at index 0, so + * every book (median 414 lines) emitted a useless `0/N (0%)`; 1,496 of the + * 3,239 ticks in the audited build carried no information at all. + * + * New behaviour: nothing at all for books of [minBookLines] lines or fewer, + * and for larger books a tick only on a multiple of [step] that is also at + * least [minPercent] of the book past the previous tick — or [everyMs] later, + * so a genuinely slow book still reports. + */ +internal class LineTickGate( + private val totalLines: Int, + private val nowMs: () -> Long = monotonicMillis, + minBookLines: Int = LINE_TICK_MIN_BOOK_LINES, + private val step: Int = LINE_TICK_STEP, + minPercent: Int = LINE_TICK_MIN_PERCENT, + private val everyMs: Long = LINE_TICK_EVERY_MS, +) { + /** Books at or below the threshold never tick — checked before any clock read. */ + val enabled: Boolean = totalLines > minBookLines + + private val minLinesBetweenTicks: Int = + if (!enabled) 0 else maxOf(step, (totalLines.toLong() * minPercent / 100L).toInt()) + + private var lastTickLine: Int = 0 + private var lastTickMs: Long = if (enabled) nowMs() else 0L + + /** + * Whether [lineIndex] should emit a tick. Advances internal state when it + * returns `true`, so call it exactly once per line index. + */ + fun shouldTick(lineIndex: Int): Boolean { + if (!enabled) return false + if (lineIndex <= 0) return false + if (lineIndex % step != 0) return false + val now = nowMs() + val dueByLines = lineIndex - lastTickLine >= minLinesBetweenTicks + val dueByTime = everyMs > 0 && now - lastTickMs >= everyMs + if (!dueByLines && !dueByTime) return false + lastTickLine = lineIndex + lastTickMs = now + return true + } + + /** `Book 5848 'X': 12000/190760 lines (6%)` — unchanged from the old format. */ + fun tickLine(bookId: Long, bookTitle: String, lineIndex: Int): String { + val pct = if (totalLines > 0) (lineIndex.toLong() * 100L / totalLines) else 0L + return "Book $bookId '$bookTitle': $lineIndex/$totalLines lines (${pct}%)" + } +} diff --git a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/DbPublish.kt b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/DbPublish.kt new file mode 100644 index 00000000..9a183908 --- /dev/null +++ b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/DbPublish.kt @@ -0,0 +1,187 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import co.touchlab.kermit.Logger +import java.nio.file.AtomicMoveNotSupportedException +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.StandardCopyOption +import java.sql.DriverManager + +/** + * How the Otzaria phases put a freshly built database in place of the previous + * one. + * + * Both phases run against `:memory:` and finish with `VACUUM INTO`, whose target + * must not exist — so the code used to **delete the target first** and vacuum + * into the hole. In the release path `baseDb == persistDb` + * (build.gradle.kts:appendOtzariaLines / appendOtzariaLinks), i.e. the file it + * deletes is the 7 GiB DB the run was seeded from, and anything that stops the + * process in between (OOM, a full disk, a cancelled job) leaves **no database at + * all** and nothing to retry from. + * + * [publishAtomically] closes that window: vacuum into `.candidate` in the + * same directory, check it, then rename it over the target in one step. Either + * the previous file or the new one is there at every instant. + */ +internal object DbPublish { + + /** Suffix of the file a publish writes before it renames it over the target. */ + const val CANDIDATE_SUFFIX = ".candidate" + + /** + * Runs [writeInto] against `$CANDIDATE_SUFFIX`, verifies what it + * produced, and ATOMIC_MOVEs it over [target]. On any failure the candidate + * is removed and the previous [target] is left exactly as it was; the + * throwable propagates so the caller fails the build. + * + * The candidate deliberately lives in the target's own directory: an + * ATOMIC_MOVE across filesystems is refused, and the copy it would fall back + * to is the very window this removes. + * + * **Space.** Holding the old file while the new one is written costs a + * second full copy, and the release runner builds on a 16 GiB tmpfs that + * already carries the ~7 GiB DB (manual-generate-release.yml, "RAM-backed + * build dir"). Turning a working build into an ENOSPC would be a worse bug + * than the one this fixes, so when there is not room for a second copy the + * publish falls back to the previous in-place behaviour and says so in the + * log. On that runner the fallback costs nothing real: build/ is a tmpfs + * that is unmounted whatever the job's outcome, so a DB lost mid-write is a + * DB that would have been discarded anyway. On a developer's disk — where + * the file IS the only copy — the atomic path is the one that runs. + */ + suspend fun publishAtomically(target: Path, logger: Logger, writeInto: suspend (Path) -> Unit): Path { + val absolute = target.toAbsolutePath() + absolute.parent?.let { Files.createDirectories(it) } + val candidate = absolute.resolveSibling(absolute.fileName.toString() + CANDIDATE_SUFFIX) + // A candidate left by a killed previous run is stale by definition. + Files.deleteIfExists(candidate) + deleteSqliteSidecars(candidate, logger) + + tooTightForACandidate(absolute)?.let { reason -> + logger.w { + "Publishing ${absolute.fileName} in place ($reason). " + + "A crash during the write therefore leaves no database at $absolute." + } + deleteSqliteSidecars(absolute, logger) + Files.deleteIfExists(absolute) + writeInto(absolute) + verifyPublishable(absolute) + logger.i { "Published ${absolute.fileName} in place (${Files.size(absolute)} bytes)" } + return absolute + } + + try { + writeInto(candidate) + verifyPublishable(candidate) + // The previous file's journal/WAL belong to the previous file. Removed + // BEFORE the rename: left beside the new DB, SQLite would replay them + // into it on the next open (it trusts a `-wal` it finds next to a DB, + // whatever the DB header says), and the phase after this one opens the + // published file straight away. Removed after the rename, a crash in + // between leaves exactly that pair on disk. + deleteSqliteSidecars(absolute, logger) + moveOver(candidate, absolute) + } catch (error: Throwable) { + runCatching { Files.deleteIfExists(candidate) } + throw error + } + logger.i { "Published ${absolute.fileName} atomically (${Files.size(absolute)} bytes)" } + return absolute + } + + private const val MIB = 1024L * 1024L + + /** + * Why a second copy of [target] does not fit beside it, or null when it does. + * + * Sized off the file being replaced (the new DB is within a few percent of + * it), plus 12.5% and 64 MiB of slack — 875 MiB of headroom at the 7 GiB + * this actually guards, and a negligible floor for a small DB. With no previous file there is + * nothing to keep alive, so the candidate costs nothing extra and always + * runs — which is also what makes the first build of a fresh checkout + * atomic. An unreadable filestore is treated as "room": a candidate that + * runs out of space fails the build with the previous DB intact, which is + * the safer way to be wrong. + */ + private fun tooTightForACandidate(target: Path): String? { + val existing = if (Files.isRegularFile(target)) Files.size(target) else 0L + if (existing == 0L) return null + val needed = existing + existing / 8 + 64 * MIB + val usable = runCatching { Files.getFileStore(target.parent).usableSpace }.getOrNull() ?: return null + if (usable >= needed) return null + return "${usable / MIB} MiB free on its filesystem, ~${needed / MIB} MiB needed to keep the current " + + "${existing / MIB} MiB file while the new one is written" + } + + /** + * The files SQLite keeps beside a database: rollback journal, WAL and its + * shared-memory index. Every Otzaria stage that writes the on-disk file + * directly opens it in WAL mode (SeforimRepository.init, GenerateHavroutaLinks), + * so a stage killed mid-run leaves `-wal`/`-shm` behind; a fresh DB + * renamed under them inherits somebody else's frames. + */ + internal val SQLITE_SIDECAR_SUFFIXES = listOf("-journal", "-wal", "-shm") + + private fun deleteSqliteSidecars(db: Path, logger: Logger) { + for (suffix in SQLITE_SIDECAR_SUFFIXES) { + val sidecar = db.resolveSibling(db.fileName.toString() + suffix) + if (Files.deleteIfExists(sidecar)) { + logger.w { "Removed stale ${sidecar.fileName} left beside ${db.fileName} by an earlier run" } + } + } + } + + private fun moveOver(candidate: Path, target: Path) { + try { + Files.move(candidate, target, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE) + } catch (_: AtomicMoveNotSupportedException) { + // Same directory, so this should not happen; if the filesystem + // refuses it anyway, a plain replace still beats delete-then-write. + Files.move(candidate, target, StandardCopyOption.REPLACE_EXISTING) + } + } + + /** + * The candidate exists, is not empty, and opens as a SQLite database that + * carries a schema. + * + * Deliberately NOT `PRAGMA integrity_check`: that is a full read of a ~7 GiB + * file on the release runner, and it is not what makes this safe — the + * atomic rename is. A torn candidate can only come from a process that died + * mid-vacuum, and such a process never reaches the rename. + */ + private fun verifyPublishable(candidate: Path) { + check(Files.isRegularFile(candidate)) { "VACUUM INTO left no file at $candidate" } + val size = Files.size(candidate) + check(size > 0) { "VACUUM INTO produced an empty file at $candidate" } + runCatching { Class.forName("org.sqlite.JDBC") } + val objects = DriverManager.getConnection("jdbc:sqlite:$candidate").use { conn -> + conn.createStatement().use { st -> + st.executeQuery("SELECT COUNT(*) FROM sqlite_master").use { rs -> + rs.next() + rs.getLong(1) + } + } + } + check(objects > 0) { "VACUUM INTO produced a database with no schema at $candidate ($size bytes)" } + } + + /** + * `-PallowEmptyBase=true` / `ALLOW_EMPTY_BASE=true`: the ONLY way to run an + * append phase without the DB it appends to. Absence of the file is not a + * mode — a phase that appends to a base and cannot find one has lost its + * input, and continuing publishes an empty DB as a success. + */ + fun allowEmptyBase(): Boolean = listOf( + System.getProperty("allowEmptyBase"), + System.getenv("ALLOW_EMPTY_BASE"), + ).firstOrNull { !it.isNullOrBlank() } + ?.let { it.equals("true", ignoreCase = true) || it == "1" } + ?: false + + /** The message every "the base is missing" abort shares. */ + fun missingBaseMessage(phase: String, baseDb: String, selector: String): String = + "$phase appends to an existing DB and refuses to create one, but no base DB exists at $baseDb. " + + "Point $selector at the database the previous phase produced, " + + "or pass -PallowEmptyBase=true (ALLOW_EMPTY_BASE=true) to accept an empty one." +} diff --git a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateHavroutaLinks.kt b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateHavroutaLinks.kt index 88fedb11..1e0e2d61 100644 --- a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateHavroutaLinks.kt +++ b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateHavroutaLinks.kt @@ -4,6 +4,7 @@ import app.cash.sqldelight.db.QueryResult import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocatorBindings import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator import io.github.kdroidfilter.seforimlibrary.core.models.ConnectionType @@ -90,17 +91,28 @@ fun main(args: Array) = runBlocking { repository.executeRawQuery("PRAGMA synchronous = NORMAL") repository.executeRawQuery("PRAGMA journal_mode = WAL") - // Persist build_state so subsequent runs preserve link ids. + // Persist build_state so subsequent runs preserve link ids. This stage + // writes straight to the on-disk DB (no VACUUM INTO), so there is no + // persist step for the snapshot to run ahead of. + val buildStateMeta = mapOf( + "generator" to "havroutalinks", + "generated_at" to java.time.Instant.now().toString(), + ) runCatching { - allocator.snapshotTo( - target = buildStatePath, - extraMeta = mapOf( - "generator" to "havroutalinks", - "generated_at" to java.time.Instant.now().toString(), - ), - ) - }.onFailure { logger.w(it) { "Failed to write build_state to $buildStatePath" } } - Unit + allocator.snapshotTo(target = buildStatePath, extraMeta = buildStateMeta) + }.onFailure { e -> + // Fail closed: a build that cannot write its allocator state would + // publish last week's — and the build after it would re-issue ids + // this one already handed out. + logger.e(e) { "Failed to write build_state to $buildStatePath" } + throw e + } + BuildStateVerifier.verifyFreshSnapshot( + buildStatePath = buildStatePath, + dbPath = Paths.get(dbPath), + expectedMeta = buildStateMeta, + logger = logger, + ) } catch (e: Exception) { logger.e(e) { "Error generating Havrouta links" } throw e @@ -174,8 +186,12 @@ private fun isSectionHeader(content: String): Boolean { /** * Generates links between Havrouta books and their corresponding Talmud tractates. + * + * `internal` rather than private so the found-vs-processed accounting can be + * tested: the audited build found 38 Havrouta books and processed 37 without + * saying which one it dropped or why. */ -private suspend fun generateHavroutaLinks( +internal suspend fun generateHavroutaLinks( repository: SeforimRepository, bindings: IdAllocatorBindings, logger: Logger @@ -206,16 +222,28 @@ private suspend fun generateHavroutaLinks( } logger.i { "Deleted existing Havrouta links" } + // "Found N Havrouta books" followed by fewer "Processing:" lines used to be + // the only trace that a book was dropped; the reason is named per book below + // and the gap is closed by an explicit N/M line after the loop. + var processedBooks = 0 + val unmatched = mutableListOf() + for (havroutaBook in havroutaBooks) { val tractateName = havroutaBook.title.removePrefix("חברותא על ") val talmudTractateName = tractateNameMapping[tractateName] ?: tractateName val talmudBook = talmudBooks[talmudTractateName] if (talmudBook == null) { - logger.w { "No Talmud match for: ${havroutaBook.title}" } + unmatched += havroutaBook.title + logger.w { + "No Talmud match for: ${havroutaBook.title} — no book titled '$talmudTractateName' " + + "among the Bavli tractates (sourceId=1, excluding משנה / תלמוד ירושלמי / תוספתא); " + + "no links created for it" + } continue } + processedBooks++ logger.i { "Processing: ${havroutaBook.title} -> ${talmudBook.title}" } val linksForBook = processBookPair( @@ -233,6 +261,15 @@ private suspend fun generateHavroutaLinks( totalLinksCreated += linksForBook } + if (unmatched.isEmpty()) { + logger.i { "Havrouta-Talmud: $processedBooks/${havroutaBooks.size} books processed" } + } else { + logger.w { + "Havrouta-Talmud: $processedBooks/${havroutaBooks.size} books processed, " + + "${unmatched.size} skipped with no matching Talmud tractate: ${unmatched.joinToString()}" + } + } + // Update book_has_links table updateBookHasLinks(repository, logger) diff --git a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLines.kt b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLines.kt index 26498ffb..d91b5dc5 100644 --- a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLines.kt +++ b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLines.kt @@ -6,6 +6,7 @@ import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity import io.github.kdroidfilter.seforimlibrary.common.db.SEFORIM_DB_PAGE_SIZE_PRAGMA +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier import io.github.kdroidfilter.seforimlibrary.common.buildstate.IdTable import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository @@ -70,8 +71,20 @@ fun main(args: Array) = runBlocking { val dbFile = File(dbPath) if (dbFile.exists()) { logger.i { "Appending to existing DB at ${dbFile.absolutePath}" } + } else if (DbPublish.allowEmptyBase()) { + logger.w { "appendExistingDb enabled but no DB found at $dbPath; -PallowEmptyBase is set, a new DB will be created" } } else { - logger.i { "appendExistingDb enabled but no DB found at $dbPath; a new DB will be created" } + // Same rule as the in-memory seed below: appendExistingDb is an + // explicit opt-in, so a DB that is not there is a lost input, not a + // first build. Thrown before the driver opens — and thereby creates — + // the file, so nothing is left at dbPath. + throw IllegalStateException( + DbPublish.missingBaseMessage( + "phase 1 (lines) with appendExistingDb", + dbFile.absolutePath, + "-PseforimDb / SEFORIM_DB", + ) + ) } } @@ -96,6 +109,8 @@ fun main(args: Array) = runBlocking { val baseFile = File(baseDbPath) if (baseFile.exists()) { logger.i { "Seeding in-memory DB from base file: ${baseFile.absolutePath}" } + // Names the table the copy died on; see the fail-closed note below. + var copyingTable: String? = null runCatching { repository.executeRawQuery("PRAGMA foreign_keys=OFF") val escaped = baseFile.absolutePath.replace("'", "''") @@ -113,20 +128,57 @@ fun main(args: Array) = runBlocking { 0 ).value for (t in tables) { + copyingTable = t repository.executeRawQuery("DELETE FROM \"$t\"") repository.executeRawQuery("INSERT INTO \"$t\" SELECT * FROM disk.\"$t\"") } + copyingTable = null repository.executeRawQuery("DETACH DATABASE disk") repository.executeRawQuery("PRAGMA foreign_keys=ON") logger.i { "Seeding completed. Imported ${tables.size} tables." } }.onFailure { e -> - logger.e(e) { "Failed to seed in-memory DB from $baseDbPath; continuing with empty DB." } + // Fail closed. In the release path baseDb and persistDb are the + // SAME file (build.gradle.kts:appendOtzariaLines), so "continuing + // with empty DB" meant the VACUUM INTO below would delete the DB + // this run produced and replace it with an empty one — reported + // as a success. A half-copied seed is just as fatal. + val where = copyingTable?.let { " while copying table \"$it\"" } ?: "" + logger.e(e) { "Failed to seed in-memory DB from $baseDbPath$where; aborting phase 1." } + // This seed runs BEFORE the try/finally below, which is what + // would otherwise close the repository on the way out. + runCatching { repository.close() } + throw e } + } else if (DbPublish.allowEmptyBase()) { + logger.w { "appendExistingDb enabled but base DB not found at $baseDbPath; -PallowEmptyBase is set, starting from an empty in-memory DB" } } else { - logger.w { "appendExistingDb enabled but base DB not found at $baseDbPath; starting from empty in-memory DB" } + // appendExistingDb is an EXPLICIT opt-in (property/env, default + // false): a first-ever build simply does not set it and takes + // the rotate path above. Once it IS set, the base is this run's + // input, and since baseDb == persistDb in the release path + // (build.gradle.kts:appendOtzariaLines) continuing meant + // vacuuming an empty DB over the target and exiting 0. + runCatching { repository.close() } + throw IllegalStateException( + DbPublish.missingBaseMessage( + "phase 1 (lines) with appendExistingDb", + baseDbPath, + "-PbaseDb / SEFORIM_DB_BASE / -PseforimDb", + ) + ) } } else { - logger.w { "appendExistingDb enabled in-memory but no base DB path provided; starting from empty DB" } + if (!DbPublish.allowEmptyBase()) { + runCatching { repository.close() } + throw IllegalStateException( + DbPublish.missingBaseMessage( + "phase 1 (lines) with appendExistingDb", + baseDbPath, + "-PbaseDb / SEFORIM_DB_BASE / -PseforimDb", + ) + ) + } + logger.w { "appendExistingDb enabled in-memory but no base DB path provided; -PallowEmptyBase is set, starting from an empty DB" } } } @@ -165,17 +217,15 @@ fun main(args: Array) = runBlocking { ) generator.generateLinesOnly() if (useMemoryDb) { - // Persist in-memory DB to disk using VACUUM INTO (target must not exist) + // VACUUM INTO a candidate beside the target, then rename it over the + // target in one step — never delete-then-write, which left no DB at + // all if the process died in between (see [DbPublish]). runCatching { - val outFile = File(persistDbPath) - outFile.parentFile?.mkdirs() - if (outFile.exists()) { - outFile.delete() - logger.i { "Existing DB removed to allow VACUUM INTO" } - } - val escaped = persistDbPath.replace("'", "''") logger.i { "Persisting in-memory DB to $persistDbPath via VACUUM INTO..." } - repository.executeRawQuery("VACUUM INTO '$escaped'") + DbPublish.publishAtomically(Paths.get(persistDbPath), logger) { candidate -> + val escaped = candidate.toString().replace("'", "''") + repository.executeRawQuery("VACUUM INTO '$escaped'") + } logger.i { "In-memory DB persisted to $persistDbPath" } }.onFailure { e -> logger.e(e) { "Failed to persist in-memory DB to $persistDbPath" } @@ -183,16 +233,27 @@ fun main(args: Array) = runBlocking { } } // Persist build_state so subsequent phases/builds reuse the same ids. + // Written after the persist above on purpose: a failed VACUUM INTO must + // not leave an advanced buildstate beside a DB that was never written. + val buildStateMeta = mapOf( + "generator" to "otzariasqlite/generateLines", + "generated_at" to java.time.Instant.now().toString(), + ) runCatching { - allocator.snapshotTo( - target = buildStatePath, - extraMeta = mapOf( - "generator" to "otzariasqlite/generateLines", - "generated_at" to java.time.Instant.now().toString(), - ), - ) - }.onFailure { logger.w(it) { "Failed to write build_state to $buildStatePath" } } - Unit + allocator.snapshotTo(target = buildStatePath, extraMeta = buildStateMeta) + }.onFailure { e -> + // Fail closed: a build that cannot write its allocator state would + // publish last week's — and the build after it would re-issue ids + // this one already handed out. + logger.e(e) { "Failed to write build_state to $buildStatePath" } + throw e + } + BuildStateVerifier.verifyFreshSnapshot( + buildStatePath = buildStatePath, + dbPath = Paths.get(if (useMemoryDb) persistDbPath else dbPath), + expectedMeta = buildStateMeta, + logger = logger, + ) logger.i { "Phase 1 completed successfully. DB at ${if (useMemoryDb) persistDbPath else dbPath}" } } catch (e: Exception) { logger.e(e) { "Error during phase 1 generation" } diff --git a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLinks.kt b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLinks.kt index ac5976cd..dc59323e 100644 --- a/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLinks.kt +++ b/generator/otzariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GenerateLinks.kt @@ -5,6 +5,7 @@ import app.cash.sqldelight.db.SqlCursor import app.cash.sqldelight.db.QueryResult import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier import io.github.kdroidfilter.seforimlibrary.common.db.SEFORIM_DB_PAGE_SIZE_PRAGMA import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository @@ -56,6 +57,8 @@ fun main(args: Array) = runBlocking { val baseFile = java.io.File(baseDb) if (baseFile.exists()) { logger.i { "Seeding in-memory DB from base file: $baseDb" } + // Names the table the copy died on; see the fail-closed note below. + var copyingTable: String? = null runCatching { repository.executeRawQuery("PRAGMA foreign_keys=OFF") val escaped = baseDb.replace("'", "''") @@ -74,17 +77,37 @@ fun main(args: Array) = runBlocking { // Copy data for each table into main for (t in tables) { val tn = t + copyingTable = tn repository.executeRawQuery("DELETE FROM \"$tn\"") repository.executeRawQuery("INSERT INTO \"$tn\" SELECT * FROM disk.\"$tn\"") } + copyingTable = null repository.executeRawQuery("DETACH DATABASE disk") repository.executeRawQuery("PRAGMA foreign_keys=ON") - logger.i { "Seeding completed. Imported ${'$'}{tables.size} tables." } + logger.i { "Seeding completed. Imported ${tables.size} tables." } }.onFailure { e -> - logger.e(e) { "Failed to seed in-memory DB from $baseDb. Links may not be processed." } + // Fail closed. In the release path baseDb and persistDb are the + // SAME file (build.gradle.kts:appendOtzariaLinks), so continuing + // here would let the VACUUM INTO below delete the 7 GiB DB this + // run produced and replace it with an empty one — reported as a + // success. A half-copied seed is just as fatal: the tables + // enumerated before the failure are populated, the rest empty. + val where = copyingTable?.let { " while copying table \"$it\"" } ?: "" + logger.e(e) { "Failed to seed in-memory DB from $baseDb$where; aborting phase 2." } + throw e } + } else if (DbPublish.allowEmptyBase()) { + logger.w { "Base DB not found at $baseDb; -PallowEmptyBase is set, so phase 2 starts from an empty DB" } } else { - logger.w { "Base DB not found at $baseDb; running with empty in-memory DB" } + // Fail BEFORE anything touches the target. The absence of the + // base used to be a warning, and since baseDb == persistDb in + // the release path (build.gradle.kts:appendOtzariaLinks) the + // run then vacuumed an empty in-memory DB over the target and + // exited 0 — an empty database published as a success. Only the + // explicit opt-in above may do that. + throw IllegalStateException( + DbPublish.missingBaseMessage("phase 2 (links)", baseDb, "-PbaseDb / SEFORIM_DB_BASE") + ) } } @@ -103,38 +126,44 @@ fun main(args: Array) = runBlocking { allocator = allocator, ) generator.generateLinksOnly() - runCatching { - allocator.snapshotTo( - target = buildStatePath, - extraMeta = mapOf( - "generator" to "otzariasqlite/generateLinks", - "generated_at" to java.time.Instant.now().toString(), - ), - ) - }.onFailure { logger.w(it) { "Failed to write build_state to $buildStatePath" } } - Unit if (useMemoryDb) { - // Persist in-memory DB to disk using VACUUM INTO (target must not exist) + // VACUUM INTO a candidate beside the target, then rename it over the + // target in one step — never delete-then-write, which left no DB at + // all if the process died in between (see [DbPublish]). runCatching { - val outFile = java.io.File(persistDbPath) - outFile.parentFile?.mkdirs() - if (outFile.exists()) { - // No backup required: remove existing file to allow VACUUM INTO - val deleted = runCatching { java.nio.file.Files.deleteIfExists(outFile.toPath()) }.getOrDefault(false) - if (!deleted) { - throw IllegalStateException("Cannot remove existing DB at ${outFile.absolutePath} before persisting") - } - logger.i { "Removed existing DB at ${outFile.absolutePath}" } - } - val escaped = persistDbPath.replace("'", "''") logger.i { "Persisting in-memory DB to $persistDbPath via VACUUM INTO..." } - repository.executeRawQuery("VACUUM INTO '$escaped'") + DbPublish.publishAtomically(Paths.get(persistDbPath), logger) { candidate -> + val escaped = candidate.toString().replace("'", "''") + repository.executeRawQuery("VACUUM INTO '$escaped'") + } logger.i { "In-memory DB persisted to $persistDbPath" } }.onFailure { e -> logger.e(e) { "Failed to persist in-memory DB to $persistDbPath" } throw e } } + // Persist build_state AFTER the DB itself is on disk (this is also the + // order GenerateLines uses): written first, a failed VACUUM INTO would + // leave an advanced buildstate beside a DB that was never written. + val buildStateMeta = mapOf( + "generator" to "otzariasqlite/generateLinks", + "generated_at" to java.time.Instant.now().toString(), + ) + runCatching { + allocator.snapshotTo(target = buildStatePath, extraMeta = buildStateMeta) + }.onFailure { e -> + // Fail closed: a build that cannot write its allocator state would + // publish last week's — and the build after it would re-issue ids + // this one already handed out. + logger.e(e) { "Failed to write build_state to $buildStatePath" } + throw e + } + BuildStateVerifier.verifyFreshSnapshot( + buildStatePath = buildStatePath, + dbPath = Paths.get(if (useMemoryDb) persistDbPath else dbPath), + expectedMeta = buildStateMeta, + logger = logger, + ) logger.i { "Phase 2 completed successfully. Links processed. DB at ${if (useMemoryDb) persistDbPath else dbPath}" } } catch (e: Exception) { logger.e(e) { "Error during phase 2 (links)" } diff --git a/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgressTest.kt b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgressTest.kt new file mode 100644 index 00000000..eef5f47e --- /dev/null +++ b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/GeneratorProgressTest.kt @@ -0,0 +1,289 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertNotNull +import kotlin.test.assertNull +import kotlin.test.assertTrue + +/** + * Cadence + formatting of the generator's progress output. + * + * These gates replace ~18k INFO lines per build with a throttled progress line + * and a summary, so their behaviour is pinned here against a fake clock rather + * than against a 35-minute build. + */ +class GeneratorProgressTest { + + /** Fake clock: advances only when the test says so. */ + private class FakeClock(var nowMs: Long = 0L) { + val reader: () -> Long = { nowMs } + fun advance(ms: Long) { nowMs += ms } + } + + // ─── formatters ──────────────────────────────────────────────────────── + + @Test + fun formatCountIsCompactAndLocaleIndependent() { + assertEquals("0", formatCount(0)) + assertEquals("934", formatCount(934)) + assertEquals("9999", formatCount(9_999)) + assertEquals("10.0K", formatCount(10_000)) + assertEquals("372.9K", formatCount(372_904)) + assertEquals("999.9K", formatCount(999_999)) + assertEquals("1.0M", formatCount(1_000_000)) + assertEquals("5.2M", formatCount(5_234_567)) + assertEquals("2.2M", formatCount(2_250_736)) + } + + @Test + fun formatDurationUsesMmSsBelowAnHourAndHMmSsAbove() { + assertEquals("00:00", formatDuration(0)) + assertEquals("00:00", formatDuration(-5)) + assertEquals("00:09", formatDuration(9_400)) + assertEquals("06:00", formatDuration(360_000)) + assertEquals("27:10", formatDuration(1_630_000)) + assertEquals("59:59", formatDuration(3_599_000)) + assertEquals("1:00:00", formatDuration(3_600_000)) + assertEquals("2:05:03", formatDuration(7_503_000)) + } + + @Test + fun formatPercentKeepsOneDecimalAndToleratesUnknownTotals() { + assertEquals("82.2", formatPercent(1234, 1501)) + assertEquals("0.0", formatPercent(0, 1501)) + assertEquals("100.0", formatPercent(1501, 1501)) + assertEquals("?", formatPercent(3, 0)) + assertEquals("?", formatPercent(3, -1)) + } + + // ─── book progress cadence ───────────────────────────────────────────── + + @Test + fun bookProgressEmitsEveryNBooksAndNotInBetween() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 100, everyMs = 60_000) + reporter.start(totalBooks = 1501) + + val emitted = mutableListOf() + repeat(250) { reporter.onBookFinished(lines = 10)?.let(emitted::add) } + + // 100th and 200th book only — 248 of the 250 books stay silent. + assertEquals(2, emitted.size) + assertTrue(emitted[0].startsWith("books 100/1501 (6.6%)"), emitted[0]) + assertTrue(emitted[1].startsWith("books 200/1501 (13.3%)"), emitted[1]) + } + + @Test + fun bookProgressAlsoEmitsOnTheTimeCadenceForSlowBooks() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 100, everyMs = 60_000) + reporter.start(totalBooks = 1000) + + // 10 very slow books: never 100 books, but each crosses a minute. + val emitted = mutableListOf() + repeat(10) { + clock.advance(61_000) + reporter.onBookFinished(lines = 1)?.let(emitted::add) + } + assertEquals(10, emitted.size) + + // The time budget resets on every emission: fast books stay silent again. + val quiet = (1..50).mapNotNull { reporter.onBookFinished(lines = 1) } + assertTrue(quiet.isEmpty(), "expected silence, got $quiet") + } + + @Test + fun bookProgressLineCarriesCountersElapsedAndEta() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 1000, everyMs = 0) + reporter.start(totalBooks = 1501) + repeat(1234) { + clock.advance(1_320) // 1234 books ≈ 27:08 + reporter.onBookFinished(lines = 4_216, tocEntries = 244) + } + val line = reporter.progressLine() + assertEquals( + "books 1234/1501 (82.2%) · lines 5.2M · toc 301.0K · elapsed 27:08 · eta ~05:52", + line, + ) + } + + @Test + fun bookProgressOmitsEtaWhenTotalIsUnknownOrComplete() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 1, everyMs = 0) + + reporter.start(totalBooks = 0) + clock.advance(1_000) + val unknown = assertNotNull(reporter.onBookFinished(lines = 5)) + assertEquals("books 1/0 (?%) · lines 5 · toc 0 · elapsed 00:01", unknown) + + reporter.start(totalBooks = 2) + clock.advance(1_000) + reporter.onBookFinished(lines = 5) + clock.advance(1_000) + val complete = assertNotNull(reporter.onBookFinished(lines = 5)) + assertEquals("books 2/2 (100.0%) · lines 10 · toc 0 · elapsed 00:02", complete) + } + + @Test + fun summaryLineCarriesExactCountersAndExtrasInOrder() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 100_000, everyMs = 0) + reporter.start(totalBooks = 1501) + reporter.onBookFinished(lines = 2_250_736, tocEntries = 372_904) + clock.advance(211_000) + + assertEquals( + "Otzaria book import (phase 1): books=1/1501 lines=2250736 tocEntries=372904 " + + "directories=270 skipped=66 acronymBooks=715 hearotMerged=60 elapsed=03:31", + reporter.summaryLine( + "Otzaria book import (phase 1)", + mapOf( + "directories" to 270L, + "skipped" to 66L, + "acronymBooks" to 715L, + "hearotMerged" to 60L, + ), + ), + ) + } + + @Test + fun startResetsCountersBetweenPhases() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 100_000, everyMs = 0) + reporter.start(totalBooks = 10) + reporter.onBookFinished(lines = 7, tocEntries = 3) + clock.advance(5_000) + + reporter.start(totalBooks = 20) + assertEquals(0, reporter.booksDone) + assertEquals(0L, reporter.linesInserted) + assertEquals(0L, reporter.tocEntries) + assertEquals(0L, reporter.elapsedMs()) + assertEquals(20, reporter.totalBooks) + } + + // ─── per-book line ticks ─────────────────────────────────────────────── + + @Test + fun lineTicksNeverFireAtIndexZero() { + // The old `lineIndex % 1000 == 0` gate emitted `0/N (0%)` for every + // single book — 1,496 of 3,239 ticks in the audited build. + val clock = FakeClock() + val small = LineTickGate(totalLines = 869, nowMs = clock.reader) + val large = LineTickGate(totalLines = 190_760, nowMs = clock.reader) + assertFalse(small.shouldTick(0)) + assertFalse(large.shouldTick(0)) + } + + @Test + fun smallBooksNeverTick() { + val clock = FakeClock() + val gate = LineTickGate(totalLines = 5_000, nowMs = clock.reader) + assertFalse(gate.enabled) + val ticks = (0 until 5_000).count { gate.shouldTick(it) } + assertEquals(0, ticks) + } + + @Test + fun largeBooksTickOnThousandsAtLeastTenPercentApart() { + val clock = FakeClock() + val gate = LineTickGate(totalLines = 50_000, nowMs = clock.reader) + assertTrue(gate.enabled) + val ticked = (0 until 50_000).filter { gate.shouldTick(it) } + // 10% of 50k is 5k: ticks at 5000, 10000, … 45000 — nine, not fifty. + assertEquals(listOf(5_000, 10_000, 15_000, 20_000, 25_000, 30_000, 35_000, 40_000, 45_000), ticked) + } + + @Test + fun justAboveTheThresholdTicksOnEveryThousand() { + // 10% of 5,001 is below the 1,000 step, so the step is the floor. + val clock = FakeClock() + val gate = LineTickGate(totalLines = 5_001, nowMs = clock.reader) + val ticked = (0 until 5_001).filter { gate.shouldTick(it) } + assertEquals(listOf(1_000, 2_000, 3_000, 4_000, 5_000), ticked) + } + + @Test + fun aSlowBookStillTicksOnTheTimeCadence() { + val clock = FakeClock() + val gate = LineTickGate(totalLines = 100_000, nowMs = clock.reader, everyMs = 60_000) + // 10% is 10k lines, but a minute passes every 1,000 lines. + val ticked = (0 until 20_000).filter { index -> + if (index % 1_000 == 0) clock.advance(61_000) + gate.shouldTick(index) + } + assertEquals(19, ticked.size) + assertEquals(1_000, ticked.first()) + assertEquals(19_000, ticked.last()) + } + + @Test + fun tickLineKeepsTheHistoricalFormat() { + val gate = LineTickGate(totalLines = 190_760, nowMs = { 0L }) + assertEquals( + "Book 5848 'דרך החיים': 19000/190760 lines (9%)", + gate.tickLine(5848L, "דרך החיים", 19_000), + ) + } + + @Test + fun aZeroLineBookIsSilentAndNeverDividesByZero() { + val gate = LineTickGate(totalLines = 0, nowMs = { 0L }) + assertFalse(gate.enabled) + assertFalse(gate.shouldTick(0)) + assertEquals("Book 1 'X': 0/0 lines (0%)", gate.tickLine(1L, "X", 0)) + } + + // ─── whole-corpus behaviour ──────────────────────────────────────────── + + @Test + fun aWholeCorpusOfBooksTicksOnlyForTheLargeOnes() { + // Shape only, not the audited run's exact sizes: a long tail of small + // books (the real median is 414 lines) plus a handful of large ones. + // The measured cut on the real distribution of run 34024655297 — + // 3,239 ticks down to 507 — is recorded in the task notes. + val sizes = (1..1_400).map { 100 + (it * 7) % 4_500 } + + listOf(190_760, 137_000, 91_000, 60_000, 25_000, 12_000, 8_000, 5_400, 5_001) + + val oldTicks = sizes.sumOf { n -> (n - 1) / 1_000 + 1 } + val newTicks = sizes.sumOf { n -> + val gate = LineTickGate(totalLines = n, nowMs = { 0L }) + (0 until n).count { gate.shouldTick(it) } + } + assertTrue(newTicks * 6 < oldTicks, "expected a >6x cut, got $oldTicks -> $newTicks") + + // Every book used to emit `0/N (0%)`; none does now. + assertEquals(0, sizes.count { n -> LineTickGate(totalLines = n, nowMs = { 0L }).shouldTick(0) }) + + // Books at or below the threshold are completely silent. + assertEquals( + 0, + sizes.filter { it <= LINE_TICK_MIN_BOOK_LINES }.sumOf { n -> + val gate = LineTickGate(totalLines = n, nowMs = { 0L }) + (0 until n).count { gate.shouldTick(it) } + }, + ) + } + + @Test + fun fifteenHundredBooksProduceFifteenProgressLinesNotFifteenHundred() { + val clock = FakeClock() + val reporter = BookProgressReporter(nowMs = clock.reader, everyNBooks = 100, everyMs = 60_000) + reporter.start(totalBooks = 1_501) + // 1,501 books over the ~211 s the book loop took in run 34024655297. + val emitted = (1..1_501).mapNotNull { + clock.advance(140) + reporter.onBookFinished(lines = 1_500, tocEntries = 249) + } + // The count cadence carries it: no book here takes a minute. + assertEquals(15, emitted.size) + assertEquals("books 100/1501 (6.6%)", emitted.first().substringBefore(" · ")) + assertEquals("books 1500/1501 (99.9%)", emitted.last().substringBefore(" · ")) + assertNull(reporter.onBookFinished(lines = 0)) + } +} diff --git a/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/HavroutaUnmatchedTractateTest.kt b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/HavroutaUnmatchedTractateTest.kt new file mode 100644 index 00000000..c6e2b6ad --- /dev/null +++ b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/HavroutaUnmatchedTractateTest.kt @@ -0,0 +1,96 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver +import co.touchlab.kermit.LogWriter +import co.touchlab.kermit.Logger +import co.touchlab.kermit.Severity +import co.touchlab.kermit.StaticConfig +import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocatorBindings +import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import io.github.kdroidfilter.seforimlibrary.core.models.Book +import io.github.kdroidfilter.seforimlibrary.core.models.Category +import io.github.kdroidfilter.seforimlibrary.core.models.ConnectionType +import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository +import io.github.kdroidfilter.seforimlibrary.db.SeforimDb +import kotlinx.coroutines.runBlocking +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * `Found 38 Havrouta books` followed by 37 `Processing:` lines was the only + * trace that one book had been dropped — no name, no reason, no closing count. + * The gap is now stated outright and the drop carries its cause. + */ +class HavroutaUnmatchedTractateTest { + + private class Capture : LogWriter() { + val lines = mutableListOf>() + override fun log(severity: Severity, message: String, tag: String, throwable: Throwable?) { + lines += severity to message + } + } + + private fun run(havroutaTitles: List, talmudTitles: List): Capture = runBlocking { + val driver = JdbcSqliteDriver(url = "jdbc:sqlite::memory:") + SeforimDb.Schema.create(driver) + val repo = SeforimRepository(":memory:", driver) + val bavli = repo.insertSource("Sefaria") + val otzaria = repo.insertSource("Otzaria") + val catId = repo.insertCategory(Category(0, null, "תלמוד", level = 0, order = 1)) + // The tractate filter keys on sourceId == 1, which is the first source row. + assertEquals(1L, bavli, "the Talmud side is selected by sourceId == 1") + + talmudTitles.forEach { + repo.insertBook(Book(categoryId = catId, sourceId = bavli, title = it, heRef = it)) + } + havroutaTitles.forEach { + repo.insertBook(Book(categoryId = catId, sourceId = otzaria, title = it, heRef = it)) + } + + val bindings = IdAllocatorBindings(InMemoryIdAllocator.load(path = null), repo) + ConnectionType.entries.forEach { bindings.upsertConnectionType(it.name) } + + val capture = Capture() + generateHavroutaLinks(repo, bindings, Logger(StaticConfig(Severity.Verbose, listOf(capture)), "t")) + capture + } + + @Test + fun `a Havrouta book with no matching tractate is named, with its reason, and counted`() { + val capture = run( + havroutaTitles = listOf("חברותא על ברכות", "חברותא על מסכת שאיננה"), + talmudTitles = listOf("ברכות"), + ) + val warnings = capture.lines.filter { it.first == Severity.Warn }.map { it.second } + + val perBook = warnings.single { it.startsWith("No Talmud match for:") } + assertContains(perBook, "חברותא על מסכת שאיננה") + assertContains(perBook, "no book titled 'מסכת שאיננה' among the Bavli tractates") + assertContains(perBook, "no links created for it") + + assertEquals( + "Havrouta-Talmud: 1/2 books processed, 1 skipped with no matching Talmud tractate: " + + "חברותא על מסכת שאיננה", + warnings.single { it.startsWith("Havrouta-Talmud:") }, + "the found-vs-processed gap must be closed explicitly, not left to be inferred", + ) + } + + @Test + fun `a fully matched run closes the count at INFO with nothing to warn about`() { + val capture = run( + havroutaTitles = listOf("חברותא על ברכות"), + talmudTitles = listOf("ברכות"), + ) + assertEquals( + listOf("Havrouta-Talmud: 1/1 books processed"), + capture.lines.filter { it.second.startsWith("Havrouta-Talmud:") }.map { it.second }, + ) + assertTrue( + capture.lines.none { it.first == Severity.Warn }, + "nothing to warn about: ${capture.lines.filter { it.first == Severity.Warn }}", + ) + } +} diff --git a/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaBuildFailClosedTest.kt b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaBuildFailClosedTest.kt new file mode 100644 index 00000000..416bbc9b --- /dev/null +++ b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaBuildFailClosedTest.kt @@ -0,0 +1,466 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import co.touchlab.kermit.Logger +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier +import io.github.kdroidfilter.seforimlibrary.common.buildstate.IdTable +import kotlinx.coroutines.runBlocking +import java.lang.reflect.InvocationTargetException +import java.nio.file.Files +import java.nio.file.Path +import java.sql.DriverManager +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertContentEquals +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * The Otzaria phases used to keep going after a failed seed and after a failed + * build_state write, and to report success either way. + * + * That matters because `appendOtzariaLines` / `appendOtzariaLinks` pass the SAME + * path as `baseDb` and `persistDb` (build.gradle.kts): the seed reads + * build/seforim.db and the VACUUM INTO at the end of the phase deletes and + * rewrites it. A swallowed seed failure therefore replaced the DB the run had + * just produced with an empty one — and a swallowed snapshot failure published + * the previous release's id allocator state beside it. + * + * These drive the real entry points (through their file facades, since all three + * declare `main` in this package) and assert the phase now stops. + */ +class OtzariaBuildFailClosedTest { + + private val savedProperties = mutableMapOf() + private val previousSeverity = Logger.config.minSeverity + + @AfterTest + fun restoreGlobals() { + savedProperties.forEach { (key, value) -> + if (value == null) System.clearProperty(key) else System.setProperty(key, value) + } + savedProperties.clear() + Logger.setMinSeverity(previousSeverity) + } + + private fun setProperty(key: String, value: String) { + if (key !in savedProperties) savedProperties[key] = System.getProperty(key) + System.setProperty(key, value) + } + + /** Runs a generator entry point; unwraps the reflective invocation. */ + private fun runGenerator(facade: String) { + val main = Class.forName("io.github.kdroidfilter.seforimlibrary.otzariasqlite.$facade") + .getMethod("main", Array::class.java) + try { + main.invoke(null, arrayOf() as Any) + } catch (e: InvocationTargetException) { + throw e.targetException + } + } + + /** + * Pins the abort to the ATTACH of the corrupt base — not to some later step + * that happens to fail too, which would make these tests pass on the code + * that swallowed the seed failure. + */ + private fun assertSeedFailure(failure: Throwable) { + val text = generateSequence(failure) { it.cause }.mapNotNull { it.message }.joinToString(" | ") + assertTrue( + text.contains("not a database", ignoreCase = true), + "expected the corrupt base DB to be the cause, got: $text", + ) + } + + private fun corruptDb(dir: Path): Path { + val file = dir.resolve("corrupt-base.db") + Files.write(file, "this is emphatically not a SQLite database".repeat(8).toByteArray()) + return file + } + + /** A stand-in for the DB a previous phase produced: three books, on disk. */ + private fun populatedTargetDb(dir: Path): Path { + val file = dir.resolve("seforim.db") + Class.forName("org.sqlite.JDBC") + DriverManager.getConnection("jdbc:sqlite:${file.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeUpdate("CREATE TABLE book (id INTEGER PRIMARY KEY NOT NULL, title TEXT)") + st.executeUpdate("INSERT INTO book(id, title) VALUES (1, 'א'), (2, 'ב'), (3, 'ג')") + } + } + return file + } + + private fun countBooks(db: Path): Long { + DriverManager.getConnection("jdbc:sqlite:${db.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeQuery("SELECT COUNT(*) FROM book").use { rs -> + rs.next() + return rs.getLong(1) + } + } + } + } + + private fun maxId(db: Path, table: String): Long { + DriverManager.getConnection("jdbc:sqlite:${db.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeQuery("SELECT COALESCE(MAX(id), 0) FROM \"$table\"").use { rs -> + rs.next() + return rs.getLong(1) + } + } + } + } + + // ─── (1) seed fail-closed ────────────────────────────────────────────── + + @Test + fun `phase 2 aborts on an unreadable base DB instead of overwriting the target`() { + val dir = Files.createTempDirectory("s13-links-seed") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + + setProperty("seforimDb", ":memory:") + setProperty("persistDb", target.toAbsolutePath().toString()) + setProperty("baseDb", corruptDb(dir).toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("buildStatePath", dir.resolve("seforim.db.buildstate").toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateLinksKt") } + assertSeedFailure(failure) + + assertContentEquals(before, Files.readAllBytes(target), "the target DB must survive a failed seed") + assertEquals(3, countBooks(target)) + assertTrue(Files.notExists(dir.resolve("seforim.db.buildstate")), "no buildstate for a phase that failed") + } + + @Test + fun `phase 1 aborts on an unreadable base DB instead of overwriting the target`() { + val dir = Files.createTempDirectory("s13-lines-seed") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + + setProperty("seforimDb", ":memory:") + setProperty("appendExistingDb", "true") + setProperty("persistDb", target.toAbsolutePath().toString()) + setProperty("baseDb", corruptDb(dir).toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("acronymDb", dir.resolve("acronymizer.db").toAbsolutePath().toString()) + setProperty("buildStatePath", dir.resolve("seforim.db.buildstate").toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateLinesKt") } + assertSeedFailure(failure) + + assertContentEquals(before, Files.readAllBytes(target), "the target DB must survive a failed seed") + assertEquals(3, countBooks(target)) + } + + // ─── (1b) a MISSING base is fail-closed too ──────────────────────────── + // + // The absence of the base used to be a WARN and the phase continued: with + // baseDb == persistDb (the release path) that vacuumed an empty in-memory DB + // over the target and exited 0 — an empty database published as a success. + + private fun assertNamesTheMissingBase(failure: Throwable, missing: Path) { + val text = generateSequence(failure) { it.cause }.mapNotNull { it.message }.joinToString(" | ") + assertTrue(text.contains(missing.toAbsolutePath().toString()), "the abort must name the base path, got: $text") + assertTrue(text.contains("allowEmptyBase"), "the abort must name the opt-in that overrides it, got: $text") + } + + @Test + fun `phase 2 refuses a missing base DB before it touches the target`() { + val dir = Files.createTempDirectory("s16-links-missing-base") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + val missing = dir.resolve("nowhere").resolve("base.db") + + setProperty("seforimDb", ":memory:") + setProperty("persistDb", target.toAbsolutePath().toString()) + setProperty("baseDb", missing.toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("buildStatePath", dir.resolve("seforim.db.buildstate").toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateLinksKt") } + assertNamesTheMissingBase(failure, missing) + + assertContentEquals(before, Files.readAllBytes(target), "the target must be byte-identical") + assertEquals(3, countBooks(target)) + assertTrue(Files.notExists(dir.resolve("seforim.db.buildstate")), "no buildstate for a phase that failed") + assertNoCandidateBeside(target) + } + + @Test + fun `phase 1 with appendExistingDb refuses a missing base DB before it touches the target`() { + val dir = Files.createTempDirectory("s16-lines-missing-base") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + val missing = dir.resolve("nowhere").resolve("base.db") + + setProperty("seforimDb", ":memory:") + setProperty("appendExistingDb", "true") + setProperty("persistDb", target.toAbsolutePath().toString()) + setProperty("baseDb", missing.toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("acronymDb", dir.resolve("acronymizer.db").toAbsolutePath().toString()) + setProperty("buildStatePath", dir.resolve("seforim.db.buildstate").toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateLinesKt") } + assertNamesTheMissingBase(failure, missing) + + assertContentEquals(before, Files.readAllBytes(target), "the target must be byte-identical") + assertEquals(3, countBooks(target)) + assertTrue(Files.notExists(dir.resolve("seforim.db.buildstate")), "no buildstate for a phase that failed") + assertNoCandidateBeside(target) + } + + @Test + fun `phase 1 on disk with appendExistingDb refuses a missing DB before the driver creates it`() { + // The on-disk append path has no seed and no VACUUM INTO: the JDBC driver + // simply creates an empty file at dbPath and the phase runs on it. Same + // explicit flag, same rule — and the check has to come before the driver + // opens the URL, or the "missing" DB exists by the time it is reported. + val dir = Files.createTempDirectory("s16-lines-disk-missing") + val missing = dir.resolve("nowhere").resolve("seforim.db") + + setProperty("seforimDb", missing.toAbsolutePath().toString()) + setProperty("appendExistingDb", "true") + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("acronymDb", dir.resolve("acronymizer.db").toAbsolutePath().toString()) + setProperty("buildStatePath", dir.resolve("seforim.db.buildstate").toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateLinesKt") } + assertNamesTheMissingBase(failure, missing) + + assertTrue(Files.notExists(missing), "no DB may be created at the path that was reported missing") + assertTrue(Files.notExists(missing.parent), "nor its directory") + assertTrue(Files.notExists(dir.resolve("seforim.db.buildstate")), "no buildstate for a phase that failed") + } + + // ─── (1c) the publish is atomic ──────────────────────────────────────── + + private fun assertNoCandidateBeside(target: Path) { + assertTrue( + Files.notExists(target.resolveSibling(target.fileName.toString() + DbPublish.CANDIDATE_SUFFIX)), + "no .candidate may survive", + ) + } + + @Test + fun `a successful publish renames the candidate and leaves none behind`() { + val dir = Files.createTempDirectory("s16-publish-ok") + val target = populatedTargetDb(dir) + + runBlocking { + DbPublish.publishAtomically(target, Logger.withTag("test")) { candidate -> + DriverManager.getConnection("jdbc:sqlite:${candidate.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeUpdate("CREATE TABLE book (id INTEGER PRIMARY KEY NOT NULL, title TEXT)") + st.executeUpdate("INSERT INTO book(id, title) VALUES (1, 'א'), (2, 'ב')") + } + } + } + } + + assertEquals(2, countBooks(target), "the target now holds what the candidate held") + assertNoCandidateBeside(target) + } + + @Test + fun `a publish that fails mid-write keeps the previous DB and discards the candidate`() { + val dir = Files.createTempDirectory("s16-publish-torn") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + + assertFailsWith { + runBlocking { + DbPublish.publishAtomically(target, Logger.withTag("test")) { candidate -> + // A half-written file, then death — exactly what an OOM or a + // cancelled job leaves behind mid-VACUUM. + Files.write(candidate, byteArrayOf(1, 2, 3)) + error("vacuum died") + } + } + } + + assertContentEquals(before, Files.readAllBytes(target), "the previous DB is still there, untouched") + assertEquals(3, countBooks(target)) + assertNoCandidateBeside(target) + } + + @Test + fun `a candidate that is not a database is never published`() { + val dir = Files.createTempDirectory("s16-publish-garbage") + val target = populatedTargetDb(dir) + val before = Files.readAllBytes(target) + + assertFailsWith { + runBlocking { + DbPublish.publishAtomically(target, Logger.withTag("test")) { candidate -> + Files.write(candidate, ByteArray(0)) + } + } + } + + assertContentEquals(before, Files.readAllBytes(target)) + assertNoCandidateBeside(target) + } + + @Test + fun `a stale candidate from a killed run is replaced, not appended to`() { + val dir = Files.createTempDirectory("s16-publish-stale") + val target = populatedTargetDb(dir) + val stale = target.resolveSibling(target.fileName.toString() + DbPublish.CANDIDATE_SUFFIX) + Files.write(stale, "left over by a killed run".toByteArray()) + + runBlocking { + DbPublish.publishAtomically(target, Logger.withTag("test")) { candidate -> + assertTrue(Files.notExists(candidate), "the stale candidate is removed before the write") + DriverManager.getConnection("jdbc:sqlite:${candidate.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeUpdate("CREATE TABLE book (id INTEGER PRIMARY KEY NOT NULL, title TEXT)") + st.executeUpdate("INSERT INTO book(id, title) VALUES (1, 'א')") + } + } + } + } + + assertEquals(1, countBooks(target)) + assertNoCandidateBeside(target) + } + + @Test + fun `a stale WAL or journal beside the target does not survive the publish`() { + // Every on-disk Otzaria stage opens seforim.db in WAL mode; one killed + // mid-run leaves seforim.db-wal/-shm behind. SQLite trusts a -wal it finds + // next to a database whatever that database's header says, so a fresh + // file renamed under a stale one has the old frames replayed into it on + // the next open. The publish must take those files with the old DB. + val dir = Files.createTempDirectory("s16-publish-stale-wal") + val target = populatedTargetDb(dir) + val stale = DbPublish.SQLITE_SIDECAR_SUFFIXES.map { suffix -> + target.resolveSibling(target.fileName.toString() + suffix).also { + Files.write(it, "frames of the previous database".toByteArray()) + } + } + assertEquals(3, stale.size, "premise: -journal, -wal and -shm are all covered") + + runBlocking { + DbPublish.publishAtomically(target, Logger.withTag("test")) { candidate -> + DriverManager.getConnection("jdbc:sqlite:${candidate.toAbsolutePath()}").use { conn -> + conn.createStatement().use { st -> + st.executeUpdate("CREATE TABLE book (id INTEGER PRIMARY KEY NOT NULL, title TEXT)") + st.executeUpdate("INSERT INTO book(id, title) VALUES (1, 'א'), (2, 'ב')") + } + } + } + } + + stale.forEach { assertTrue(Files.notExists(it), "${it.fileName} must not outlive the DB it belonged to") } + assertEquals(2, countBooks(target), "the published DB reads back as the candidate, not as a WAL replay") + assertNoCandidateBeside(target) + } + + // ─── (1d) the build script keeps the contract these rules rest on ────── + + private fun otzariaBuildScript(): String = generateSequence(Path.of("").toAbsolutePath()) { it.parent } + .map { it.resolve("generator/otzariasqlite/build.gradle.kts") } + .firstOrNull { Files.isRegularFile(it) } + ?.let { Files.readString(it) } + ?: error("could not locate generator/otzariasqlite/build.gradle.kts") + + /** The body of one `tasks.register("name")` block. */ + private fun javaExecTask(script: String, name: String): String { + val header = "tasks.register(\"$name\")" + val start = script.indexOf(header) + assertTrue(start >= 0, "the build script must still register $name") + val next = script.indexOf("tasks.register<", start + header.length) + return if (next < 0) script.substring(start) else script.substring(start, next) + } + + @Test + fun `the release tasks pass one path as both base and target and forward the opt-in`() { + val script = otzariaBuildScript() + // Every rule above rests on this: the append phases are seeded from the + // very file they publish back, so a missing base is a lost input and + // continuing would vacuum an empty DB over the release DB. If this ever + // stops being true the fail-closed rule needs rethinking, not silence. + val links = javaExecTask(script, "appendOtzariaLinks") + assertTrue( + links.contains("""systemProperty("baseDb", persistDb)"""), + "appendOtzariaLinks must seed from the file it publishes", + ) + assertTrue( + links.contains("""systemProperty("persistDb", persistDb)"""), + "appendOtzariaLinks must publish to the file it was seeded from", + ) + // -PallowEmptyBase is what every abort message advertises as the way + // past a missing base. A Gradle project property is NOT a system + // property of the forked JVM, so without this forwarding the flag would + // silently do nothing and the message would be a lie. + listOf("generateLines", "generateLinks", "appendOtzariaLines", "appendOtzariaLinks").forEach { task -> + assertTrue( + javaExecTask(script, task) + .contains("""systemProperty("allowEmptyBase", project.property("allowEmptyBase")"""), + "$task must forward -PallowEmptyBase to the JVM it starts", + ) + } + } + + // ─── (2) build_state fail-closed ─────────────────────────────────────── + + @Test + fun `a build_state that cannot be written stops the stage`() { + val dir = Files.createTempDirectory("s13-havrouta-unwritable") + val db = dir.resolve("seforim.db") + // The snapshot's parent is a regular file, so BuildStateWriter's + // createDirectories fails — the cheapest portable unwritable target. + val blocker = Files.write(dir.resolve("blocker"), byteArrayOf(0)) + val statePath = blocker.resolve("seforim.db.buildstate") + + setProperty("seforimDb", db.toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("buildStatePath", statePath.toAbsolutePath().toString()) + + val failure = assertFailsWith { runGenerator("GenerateHavroutaLinksKt") } + val text = generateSequence(failure) { it.cause }.mapNotNull { it.message }.joinToString(" | ") + assertTrue(text.contains("blocker"), "expected the unwritable build_state path in the cause, got: $text") + // It must be the snapshot's own rethrow that stops the stage, not the + // verifier noticing the file afterwards — otherwise dropping the rethrow + // would leave this test green. + assertFalse( + text.contains("was reported written but"), + "the snapshot failure must propagate itself, not be caught downstream by the verifier: $text", + ) + + assertTrue(Files.notExists(statePath)) + } + + // ─── (3) happy path + the written-state self-check ───────────────────── + + @Test + fun `a completed stage leaves a build_state whose counters lead the DB`() { + val dir = Files.createTempDirectory("s13-havrouta-ok") + val db = dir.resolve("seforim.db") + val statePath = dir.resolve("seforim.db.buildstate") + + setProperty("seforimDb", db.toAbsolutePath().toString()) + setProperty("sourceDir", Files.createDirectory(dir.resolve("source")).toAbsolutePath().toString()) + setProperty("buildStatePath", statePath.toAbsolutePath().toString()) + + runGenerator("GenerateHavroutaLinksKt") + + assertTrue(Files.exists(statePath), "the stage must leave its build_state behind") + val header = BuildStateVerifier.readHeader(statePath) + assertEquals("havroutalinks", header.meta["generator"]) + // The stage pre-registers every ConnectionType through the allocator, so + // its counter has to sit above what the DB now holds. + val connectionTypes = maxId(db, "connection_type") + assertTrue(connectionTypes > 0, "the stage inserted connection types") + assertTrue( + header.counters.getValue(IdTable.CONNECTION_TYPE) > connectionTypes, + "next_id must lead MAX(id)", + ) + } +} diff --git a/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaGeneratorDiagnosticsTest.kt b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaGeneratorDiagnosticsTest.kt new file mode 100644 index 00000000..4b27fc7b --- /dev/null +++ b/generator/otzariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/otzariasqlite/OtzariaGeneratorDiagnosticsTest.kt @@ -0,0 +1,349 @@ +package io.github.kdroidfilter.seforimlibrary.otzariasqlite + +import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver +import co.touchlab.kermit.LogWriter +import co.touchlab.kermit.Logger +import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BookKey +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BookSourceHash +import io.github.kdroidfilter.seforimlibrary.common.ids.AllocatorStats +import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocator +import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport +import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository +import io.github.kdroidfilter.seforimlibrary.db.SeforimDb +import kotlinx.coroutines.runBlocking +import java.nio.file.Files +import java.nio.file.Path +import kotlin.io.path.readText +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertTrue + +/** + * The audited generation step emitted 24,207 lines, 94.8% of them boilerplate, + * while four real findings — a dead priority list, a fifth of the manual links + * dropped, 37 heading-map files fed to the link parser, and books with no source + * hash — were either invisible or drowned in it. + * + * These tests drive the real generator over a small library fixture and assert + * on what it says: each noise cut collapses into one summary line that carries + * the counts, and the full list lands in a report file. + */ +class OtzariaGeneratorDiagnosticsTest { + + // ─── log capture ─────────────────────────────────────────────────────── + // DatabaseGenerator logs through `Logger.withTag(...)`, which shares the + // global mutable kermit config, so the capture is installed globally and + // restored afterwards. Note SeforimRepository's init drops the global + // severity to Assert — the capture must be installed AFTER the repository. + + private class Capture : LogWriter() { + val lines = mutableListOf>() + override fun log(severity: Severity, message: String, tag: String, throwable: Throwable?) { + lines += severity to message + } + + fun matching(prefix: String): List = + lines.map { it.second }.filter { it.startsWith(prefix) } + + fun warnings(): List = lines.filter { it.first == Severity.Warn }.map { it.second } + } + + private val previousWriters = Logger.config.logWriterList + private val previousSeverity = Logger.config.minSeverity + private val previousReportDir: String? = System.getProperty(GeneratorReport.DIR_PROPERTY) + + @AfterTest + fun restoreGlobals() { + Logger.setLogWriters(previousWriters) + Logger.setMinSeverity(previousSeverity) + if (previousReportDir == null) System.clearProperty(GeneratorReport.DIR_PROPERTY) + else System.setProperty(GeneratorReport.DIR_PROPERTY, previousReportDir) + } + + private fun installCapture(): Capture { + val capture = Capture() + Logger.setLogWriters(listOf(capture)) + Logger.setMinSeverity(Severity.Verbose) + return capture + } + + private fun reportDir(): Path { + val dir = Files.createTempDirectory("generator-reports") + System.setProperty(GeneratorReport.DIR_PROPERTY, dir.toAbsolutePath().toString()) + return dir + } + + // ─── fixture ─────────────────────────────────────────────────────────── + + private fun newRepo(): SeforimRepository { + val driver = JdbcSqliteDriver(url = "jdbc:sqlite::memory:") + SeforimDb.Schema.create(driver) + return SeforimRepository(":memory:", driver) + } + + private fun hash(seed: Char) = seed.toString().repeat(64) + + /** + * A library with one ordinary book, one book dropped at the library root for + * having no category, and a manifest entry for a book that is not on disk at + * all — i.e. one instance of each source-hash class the importer can name. + */ + private fun writeLibrary( + linksJson: String? = null, + headingsJson: String? = null, + ): Path { + val root = Files.createTempDirectory("otzaria-diagnostics") + val musar = Files.createDirectories(root.resolve("אוצריא").resolve("מוסר")) + Files.writeString(musar.resolve("ספר א.txt"), "

ספר א

\nשורה ראשונה\nשורה שניה") + Files.writeString(root.resolve("אוצריא").resolve("יתום.txt"), "

יתום

\nשורה") + Files.writeString( + root.resolve("files_manifest.json"), + """ + { + "אוצריא/מוסר/ספר א.txt": { "hash": "${hash('a')}" }, + "אוצריא/יתום.txt": { "hash": "${hash('b')}" }, + "אוצריא/מוסר/רפאים.txt": { "hash": "${hash('c')}" } + } + """.trimIndent(), + ) + if (linksJson != null || headingsJson != null) { + val linksDir = Files.createDirectories(root.resolve("links")) + if (linksJson != null) Files.writeString(linksDir.resolve("ספר א_links.json"), linksJson) + if (headingsJson != null) Files.writeString(linksDir.resolve("ברכות_headings.json"), headingsJson) + } + return root + } + + /** The two missing target books of the audited build, with their real link counts scaled down. */ + private val kookPath = "אוצריא\\מחשבת ישראל\\כתבי הרב קוק\\עולת ראיה.txt" + private val kapachPath = "אוצריא\\הלכה\\מלחמות השם יחיא קאפח.txt" + + /** Three manual links at one missing target book and one at another. */ + private fun manualLinks(): String { + // otzaria-library writes Windows separators in `path_2`, so every + // backslash is escaped for JSON and arrives single in the parsed value. + fun entry(i1: Int, target: String) = + """{"line_index_1": $i1, "heRef_2": "x", "path_2": "${target.replace("\\", "\\\\")}", + |"line_index_2": 2, "Conection Type": "commentary"}""".trimMargin() + return "[" + listOf( + entry(2, kookPath), + entry(3, kookPath), + entry(2, kookPath), + entry(3, kapachPath), + ).joinToString(",") + "]" + } + + // ─── 1. `_headings` files never reach the link parser ────────────────── + + @Test + fun `heading maps are excluded at discovery and never parsed as links`() = runBlocking { + val root = writeLibrary( + linksJson = manualLinks(), + // The real shape: a heading → line-number map, which the link parser + // rejects with `Expected start of the array '['` and a JSON dump. + headingsJson = """{"ברכות": 1, "דף ב.": 2, "דף ג.": 14}""", + ) + val repo = newRepo() + reportDir() + val capture = installCapture() + + val generator = DatabaseGenerator(sourceDirectory = root, repository = repo) + generator.generateLinesOnly() + generator.generateLinksOnly() + + assertTrue( + capture.lines.none { "Failed to parse links" in it.second }, + "a heading map must never be handed to the link parser: ${capture.warnings()}", + ) + assertTrue( + capture.lines.none { "Source book not found for links" in it.second }, + "excluding the file also removes its follow-on warning: ${capture.warnings()}", + ) + assertTrue( + capture.lines.none { "\"ברכות\": 1" in it.second }, + "the offending document must not be dumped into the log", + ) + // Both discovery passes (phase 1's merge planner and phase 2's link + // importer) report the exclusion once each — the file used to be parsed + // twice, so silence from either pass would hide half the problem. + assertEquals( + listOf( + "skipped 1 *_headings files (heading maps, not link files)", + "skipped 1 *_headings files (heading maps, not link files)", + ), + capture.matching("skipped 1 *_headings"), + ) + } + + // ─── 2. dropped manual links ─────────────────────────────────────────── + + @Test + fun `manual links to a missing target are summarised per target, not per link`() = runBlocking { + val root = writeLibrary(linksJson = manualLinks()) + val repo = newRepo() + val dir = reportDir() + val capture = installCapture() + + val generator = DatabaseGenerator(sourceDirectory = root, repository = repo) + generator.generateLinesOnly() + generator.generateLinksOnly() + + assertEquals(0, repo.countLinks(), "the drop behaviour itself is unchanged") + assertTrue( + capture.lines.none { it.second.startsWith("Link ") || it.second.startsWith("Original path:") }, + "the two INFO lines per dropped link are gone: ${capture.lines}", + ) + assertEquals( + listOf("manual links: 4 of 4 links dropped — 2 target book(s) not found"), + capture.matching("manual links: 4 of"), + ) + // Ordered by link count, so the worst offender is the first thing read. + assertEquals( + listOf( + "manual links: 3 links dropped — target book not found: 'עולת ראיה' ($kookPath)", + "manual links: 1 links dropped — target book not found: 'מלחמות השם יחיא קאפח' ($kapachPath)", + ), + capture.matching("manual links: ").filter { "target book not found" in it }, + ) + + val report = dir.resolve("otzaria-manual-links-dropped.json").readText() + assertContains(report, "\"droppedLinks\": 4") + assertContains(report, "\"missingTargetBooks\": 2") + assertContains(report, "\"title\": \"עולת ראיה\", \"links\": 3") + } + + // ─── 3. priority list drift ──────────────────────────────────────────── + + @Test + fun `a priority list whose entries are all missing warns once and reports the list`() = runBlocking { + val root = writeLibrary() + val repo = newRepo() + val dir = reportDir() + val capture = installCapture() + + DatabaseGenerator(sourceDirectory = root, repository = repo).generateLinesOnly() + + // None of the shipped priority.txt entries exist under this fixture, so + // the pass is a total miss — as it very nearly is in production (430/431). + val summary = capture.matching("priority list: ").single() + assertTrue( + Regex("""^priority list: 0/\d+ entries found, \d+ missing \(first: .+\) — list is """) + .containsMatchIn(summary), + "unexpected summary: $summary", + ) + assertContains(summary, "priority.txt") + assertContains(summary, "operator decision") + assertTrue( + capture.warnings().none { it.startsWith("Priority entry ") }, + "the per-entry warnings are demoted to DEBUG, not kept as 430 WARNs", + ) + + val report = dir.resolve("otzaria-priority-list-missing.json").readText() + assertContains(report, "\"found\": 0") + assertContains(report, "\"listResource\": \"otzariasqlite/src/commonMain/resources/priority.txt\"") + // The log line named three of them; the report has to carry every one, + // which is the whole point of splitting the finding in two. + val summaryMissing = Regex("""(\d+) missing""").find(summary)!!.groupValues[1].toInt() + assertContains(report, "\"missing\": $summaryMissing") + val listed = report.substringAfter("\"missingEntries\": [") + assertEquals(summaryMissing, Regex("""\.txt"""").findAll(listed).count()) + } + + // ─── 4. books with no source hash ────────────────────────────────────── + + @Test + fun `books whose source hash cannot be recorded are classified and listed`() = runBlocking { + val root = writeLibrary() + val repo = newRepo() + val dir = reportDir() + val capture = installCapture() + + DatabaseGenerator(sourceDirectory = root, repository = repo).generateLinesOnly() + + assertEquals( + listOf("Recorded source hashes for 1 / 3 Otzaria books"), + capture.matching("Recorded source hashes"), + ) + assertEquals( + listOf("source hashes: 2 of 3 Otzaria books have no source hash — they are fully reprocessed every cycle"), + capture.matching("source hashes: 2 of"), + ) + val classes = capture.matching("source hashes: 1 not recorded") + assertEquals( + setOf( + "source hashes: 1 not recorded — file has no category (library root) (יתום)", + "source hashes: 1 not recorded — not imported (reason not tracked) (רפאים)", + ), + classes.toSet(), + "each class names its own books rather than folding into one bucket", + ) + + val report = dir.resolve("otzaria-source-hashes-not-recorded.json").readText() + assertContains(report, "\"computed\": 3") + assertContains(report, "\"notRecorded\": 2") + assertContains(report, "\"title\": \"יתום\", \"reason\": \"file has no category (library root)\"") + assertContains(report, "\"title\": \"רפאים\", \"reason\": \"not imported (reason not tracked)\"") + } + + // ─── 5. the insert-path breadcrumb ───────────────────────────────────── + + /** Fails the id allocation for one named book, the way a real insert failure would. */ + private class FailingBookIdAllocator( + private val delegate: IdAllocator, + private val failFor: String, + ) : IdAllocator by delegate { + override fun bookId(sourceName: String, canonicalHeTitle: String): Long { + if (canonicalHeTitle == failFor) error("boom in bookId") + return delegate.bookId(sourceName, canonicalHeTitle) + } + + override fun stats(): AllocatorStats = delegate.stats() + override fun peekBookId(sourceName: String, canonicalHeTitle: String): Long? = + delegate.peekBookId(sourceName, canonicalHeTitle) + + override fun recordSourceHash(key: BookKey, sourceHash: BookSourceHash) = + delegate.recordSourceHash(key, sourceHash) + } + + @Test + fun `a failure on the serial insert path names the book it was on`() = runBlocking { + val root = writeLibrary() + val repo = newRepo() + reportDir() + val capture = installCapture() + + val generator = DatabaseGenerator( + sourceDirectory = root, + repository = repo, + allocator = FailingBookIdAllocator(InMemoryIdAllocator.load(path = null), failFor = "ספר א"), + ) + + // The exception is rethrown unchanged — the catch exists only to log. + val failure = assertFailsWith { generator.generateLinesOnly() } + assertEquals("boom in bookId", failure.message) + + val breadcrumb = capture.lines.map { it.second }.single { it.startsWith("Error during phase 1") } + assertContains(breadcrumb, "(last book: 'ספר א' file=ספר א.txt categoryId=") + } + + @Test + fun `a failure before any book leaves the breadcrumb out rather than lying`() = runBlocking { + // No אוצריא directory: phase 1 throws before a single book is looked at. + val root = Files.createTempDirectory("otzaria-empty") + val repo = newRepo() + reportDir() + val capture = installCapture() + + assertFailsWith { + DatabaseGenerator(sourceDirectory = root, repository = repo).generateLinesOnly() + } + + val line = capture.lines.map { it.second }.single { it.startsWith("Error during phase 1") } + assertEquals("Error during phase 1", line) + } +} diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/GenerateSefariaSqlite.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/GenerateSefariaSqlite.kt index 544c536c..a78d19aa 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/GenerateSefariaSqlite.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/GenerateSefariaSqlite.kt @@ -3,6 +3,7 @@ package io.github.kdroidfilter.seforimlibrary.sefariasqlite import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier import io.github.kdroidfilter.seforimlibrary.common.db.SEFORIM_DB_PAGE_SIZE_PRAGMA import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository @@ -152,17 +153,28 @@ fun main(args: Array) = runBlocking { } ?: logger.i { "No links phase ran — link-import metrics report not written." } // Persist build_state.db so the next build re-uses the same primary keys. + // Already ordered after the VACUUM INTO above, so a failed persist cannot + // advance the buildstate. + val buildStateMeta = mapOf( + "generator" to "sefariasqlite", + "generated_at" to java.time.Instant.now().toString(), + "build_version" to buildVersion.toString(), + ) runCatching { - allocator.snapshotTo( - target = buildStatePath, - extraMeta = mapOf( - "generator" to "sefariasqlite", - "generated_at" to java.time.Instant.now().toString(), - "build_version" to buildVersion.toString(), - ), - ) - }.onFailure { logger.w(it) { "Failed to write build_state to $buildStatePath" } } - Unit + allocator.snapshotTo(target = buildStatePath, extraMeta = buildStateMeta) + }.onFailure { e -> + // Fail closed: a build that cannot write its allocator state would + // publish last week's — and the build after it would re-issue ids + // this one already handed out. + logger.e(e) { "Failed to write build_state to $buildStatePath" } + throw e + } + BuildStateVerifier.verifyFreshSnapshot( + buildStatePath = buildStatePath, + dbPath = Paths.get(persistedDbPath), + expectedMeta = buildStateMeta, + logger = logger, + ) logger.i { "Sefaria -> SQLite completed. DB at ${if (useMemoryDb) persistDbPath else dbPath}" } } catch (e: Exception) { diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataPostProcess.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataPostProcess.kt index 29705d8e..63fccc5b 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataPostProcess.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataPostProcess.kt @@ -3,8 +3,10 @@ package io.github.kdroidfilter.seforimlibrary.sefariasqlite import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver import co.touchlab.kermit.Logger import co.touchlab.kermit.Severity +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BuildStateVerifier import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocatorBindings import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport import io.github.kdroidfilter.seforimlibrary.core.text.normalizeCategoryPath import io.github.kdroidfilter.seforimlibrary.dao.repository.CategoryDescriptionUpdate import io.github.kdroidfilter.seforimlibrary.dao.repository.SeforimRepository @@ -88,20 +90,34 @@ fun main(args: Array) = runBlocking { val categoryPlan = planCategoryDescriptionOverrides(repository, categoryOverrides) val result = applyMetadata(repository, bindings, bulk, descriptions, logger) val categoryResult = applyCategoryDescriptionPlan(repository, categoryPlan, logger) + // This stage writes straight to the on-disk DB (no VACUUM INTO), so there + // is no persist step for the snapshot to run ahead of. + val buildStateMeta = mapOf( + "generator" to "seedallmetadata", + "generated_at" to Instant.now().toString(), + ) runCatching { - allocator.snapshotTo( - target = buildStatePath, - extraMeta = mapOf( - "generator" to "seedallmetadata", - "generated_at" to Instant.now().toString(), - ), - ) - }.onFailure { logger.w(it) { "Failed to write build_state to $buildStatePath" } } + allocator.snapshotTo(target = buildStatePath, extraMeta = buildStateMeta) + }.onFailure { e -> + // Fail closed: a build that cannot write its allocator state would + // publish last week's — and the build after it would re-issue ids + // this one already handed out. The catch below turns this into + // exitProcess(1). + logger.e(e) { "Failed to write build_state to $buildStatePath" } + throw e + } + BuildStateVerifier.verifyFreshSnapshot( + buildStatePath = buildStatePath, + dbPath = dbPath, + expectedMeta = buildStateMeta, + logger = logger, + ) logger.i { "All-metadata done: updated=${result.updated} unmatched=${result.unmatched}; " + "category records=${categoryResult.records} updates=${categoryResult.updated} " + "unchanged=${categoryResult.unchanged}" } + reportUnmatchedMetadataTitles(result.unmatchedTitles, logger) } catch (e: Exception) { logger.e(e) { "Failed to seed all-metadata; aborting" } exitProcess(1) @@ -126,7 +142,39 @@ internal data class Description( val heDesc: String?, ) -internal data class MetadataResult(val updated: Int, val unmatched: Int) +internal data class MetadataResult( + val updated: Int, + val unmatched: Int, + /** The titles behind [unmatched], in iteration order. See [reportUnmatchedMetadataTitles]. */ + val unmatchedTitles: List = emptyList(), +) + +/** How many unmatched titles the summary names before deferring to the report file. */ +private const val MAX_REPORTED_UNMATCHED_TITLES = 20 + +/** + * One bounded WARN naming the ForDB metadata records that matched no book, plus + * the complete list in a report file. The records are skipped exactly as before + * — the metadata corpus legitimately covers the whole library, which is a + * superset of any single generated DB — but "unmatched=1116" alone gave nobody + * a way to tell a benign superset from a title-normalisation regression. + */ +internal fun reportUnmatchedMetadataTitles(titles: List, logger: Logger) { + if (titles.isEmpty()) return + logger.w { + "all-metadata: ${titles.size} ForDB metadata record(s) matched no book " + + "(skipped): ${titles.take(MAX_REPORTED_UNMATCHED_TITLES).joinToString()}" + + if (titles.size > MAX_REPORTED_UNMATCHED_TITLES) { + ", … and ${titles.size - MAX_REPORTED_UNMATCHED_TITLES} more" + } else { + "" + } + } + GeneratorReport.write("sefaria-all-metadata-unmatched", logger) { + put("unmatched", titles.size.toLong()) + putStrings("titles", titles) + } +} internal sealed interface DescriptionEdit { data object Keep : DescriptionEdit @@ -304,12 +352,18 @@ internal suspend fun applyMetadata( val bookIdsByTitle = repository.getAllBookTitleIds().groupBy({ it.second }, { it.first }) var updated = 0 - var unmatched = 0 + // Names, not just a count: `unmatched=1116` (24.5% of the ForDB metadata + // corpus) said nothing about WHICH books were missing their publication + // data, so nobody downstream — otzaria-library's own metadata pass in + // particular — could act on it. Iteration order is the deterministic + // `bulk.keys + descriptions.keys` set order, so the list is stable + // build-to-build. + val unmatchedTitles = ArrayList() for (title in bulk.keys + descriptions.keys) { val ids = bookIdsByTitle[title] if (ids == null) { - unmatched++ + unmatchedTitles += title continue } if (ids.size > 1) { @@ -335,7 +389,7 @@ internal suspend fun applyMetadata( } updated++ } - return MetadataResult(updated, unmatched) + return MetadataResult(updated, unmatchedTitles.size, unmatchedTitles) } private fun JsonObject.string(key: String): String? = diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedGenerationsPostProcess.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedGenerationsPostProcess.kt index a6da7893..35071bf1 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedGenerationsPostProcess.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedGenerationsPostProcess.kt @@ -159,7 +159,12 @@ internal fun applyGenerations( unmatchedTitles.take(20).joinToString() } } - return GenerationApplyResult(generationsCreated, linksCreated, 0) + // The count MUST be the one the warning above reports. It used to be + // hard-coded to 0, so the same run printed + // `Generation CSV has 13 unmatched book title(s)` and then + // `Generations done: … unmatched=0` — the second line is the one a reader + // scanning the summary sees, and it said the opposite of the truth. + return GenerationApplyResult(generationsCreated, linksCreated, unmatchedTitles.size) } // `book.title` is not UNIQUE in the schema, so even an exact match can return diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaBlacklists.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaBlacklists.kt index 1dede426..9baf28f5 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaBlacklists.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaBlacklists.kt @@ -25,7 +25,16 @@ internal data class BlacklistFilterResult( val skippedByAuthor: Int, val skippedBookExamples: List, val skippedAuthorExamples: List, - val skippedNormalizedPaths: Set + val skippedNormalizedPaths: Set, + /** + * `heTitle` of every skipped payload — the same string + * `SefariaDirectImporter.canonicalHeTitle` uses as the book's natural key, + * and therefore the key `SefariaSourceHashComputer` builds its `BookKey` + * from. Lets the end-of-import source-hash accounting name blacklisting as + * the reason a computed hash never reached the allocator, instead of + * reporting a bare `5825 / 6216`. Diagnostics only — nothing filters on it. + */ + val skippedHeTitles: Set = emptySet(), ) /** Blacklist of book editions (book_version). Format rules: see black_versions.txt. */ @@ -113,6 +122,7 @@ internal fun filterBlacklistedPayloads( val skippedBookExamples = ArrayList(5) val skippedAuthorExamples = ArrayList(5) val skippedNormalizedPaths = LinkedHashSet() + val skippedHeTitles = LinkedHashSet() val filtered = payloads.filter { payload -> val bookBlacklisted = isBookBlacklisted(payload, blacklists) @@ -121,6 +131,7 @@ internal fun filterBlacklistedPayloads( if (bookBlacklisted || authorBlacklisted) { skippedTotal++ skippedNormalizedPaths += normalizedBookPath(payload.categoriesHe, payload.heTitle) + skippedHeTitles += payload.heTitle if (bookBlacklisted) { skippedByBook++ @@ -147,7 +158,8 @@ internal fun filterBlacklistedPayloads( skippedByAuthor = skippedByAuthor, skippedBookExamples = skippedBookExamples, skippedAuthorExamples = skippedAuthorExamples, - skippedNormalizedPaths = skippedNormalizedPaths + skippedNormalizedPaths = skippedNormalizedPaths, + skippedHeTitles = skippedHeTitles, ) } diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaDirectImporter.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaDirectImporter.kt index b8300af3..32b66166 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaDirectImporter.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaDirectImporter.kt @@ -7,6 +7,7 @@ import io.github.kdroidfilter.seforimlibrary.common.changes.TouchedBookDetector import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocator import io.github.kdroidfilter.seforimlibrary.common.ids.IdAllocatorBindings import io.github.kdroidfilter.seforimlibrary.common.ids.InMemoryIdAllocator +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport import io.github.kdroidfilter.seforimlibrary.core.models.Author import io.github.kdroidfilter.seforimlibrary.core.models.Book import io.github.kdroidfilter.seforimlibrary.core.models.Category @@ -636,13 +637,23 @@ class SefariaDirectImporter( // (i.e. whose natural key now exists in the allocator), so books that were // filtered out (blacklists, dedup vs Sefaria) don't get spurious hashes. var recorded = 0 + // A computed hash whose natural key never reached the allocator has no + // book to attach to, so the next build sees that book as `added` and + // reprocesses it in full — every cycle, forever. 391 of 6,216 in the + // audited build, reported only as a bare `5825 / 6216`. Classify them. + val unrecordedByReason = LinkedHashMap>() for ((key, hash) in currentSourceHashes) { if (allocator.peekBookId(key.sourceName, key.canonicalHeTitle) != null) { allocator.recordSourceHash(key, hash) recorded++ + } else { + unrecordedByReason + .getOrPut(classifyUnimportedSefariaBook(key, blacklistResult)) { mutableListOf() } + .add(key.canonicalHeTitle) } } logger.i { "Recorded source hashes for $recorded / ${currentSourceHashes.size} Sefaria books" } + reportUnrecordedSefariaSourceHashes(currentSourceHashes.size, unrecordedByReason, logger) logger.i { "Category descriptions: parsed=${categoryDescriptions.size}, " + @@ -662,6 +673,78 @@ class SefariaDirectImporter( */ private fun canonicalHeTitle(payload: BookPayload): String = payload.heTitle +/** How many book titles each source-hash class names before deferring to the report file. */ +private const val MAX_REPORTED_UNRECORDED_HASHES = 20 + +/** + * Why a Sefaria book whose source hash was computed never reached the allocator. + * + * The hash computer walks `json/**/merged.json` and keys every book it finds by + * its `heTitle` — exactly [canonicalHeTitle] — but it runs BEFORE the blacklist + * filter, so every blacklisted book contributes a hash that no book id will ever + * claim. In the audited build that is essentially the whole gap: 395 payloads + * skipped by blacklist against 391 unrecorded hashes (the difference being books + * blacklisted only recently, whose ids the allocator still carries from an + * earlier cycle, so `peekBookId` still answers). + * + * Anything the blacklist does not explain is a genuinely unaccounted book and + * says so rather than being folded into a plausible-looking bucket. + */ +internal fun classifyUnimportedSefariaBook(key: BookKey, blacklist: BlacklistFilterResult): String = + if (key.canonicalHeTitle in blacklist.skippedHeTitles) { + "skipped by the book/author blacklist" + } else { + "not imported (reason not tracked)" + } + +/** + * One WARN per class with counts and a bounded, sorted name list; the complete + * list goes to the report file. Recording itself is unchanged — which books get + * a hash decides which books the NEXT build treats as touched, so a "fix" here + * would change that build's content, not this one's logging. + */ +internal fun reportUnrecordedSefariaSourceHashes( + computed: Int, + byReason: Map>, + logger: Logger, +) { + if (byReason.isEmpty()) return + val total = byReason.values.sumOf { it.size } + logger.w { + "source hashes: $total of $computed Sefaria books have no source hash — " + + "they are fully reprocessed every cycle" + } + val ordered = byReason.entries.sortedWith( + compareByDescending>> { it.value.size }.thenBy { it.key }, + ) + for ((reason, titles) in ordered) { + val names = titles.sorted() + logger.w { + "source hashes: ${titles.size} not recorded — $reason " + + "(${names.take(MAX_REPORTED_UNRECORDED_HASHES).joinToString()}" + + if (names.size > MAX_REPORTED_UNRECORDED_HASHES) { + ", … and ${names.size - MAX_REPORTED_UNRECORDED_HASHES} more)" + } else { + ")" + } + } + } + GeneratorReport.write("sefaria-source-hashes-not-recorded", logger) { + put("computed", computed.toLong()) + put("notRecorded", total.toLong()) + putRows( + "byReason", + ordered.map { (reason, titles) -> mapOf("reason" to reason, "books" to titles.size) }, + ) + putRows( + "books", + ordered.flatMap { (reason, titles) -> + titles.sorted().map { mapOf("title" to it, "reason" to reason) } + }, + ) + } +} + /** * One book's contribution to the title→bookId index, in priority order. * [primaryTitles] are raw titles (heTitle/enTitle, normalized on build); diff --git a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaLinksImporter.kt b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaLinksImporter.kt index 329024c4..12fe80a6 100644 --- a/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaLinksImporter.kt +++ b/generator/sefariasqlite/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaLinksImporter.kt @@ -304,18 +304,20 @@ internal class SefariaLinksImporter( // Per-connection-type importer summary (QA plan §10.5); semantics in // [LinkImportTypeMetrics]. + // + // The three read-side counters and the write-side one are keyed by + // DIFFERENT types — see [LinkImportTypeMetrics] — and printing them + // unqualified side by side made the line read as a contradiction: + // `type=REFERENCE rowsRead=381152 dropped=376731 resolvedPairs=4421 + // written=106912` looks like 24 rows written per resolved pair. It is + // not: 4,421 is how many pairs came from rows the CSV *typed* REFERENCE, + // 106,912 is how many links were *stored* as REFERENCE — nearly all of + // them from blank-typed rows that [inferBlankConnectionType] resolved to + // REFERENCE. The write path is correct and is left untouched; the field + // names now say which keying each number uses, and the totals line gives + // the one comparison that is meaningful across the two keyings. val metrics = metricsSnapshot() - logger.i { - buildString { - append("Sefaria links importer per-type counters:") - for ((name, t) in metrics.insertedByType) { - append("\ntype=$name rowsRead=${t.rowsRead}") - append(" dropped=${t.dropped}") - append(" resolvedPairs=${t.resolvedPairs}") - append(" written=${t.written}") - } - } - } + logger.i { formatPerTypeCounters(metrics) } } /** @@ -1107,6 +1109,41 @@ internal fun parseSuppressionMask(cell: String?, source: String): Int { return mask } +/** + * Renders the per-type importer summary. + * + * The prefix `Sefaria links importer per-type counters` is load-bearing: + * pipeline-monitor's `generate_phases.sh` anchors its `links_post` phase marker + * on it. Everything after it is free-form. + */ +internal fun formatPerTypeCounters(metrics: LinkImportMetrics): String = buildString { + append( + "Sefaria links importer per-type counters " + + "(csv* keyed by the CSV's `Conection Type`; storedWritten keyed by the type " + + "the link was STORED under, after blank-type inference and base→dependant " + + "direction normalisation — the two keyings describe different row sets, so " + + "only the totals are comparable):" + ) + var rowsRead = 0L + var dropped = 0L + var resolvedPairs = 0L + var written = 0L + for ((name, t) in metrics.insertedByType) { + append("\ntype=$name csvRowsRead=${t.rowsRead}") + append(" csvDropped=${t.dropped}") + append(" csvResolvedPairs=${t.resolvedPairs}") + append(" storedWritten=${t.written}") + rowsRead += t.rowsRead + dropped += t.dropped + resolvedPairs += t.resolvedPairs + written += t.written + } + append("\ntotals csvRowsRead=$rowsRead csvDropped=$dropped") + append(" csvResolvedPairs=$resolvedPairs storedWritten=$written") + append(" (resolvedPairs-written=${resolvedPairs - written}: pairs dropped by the") + append(" heading/self-link filters or collapsed by INSERT OR IGNORE)") +} + internal fun mapCsvConnectionType(raw: String, source: String): ConnectionType { val type = ConnectionType.fromKnownStringOrNull(raw) ?: error("Unmapped Sefaria connection type '$raw' in $source") diff --git a/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataSourceTest.kt b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataSourceTest.kt index 1f026937..5b064825 100644 --- a/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataSourceTest.kt +++ b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SeedAllMetadataSourceTest.kt @@ -35,4 +35,34 @@ class SeedAllMetadataSourceTest { assertEquals(1, result.updated) assertEquals(natlId, repo.getBook(bookId)?.sourceId, "seedAllMetadata must not overwrite the book source") } + + /** + * `unmatched=1116` named none of the 1,116, so a benign superset and a + * title-normalisation regression looked identical in the log. The result now + * carries the titles themselves. + */ + @Test + fun applyMetadata_namesTheRecordsThatMatchedNoBook() = runBlocking { + val driver = JdbcSqliteDriver(url = "jdbc:sqlite::memory:") + SeforimDb.Schema.create(driver) + val repo = SeforimRepository(":memory:", driver) + + val sourceId = repo.insertSource("Sefaria") + val catId = repo.insertCategory(Category(0, null, "הלכה", level = 0, order = 1)) + repo.insertBook(Book(categoryId = catId, sourceId = sourceId, title = "תבונה", heRef = "תבונה")) + + val bulk = linkedMapOf( + "תבונה" to BulkMetadata(pubDates = listOf(1900), pubPlaceHe = "ירושלים"), + "אבן הראשה" to BulkMetadata(pubDates = emptyList(), pubPlaceHe = null), + "קול התור" to BulkMetadata(pubDates = emptyList(), pubPlaceHe = null), + ) + val bindings = IdAllocatorBindings(InMemoryIdAllocator.load(path = null), repo) + + val result = applyMetadata(repo, bindings, bulk, emptyMap(), Logger.withTag("test")) + + assertEquals(1, result.updated) + assertEquals(2, result.unmatched) + assertEquals(listOf("אבן הראשה", "קול התור"), result.unmatchedTitles) + assertEquals(result.unmatched, result.unmatchedTitles.size, "the count must be the list's size") + } } diff --git a/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaGeneratorDiagnosticsTest.kt b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaGeneratorDiagnosticsTest.kt new file mode 100644 index 00000000..47e54ab3 --- /dev/null +++ b/generator/sefariasqlite/src/jvmTest/kotlin/io/github/kdroidfilter/seforimlibrary/sefariasqlite/SefariaGeneratorDiagnosticsTest.kt @@ -0,0 +1,294 @@ +package io.github.kdroidfilter.seforimlibrary.sefariasqlite + +import co.touchlab.kermit.LogWriter +import co.touchlab.kermit.Logger +import co.touchlab.kermit.Severity +import co.touchlab.kermit.StaticConfig +import io.github.kdroidfilter.seforimlibrary.common.buildstate.BookKey +import io.github.kdroidfilter.seforimlibrary.common.reports.GeneratorReport +import java.nio.file.Files +import java.sql.DriverManager +import kotlin.io.path.readText +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * The Sefaria-stage half of the generation-step audit: a per-type counter block + * that read as a contradiction, a "done" summary that disagreed with its own + * warning, and two "N of M" lines that named none of the M − N. + */ +class SefariaGeneratorDiagnosticsTest { + + private class Capture : LogWriter() { + val lines = mutableListOf>() + override fun log(severity: Severity, message: String, tag: String, throwable: Throwable?) { + lines += severity to message + } + + fun warnings(): List = lines.filter { it.first == Severity.Warn }.map { it.second } + } + + private fun capturingLogger(capture: Capture) = + Logger(StaticConfig(Severity.Verbose, listOf(capture)), "test") + + private val previousReportDir: String? = System.getProperty(GeneratorReport.DIR_PROPERTY) + + @AfterTest + fun restoreReportDir() { + if (previousReportDir == null) System.clearProperty(GeneratorReport.DIR_PROPERTY) + else System.setProperty(GeneratorReport.DIR_PROPERTY, previousReportDir) + } + + private fun reportDir() = Files.createTempDirectory("generator-reports").also { + System.setProperty(GeneratorReport.DIR_PROPERTY, it.toAbsolutePath().toString()) + } + + // ─── per-type counters ───────────────────────────────────────────────── + + private fun metrics(vararg rows: Pair) = + LinkImportMetrics(linkedMapOf(*rows)) + + private fun typeMetrics(rowsRead: Long, dropped: Long, resolvedPairs: Long, written: Long) = + LinkImportTypeMetrics(rowsRead, dropped, resolvedPairs, written) + + @Test + fun `each counter names the keying it uses so the REFERENCE skew stops reading as a bug`() { + // The audited numbers. rowsRead/dropped/resolvedPairs are keyed by the + // CSV's declared type, `written` by the type the row was stored under — + // so 4,421 resolved pairs against 106,912 written rows is not 24 writes + // per pair, it is two different row sets sharing a name. + val text = formatPerTypeCounters( + metrics( + "REFERENCE" to typeMetrics(381152, 376731, 4421, 106912), + "OTHER" to typeMetrics(2035716, 367327, 1668400, 1590315), + ), + ) + + assertTrue( + text.startsWith("Sefaria links importer per-type counters "), + "pipeline-monitor's links_post marker anchors on this prefix: $text", + ) + assertContains(text, "csv* keyed by the CSV's `Conection Type`") + assertContains(text, "storedWritten keyed by the type the link was STORED under") + assertContains( + text, + "type=REFERENCE csvRowsRead=381152 csvDropped=376731 csvResolvedPairs=4421 storedWritten=106912", + ) + assertTrue( + "type=REFERENCE rowsRead=" !in text && " written=" !in text, + "no unqualified counter name may survive: $text", + ) + } + + @Test + fun `the totals line gives the one comparison that crosses both keyings`() { + val text = formatPerTypeCounters( + metrics( + "REFERENCE" to typeMetrics(381152, 376731, 4421, 106912), + "OTHER" to typeMetrics(2035716, 367327, 1668400, 1590315), + ), + ) + assertContains( + text, + "totals csvRowsRead=2416868 csvDropped=744058 csvResolvedPairs=1672821 storedWritten=1697227", + ) + // Σ written may exceed Σ resolvedPairs in a partial sample; on a full + // build it is the other way round and the gap is the filtered pairs. + assertContains(text, "resolvedPairs-written=-24406") + } + + @Test + fun `an empty import still renders a well-formed summary`() { + val text = formatPerTypeCounters(LinkImportMetrics(emptyMap())) + assertContains(text, "totals csvRowsRead=0 csvDropped=0 csvResolvedPairs=0 storedWritten=0") + assertContains(text, "resolvedPairs-written=0") + } + + // ─── generations: the warning and the summary must agree ─────────────── + + private fun generationsDb() = DriverManager.getConnection("jdbc:sqlite::memory:").apply { + createStatement().use { st -> + st.execute("CREATE TABLE book (id INTEGER PRIMARY KEY, title TEXT NOT NULL)") + st.execute("CREATE TABLE generation (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE)") + st.execute( + "CREATE TABLE book_generation (bookId INTEGER NOT NULL, generationId INTEGER NOT NULL, " + + "PRIMARY KEY (bookId, generationId))", + ) + st.execute("INSERT INTO book VALUES (1, 'ברכות')") + } + } + + @Test + fun `the generations summary reports the same unmatched count as its warning`() { + generationsDb().use { conn -> + val capture = Capture() + val result = applyGenerations( + conn, + listOf( + "ברכות" to "אמוראים", + "אבן הראשה" to "ראשונים", + "קול התור" to "אחרונים", + ), + capturingLogger(capture), + ) + + assertEquals(1, result.linksCreated) + // Was hard-coded to 0, so `Generations done: … unmatched=0` flatly + // contradicted `Generation CSV has 13 unmatched book title(s)`. + assertEquals(2, result.unmatched) + val warning = capture.warnings().single() + assertContains(warning, "Generation CSV has 2 unmatched book title(s)") + assertEquals( + result.unmatched, + Regex("""has (\d+) unmatched""").find(warning)!!.groupValues[1].toInt(), + "the summary count and the warning count must be the same number", + ) + } + } + + @Test + fun `a fully matched generations CSV reports no unmatched titles and warns about nothing`() { + generationsDb().use { conn -> + val capture = Capture() + val result = applyGenerations(conn, listOf("ברכות" to "אמוראים"), capturingLogger(capture)) + assertEquals(0, result.unmatched) + assertEquals(emptyList(), capture.warnings()) + } + } + + // ─── all-metadata: name the records that matched no book ─────────────── + + @Test + fun `unmatched metadata titles are named in the log and listed in full in the report`() { + val dir = reportDir() + val capture = Capture() + val titles = (1..25).map { "ספר $it" } + + reportUnmatchedMetadataTitles(titles, capturingLogger(capture)) + + val warning = capture.warnings().single() + assertContains(warning, "all-metadata: 25 ForDB metadata record(s) matched no book") + assertContains(warning, "ספר 1") + assertContains(warning, "… and 5 more") + assertTrue("ספר 21" !in warning, "the log line stays bounded at 20 names: $warning") + + val report = dir.resolve("sefaria-all-metadata-unmatched.json").readText() + assertContains(report, "\"unmatched\": 25") + // The five names the bounded log line had to leave out are all here. + titles.forEach { assertContains(report, "\"$it\"") } + } + + @Test + fun `no unmatched metadata means no warning and no report file`() { + val dir = reportDir() + val capture = Capture() + reportUnmatchedMetadataTitles(emptyList(), capturingLogger(capture)) + assertEquals(emptyList(), capture.warnings()) + assertTrue(Files.list(dir).use { it.findAny().isEmpty }, "an empty finding writes nothing") + } + + // ─── Sefaria source hashes ───────────────────────────────────────────── + + private fun blacklistResult(skipped: Set) = BlacklistFilterResult( + payloads = emptyList(), + skippedTotal = skipped.size, + skippedByBook = skipped.size, + skippedByAuthor = 0, + skippedBookExamples = skipped.toList(), + skippedAuthorExamples = emptyList(), + skippedNormalizedPaths = emptySet(), + skippedHeTitles = skipped, + ) + + @Test + fun `a blacklisted book explains its missing source hash, anything else says it does not`() { + val blacklist = blacklistResult(setOf("קדמוניות היהודים")) + assertEquals( + "skipped by the book/author blacklist", + classifyUnimportedSefariaBook(BookKey("Sefaria", "קדמוניות היהודים"), blacklist), + ) + assertEquals( + "not imported (reason not tracked)", + classifyUnimportedSefariaBook(BookKey("Sefaria", "ספר אחר"), blacklist), + ) + } + + @Test + fun `unrecorded Sefaria hashes are reported per class, biggest first, with the full list on disk`() { + val dir = reportDir() + val capture = Capture() + + reportUnrecordedSefariaSourceHashes( + computed = 6216, + byReason = linkedMapOf( + "not imported (reason not tracked)" to listOf("ב", "א"), + "skipped by the book/author blacklist" to (1..21).map { "חסום $it" }, + ), + logger = capturingLogger(capture), + ) + + val warnings = capture.warnings() + assertEquals( + "source hashes: 23 of 6216 Sefaria books have no source hash — " + + "they are fully reprocessed every cycle", + warnings.first(), + ) + // Biggest class first, names sorted so two builds' logs are comparable. + assertTrue( + warnings[1].startsWith("source hashes: 21 not recorded — skipped by the book/author blacklist ("), + warnings[1], + ) + assertContains(warnings[1], "… and 1 more)") + assertEquals("source hashes: 2 not recorded — not imported (reason not tracked) (א, ב)", warnings[2]) + + val report = dir.resolve("sefaria-source-hashes-not-recorded.json").readText() + assertContains(report, "\"computed\": 6216") + assertContains(report, "\"notRecorded\": 23") + assertContains(report, "{ \"reason\": \"skipped by the book/author blacklist\", \"books\": 21 }") + assertContains(report, "{ \"title\": \"א\", \"reason\": \"not imported (reason not tracked)\" }") + } + + @Test + fun `nothing unrecorded means nothing logged`() { + val capture = Capture() + reportUnrecordedSefariaSourceHashes(6216, emptyMap(), capturingLogger(capture)) + assertEquals(emptyList(), capture.lines.map { it.second }) + } + + // ─── the blacklist filter feeds the classification ───────────────────── + + private fun payload(heTitle: String) = BookPayload( + heTitle = heTitle, + enTitle = heTitle, + categoriesHe = listOf("תנך"), + lines = listOf("שורה"), + refEntries = emptyList(), + headings = emptyList(), + authors = emptyList(), + description = null, + heShortDesc = null, + pubDates = emptyList(), + altStructures = emptyList(), + ) + + @Test + fun `the blacklist filter records the heTitles it skipped`() { + val kept = payload("ברכות") + val dropped = payload("קדמוניות היהודים") + val result = filterBlacklistedPayloads( + listOf(kept, dropped), + SefariaBlacklists( + authorKeys = emptySet(), + bookTitleKeys = setOfNotNull(normalizeTitleKey("קדמוניות היהודים")), + bookPathKeys = emptySet(), + ), + ) + assertEquals(listOf(kept), result.payloads) + // The heTitle IS the natural-key half the source-hash accounting joins + // on, so this set is what turns "391 unrecorded" into "391 blacklisted". + assertEquals(setOf("קדמוניות היהודים"), result.skippedHeTitles) + } +} From fd0e58e1156ab8266b28180789119d3f80ac6e1d Mon Sep 17 00:00:00 2001 From: ypl <7353755@gmail.com> Date: Wed, 9 Sep 2026 12:13:34 +0300 Subject: [PATCH 3/7] fix(qa): a Sefaria schema that parses but is not a book is counted, not silently skipped; QA drift is gated and observable Squashed from the audit branch by file set (1 original commits contributed; their messages follow). --- 9b586e0 fix(qa): a Sefaria schema that parses but is not a book is counted, not silently skipped load_schema_books failed on unreadable JSON but silently `continue`d over valid JSON whose top level is not an object or that has no object-valued `schema` key, so such a book vanished from both the expected set and the DB and could slip under the 2%/10-book drift gate (--expect-snapshot runs in no workflow). Both shapes now land in the same `unreadable` accounting: counted, named with the reason, allowlistable through KNOWN_UNREADABLE_SCHEMAS like Sheet.json, and otherwise a failure. Measured against the 2026-09-01 export (6,601 schema files): zero files take either branch, so this is a no-op today and fail-closed for the next export that ships a different shape. Co-Authored-By: Claude Fable 5.1 --- scripts/qa/README.md | 33 +++- scripts/qa/check1_dependence_count.py | 9 +- scripts/qa/check2_book_base_text.py | 5 +- scripts/qa/check7_provenance.py | 5 +- scripts/qa/common.py | 133 ++++++++++++++-- scripts/qa/run_all.py | 16 +- scripts/qa/tests/test_qa_synthetic.py | 212 +++++++++++++++++++++++++- 7 files changed, 394 insertions(+), 19 deletions(-) diff --git a/scripts/qa/README.md b/scripts/qa/README.md index 0f462314..4dc2837d 100644 --- a/scripts/qa/README.md +++ b/scripts/qa/README.md @@ -20,9 +20,36 @@ generator/sefariasqlite/build/sefaria/export/schemas/*.json `` עצמו. כל מועמד עם קובצי `*.json` מאומת שהוא **באמת** תיקיית schemas (לפחות קובץ אחד נטען ל-dict עם אובייקט `schema` מקונן); אם למועמד יש `*.json` אך אף לא schema אחד (למשל `export/` המכיל רק `table_of_contents.json`) — ממשיכים למועמד הבא; -אם אף מועמד לא מספק schema — כשל בקול עם רשימת הנתיבים שנוסו. קובץ schema לא-קריא -(למשל `Sheet.json` בגודל 0 בייצוא האמיתי) מדולג עם אזהרה — שכפול דטרמיניסטי של -`runCatching` פר-קובץ ב-`SefariaBookPayloadReader.kt:33-49`, לא היוריסטיקה. +אם אף מועמד לא מספק schema — כשל בקול עם רשימת הנתיבים שנוסו. קובץ schema שאינו +נטען לספר — בין שאינו נפרס כלל, בין שהוא JSON תקין שאינו אובייקט בשורש, ובין שאין +בו אובייקט `schema` מקונן — מדולג רק אם הוא ברשימה המפורשת +`KNOWN_UNREADABLE_SCHEMAS` ב-`common.py`, ואז נרשמת +שורת `INFO:` אחת עם הסיבה. כרגע ברשימה `Sheet.json` בלבד: ה-pseudo-index של +דפי-המקורות בספריא, שהייצוא שולח כקובץ באורך 0 — לא ספר, ואין לו שורה ב-`book`. +זה שכפול דטרמיניסטי של `runCatching` פר-קובץ ב-`SefariaBookPayloadReader.kt:33-49`, +לא היוריסטיקה. כל קובץ אחר שאינו נטען לספר הוא נזק בארכיון מקובע ומאומת-digest: +הוא נספר, נוקב בשם ובסיבה, ומפיל את הבדיקה — במקום לצמצם את כיסויה בשקט ולדווח +`PASS`. + +### שער סחיפה מול ה-reference snapshot + +בדיקות 1/2/7 נושאות baselines קשיחים. בנוסף להשוואה הפנימית (DB מול schemas) הן +מעבירות כל מדד סָפוּר דרך `gate_snapshot_drift`: התכווצות מעבר ל- +`QA_DRIFT_MAX_SHRINK_PCT` אחוזים מה-baseline היא `::error::` ויציאה 1; כל הפרש אחר +(התכווצות קטנה יותר, או גדילה) הוא `::warning::` עם המספרים. + +כשהמשתנה אינו מוגדר (או ריק) **השער כבוי לגמרי ואינו מדפיס דבר** — ה-baselines +משמעותיים רק מול הקורפוס המקובע, לא מול פיקסטורה סינתטית או DB חלקי בהרצת אד-הוק. +אין לכך ברירת-מחדל מובלעת: ה-workflow השבועי קובע `QA_DRIFT_MAX_SHRINK_PCT: "2"` +בצעד ה-QA, ו-`run_all.py --require-all` (מצב release) נכשל מיד אם המשתנה חסר, כדי +שהרצת release לא תוכל לרוץ בלי שער. ערך לא-מספרי או שלילי הוא כשל, לא שער כבוי. +אין בכך רענון של ה-baseline — `--expect-snapshot` ממשיך לאכוף את המספרים המדויקים. + +השער אינו אחוזי בלבד: התכווצות היא `::error::` רק כשהיא חוצה **גם** את האחוז +**וגם** את הרצפה המוחלטת `QA_DRIFT_MIN_SHRINK_ABS` (ברירת מחדל 10, ה-workflow +קובע אותה במפורש). על מדד קטן (`guides`=2, `midrash`=5, זוגות מוסקים=20, +`targum`=45) ירידה של 1 היא 2%..50% — בלי הרצפה ספר יחיד שנעלם מהייצוא היה מפיל +בנייה שבועית; עם הרצפה הוא `::warning::` עם המספרים, וקריסה אמיתית עדיין נכשלת. הרצת הכול: diff --git a/scripts/qa/check1_dependence_count.py b/scripts/qa/check1_dependence_count.py index a707a418..e9a7c965 100755 --- a/scripts/qa/check1_dependence_count.py +++ b/scripts/qa/check1_dependence_count.py @@ -6,7 +6,8 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from common import (load_schema_books, normalize_title_key, open_db, - require_columns, resolve_schemas_dir, sefaria_source_id, die) + require_columns, resolve_schemas_dir, sefaria_source_id, die, + gate_snapshot_drift) SNAPSHOT_TOTAL = 4941 SNAPSHOT_BREAKDOWN = {"commentary": 4889, "targum": 45, "midrash": 5, "guides": 2} @@ -61,6 +62,12 @@ def main(): for k in sorted(keys) if db_breakdown.get(k, 0) != exp_breakdown.get(k, 0)} die(f"DB לא תואם ל-schemas (סוג: DB,expected): {diff}") + # שער הסחיפה: הסך וכל סוג בנפרד, כדי שהחלפה מקזזת (סוג יורד, אחר עולה) לא + # תעבור מתחת לסך יציב. + gate_snapshot_drift("dependenceType total", db_total, SNAPSHOT_TOTAL) + for kind, expected in sorted(SNAPSHOT_BREAKDOWN.items()): + gate_snapshot_drift(f"dependenceType {kind}", db_breakdown.get(kind, 0), expected) + if args.expect_snapshot: if db_total != SNAPSHOT_TOTAL: die(f"snapshot: total={db_total} != {SNAPSHOT_TOTAL}") diff --git a/scripts/qa/check2_book_base_text.py b/scripts/qa/check2_book_base_text.py index b452cd8f..3b22d711 100755 --- a/scripts/qa/check2_book_base_text.py +++ b/scripts/qa/check2_book_base_text.py @@ -8,7 +8,8 @@ from common import (load_schema_books, normalize_title_key, open_db, require_columns, resolve_schemas_dir, sefaria_source_id, die, default_priority_list_path, load_priority_list, - order_books_by_priority, build_normalized_title_to_bookid) + order_books_by_priority, build_normalized_title_to_bookid, + gate_snapshot_drift) # ‏5,426 הוא ה-baseline הנכון (אחרי תיקון היבואן ב-8358a16). ‏5,437 שנצפה קודם היה תוצר # באג ביבואן — alias של ספר מוקדם (מ"מעילה") האפיל על פרימרי של ספר מאוחר, וכפל 11 זוגות. @@ -76,6 +77,8 @@ def main(): print(f"DB∖expected (עד 10): {sorted(extra)[:10]}", file=sys.stderr) die(f"אי-התאמת זוגות: {len(missing)} חסרים, {len(extra)} עודפים") + gate_snapshot_drift("book_base_text rows", len(db_pairs), SNAPSHOT_ROWS) + if args.expect_snapshot and len(db_pairs) != SNAPSHOT_ROWS: die(f"snapshot: rows={len(db_pairs)} != {SNAPSHOT_ROWS}") diff --git a/scripts/qa/check7_provenance.py b/scripts/qa/check7_provenance.py index 57335d4c..f7dd8403 100755 --- a/scripts/qa/check7_provenance.py +++ b/scripts/qa/check7_provenance.py @@ -10,7 +10,7 @@ import os sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from common import open_db, require_columns, die +from common import open_db, require_columns, die, gate_snapshot_drift SNAPSHOT_INFERRED = 13056 # baseProvenance=1 SNAPSHOT_INFERRED_PAIRS = 20 # זוגות ספרים מוסקים @@ -53,6 +53,9 @@ def main(): big = inferred_pairs[0] print(f" הזוג הגדול: source={big['s']} target={big['t']} ({big['c']} קישורים)") print(f"reference snapshot: {SNAPSHOT_INFERRED} קישורים, {SNAPSHOT_INFERRED_PAIRS} זוגות") + gate_snapshot_drift("baseProvenance=1 links", inferred_total, SNAPSHOT_INFERRED) + gate_snapshot_drift("baseProvenance=1 book pairs", len(inferred_pairs), + SNAPSHOT_INFERRED_PAIRS) # (ב) baseProvenance=2 (SEFARIA_DECLARED): לא-ריק ועקבי עם book_base_text. # book_base_text מאוחסן (bookId=תלוי, baseBookId=בסיס); קישור מוצהר מאוחסן base→dependant diff --git a/scripts/qa/common.py b/scripts/qa/common.py index 7e212bed..d0f04bcb 100755 --- a/scripts/qa/common.py +++ b/scripts/qa/common.py @@ -216,10 +216,23 @@ def _file_title(path): return base.replace("_", " ") +# קובצי schema שהייצוא המקובע של ספריא שולח בכוונה כך שאינם נטענים לספר, עם הסיבה. +# Sheet.json הוא ה-pseudo-index של "דפי מקורות" (Sheets) — לא ספר: הייצוא כותב +# אותו כקובץ באורך 0, ולכן אין לו schema, אין לו heTitle ואין לו שורה ב-book. +# היבואן עצמו מפיל אותו באותה הדרך בדיוק (SefariaBookPayloadReader.buildSchemaLookup, +# runCatching פר-קובץ), ולכן דילוג עליו כאן אינו מקטין כיסוי — הוא משחזר את התנהגות +# היבואן. כל קובץ אחר שאינו נטען לספר הוא נזק בארכיון מקובע ומאומת-digest, ונכשל בקול. +KNOWN_UNREADABLE_SCHEMAS = { + "Sheet.json": "ה-pseudo-index של דפי-מקורות בספריא, נשלח כקובץ באורך 0; " + "היבואן מפיל אותו זהה, ואין לו ספר ב-DB", +} + + def load_schema_books(schemas_dir): # שכפול קריאת SefariaBookPayloadReader.kt: כותרות מ-schema המקונן, dependence/base מ-top עם fallback. books = [] - skipped = 0 + known_skipped = [] + unreadable = [] for fn in sorted(os.listdir(schemas_dir)): if not fn.endswith(".json"): continue @@ -229,14 +242,26 @@ def load_schema_books(schemas_dir): top = json.load(fh) except (ValueError, OSError) as e: # שכפול דטרמיניסטי של runCatching פר-קובץ ב-SefariaBookPayloadReader.kt:33-49 - # (buildSchemaLookup בולע קובץ לא-פריס בשקט; למשל Sheet.json ריק בייצוא האמיתי). - print(f"אזהרה: schema לא-קריא, מדולג: {fn} ({e})", file=sys.stderr) - skipped += 1 + # (buildSchemaLookup בולע קובץ לא-פריס בשקט). + if fn in KNOWN_UNREADABLE_SCHEMAS: + known_skipped.append(fn) + continue + unreadable.append((fn, str(e))) continue + # JSON תקין אך לא-שמיש הוא בדיוק אותו נזק כמו JSON לא-פריס: היבואן מפיל גם + # אותו (בלי אובייקט schema מקונן אין payload), הספר נעדר גם מהצפי וגם מה-DB, + # וכל בדיקות ההשוואה עוברות על קבוצה שהצטמצמה בשקט מתחת לשער הסחיפה. לכן + # אותה הנהלת-חשבונות בדיוק: החרגה מנומקת ב-KNOWN_UNREADABLE_SCHEMAS, או כשל. if not isinstance(top, dict): - continue - schema = top.get("schema") - if not isinstance(schema, dict): + unusable = f"top-level JSON is {type(top).__name__}, not an object" + else: + schema = top.get("schema") + unusable = None if isinstance(schema, dict) else "no object-valued 'schema' key" + if unusable is not None: + if fn in KNOWN_UNREADABLE_SCHEMAS: + known_skipped.append(fn) + continue + unreadable.append((fn, unusable)) continue en = _str_or_none(schema.get("title")) or _file_title(path) he = _str_or_none(schema.get("heTitle")) or en @@ -264,8 +289,19 @@ def load_schema_books(schemas_dir): if b.collective_en == "": b.collective_en = None books.append(b) - if skipped: - print(f"skipped {skipped} unreadable schema files", file=sys.stderr) + # דילוג ידוע = שורת INFO אחת שמסבירה למה, במקום אזהרה + שורת-ספירה שאיש אינו פועל לפיהן. + if known_skipped: + print("INFO: schemas ידועים שאינם ספרים, מדולגים: " + "; ".join( + f"{fn} ({KNOWN_UNREADABLE_SCHEMAS[fn]})" for fn in known_skipped)) + # כל schema אחר שאינו נטען לספר בארכיון ספריא המקובע (שה-digest שלו אומת) הוא + # נזק אמיתי — לא-פריס, או פריס אך בלי אובייקט schema: דילוג שקט עליו היה מצמצם + # את כיסוי הבדיקה ועדיין מדווח PASS. כשל רועש. + if unreadable: + for fn, msg in unreadable[:10]: + print(f" schema לא-שמיש: {fn} ({msg})", file=sys.stderr) + die(f"{len(unreadable)} קובצי schema לא-שמישים בארכיון ספריא המקובע " + f"(מעבר ל-{sorted(KNOWN_UNREADABLE_SCHEMAS)} הידועים) — " + "כיסוי הבדיקה היה מצטמצם בשקט") return books @@ -305,3 +341,82 @@ def die(msg): def ok(msg): print(f"PASS: {msg}") sys.exit(0) + + +# ─── שער סחיפה מול ה-reference snapshot ──────────────────────────────────── +# עד כאן כל בדיקה שנושאת baseline קשיח רק הדפיסה אותו לצד הערך הנמדד ועברה בכל +# מקרה. הרצה 34024655297 פרסמה DB עם dependenceType 4940 מול snapshot 4941, +# book_base_text 5425 מול 5426 ו-baseProvenance=1 12970 מול 13056 (‎−0.659%) — +# שלושתן PASS. סחיפה איטית הייתה נראית רק לאדם שקורא את הלוג. +# +# השער חד-צדדי ונדיב בכוונה: +# * התכווצות גדולה מ-QA_DRIFT_MAX_SHRINK_PCT מה-snapshot → ‎::error:: ויציאה 1 +# * כל הפרש אחר (התכווצות קטנה יותר, או גדילה) → ‎::warning:: עם המספרים, +# כי הקורפוס אכן זז בין ייצוא לייצוא וזה לא יהפוך לאזעקת-שווא שבועית. +# הסף מגיע מהסביבה: QA_DRIFT_MAX_SHRINK_PCT, וברירת המחדל שלו נקבעת ב-workflow +# השבועי (צעד ה-QA). כשהמשתנה אינו מוגדר השער כבוי — ה-baselines משמעותיים רק מול +# הקורפוס המקובע האמיתי, לא מול פיקסטורה סינתטית או DB חלקי בהרצת אד-הוק. כדי +# שהשער לא ייעלם בשקט מהרצת release, run_all.py ‎--require-all נכשל כשהוא אינו +# מוגדר. אין בכך רענון snapshot: ‎--expect-snapshot ממשיך לאכוף את המספרים המדויקים. +DRIFT_ENV = "QA_DRIFT_MAX_SHRINK_PCT" +# רצפה מוחלטת: התכווצות היא ‎::error:: רק כשהיא חוצה גם את האחוז וגם את המספר +# הזה. בלעדיה, על מדד קטן (guides=2, midrash=5, זוגות מוסקים=20, targum=45) +# ירידה של 1 = 2%..50% והייתה מפילה בנייה שבועית על ספר יחיד שנעלם מהייצוא. +# ברירת המחדל 10 חלה גם כשהמשתנה אינו מוגדר; ה-workflow קובע אותו במפורש. +DRIFT_ABS_ENV = "QA_DRIFT_MIN_SHRINK_ABS" +DRIFT_ABS_DEFAULT = 10 + + +def drift_max_shrink_pct(): + """הסף באחוזים, או None כשהשער כבוי (המשתנה אינו מוגדר/ריק).""" + raw = os.environ.get(DRIFT_ENV, "").strip() + if raw == "": + return None + try: + value = float(raw) + except ValueError: + die(f"{DRIFT_ENV} אינו מספר: {raw!r}") + if value < 0: + die(f"{DRIFT_ENV} חייב להיות ≥ 0: {raw!r}") + return value + + +def drift_min_shrink_abs(): + """הרצפה המוחלטת (מספר שלם ≥ 0); ברירת מחדל DRIFT_ABS_DEFAULT כשלא מוגדר.""" + raw = os.environ.get(DRIFT_ABS_ENV, "").strip() + if raw == "": + return DRIFT_ABS_DEFAULT + try: + value = int(raw) + except ValueError: + die(f"{DRIFT_ABS_ENV} אינו מספר שלם: {raw!r}") + if value < 0: + die(f"{DRIFT_ABS_ENV} חייב להיות ≥ 0: {raw!r}") + return value + + +def gate_snapshot_drift(label, observed, snapshot): + """משווה מדד סָפוּר אחד ל-reference snapshot שלו ומחיל את השער. + + יוצא 1 (עם ‎::error::) כשההתכווצות עוברת את הסף; אחרת מדפיס שורה אחת בלבד. + כשהשער כבוי אינו מדפיס דבר — הערך וה-snapshot כבר הודפסו על ידי הבדיקה עצמה. + """ + limit = drift_max_shrink_pct() + if limit is None: + return + delta = observed - snapshot + if delta == 0: + print(f"drift {label}: 0 (snapshot {snapshot})") + return + pct = (abs(delta) * 100.0 / snapshot) if snapshot else float("inf") + detail = f"{label}: DB={observed} snapshot={snapshot} delta={delta:+d} ({pct:.3f}%)" + if delta < 0 and pct > limit: + floor = drift_min_shrink_abs() + if abs(delta) >= floor: + print(f"::error::drift {detail} — התכווצות מעבר ל-{DRIFT_ENV}={limit}% " + f"וגם ל-{DRIFT_ABS_ENV}={floor}", file=sys.stderr) + sys.exit(1) + print(f"::warning::drift {detail} (מעבר ל-{DRIFT_ENV}={limit}%, אך מתחת " + f"לרצפה המוחלטת {DRIFT_ABS_ENV}={floor})") + return + print(f"::warning::drift {detail} (בתוך {DRIFT_ENV}={limit}%)") diff --git a/scripts/qa/run_all.py b/scripts/qa/run_all.py index 23c804a5..911b0db4 100755 --- a/scripts/qa/run_all.py +++ b/scripts/qa/run_all.py @@ -4,6 +4,8 @@ ברירת מחדל: בדיקה שאין לה את הארגומנט הנדרש (למשל --metrics) מדולגת (SKIP) — נוח להרצות אד-הוק. בהרצת release יש להעביר --require-all: אז כל דילוג הוא כשל (יציאה שונה מ-0), כדי שבנייה ששכחה --metrics לא "תעבור" בשקט ותפספס בדיקה 10.5. +‏--require-all גם דורש QA_DRIFT_MAX_SHRINK_PCT בסביבה, אחרת שער הסחיפה מול ה- +reference snapshot (common.gate_snapshot_drift) היה כבוי בהרצת release — בשקט. """ import argparse import os @@ -11,6 +13,8 @@ import sys HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +from common import DRIFT_ENV # noqa: E402 # (script, needs_sefaria_dir, accepts_expect_snapshot, needs_metrics) SCRIPTS = [("check3_elucidation.py", False, False, False), @@ -35,10 +39,20 @@ def main(): "מחמיר את הצלבת ה-DB בבדיקה 5 מ-≥ ל-==") ap.add_argument("--expect-snapshot", action="store_true") ap.add_argument("--require-all", action="store_true", - help="הרצת release: כל דילוג הוא כשל (יציאה!=0). ברירת המחדל " + help="הרצת release: כל דילוג הוא כשל (יציאה!=0), ו-" + "QA_DRIFT_MAX_SHRINK_PCT חייב להיות מוגדר. ברירת המחדל " "מתירה דילוגים להרצות אד-הוק.") args = ap.parse_args() + # ‏--require-all הוא מצב release. שער הסחיפה מול ה-reference snapshot + # (common.gate_snapshot_drift) מופעל רק כשהסף מוגדר בסביבה, ולכן הרצת release + # שאיבדה את המשתנה הייתה מריצה את הבדיקות בלי שער — בדיוק המצב שהשער בא לתקן, + # ובשקט. נכשל מיד, עם השם המדויק של מה שחסר. + if args.require_all and os.environ.get(DRIFT_ENV, "").strip() == "": + print(f"::error::--require-all (הרצת release) ללא {DRIFT_ENV} — שער הסחיפה " + "מול ה-reference snapshot היה כבוי", file=sys.stderr) + sys.exit(1) + results = [] for name, needs_sefaria, accepts_snapshot, needs_metrics in SCRIPTS: if needs_sefaria and not args.sefaria_dir: diff --git a/scripts/qa/tests/test_qa_synthetic.py b/scripts/qa/tests/test_qa_synthetic.py index 442d3bbe..16776e7d 100644 --- a/scripts/qa/tests/test_qa_synthetic.py +++ b/scripts/qa/tests/test_qa_synthetic.py @@ -4,6 +4,8 @@ הרצה: python3 scripts/qa/tests/test_qa_synthetic.py (יציאה 0 = הכול עבר). כל בדיקה בונה schemas/DB מינימליים ומריצה את הסקריפט האמיתי כתת-תהליך. """ +import contextlib +import io import json import os import sqlite3 @@ -26,9 +28,17 @@ def _check(name, cond, detail=""): _FAILURES.append(name) -def _run(script, *args): +def _run(script, *args, env_extra=None): + # הפיקסטורות כאן הן DB-ים בני ארבע שורות, לא הקורפוס המקובע — ולכן שער הסחיפה + # מול ה-reference snapshot (common.gate_snapshot_drift) חייב להיות כבוי בהן. + # הוא כבוי כברירת מחדל בדיוק לשם כך: QA_DRIFT_MAX_SHRINK_PCT אינו מוגדר, וה- + # workflow השבועי הוא זה שקובע אותו. מנקים אותו מהסביבה כדי שהרצה מקומית + # שהגדירה אותו לא תשנה את התוצאה. + env = dict(os.environ) + env.pop("QA_DRIFT_MAX_SHRINK_PCT", None) + env.update(env_extra or {}) p = subprocess.run([sys.executable, os.path.join(_QA, script), *args], - capture_output=True, text=True) + capture_output=True, text=True, env=env) return p.returncode, p.stdout + p.stderr @@ -397,17 +407,213 @@ def test_run_all_require_all(): _check("run_all ברירת מחדל: דילוג מותר (יציאה 0)", rc == 0, out.strip().splitlines()[-3:]) _check("run_all ברירת מחדל: מנסח 'דולגו' ולא 'כל הבדיקות עברו'", "דולגו" in out and "כל הבדיקות עברו" not in out, out.strip().splitlines()[-3:]) - rc, out = _run("run_all.py", "--db", db, "--require-all") + # הרצת release אמיתית מגדירה את הסף; כאן 100% כדי שהשער עצמו לא יפיל + # פיקסטורה בת ארבע שורות — הנבדק הוא התנהגות הדילוגים. + rc, out = _run("run_all.py", "--db", db, "--require-all", + env_extra={"QA_DRIFT_MAX_SHRINK_PCT": "100"}) _check("run_all --require-all: דילוג → כשל (יציאה!=0)", rc != 0, out.strip().splitlines()[-3:]) _check("run_all --require-all: מפרט את הארגומנט המפעיל (--metrics)", "--metrics" in out, out.strip().splitlines()[-4:]) + # ‏--require-all ללא הסף = הרצת release בלי שער סחיפה. חייב להיכשל מיד, + # ולנקוב בשם המשתנה החסר. + rc, out = _run("run_all.py", "--db", db, "--require-all") + _check("run_all --require-all ללא QA_DRIFT_MAX_SHRINK_PCT → כשל", rc != 0, + out.strip().splitlines()[-2:]) + _check("run_all: הכשל נוקב בשם QA_DRIFT_MAX_SHRINK_PCT", + "QA_DRIFT_MAX_SHRINK_PCT" in out, out.strip().splitlines()[-2:]) + + +# --- שער הסחיפה מול ה-reference snapshot ----------------------------------------- +def test_snapshot_drift_gate(): + print("gate_snapshot_drift: כבוי כברירת מחדל, ‎::warning:: בתוך הסף, ‎::error:: מעבר לו") + with tempfile.TemporaryDirectory() as tmp: + schemas = os.path.join(tmp, "schemas") + os.makedirs(schemas) + _write_schema(schemas, "dep.json", "Commentary A", "פירוש א", base_he="בסיס") + _write_schema(schemas, "base.json", "Base", "בסיס") + db = os.path.join(tmp, "ok.db") + _make_db(db, + books=[(10, "פירוש א", None, None, None, 0, 999, SRC_SEFARIA), + (20, "בסיס", None, None, None, 1, 1, SRC_SEFARIA)], + bbt=[(10, 20)]) + # הפיקסטורה נושאת שורה אחת מול baseline של 5,426 — התכווצות של ~99.98%. + rc, out = _run("check2_book_base_text.py", "--db", db, "--sefaria-dir", schemas) + _check("שער כבוי (המשתנה אינו מוגדר): עובר ואינו מדפיס drift", + rc == 0 and "drift " not in out, out.strip().splitlines()[-2:]) + rc, out = _run("check2_book_base_text.py", "--db", db, "--sefaria-dir", schemas, + env_extra={"QA_DRIFT_MAX_SHRINK_PCT": "100"}) + _check("סף רחב: עובר עם ‎::warning:: הנוקב במספרים", + rc == 0 and "::warning::drift" in out and "snapshot=5426" in out, + out.strip().splitlines()[-3:]) + rc, out = _run("check2_book_base_text.py", "--db", db, "--sefaria-dir", schemas, + env_extra={"QA_DRIFT_MAX_SHRINK_PCT": "2"}) + _check("סף 2% (ברירת המחדל של ה-workflow): ‎::error:: ויציאה!=0", + rc != 0 and "::error::drift" in out, out.strip().splitlines()[-2:]) + rc, out = _run("check2_book_base_text.py", "--db", db, "--sefaria-dir", schemas, + env_extra={"QA_DRIFT_MAX_SHRINK_PCT": "לא-מספר"}) + _check("סף לא-מספרי: כשל, לא שער כבוי בשקט", rc != 0, + out.strip().splitlines()[-2:]) + # ארבעת המקרים של השער עצמו, בתהליך, בלי לבנות DB לכל אחד. + def _gate(observed, snapshot, limit, floor=None): + previous = os.environ.get("QA_DRIFT_MAX_SHRINK_PCT") + previous_floor = os.environ.get("QA_DRIFT_MIN_SHRINK_ABS") + if limit is None: + os.environ.pop("QA_DRIFT_MAX_SHRINK_PCT", None) + else: + os.environ["QA_DRIFT_MAX_SHRINK_PCT"] = limit + if floor is None: + os.environ.pop("QA_DRIFT_MIN_SHRINK_ABS", None) + else: + os.environ["QA_DRIFT_MIN_SHRINK_ABS"] = floor + buf = io.StringIO() + code = 0 + try: + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + common.gate_snapshot_drift("m", observed, snapshot) + except SystemExit as exit_code: + code = exit_code.code + finally: + os.environ.pop("QA_DRIFT_MAX_SHRINK_PCT", None) + if previous is not None: + os.environ["QA_DRIFT_MAX_SHRINK_PCT"] = previous + os.environ.pop("QA_DRIFT_MIN_SHRINK_ABS", None) + if previous_floor is not None: + os.environ["QA_DRIFT_MIN_SHRINK_ABS"] = previous_floor + return code, buf.getvalue() + + code, out = _gate(100, 100, "2") + _check("דלתא 0: שורה חיובית אחת, לא אזהרה", + code == 0 and out.strip() == "drift m: 0 (snapshot 100)", out) + code, out = _gate(99, 100, "2") + _check("התכווצות בתוך הסף: ‎::warning:: עם המספרים", + code == 0 and "::warning::drift" in out and "delta=-1" in out, out) + code, out = _gate(150, 100, "2") + _check("גדילה: ‎::warning:: בלבד, לעולם לא כשל", + code == 0 and "::warning::drift" in out and "delta=+50" in out, out) + code, out = _gate(97, 100, "2", "0") + _check("התכווצות מעבר לסף (רצפה 0): ‎::error:: ויציאה 1", + code == 1 and "::error::drift" in out, out) + code, out = _gate(97, 100, "2") + _check("מעבר לאחוז אך מתחת לרצפה 10 (ברירת מחדל): ‎::warning:: בלבד", + code == 0 and "::warning::drift" in out and "QA_DRIFT_MIN_SHRINK_ABS=10" in out, out) + code, out = _gate(1, 2, "2") + _check("guides 2→1 (50%): אזהרה, לא כשל", code == 0 and "::warning::drift" in out, out) + code, out = _gate(80, 100, "2") + _check("התכווצות 20 ≥ רצפה 10 ומעבר ל-2%: ‎::error::", + code == 1 and "::error::drift" in out, out) + code, out = _gate(80, 100, "2", "50") + _check("רצפה מפורשת 50: התכווצות 20 היא אזהרה", code == 0 and "::warning::drift" in out, out) + code, out = _gate(80, 100, "2", "לא-מספר") + _check("רצפה לא-מספרית: כשל", code not in (0, None) and "QA_DRIFT_MIN_SHRINK_ABS" in out, out) + code, out = _gate(1, 100, None) + _check("שער כבוי: שקט מוחלט", code == 0 and out == "", out) + + +# --- מדיניות schema לא-קריא: Sheet.json מוחרג במפורש, כל השאר נכשל ---------------- +def test_unreadable_schema_policy(): + print("schemas לא-קריאים: Sheet.json = INFO מוסבר, כל קובץ אחר = כשל") + with tempfile.TemporaryDirectory() as tmp: + known = os.path.join(tmp, "known") + os.makedirs(known) + _write_schema(known, "a.json", "A", "ספר א", dependence="commentary") + # בדיוק מה שהייצוא האמיתי שולח: Sheet.json באורך 0. + open(os.path.join(known, "Sheet.json"), "w", encoding="utf-8").close() + db = os.path.join(tmp, "d.db") + _make_db(db, books=[(1, "ספר א", "commentary", None, None, 0, 1, SRC_SEFARIA)]) + rc, out = _run("check6_metadata_rowbyrow.py", "--db", db, "--sefaria-dir", known) + _check("Sheet.json ריק: עובר", rc == 0, out.strip().splitlines()[-2:]) + _check("Sheet.json ריק: שורת INFO אחת שמסבירה למה", + out.count("INFO: schemas ידועים") == 1 and "Sheet.json" in out, + out.strip().splitlines()[-3:]) + _check("Sheet.json ריק: אין יותר 'אזהרה: schema לא-קריא' ולא שורת-ספירה", + "אזהרה: schema לא-קריא" not in out + and "unreadable schema files" not in out, + out.strip().splitlines()[-3:]) + # קובץ לא-פריס אחר בארכיון מקובע ומאומת-digest = נזק, לא דילוג שקט. + bogus = os.path.join(tmp, "bogus") + os.makedirs(bogus) + _write_schema(bogus, "a.json", "A", "ספר א", dependence="commentary") + with open(os.path.join(bogus, "Broken.json"), "w", encoding="utf-8") as fh: + fh.write("not json") + rc, out = _run("check6_metadata_rowbyrow.py", "--db", db, "--sefaria-dir", bogus) + _check("schema לא-קריא שאינו ברשימה: כשל", rc != 0, out.strip().splitlines()[-2:]) + _check("הכשל נוקב בקובץ", "Broken.json" in out, out.strip().splitlines()[-3:]) + + +# --- schema פריס אך לא-שמיש: מערך בשורש / אובייקט בלי schema מקונן --------------- +def test_unusable_schema_shape_policy(): + print("schemas פריסים אך לא-שמישים: נספרים, ניתנים להחרגה מפורשת, אחרת כשל") + before = len(_FAILURES) + + def _load(d): + # load_schema_books בתהליך: die() הוא sys.exit ולכן נתפס כאן כ-SystemExit. + buf = io.StringIO() + code, books = 0, [] + try: + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + books = common.load_schema_books(d) + except SystemExit as exit_code: + code = exit_code.code or 1 + return code, buf.getvalue(), books + + with tempfile.TemporaryDirectory() as tmp: + d = os.path.join(tmp, "schemas") + os.makedirs(d) + _write_schema(d, "Good.json", "Good", "ספר טוב", dependence="commentary") + # שני הקבצים האלה הם JSON תקין לחלוטין — ובדיוק לכן דולגו בשקט עד כה. + with open(os.path.join(d, "Array.json"), "w", encoding="utf-8") as fh: + json.dump([{"schema": {"title": "A", "heTitle": "א"}}], fh, ensure_ascii=False) + with open(os.path.join(d, "NoSchema.json"), "w", encoding="utf-8") as fh: + json.dump({"title": "B", "heTitle": "ב"}, fh, ensure_ascii=False) + + code, out, books = _load(d) + _check("מערך בשורש + אובייקט בלי schema: כשל, לא דילוג שקט", code != 0, out) + _check("הכשל נוקב בשני הקבצים ובסיבת כל אחד", + "Array.json" in out and "top-level JSON is list, not an object" in out + and "NoSchema.json" in out and "no object-valued 'schema' key" in out, out) + _check("שניהם נספרים (2), לא נבלעים", "2 קובצי schema לא-שמישים" in out, out) + + # אותם קבצים בדיוק, מוחרגים בשם ובסיבה כמו Sheet.json: דילוג מוסבר, לא כשל. + original = common.KNOWN_UNREADABLE_SCHEMAS + try: + common.KNOWN_UNREADABLE_SCHEMAS = dict( + original, **{"Array.json": "פיקסטורה: אינו ספר", + "NoSchema.json": "פיקסטורה: אינו ספר"}) + code, out, books = _load(d) + finally: + common.KNOWN_UNREADABLE_SCHEMAS = original + _check("מוחרגים ברשימה המפורשת: עוברים", code == 0, out) + _check("מוחרגים ברשימה: שורת INFO אחת עם שני השמות", + out.count("INFO: schemas ידועים") == 1 + and "Array.json" in out and "NoSchema.json" in out, out) + _check("מוחרגים ברשימה: הספר התקין עדיין נטען", + [b.he_title for b in books] == ["ספר טוב"], [b.he_title for b in books]) + + # ומקצה-לקצה: בדיקה אמיתית נכשלת, במקום לעבור על קבוצת-צפי שהצטמצמה בשקט. + with tempfile.TemporaryDirectory() as tmp: + d = os.path.join(tmp, "schemas") + os.makedirs(d) + _write_schema(d, "a.json", "A", "ספר א", dependence="commentary") + with open(os.path.join(d, "NoSchema.json"), "w", encoding="utf-8") as fh: + json.dump({"title": "B"}, fh) + db = os.path.join(tmp, "d.db") + _make_db(db, books=[(1, "ספר א", "commentary", None, None, 0, 1, SRC_SEFARIA)]) + rc, out = _run("check6_metadata_rowbyrow.py", "--db", db, "--sefaria-dir", d) + _check("check6 מקצה-לקצה: נכשל ונוקב בקובץ", rc != 0 and "NoSchema.json" in out, + out.strip().splitlines()[-3:]) + + # תחת pytest אין מי שקורא את _FAILURES (main אינו רץ), ולכן חוסמים כאן במפורש. + if "pytest" in sys.modules: + assert len(_FAILURES) == before, _FAILURES[before:] def main(): for t in (test_primary_beats_earlier_alias, test_resolve_order, test_check2, test_check6, test_check7, test_source_filter_regression, test_check5, + test_snapshot_drift_gate, test_unreadable_schema_policy, + test_unusable_schema_shape_policy, test_run_all_require_all): t() print() From c2b88e2685d28dff945c384dddd0b01759bc6aa6 Mon Sep 17 00:00:00 2001 From: ypl <7353755@gmail.com> Date: Wed, 9 Sep 2026 12:13:34 +0300 Subject: [PATCH 4/7] fix(ci): durable, digest-verified patch-fan anchor cache ranked by a locked monotonic use stamp; progress and named failures in the patch fan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed from the audit branch by file set (4 original commits contributed; their messages follow). --- 06b5d0b fix(ci): durable, digest-verified patch-fan anchor cache; progress and named failures in the patch fan Audit of cycle 33987355439 (item S8, reports 07/08). Prefetched patch-fan anchors (~1.3 GB seforim.db.zst each, ~5.2 GB) were downloaded in the background, deleted by the cleanup step's `rm -rf … prefetch` when attempt 34021998271 failed 45 min later, and re-downloaded in full by the successful retry (237+241+232+243 = 953 s, ~249 s wall at parallel=5, plus downlink contention with the generator). Anchors are immutable content-addressed release assets. - prefetch_patch_anchors.sh: durable cache at ${XDG_CACHE_HOME:-$HOME/.cache}/seforimlibrary/patch-anchors (the same per-runner root the image embedder uses; never the tmpfs — path AND `stat -f` filesystem-type guard), keyed by release tag, sha256-verified against the release's published digest before every reuse (mismatch, republished tag, or missing digest with no recorded sha -> ::warning:: + fresh download), hardlinked in and out (copy fallback across filesystems; a partial copy is removed so gh's no-clobber download can proceed), bounded at 8 anchors / 30 days by cache_prune under a lock that goes stale after 600 s (a pruner killed mid-run no longer disables the bound), applied on all-hit runs too. Tag names are validated as single path components (`..` refused). One line per anchor: `reused from cache (sha256 ok)` / `downloaded in Ns`; unwritable cache -> warn and download to the run dir. The cleanup step only reports on the cache; S2's temp sweep is disjoint by root and by name; `rm -rf prefetch` drops only hardlinks. - The five bare `ANCHOR` lines are `anchor (offset N): `; the abort paths of prefetch_patch_anchors.sh and upload_early_release_assets.sh say what they aborted and why (still never failing the job); a download failure names the missing asset. - "Produce + verify patch fan" was dark for 22 minutes: numbered per-anchor start lines, a per-anchor end line (elapsed, patch size, verify result, the folded full-snapshot columns) and a 5-minute heartbeat while producing. The launcher command line and anchor selection are unchanged. - QA fixes: the new cache-report call in the if: always() cleanup step is guarded (a job dying before the pipeline-control checkout would have gone red on exit 127); cache-store failures are ::warning::. Tests: test_prefetch_patch_anchors.py (cold/warm/corrupted/bound/stale lock/abort + 8 edge cases: republished tag, absent digest, no-hardlink fs, partial copy, unwritable root, hardlink survival, age bound, set -u without HOME), workflow-contract tests for the cache path, cleanup exclusion, sweep exclusion, progress lines and guarded helper calls; 123 pytest; 45/45 step bodies bash -n; 0 CR in the diff. Known limit: the heartbeat lists anchors in batch order, so an anchor that finishes out of order stays listed until its predecessor lands (over-inclusive, never a false all-clear). --- 2350a6c fix(ci): rank the patch-anchor cache by a monotonic use stamp, not by whole-second mtime prefetch_patch_anchors.sh pruned by `stat %Y` + `sort -rn`: entries stored or refreshed within the same second tied, sort fell back to comparing the rest of the line, and the entry just downloaded (v23-x) could rank below older ones and be evicted. In the reconcile job the eviction test flaked 0/6 green; in production the cache could discard the anchor it had spent ~16 minutes fetching. Each entry now records used_ns = max(now_ns, highest used_ns in the cache + 1) in .meta on store and on every hit (a Lamport stamp: later writes always rank higher, whatever the clock or filesystem resolution), and the pruner sorts by that key with the .meta mtime as the tie-break for pre-fix entries. The refresh rewrites .meta via temp+rename under $BASHPID (the five parallel fetches are subshells sharing $$). Keep count, age cutoff, eviction report line and lock protocol are unchanged; the new tests fail on the previous script with the defect itself. --- 1ba52c6 test(ci): gate the anchor-cache sandbox suites on a capability probe instead of failing on BSD tools The three sandbox classes in test_prefetch_patch_anchors.py drive prefetch_patch_anchors.sh and the fan step through a real bash and need mapfile (bash 4), GNU stat --format, GNU sed -i and sha256sum; on macOS they failed instead of skipping. One cached bash probe at import now decides HAS_GNU_SANDBOX and the classes skip with the missing capability named. The probe is by feature, not platform: Windows Git Bash still runs all 22 tests, and on Linux a probe that comes back false raises instead of skipping, so a broken probe cannot silently retire the S8/S12 coverage in the reconcile and contracts jobs. --- e4956d8 fix(ci): serialise the anchor-cache use stamp under flock; the prune never falls back to tag order Round 3 of the audit of cycle 33987355439 (item S18). The five parallel prefetchers allocated `used_ns` as read-max → write with no lock, so two could get one stamp (reachable on the highest+1 branch or with a %N-less date), and a full tie made `sort` decide by whole line, i.e. by tag — the freshest anchor could still be evicted. Allocation now runs under `flock -w 30` on `/.stamp.lock`, the floor is persisted in `/.stamp` so a `.meta` not yet written cannot be overtaken, the fallback key is `.` when flock is unavailable, and the prune sorts stably on three numeric keys. Tests: STAMP_ALLOC_RACE phase (20 concurrent marks → distinct, strictly increasing keys; the last-marked anchor survives a keep-1 prune under forced future stamps) and a rank-contract test on the flock/sort text. Prefetch suite 26/26 on a CR-normalised copy. Co-Authored-By: Claude Fable 5.1 --- .github/scripts/patch_fan_lib.sh | 355 ++++ .github/scripts/prefetch_patch_anchors.sh | 529 +++++- .../scripts/test_prefetch_patch_anchors.py | 1421 +++++++++++++++++ .../scripts/upload_early_release_assets.sh | 32 +- 4 files changed, 2278 insertions(+), 59 deletions(-) create mode 100644 .github/scripts/patch_fan_lib.sh create mode 100644 .github/scripts/test_prefetch_patch_anchors.py diff --git a/.github/scripts/patch_fan_lib.sh b/.github/scripts/patch_fan_lib.sh new file mode 100644 index 00000000..b9e1d069 --- /dev/null +++ b/.github/scripts/patch_fan_lib.sh @@ -0,0 +1,355 @@ +#!/usr/bin/env bash +# Shared machinery for the weekly DB release's "Produce + verify patch fan" +# step: one anchor start to finish, the heartbeat that keeps the step from +# going dark, and the batch driver's drain. Sourced, not run. +# +# WHY THIS EXISTS +# --------------- +# GitHub Actions refuses to LOAD a workflow whose single `run:` template is +# longer than 21,000 characters. Run 34195296928 never started a job: +# "Invalid workflow file … (Line: 1974, Col: 14): Exceeded max expression +# length 21000" — the fan's body had grown to 22,592 characters. Taking the +# function definitions out leaves the step body as the driver alone. +# +# SHELL SEMANTICS +# --------------- +# Sourced into the step's own shell, so nothing about how any of this runs +# changes: the caller's `set -euo pipefail` is inherited verbatim (that is why +# this is sourced and not executed), the variables below are the step's own +# globals, and `produce_anchor` still forks from the step's shell when the +# driver backgrounds it. Deliberately NO `set` line here — re-issuing the +# options would be a no-op for the one caller and would silently impose them +# on anyone else who sources this. +# +# Read at SOURCE time: RUNNER_TEMP, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT and the +# optional PATCH_FAN_HEARTBEAT_SECONDS (the heartbeat's cadence and paths). +# Read at CALL time, from the driver's globals: THIS_VER, THIS_SCHEMA, +# PREFETCH_DIR, PREFETCH_WAIT_SECONDS, PATCH_MAIN_CLASS, PATCH_JVM_ARGS, +# PATCH_CLASSPATH and the BATCH_* arrays. + +# קורא db_schema_version מ-DB; טבלה/שורה חסרה → ERROR (release DB חייב מוטבע). +# sqlite3 זמין ברנרים ה-self-hosted (בשימוש גם בוורקפלואי delta-real-diff-*). +read_schema() { + local db="$1" v + if ! v=$(sqlite3 "$db" "SELECT value FROM schema_meta WHERE key='db_schema_version'" 2>/dev/null); then + echo "::error::schema_meta missing/unreadable in $db (release DB must be stamped)" >&2 + return 1 + fi + [ -n "$v" ] || { echo "::error::db_schema_version row missing in $db (release DB must be stamped)" >&2; return 1; } + printf '%s' "$v" +} + +# One anchor, start to finish: pre-check, reconstitute, produce, +# verify, clean up. Every path it touches is scoped to its own offset +# so two of these can run side by side, it writes to its own log, and +# its exit status is the only channel back to the driver: 0 for a +# produced patch AND for every legitimate skip, non-zero only for a +# genuine failure — which fails the step exactly as `set -e` inside +# the serial loop always did. +produce_anchor() { # + local OFFSET="$1" TARGET_VER="$2" TAG="$3" + local ANCHOR_DIR="prev-dbs/anchor-$OFFSET" + local META_DIR="prev-meta/anchor-$OFFSET" + local PREV_DB="$ANCHOR_DIR/seforim.db" + local PATCH_OUT="$PWD/patches/patch-v${TARGET_VER}-v${THIS_VER}.db" + local PRECHECK WAIT_BUDGET PREFETCH_STATE PREFETCH_WAITED + local PREV_SCHEMA PRODUCE_RC REASON + local T_START T_DOWNLOADED T_EXTRACTED T_DONE + echo "=== Producing patch v${TARGET_VER} → v${THIS_VER} (offset $OFFSET, tag=$TAG) ===" + + # Reconstituting an anchor costs 110–135 s of download plus a + # decompress; run 33865604251 paid ~140 s for the v10 anchor only + # to be told it was unpatchable. patch_anchor_schema.py answers the + # same question first, from the anchor's own build_provenance.json + # `db_schema` block (a few KB, exact from db_version 27 on — the + # published v26 provenance is schema_version 3 and carries none) or — + # for releases that predate that block — from a documented list of + # db_versions proven unpatchable by this repository's history. + # It is advisory in one direction only: PROCEED is NOT a + # patchability claim, the producer's exit-3 + ".unpatchable" + # marker below stays the sole authority, and the pre-check itself + # can never fail the release (any surprise degrades to PROCEED). + # --contract-tables scopes the column comparison to the producer's + # own table list. It comes from the PAYLOAD checkout — the same + # commit whose PatchDbProducer runs below, not the control-plane + # checkout — because a comparison wider than that list would skip + # anchors the producer would still have patched. + rm -rf "$META_DIR" + mkdir -p "$META_DIR" + # An anchor without provenance is normal (every release before + # v27 carries none) but gh's bare "no assets match the file + # pattern" never says which asset was missing — this does. + if ! gh release download "$TAG" \ + --pattern 'build_provenance.json' \ + --dir "$META_DIR" 2>"$META_DIR/gh.err"; then + echo "anchor v${TARGET_VER} ($TAG): release carries no build_provenance.json ($(tr -d '\r' < "$META_DIR/gh.err" | head -n1)) — the pre-check falls back to the documented unpatchable list" + fi + rm -f "$META_DIR/gh.err" + PRECHECK=$(python3 .pipeline-control/.github/scripts/patch_anchor_schema.py check \ + --this-schema build/db_schema.json \ + --anchor-version "$TARGET_VER" \ + --anchor-provenance "$META_DIR/build_provenance.json" \ + --contract-tables generator/common/src/jvmTest/resources/patch_tables_contract.json) \ + || PRECHECK="PROCEED pre-check did not run — deferring to the producer" + rm -rf "$META_DIR" + echo "pre-check: $PRECHECK" + if [ "${PRECHECK%% *}" = UNPATCHABLE ]; then + echo "::warning::anchor v${TARGET_VER} ($TAG): ${PRECHECK#* } — pre-download schema check declared the anchor unpatchable; skip anchor" + return 0 + fi + + rm -rf "$ANCHOR_DIR" + mkdir -p "$ANCHOR_DIR" + # A1: this anchor was very probably downloaded and verified (size, + # and digest when GitHub published one) while the DB was being + # generated. Wait for THIS tag's marker; anything short of a + # verified hit — no marker inside the budget, a failed or + # unpatchable verdict, a vanished file — falls straight back to the + # unchanged serial download below. A missing marker also aborts the + # prefetch, so it cannot keep competing for the downlink. + # The "stop waiting" verdict has to outlive this anchor: the ones + # still to come run in their own subshells, where a shell variable + # could not reach them, so it is recorded as a file next to the + # prefetch state. + WAIT_BUDGET="$PREFETCH_WAIT_SECONDS" + if [ -f "$PREFETCH_DIR/.abandoned" ]; then + WAIT_BUDGET=0 + fi + PREFETCH_STATE=absent + if [ -f "$PREFETCH_DIR/.started" ]; then + PREFETCH_WAITED=0 + while [ ! -f "$PREFETCH_DIR/$TAG/.done" ] && [ ! -f "$PREFETCH_DIR/.abandoned" ] && [ "$PREFETCH_WAITED" -lt "$WAIT_BUDGET" ]; do + sleep 5 + PREFETCH_WAITED=$((PREFETCH_WAITED + 5)) + done + if [ -f "$PREFETCH_DIR/$TAG/.done" ]; then + PREFETCH_STATE=$(head -n1 "$PREFETCH_DIR/$TAG/.done") + sed -n '2,$p' "$PREFETCH_DIR/$TAG/.done" + else + echo "::warning::anchor v${TARGET_VER} ($TAG): no prefetch marker after ${PREFETCH_WAITED}s — falling back to the serial download" + bash .pipeline-control/.github/scripts/prefetch_patch_anchors.sh abort "$PREFETCH_DIR" \ + || echo "::warning::anchor v${TARGET_VER} ($TAG): the prefetch abort helper itself failed — continuing with the serial download" + : > "$PREFETCH_DIR/.abandoned" + fi + fi + T_START=$(date +%s) + if [ "$PREFETCH_STATE" = ok ] && [ -s "$PREFETCH_DIR/$TAG/seforim.db.zst" ]; then + mv "$PREFETCH_DIR/$TAG/seforim.db.zst" "$ANCHOR_DIR/seforim.db.zst" + echo "anchor v${TARGET_VER} ($TAG): reused the prefetched DB" + else + # Under `set -e` a failed download used to end this subshell with + # nothing but gh's own bare stderr, and the driver could only say + # "exit code 1". Name the release, the asset and gh's reason. + if ! gh release download "$TAG" \ + --pattern 'seforim.db.zst' \ + --dir "$ANCHOR_DIR" 2>"$ANCHOR_DIR/gh.err"; then + echo "::error::anchor v${TARGET_VER} ($TAG): could not download seforim.db.zst from that release ($(tr -d '\r' < "$ANCHOR_DIR/gh.err" | head -n1)) — no patch can be produced against this anchor" + rm -rf "$ANCHOR_DIR" + return 1 + fi + rm -f "$ANCHOR_DIR/gh.err" + fi + T_DOWNLOADED=$(date +%s) + if ! unzstd -c "$ANCHOR_DIR/seforim.db.zst" > "$PREV_DB"; then + echo "::error::anchor v${TARGET_VER} ($TAG): seforim.db.zst from that release could not be decompressed — no patch can be produced against this anchor" + rm -rf "$ANCHOR_DIR" + return 1 + fi + T_EXTRACTED=$(date +%s) + test -s "$PREV_DB" || { echo "::error::seforim.db.zst not found in release $TAG"; return 1; } + + # Explicitly supported schema transitions. The producer derives + # contract promotions from both signed schema versions, rebuilds + # newly tracked tables from a full snapshot, and verifies a real + # apply. Unknown transitions remain fail-closed per anchor. + PREV_SCHEMA=$(read_schema "$PREV_DB") || return 1 + if [ "$PREV_SCHEMA" != "$THIS_SCHEMA" ]; then + if { [ "$PREV_SCHEMA" = 2 ] && [ "$THIS_SCHEMA" = 3 ]; } || \ + { [ "$PREV_SCHEMA" = 1 ] && [ "$THIS_SCHEMA" = 4 ]; } || \ + { [ "$PREV_SCHEMA" = 2 ] && [ "$THIS_SCHEMA" = 4 ]; } || \ + { [ "$PREV_SCHEMA" = 3 ] && [ "$THIS_SCHEMA" = 4 ]; } || \ + { [ "$PREV_SCHEMA" = 1 ] && [ "$THIS_SCHEMA" = 5 ]; } || \ + { [ "$PREV_SCHEMA" = 2 ] && [ "$THIS_SCHEMA" = 5 ]; } || \ + { [ "$PREV_SCHEMA" = 3 ] && [ "$THIS_SCHEMA" = 5 ]; } || \ + { [ "$PREV_SCHEMA" = 4 ] && [ "$THIS_SCHEMA" = 5 ]; }; then + echo "schema $PREV_SCHEMA → $THIS_SCHEMA — producing the supported cross-schema delta" + else + echo "schema $PREV_SCHEMA → $THIS_SCHEMA is unsupported — skip anchor" + rm -rf "$ANCHOR_DIR" + return 0 + fi + fi + + # A column added without a db_schema_version bump (e.g. + # category.heShortDesc, 2026-07-16, still schema 1) no longer costs + # us the anchor: PatchDbProducer emits an + # `ALTER TABLE … ADD COLUMN` migration — plain SQL that every + # released PatchApplier already runs before the upserts — and ships + # every row whose new value is non-NULL. The shell pre-check that + # compared PRAGMA table_info is therefore gone; the producer itself + # is now the single authority on what is patchable. + # + # It declares an anchor unpatchable only when no patch could ever + # express the change (a PRIMARY KEY column missing from prev, or a + # column dropped without a bump): PatchPipelineCli then exits 3 + # (UnpatchableAnchor) and drops ".unpatchable" carrying the + # reason, and this anchor is skipped for that one code. + # Every other non-zero exit still fails the release. + rm -f "$PATCH_OUT.unpatchable" + PRODUCE_RC=0 + if [ -n "$PATCH_MAIN_CLASS" ]; then + # Exactly what `gradle :generator-common:producePatchAndVerify` + # forks: main class, -Xmx/GC flags and runtime classpath all come + # from that task's own definition (see patchPipelineLauncher in + # generator/common/build.gradle.kts), and its -P properties are + # the -D system properties the task sets on the fork. ZSTD_LEVEL + # reaches the CLI through the step env either way. Keeping Gradle + # out of the loop is what lets two of these run at once. + java $PATCH_JVM_ARGS -cp "$PATCH_CLASSPATH" \ + -DprevDb=$PWD/$PREV_DB \ + -DnewDb=$PWD/build/seforim.db \ + -Dout=$PATCH_OUT \ + -DfromVersion=$TARGET_VER -DtoVersion=$THIS_VER \ + "$PATCH_MAIN_CLASS" || PRODUCE_RC=$? + else + gradle :generator-common:producePatchAndVerify \ + -PprevDb=$PWD/$PREV_DB \ + -PnewDb=$PWD/build/seforim.db \ + -Pout=$PATCH_OUT \ + -PfromVersion=$TARGET_VER -PtoVersion=$THIS_VER \ + --no-daemon --stacktrace || PRODUCE_RC=$? + fi + # The Gradle task maps the CLI's exit 3 onto a warning and a + # successful build; run directly, that 3 arrives here. Both funnel + # into the one marker path below, and every OTHER non-zero exit + # fails this anchor — which fails the whole step, as before. + if [ "$PRODUCE_RC" -ne 0 ] && [ "$PRODUCE_RC" -ne 3 ]; then + echo "::error::anchor v${TARGET_VER} ($TAG): producePatchAndVerify failed with exit code $PRODUCE_RC" + rm -rf "$ANCHOR_DIR" + return "$PRODUCE_RC" + fi + if [ "$PRODUCE_RC" -eq 3 ] || [ -f "$PATCH_OUT.unpatchable" ]; then + REASON=$(cat "$PATCH_OUT.unpatchable" 2>/dev/null || true) + echo "::warning::anchor v${TARGET_VER} ($TAG): ${REASON:-see PatchPipelineCli output} — producer declared the anchor unpatchable; skip anchor" + # Leave nothing behind: the marker plus the producer's half-built + # .tmp (and any stale .db) must not clutter patches/, which the + # release staging step globs. + rm -f "$PATCH_OUT.unpatchable" "$PATCH_OUT" "$PATCH_OUT.tmp" + rm -rf "$ANCHOR_DIR" + return 0 + fi + T_DONE=$(date +%s) + echo "anchor v${TARGET_VER} timings: download=$((T_DOWNLOADED - T_START))s extract=$((T_EXTRACTED - T_DOWNLOADED))s produce+verify+compress=$((T_DONE - T_EXTRACTED))s total=$((T_DONE - T_START))s" + + # Free disk for the anchors still to come — only the .zst + manifest + # need to ship; the .db was the verify ground-truth. + rm -rf "$ANCHOR_DIR" + rm -f "patches/patch-v${TARGET_VER}-v${THIS_VER}.db" \ + "patches/verify-patch-v${TARGET_VER}-v${THIS_VER}.db" + df -h / +} + +# Batch-buffered logging is why the two longest gaps in run +# 34024655297 (1318 s and 1076 s) were both inside this step with +# nothing on stdout: a hung producer looked exactly like a working +# one for 22 minutes. PatchPipelineCli emits nothing between +# "Producing patch" and its result — no progress to forward — so the +# heartbeat is the shell's: one line every HEARTBEAT_SECONDS naming +# the anchors still in flight, and one end line per anchor below. +# PATCH_FAN_HEARTBEAT_SECONDS exists so a test can drive this loop in +# seconds instead of in five-minute steps; the default is the one the +# runner uses. +HEARTBEAT_SECONDS="${PATCH_FAN_HEARTBEAT_SECONDS:-300}" +case "$HEARTBEAT_SECONDS" in ''|*[!0-9]*|0) HEARTBEAT_SECONDS=300 ;; esac +HEARTBEAT_POLL=5 +[ "$HEARTBEAT_POLL" -le "$HEARTBEAT_SECONDS" ] || HEARTBEAT_POLL="$HEARTBEAT_SECONDS" +HEARTBEAT_PID="" +HEARTBEAT_FLAG="$RUNNER_TEMP/patch-fan-heartbeat-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" +# Which anchors the heartbeat is allowed to claim are still running. +# A file, because the heartbeat is a subshell: drain_batch rewrites it +# as each anchor lands, so the line never names an anchor that has +# already finished. +HEARTBEAT_INFLIGHT="$HEARTBEAT_FLAG.inflight" +heartbeat_inflight() { #