Release OSINTai 4.2.0 reliability enhancements

.github/workflows/release-gate.yml

@@ -13,8 +13,13 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  release-gate:
-    runs-on: ubuntu-latest
+  platform-gate:
+    name: release-gate (${{ matrix.os }})
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
     timeout-minutes: 10
     steps:
       - name: Check out repository
@@ -41,7 +46,7 @@ jobs:
         run: python -W error::ResourceWarning -m unittest discover -s tests -v
 
       - name: Compile Python sources
-        run: python -m py_compile run_osintai.py src/osintai/*.py tests/*.py
+        run: python -m compileall -q src tests run_osintai.py
 
       - name: Run security analysis
         run: bandit -q -r src run_osintai.py -ll -ii
@@ -51,5 +56,16 @@ jobs:
 
       - name: Verify release identity
         run: |
-          python run_osintai.py --version | grep -Fx "OSINTai 4.0.0"
-          python run_osintai.py --help >/dev/null
+          python -c "import sys; sys.path.insert(0, 'src'); from osintai import __version__; assert f'# OSINTai v{__version__} ' in open('README.md', encoding='utf-8').read()"
+          python run_osintai.py --help
+
+  release-gate:
+    name: release-gate
+    if: ${{ always() }}
+    needs: platform-gate
+    runs-on: ubuntu-latest
+    steps:
+      - name: Require every platform gate
+        env:
+          PLATFORM_GATE_RESULT: ${{ needs.platform-gate.result }}
+        run: test "$PLATFORM_GATE_RESULT" = "success"

.gitignore

@@ -27,8 +27,7 @@ id_ed25519
 *.ovpn
 
 # Local OSINT inputs/outputs may contain PII, targets, proxies, or findings
-data/runs/
-data/cache/
+/data/
 cases/
 /OSINTAI_SYNTHESIS/
 /TODO
@@ -62,6 +61,7 @@ build/
 .pytest_cache/
 .mypy_cache/
 .tox/
+.test-tmp/
 
 # IDEs
 .vscode/

BENCHMARKS.md

@@ -0,0 +1,62 @@
+# OSINTai 4.2.0 offline benchmarks
+
+Measured on 2026-09-10 using Python 3.12.13, Darwin
+arm64. These are single-run observations, not statistical guarantees.
+No page fetching or model calls were performed. Source captures and existing reports
+were preserved; the benchmark produced new extraction checkpoints and report bundles.
+
+## Saved crawls
+
+Default analysis settings: 200,000 characters per page; 16 MB text cache per reader;
+100,000 co-occurrence candidate-pair budget; 30-second page and 120-second stage deadlines.
+Timing includes source hashing, worker startup, analysis, report validation, and publication.
+
+| Saved run | Cache | Pages | Elapsed | Cache hits/misses | Parent peak RSS | Maximum child RSS |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| 20260909_212714 | cold | 152 | 18.133 s | 0/152 | 91.9 MiB | 84.4 MiB |
+| 20260909_212714 | warm | 152 | 1.644 s | 152/0 | 86.8 MiB | 84.4 MiB |
+| 20260717_144145 | cold | 159 | 18.923 s | 2/157 | 85.5 MiB | 92.6 MiB |
+
+The 159-page cold run reused two identical page contents within the same invocation.
+RSS is reported separately for the parent and the largest individual child; summing the
+columns is not a measurement of simultaneous process-tree memory. Warm means extraction
+checkpoints were reused, not that every filesystem or OS cache was controlled.
+
+Both crawls explicitly report partial analytical coverage. The 152-page crawl excluded
+92,369 potential pairs on pages exceeding the 60-identifier pairing threshold. The
+159-page crawl excluded 161,328 such pairs, omitted 25,125 low-ranked correlation rows
+at the output cap, and recorded 46 temporal parsing errors from source date values.
+Neither run exhausted its candidate-pair budget. Publication succeeded and these limits
+and errors remain visible in each bundle's summary and report.
+
+## Boilerplate fixture
+
+The synthetic fixture has 10,000 pages, four shared footer identifiers, and one
+unique email/handle pair per page. All four footer identifiers are excluded from pairing.
+
+- Entity indexing: **0.053 seconds**.
+- Correlation: **0.079 seconds**.
+- Examined co-occurrence pairs: **10,000**.
+- Pairs omitted by budget: **0**.
+- Parent peak RSS: **60.8 MiB**.
+
+## Reproduction
+
+Run from the repository using its Python environment. Benchmark memory collection uses
+`resource`, available on macOS/Linux; the unit-test CI matrix also covers Windows.
+
+```bash
+python tests/benchmark_analysis.py RUN_ID --output .test-tmp/benchmark-cold.json
+python tests/benchmark_analysis.py RUN_ID --output .test-tmp/benchmark-warm.json
+python tests/benchmark_correlation.py --pages 10000 --pair-budget 100000
+```
+
+A run is cold only if no matching extractor-version/content checkpoints exist. No
+benchmark deletes prior checkpoints or source material to manufacture a cold run.
+
+## Verification
+
+All **119 offline tests** passed locally with `ResourceWarning` treated as an error.
+Correctness lint, Bandit at the release gate's severity/confidence thresholds, compilation,
+CLI version/help checks, and whitespace checks passed. The release workflow runs the same
+gate on Linux, macOS, and Windows.

ENHANCEMENTS.md

@@ -0,0 +1,96 @@
+# OSINTai enhancements
+
+## Integrated in 4.2.0
+
+All seven enhancement areas from the 4.1.0 follow-up list are implemented:
+
+1. **Process-isolated deadlines.** Live HTML parsing, indicator extraction, hunt matching,
+   and Simhash run in disposable spawn workers. Saved-page extraction, entity indexing,
+   deterministic/model stages, and training export have deadlines. CPU timeouts, crashed
+   children, cancellation, and large IPC results have offline tests. Workers are terminated
+   and reaped. Raw HTML is retained before parsing, and failures are recorded. CLI controls:
+   `--analysis-page-timeout` (30 seconds), `--analysis-stage-timeout` (120 seconds).
+2. **Content-addressed extraction checkpoints.** `.extraction_cache/` keys include scanned
+   text SHA-256, extractor implementation hash, character limit, and truncation state.
+   Atomic JSON checkpoints carry schema/checksum validation and per-kind coverage counts.
+   Repeated pages and resumed analyses reuse valid results. Cache writes failing does not
+   discard successful extraction. Secret values and arbitrary JWT claims are excluded;
+   secret-shaped values are also removed from cached indicator fields, including values
+   beyond the findings cap. Reports reconstruct findings from counts and decode validity.
+3. **Adversarial regex audit.** Email, ASCII-domain, credential, and JWT extraction consume
+   maximal tokens or lines before bounded validation. Oversized malformed candidates cannot
+   restart an unbounded suffix search. Unicode scanning remains linear. External-deadline
+   tests cover repeated punctuation, failed domain/email suffixes, JWT-like strings, and
+   long credential lines. Fixed detector limits and their omissions are reported.
+4. **Model quality and bounded retry.** Results distinguish `ok`, `empty`, `invalid`,
+   `missing`, `timed_out`, `error`, and `skipped`. Page attribution comes from saved file
+   identity rather than model assertions. Manifests record model names, validated page
+   counts, and optional stage response counts. `--retry-model RUN_ID --retry-limit 20
+   --retry-timeout 60` retries unsuccessful/skipped saved pages through local Ollama,
+   without fetching pages or overwriting original analyses. Later offline runs use the
+   published `model_retry_latest.json` overlay only when its resolved paths stay inside
+   the run, its manifest is completed, and its source hashes still match current inputs.
+5. **Hunt offsets and URL provenance.** Unicode expansion maps matches back to original
+   start/end offsets. URL detection uses original text and rejects URLs crossing a snippet
+   boundary. Indicator provenance identifies `html_attribute`, `page_prose`, and
+   `html_source`; resolved relative HTML links are included.
+6. **Bounded correlation and text caching.** `--correlation-pair-budget` defaults to
+   100,000 examined co-occurrence pairs. Omitted pairs, oversized-page exclusions, and
+   output truncation are explicit. Source membership uses sets, and identity evidence
+   avoids repeatedly scanning a common handle's entire source list. Each text reader has
+   a `--text-cache-bytes` LRU budget (16 MB default), accounting for string and entry
+   overhead. Cache peaks, evictions, missing files, and truncation are reported.
+7. **Transactional analysis publication.** A hidden incomplete directory is populated
+   and validated before an atomic rename publishes an immutable `analysis_*/` bundle.
+   The bundle includes the human report, JSON/JSONL results, summary, manifest, and optional
+   training export. `analysis_latest.json` advances only after successful publication.
+   Attempt status records running/failed/completed state, and manifests retain full source
+   hashes and options. Source changes during analysis prevent publication. File contents
+   are flushed; POSIX directory metadata is fsynced. Original captures and prior bundles
+   remain intact. Completed publication may still report partial analytical coverage.
+
+The existing 4.1.0 fixes are retained: Unicode scanning, offline recovery, bounded text
+reads, failure isolation, clean interruption, HTML text boundaries, complete summaries,
+empty artifacts, numeric validation, and explicit profile overrides.
+
+## Verification and performance
+
+See `tests/test_enhancements.py` for deterministic offline acceptance checks and
+`tests/benchmark_analysis.py` for saved-crawl runtime and memory measurements. CI now
+runs the release gate on macOS, Linux, and Windows. Local verification was performed on
+macOS with Python 3.12.
+
+Measured results are recorded in `BENCHMARKS.md`. RSS figures separate parent memory
+from the maximum child RSS; they are not a measured concurrent process-tree total.
+
+Scanning is O(N) for bounded token validators and fixed extraction caps. Entity indexing
+is expected O(S), with S source/indicator observations. Correlation generation is bounded
+by O(S + B), followed by O(K log K) ordering of K candidates, where B is the pair budget.
+Core retained analysis memory is O(S + E + B + C + R + L): E entities, C the text-cache budget,
+R retained result rows, and L the per-job input bound. Disposable workers add serialization copies and startup cost.
+Hunt searches cost O(TN) for T terms, with at most 500 reported hits; Unicode offset mapping
+uses O(N) space only when lowercasing expands characters. Full source hashing streams
+files with a 1 MB buffer. Thread timeouts and unbounded caches were rejected because they
+cannot provide containment and predictable memory use.
+
+## Recommended next work
+
+1. **Public-suffix-aware domain families.** `_registrable()` still uses the last two
+   labels; names beneath suffixes such as `co.uk` can be incorrectly grouped. Bundle a
+   versioned Public Suffix List for deterministic offline grouping and ownership caveats.
+2. **Cache retention and quotas.** Add explicit age/size-based pruning for extraction
+   checkpoints and old report/retry bundles, preserving referenced evidence and active
+   attempts. Current cache memory is bounded, but disk retention is intentionally additive.
+3. **Typed streaming artifact ingestion.** Stream large JSONL files and validate record
+   schemas with line-level corruption counts. Current loaders materialize metadata and
+   silently skip malformed JSON lines; text-cache limits do not bound total index memory.
+4. **Reduce process startup overhead.** Benchmark a supervised, recyclable worker design
+   that preserves hard per-job termination and isolation. The current spawn-per-job design
+   is portable and simple, but cold runs pay startup cost for every distinct page.
+5. **More precise reproducibility and memory telemetry.** Inject a reference clock for
+   temporal analysis and generated timestamps; measure concurrent process-tree RSS, and
+   add Windows benchmark memory collection. Validate filesystem crash durability on the
+   supported filesystems; Windows does not provide POSIX directory-fsync semantics here.
+6. **Raw-only parsing recovery.** Offer an explicit offline command to reparse preserved
+   HTML from live extraction failures. Current `--analyze-only` consumes saved page text
+   and reports raw-only failures rather than reconstructing missing text automatically.