Release OSINTai 4.2.0 reliability enhancements

[gs-ai]

OSINTai analysis could hang on CPU-bound extraction, repeat completed work after interruption, and publish partial report files directly into a run directory. This release adds process-isolated deadlines, secret-free content-addressed checkpoints, bounded correlation and text caching, explicit model-result quality reporting with saved-page retries, and validated immutable analysis bundles.

It also records original Unicode hunt offsets and URL provenance, replaces vulnerable indicator regex paths with bounded token scanners, updates OSINTai to 4.2.0, documents measured cold/warm saved-crawl performance, and expands the release gate to macOS, Linux, and Windows.

Validation:

• 119 offline unit and integration tests passed with ResourceWarning treated as an error.
• pip-audit found no known runtime dependency vulnerabilities.
• Ruff correctness checks, Bandit, compilation, CLI version/help checks, and git diff --check passed.
• The staged index was scanned for private keys, common provider token formats, authenticated URLs, credential filenames, and ignored tracked files. Only explicit .test/localhost rejection fixtures remain.
• /data/ is ignored as a whole; no crawl data, virtual environment, or test-temporary artifact is included.

Complexity: indicator scans are O(N), entity indexing is expected O(S), and correlation work is bounded by the configured candidate-pair budget followed by O(K log K) ordering. Memory is bounded by retained evidence/results, the text-cache budget, candidate budget, and per-worker input limit.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings remain across publication, retry validation, cancellation cleanup, CI, benchmarking, and bounded-memory handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

OSINTai 4.2.0 adds reliability improvements for bounded analysis, recovery, model-result reporting, and immutable publication.

Changes:

• Added isolated deadlines, checkpoints, bounded scanners, caches, and correlation.
• Added saved-page retries, provenance tracking, Unicode offsets, and model-quality reporting.
• Added atomic bundles, benchmarks, documentation, and cross-platform CI.

File summaries

File	Description and review notes
tests/test_enhancements.py	Acceptance tests for reliability and recovery.
tests/test_analysis_recovery.py	Regression tests for bounded extraction and CLI recovery.
tests/test_analysis_layer.py	Updated analysis-layer expectations.
tests/benchmark_correlation.py	Correlation performance benchmark.
tests/benchmark_analysis.py	Saved-run analysis benchmark. Moderate (3 votes): The documented command does not create .test-tmp before writing its result.
src/osintai/storage.py	Durable atomic JSON writes.
src/osintai/scanners.py	Bounded token and credential scanners.
src/osintai/report.py	Coverage and model statistics reporting.
src/osintai/publication.py	Immutable staged analysis publication. Critical (1 vote): Read-only descriptors are passed to fsync, which can fail on Windows before replacement.
src/osintai/pipeline.py	Checkpointed, bounded analysis pipeline. Moderate (2 votes): Retry overlays are not validated against current manifests or source hashes. Moderate (1 vote): Non-ok retry payloads lose saved-page titles. Critical (1 vote): Symlink or junction retry paths can escape the saved run.
src/osintai/patterns.py	Secret findings based on redacted counts.
src/osintai/ollama_api.py	Model response classification.
src/osintai/model_retry.py	Bounded saved-page model retries.
src/osintai/model_quality.py	Model payload validation and summaries.
src/osintai/isolation.py	Spawned worker deadlines and cleanup. Moderate (1 vote): Cleanup can replace cancellation with a worker exception.
src/osintai/hunt.py	Unicode offsets and URL handling.
src/osintai/extractor.py	Bounded extraction and URL provenance.
src/osintai/entities.py	Unicode scanning and extraction coverage.
src/osintai/crawler.py	Isolated extraction and model statuses.
src/osintai/correlation.py	Candidate budgets and bounded correlation. Moderate (1 vote): seen can grow without bound despite the per-host URL cap.
src/osintai/cli.py	Recovery and retry options. Moderate (1 vote): source_hashes() includes run_manifest.json before it is written.
src/osintai/checkpoints.py	Secret-free content-addressed checkpoints.
src/osintai/__init__.py	Version updated to 4.2.0.
README.md	User-facing reliability and recovery documentation.
ENHANCEMENTS.md	Enhancement and performance documentation.
BENCHMARKS.md	Offline benchmark results and reproduction steps.
.gitignore	Broader data and temporary-artifact exclusions.
.github/workflows/release-gate.yml	Cross-platform release gate. Moderate (1 vote): The explicit executable --version assertion was removed.

Review details

Suppressed comments (5)

.github/workflows/release-gate.yml:58

• This replacement drops the previous explicit run_osintai.py --version assertion. The gate now checks only package metadata and README text, so a regression in the executable's version output can pass as long as --help still works. Keep a subprocess assertion that the CLI output equals OSINTai {__version__}.

          python -c "import sys; sys.path.insert(0, 'src'); from osintai import __version__; assert f'# OSINTai v{__version__} ' in open('README.md', encoding='utf-8').read()"

src/osintai/cli.py:470

• source_hashes() explicitly includes the source run_manifest.json, but this analysis call completes before the CLI writes that file below. For a fresh crawl, the published bundle therefore omits a source file that later exists, so its recorded source set is not the full set used by subsequent saved-run analyses. Either finalize this metadata before hashing or exclude it consistently from the input hash contract.

            analysis_report_path = analysis_output.artifacts["analysis_report"]

src/osintai/correlation.py:86

• Although domain_map retains at most 200 URLs per host, seen keeps every distinct URL because it is updated before the cap check (in both branches below). A crawl with many URL indicators therefore still grows this set without bound, defeating the bounded-memory guarantee; only deduplicate while the 200-entry list is still growing or use another bounded structure.

    seen = defaultdict(set)

src/osintai/isolation.py:90

• Cancellation cleanup only suppresses RuntimeError. If cancellation races the worker deadline, isolated_call can finish with TimeoutError (or another worker/IPC exception), which escapes this handler and replaces the caller's CancelledError. Swallow worker exceptions during the cleanup wait, then re-raise the original cancellation.

        except RuntimeError:

src/osintai/pipeline.py:202

• Retry prompts lose the saved page title for every non-ok result: this replaces the crawler's title metadata with {}, and model_retry.py then reads row.get("title", ""). Skipped, timed-out, or failed pages therefore get an empty title on retry and the copied overlay metadata loses it as well; preserve safe metadata such as title when normalizing non-success payloads.

            row = dict(payload) if status == "ok" else {}

• Files reviewed: 27/28 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.