Repository navigation
Release OSINTai v4 - #2
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR cuts the OSINTai v4.0.0 release by adding a post-crawl, deterministic “analysis layer” (provenance-labelled findings/correlation/timeline/hypotheses/leads), tightening crawl/fetch safety policies (scope + redirects + response type/size limits + loopback-only Ollama), and introducing a locked-deps CI release gate.
Changes:
- Added offline analysis pipeline (deterministic checks, correlation, temporal analysis, optional deep/cross-check, evaluation, and inert training export) plus an analysis report format.
- Hardened crawl/runtime policies: multi-seed scoping enforcement, redirect validation before follow, HTML/XHTML-only bounded downloads, atomic JSON checkpoints, and loopback-only Ollama client.
- Added comprehensive regression/capability tests and a CI “release gate” workflow; pinned runtime dependencies.
Reviewed changes
Copilot reviewed 27 out of 29 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_seed_and_fetch_policy.py | Adds tests for seed parsing/run-id validation, scope enforcement, redirect policy, response limits, and loopback-only Ollama endpoints. |
| tests/test_analysis_layer.py | Adds extensive regression/capability coverage for prompts, provenance, entities, patterns, correlation, temporal, evaluation, and end-to-end analysis pipeline artifacts. |
| src/osintai/training_export.py | Implements portable JSON/JSONL export of evaluation tasks/preferences/scored results and a manifest. |
| src/osintai/temporal.py | Adds timeline event model, timestamp parsing, plausibility filters, and temporal findings (gaps/bursts/trailing gap). |
| src/osintai/storage.py | Adds atomic JSON writes and hardens JSON reads; marks SHA1 as legacy artifact ID (not security). |
| src/osintai/report.py | Keeps existing ranked report stable and adds a separate provenance-aware analysis report writer. |
| src/osintai/proxy_pool.py | Switches proxy selection randomness to secrets.choice. |
| src/osintai/provenance.py | Introduces provenance primitives (Finding/Hypothesis/Lead/Confidence/CheckResult) and confidence kinds. |
| src/osintai/prompts.py | Centralizes prompt profiles, preserving byte-identical “standard” prompt via regression test. |
| src/osintai/pivots.py | Adds deterministic lead generation producing lookup URLs plus false-positive risk notes. |
| src/osintai/pipeline.py | Adds post-crawl analysis orchestrator producing artifacts, optional model stages, and export gating. |
| src/osintai/patterns.py | Adds deterministic checks (homoglyphs, sensitive infra, secret presence/JWT decode, generated text, outliers). |
| src/osintai/ollama_api.py | Enforces loopback-only, credential-free base URLs; disables env proxy usage; adds health/model helpers. |
| src/osintai/normalize.py | Narrows broad exception handling to ValueError. |
| src/osintai/multimodel.py | Adds optional cross-model claim checks that preserve disagreement as findings. |
| src/osintai/hypotheses.py | Adds deterministic hypothesis generation from findings and wraps model hypotheses with provenance. |
| src/osintai/fetcher.py | Adds bounded HTML-only streaming fetch with redirect validation and response size enforcement. |
| src/osintai/extractor.py | Hardens URL extraction by validating/normalizing regex matches and avoiding malformed URL crashes. |
| src/osintai/evaluation.py | Adds deterministic rubric scoring for model output quality and weak-example selection. |
| src/osintai/entities.py | Adds entity typing/normalization/indexing and extended extraction (unicode domains/handles, dates, secrets, etc.). |
| src/osintai/crawler.py | Integrates prompt profiles, seed-host scoping, redirect policy enforcement, trust_env=False, and safer exception handling. |
| src/osintai/correlation.py | Adds evidence-backed candidate-only correlation logic and domain mapping. |
| src/osintai/cli.py | Expands CLI for v4 analysis options, run profiles, seed-file handling, run manifest, and startup Ollama health checks. |
| src/osintai/init.py | Bumps version to 4.0.0 and exports analysis-layer modules. |
| src/osintai/dedupe.py | Marks SHA1 usage as legacy artifact ID with usedforsecurity=False. |
| requirements.txt | Pins runtime dependencies to exact versions. |
| README.md | Updates documentation for v4 features, analysis layer, safety guardrails, and updated usage. |
| .gitignore | Ignores additional generated directories/files. |
| .github/workflows/release-gate.yml | Adds CI release gate: locked deps install, audit, tests, compile, bandit/ruff checks, and version assertion. |
Suppressed comments (1)
src/osintai/patterns.py:370
- The credential-shaped finding uses an evidence key named "values_recorded" while the token finding uses "value_recorded". Using one consistent key name across findings simplifies consumers.
evidence={"pair_count": len(pairs), "values_recorded": False},
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+104
to
+108
| for record in page_records or []: | ||
| url = record.get("url") | ||
| fetched_at = record.get("fetched_at") | ||
| if not url or not fetched_at: | ||
| continue |
Comment on lines
+224
to
+225
| if flag in argv: | ||
| continue |
| "issuer": str(claims.get("iss", "")) if claims else "", | ||
| "audience": str(claims.get("aud", "")) if claims else "", | ||
| "expiry": str(claims.get("exp", "")) if claims else "", | ||
| "recorded_value": False, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification