feat(ipgeolocation): add IPGeolocation.io internal enrichment connector#7013
Open
mateen993 wants to merge 2 commits into
Open
feat(ipgeolocation): add IPGeolocation.io internal enrichment connector#7013mateen993 wants to merge 2 commits into
mateen993 wants to merge 2 commits into
Conversation
Contributor License Agreement✅ CLA signed 💚 Thank you @mateen993 for signing the Contributor License Agreement! Your pull request can now be reviewed and merged. We appreciate your contribution to Filigran's open source projects! ❤️ This is an automated message from the Filigran CLA Bot. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new internal-enrichment connector under internal-enrichment/ipgeolocation that calls IPGeolocation.io v3 endpoints, computes a unified risk score, and emits STIX 2.1 objects (locations, ASN, identities, indicators, notes, relationships) for IPv4/IPv6 observables in OpenCTI.
Changes:
- Introduces the connector runtime (config, API client, enrichment orchestration) plus Docker packaging.
- Implements typed response models, risk scoring, markdown note generation, and STIX mapping/bundling.
- Adds unit tests and connector documentation (architecture, STIX mapping, risk scoring, testing guide).
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| internal-enrichment/ipgeolocation/src/api_client.py | HTTP client for IPGeolocation.io v3 endpoints with retries + enrichment modes |
| internal-enrichment/ipgeolocation/src/config.py | Typed env/YAML configuration for OpenCTI + connector + IPGeolocation settings |
| internal-enrichment/ipgeolocation/src/connector.py | OpenCTI enrichment worker: read observable → call APIs → score → map to STIX → send bundle |
| internal-enrichment/ipgeolocation/src/main.py | Entrypoint wiring connector startup/error handling |
| internal-enrichment/ipgeolocation/src/markdown_generator.py | Generates analyst-readable markdown notes from intel + risk assessment |
| internal-enrichment/ipgeolocation/src/models.py | Dataclass models + merge helpers for the various API response blocks |
| internal-enrichment/ipgeolocation/src/risk_scorer.py | Unified risk scoring + confidence derivation + explanation builder |
| internal-enrichment/ipgeolocation/src/stix_mapper.py | STIX 2.1 object factory + relationship/label/indicator/note generation |
| internal-enrichment/ipgeolocation/src/init.py | Package metadata/version |
| internal-enrichment/ipgeolocation/tests/conftest.py | Shared pytest fixtures for mapper/scorer/markdown/intel samples |
| internal-enrichment/ipgeolocation/tests/mock_responses.py | Mocked API payloads used across unit tests |
| internal-enrichment/ipgeolocation/tests/test_api_client.py | Unit tests for API client enrichment modes and error handling |
| internal-enrichment/ipgeolocation/tests/test_markdown_generator.py | Unit tests for markdown note sections/content |
| internal-enrichment/ipgeolocation/tests/test_risk_scorer.py | Unit tests for scoring thresholds, caps, confidence, and opinion mapping |
| internal-enrichment/ipgeolocation/tests/test_stix_mapper.py | Unit tests validating key STIX objects/relationships/labels/indicators/notes |
| internal-enrichment/ipgeolocation/tests/init.py | Marks tests as a package |
| internal-enrichment/ipgeolocation/requirements.txt | Python dependencies for the connector |
| internal-enrichment/ipgeolocation/README.md | Connector user docs: features, configuration reference, testing notes |
| internal-enrichment/ipgeolocation/docs/ARCHITECTURE.md | Component/data-flow documentation |
| internal-enrichment/ipgeolocation/docs/RISK_SCORING.md | Detailed scoring algorithm reference |
| internal-enrichment/ipgeolocation/docs/STIX_MAPPING.md | Mapping spec for emitted STIX objects/relationships/labels |
| internal-enrichment/ipgeolocation/docs/TESTING_GUIDE.md | End-to-end deployment/testing walkthrough |
| internal-enrichment/ipgeolocation/Dockerfile | Container build for running connector |
| internal-enrichment/ipgeolocation/entrypoint.sh | Container entrypoint script |
| internal-enrichment/ipgeolocation/docker-compose.yml | Example compose service definition for running the connector |
| internal-enrichment/ipgeolocation/config.yml.sample | Sample YAML configuration for local runs |
Comment on lines
+72
to
+82
| self._author = stix2.Identity( | ||
| id=_det_id("identity", author_name), | ||
| name=author_name, | ||
| identity_class="organization", | ||
| description=( | ||
| "IPGeolocation.io — Enterprise-grade IP geolocation " | ||
| "and threat intelligence provider." | ||
| ), | ||
| created_by_ref=None, | ||
| allow_custom=True, | ||
| ) |
Comment on lines
+386
to
+392
| return stix2.DomainName( | ||
| value=intel.hostname, | ||
| allow_custom=True, | ||
| custom_properties={ | ||
| "created_by_ref": self._author.id, | ||
| }, | ||
| ) |
Comment on lines
+522
to
+534
| custom = { | ||
| "x_opencti_score": risk.opencti_score, | ||
| "x_opencti_description": risk.explanation, | ||
| } | ||
| kwargs: dict[str, Any] = { | ||
| "id": observable_id, | ||
| "value": ip_val, | ||
| "allow_custom": True, | ||
| "custom_properties": custom, | ||
| } | ||
| if ext_refs: | ||
| kwargs["external_references"] = ext_refs | ||
|
|
| | IP Observable | `located-at` | Country | Always (if country exists) | | ||
| | City | `located-at` | Country | Always (if city exists) | | ||
| | IP Observable | `belongs-to` | ASN | Always (if ASN exists) | | ||
| | ASN | `belongs-to` | Organization | Always (if org exists) | |
| @@ -0,0 +1,5 @@ | |||
| #!/bin/bash | |||
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Author
|
Hi @ncarenton, please let us know when you are available to review this merge/pull request? |
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 26 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (2)
internal-enrichment/ipgeolocation/src/stix_mapper.py:390
- For SCOs like
domain-name, OpenCTI expects the author to be set via the extension propertyx_opencti_created_by_ref(notcreated_by_ref). Usingcreated_by_refinsidecustom_propertieswill not set the author correctly and can also break strict STIX validation when the platform re-emits the bundle.
return stix2.DomainName(
value=intel.hostname,
allow_custom=True,
custom_properties={
"created_by_ref": self._author.id,
internal-enrichment/ipgeolocation/src/stix_mapper.py:536
external_referencesis being attached to an IPv4/IPv6 SCO as a top-level property. For OpenCTI observable upserts, external refs should be provided viacustom_properties['x_opencti_external_references'](and similarly labels viax_opencti_labels, author viax_opencti_created_by_ref). As written, this is likely to raise a validation/constructor error or be ignored by the platform.
if ext_refs:
kwargs["external_references"] = ext_refs
if "IPv6" in observable_type:
return stix2.IPv6Address(**kwargs)
Comment on lines
+45
to
+59
| # --------------------------------------------------------------------------- | ||
| # TLP marking definitions (re-usable singletons) | ||
| # --------------------------------------------------------------------------- | ||
| TLP_WHITE = stix2.TLP_WHITE | ||
| TLP_GREEN = stix2.TLP_GREEN | ||
| TLP_AMBER = stix2.TLP_AMBER | ||
| TLP_RED = stix2.TLP_RED | ||
|
|
||
| TLP_MAP = { | ||
| "TLP:WHITE": TLP_WHITE, | ||
| "TLP:CLEAR": TLP_WHITE, | ||
| "TLP:GREEN": TLP_GREEN, | ||
| "TLP:AMBER": TLP_AMBER, | ||
| "TLP:RED": TLP_RED, | ||
| } |
Comment on lines
+12
to
+14
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends libmagic1 && \ | ||
| rm -rf /var/lib/apt/lists/* |
Comment on lines
+177
to
+182
| ## Testing | ||
|
|
||
| ```bash | ||
| pip install pytest | ||
| pytest tests/ -v | ||
| ``` |
Comment on lines
+65
to
+70
| ## Quick Start | ||
|
|
||
| ### Docker Compose (recommended) | ||
|
|
||
| 1. Copy `docker-compose.yml` into your OpenCTI deployment. | ||
| 2. Set environment variables (or use `.env`): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
/v3/ipgeo?include=security,abuseto minimize HTTP requests, and a dedicated-endpoint mode calling/v3/security,/v3/asn,/v3/abuseseparately for granular control.Checklist
Further comments
APIs consumed (4 endpoints, configurable independently):
/v3/ipgeo/v3/security/v3/asn/v3/abuseSTIX objects created: Location (Country), Location (City), Autonomous-System, Identity (Organization), Identity (Cloud Provider), Identity (Abuse Contact), Domain-Name (hostname), Indicator (for high-risk IPs), Note (markdown), Opinion (optional), Relationship, Labels.