feat(rstthreatlibrary): create rst threat library connector (#7021)#7032
Open
Winnie-158263 wants to merge 6 commits into
Conversation
Contributor License Agreement✅ CLA signed 💚 Thank you @Winnie-158263 for signing the Contributor License Agreement! Your pull request can now be reviewed and merged. We appreciate your contribution to Filigran's open source projects! ❤️ This is an automated message from the Filigran CLA Bot. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new external-import connector (rst-threat-library) to ingest intrusion sets, malware, tools, and campaigns from the RST Cloud Threat Library REST API into OpenCTI, with incremental polling and optional reconciliation logic to keep OpenCTI objects aligned with upstream changes.
Changes:
- Implemented the connector runtime (polling, state/cursor handling, OpenCTI push via bundle or API, optional merge/split reconciliation, and analyst-edit protection via confidence).
- Added an RST Threat Library HTTP client and STIX 2.1 conversion layer that preserves upstream
standard_id. - Added Docker packaging, sample configuration, metadata/schema/docs, and a pytest suite covering key behaviors.
Reviewed changes
Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| external-import/rst-threat-library/tests/test-requirements.txt | Adds test dependencies for the connector. |
| external-import/rst-threat-library/tests/test_main.py | Smoke tests for settings/helper/connector instantiation. |
| external-import/rst-threat-library/tests/test_connector/test_settings.py | Validates settings parsing and error handling. |
| external-import/rst-threat-library/tests/test_connector/test_merge_split.py | Tests merge/split candidate selection behavior. |
| external-import/rst-threat-library/tests/test_connector/test_converter_to_stix.py | Tests STIX conversion and ID preservation. |
| external-import/rst-threat-library/tests/test_connector/test_connector_batch_send.py | Tests bundle send retry behavior and confidence/lock flows. |
| external-import/rst-threat-library/tests/test_connector/test_confidence.py | Unit tests for confidence/lock helpers. |
| external-import/rst-threat-library/tests/test_connector/test_api_client.py | Unit tests for HTTP retry behavior in the client. |
| external-import/rst-threat-library/tests/conftest.py | Ensures src/ is importable for tests. |
| external-import/rst-threat-library/src/rst_threat_library_client/api_client.py | Implements paginated API client with retry and logging. |
| external-import/rst-threat-library/src/rst_threat_library_client/init.py | Exposes the Threat Library client as a package export. |
| external-import/rst-threat-library/src/requirements.txt | Adds connector runtime dependencies. |
| external-import/rst-threat-library/src/main.py | Connector entrypoint wiring settings → helper → run loop. |
| external-import/rst-threat-library/src/connector/utils.py | Shared constants/mappings and label merge helper. |
| external-import/rst-threat-library/src/connector/settings.py | Pydantic/connectors-sdk settings models and validation. |
| external-import/rst-threat-library/src/connector/merge_split.py | Merge/split planning logic based on alias/name ownership. |
| external-import/rst-threat-library/src/connector/converter_to_stix.py | Converts upstream objects to STIX 2.1 while preserving upstream IDs. |
| external-import/rst-threat-library/src/connector/connector.py | Main connector logic: polling, state, OpenCTI push, reconciliation. |
| external-import/rst-threat-library/src/connector/confidence.py | Confidence normalization and sync-record helpers for analyst-edit protection. |
| external-import/rst-threat-library/src/connector/init.py | Exposes connector public API (settings + connector class). |
| external-import/rst-threat-library/README.md | User documentation and recommended operational settings. |
| external-import/rst-threat-library/entrypoint.sh | Container entrypoint script. |
| external-import/rst-threat-library/Dockerfile | Builds the connector image and defines a healthcheck. |
| external-import/rst-threat-library/docker-compose.yml | Example deployment configuration via Docker Compose. |
| external-import/rst-threat-library/config.yml.sample | Sample YAML configuration for manual deployment. |
| external-import/rst-threat-library/.dockerignore | Excludes non-runtime files from Docker build context. |
| external-import/rst-threat-library/metadata/connector_manifest.json | Connector manifest metadata for the monorepo. |
| external-import/rst-threat-library/metadata/connector_config_schema.json | Generated config schema for the connector. |
| external-import/rst-threat-library/metadata/CONNECTOR_CONFIG_DOC.md | Generated environment variable documentation. |
Comment on lines
+105
to
+129
| if response.status_code == 429 or 500 <= response.status_code < 600: | ||
| raise requests.HTTPError( | ||
| f"{response.status_code} {response.reason} for {response.url}", | ||
| response=response, | ||
| ) | ||
| response.raise_for_status() | ||
| return response.json() | ||
| except Exception as exc: | ||
| last_exc = exc | ||
| if attempt < self.retry: | ||
| delay = (attempt + 1) * 2 | ||
| self.helper.connector_logger.warning( | ||
| "GET request failed; retrying", | ||
| { | ||
| "url": url, | ||
| "attempt": attempt + 1, | ||
| "max_attempts": self.retry + 1, | ||
| "error": str(exc), | ||
| "retry_in_seconds": delay, | ||
| }, | ||
| ) | ||
| time.sleep(delay) | ||
| else: | ||
| raise | ||
| raise last_exc # type: ignore[misc] |
Comment on lines
+149
to
+165
| for api_sid, api_item in api_by_sid.items(): | ||
| api_ids = identifiers_from_api_item(api_item) | ||
| if not api_ids: | ||
| continue | ||
|
|
||
| oc_duplicates: Dict[str, Dict[str, Any]] = {} | ||
| for ident in api_ids: | ||
| for oc_sid in oc_index.get(ident, set()): | ||
| if oc_sid == api_sid: | ||
| continue | ||
| entity = next( | ||
| (e for e in opencti_entities if e.get("standard_id") == oc_sid), | ||
| None, | ||
| ) | ||
| if entity: | ||
| oc_duplicates[oc_sid] = entity | ||
|
|
Comment on lines
+121
to
+126
| def build_identity(self, created_by: Dict[str, Any]) -> Optional[Any]: | ||
| sid = created_by.get("standard_id") | ||
| if not sid: | ||
| return None | ||
| name = created_by.get("name") or sid | ||
| return stix2.v21.Identity(id=sid, name=name) |
…ng exception handling and stix identity class (OpenCTI-Platform#7021)
Comment on lines
+19
to
+23
| except Exception as ex: | ||
| print(str(ex)) | ||
| traceback.print_tb(ex.__traceback__) | ||
| time.sleep(10) | ||
| sys.exit(0) |
Comment on lines
+8
to
+13
| RUN apk --no-cache add --virtual .build-deps git build-base && \ | ||
| apk --no-cache add libmagic && \ | ||
| pip3 install --no-cache-dir -r src/requirements.txt && \ | ||
| pip3 install --no-cache-dir --no-deps \ | ||
| "connectors-sdk @ git+https://github.com/OpenCTI-Platform/connectors.git@master#subdirectory=connectors-sdk" && \ | ||
| apk del .build-deps |
Comment on lines
+23
to
+24
| HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ | ||
| CMD sh -c "ps | awk '/src\\/main\\.py/ {found=1} END {exit(found?0:1)}'" |
Comment on lines
+1
to
+4
| #!/bin/sh | ||
|
|
||
| cd /opt/opencti-connector-rst-threat-library | ||
| python3 src/main.py |
|
|
||
| ### Configuration variables | ||
|
|
||
| Configuration is set either in `docker-compose.yml` (Docker) or in `config.yml` (manual deployment). See `config.yml.sample` for a full annotated example. Field descriptions and examples are also defined on the Pydantic models in `src/connector/settings.py` and mirrored in `__docs__/connector_config_schema.json`. |
…ng and dockerfile requirements (OpenCTI-Platform#7021)
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
external-import/rst-threat-library/src/connector/settings.py:121
- OpenCTI push retry settings are currently unbounded. max_retries=0 will make the connector never attempt a push, and non-positive backoff multipliers can produce no/negative delays. Add validation constraints to keep these values in a safe range.
max_retries: int = Field(
description="Maximum retries when pushing bundles to OpenCTI.",
default=3,
examples=[3],
)
Comment on lines
+74
to
+85
| for ref in refs or []: | ||
| kwargs: Dict[str, Any] = {} | ||
| if ref.get("source_name"): | ||
| kwargs["source_name"] = ref["source_name"] | ||
| if ref.get("url"): | ||
| kwargs["url"] = ref["url"] | ||
| if ref.get("external_id"): | ||
| kwargs["external_id"] = ref["external_id"] | ||
| if ref.get("description"): | ||
| kwargs["description"] = ref["description"] | ||
| out.append(stix2.v21.ExternalReference(**kwargs)) | ||
| return out |
Comment on lines
+77
to
+101
| contimeout: int = Field( | ||
| description="HTTP connect timeout in seconds.", | ||
| default=30, | ||
| examples=[30], | ||
| ) | ||
| readtimeout: int = Field( | ||
| description="HTTP read timeout in seconds.", | ||
| default=120, | ||
| examples=[120, 600], | ||
| ) | ||
| retry: int = Field( | ||
| description="Per-request HTTP retry count.", | ||
| default=2, | ||
| examples=[2, 10], | ||
| ) | ||
| ssl_verify: bool = Field( | ||
| description="Verify TLS certificates for API requests.", | ||
| default=True, | ||
| examples=[True, False], | ||
| ) | ||
| page_size: int = Field( | ||
| description="Page size (limit) for Threat Library list requests.", | ||
| default=100, | ||
| examples=[20, 100], | ||
| ) |
Comment on lines
+868
to
+871
| self.helper.api.stix.merge( | ||
| id=target_internal_id, | ||
| object_ids=chunk, | ||
| ) |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 30 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
external-import/rst-threat-library/src/connector/connector.py:1280
- The API push retry loop has the same issue as bundle mode: only built-in ConnectionError/OSError/TimeoutError are considered retryable, so requests-style transient failures may bypass retries. Additionally, a failed attempt leaves its OpenCTI work item not marked processed/in_error.
except (ConnectionError, OSError, TimeoutError) as ex:
self.helper.connector_logger.error(
f"[{obj_type}] API push attempt {attempt + 1}/{max_retries} "
f"failed: {ex}"
)
Comment on lines
+1196
to
+1219
| except (ConnectionError, OSError, TimeoutError) as ex: | ||
| self.helper.connector_logger.error( | ||
| f"[{obj_type}] push attempt {attempt + 1}/{max_retries} " | ||
| f"failed: {ex}" | ||
| ) | ||
| if attempt < max_retries - 1: | ||
| self.helper.connector_logger.info( | ||
| f"Retrying in {retry_delay} seconds..." | ||
| ) | ||
| time.sleep(retry_delay) | ||
| retry_delay = ( | ||
| int(retry_delay * self._retry_backoff_multiplier) or retry_delay | ||
| ) | ||
| else: | ||
| self.helper.connector_logger.error( | ||
| f"[{obj_type}] failed to upload bundle after " | ||
| f"{max_retries} attempts." | ||
| ) | ||
| return False | ||
|
|
||
| except Exception as ex: | ||
| error_message = f"[{obj_type}] unexpected error during upload: {ex}" | ||
| self.helper.connector_logger.error(error_message) | ||
| raise ConnectionError(error_message) from ex |
Comment on lines
+1
to
+8
| { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "title": "RST Threat Library Connector configuration", | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["connector", "rst_threat_library"], | ||
| "properties": { | ||
| "connector": { |
…rmat and connection error formats (OpenCTI-Platform#7021)
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
external-import/rst-threat-library/src/connector/connector.py:1291
- Same as bundle mode: if
RST_THREAT_LIBRARY_MAX_RETRIESis configured to 0, this loop never runs and the connector never attempts an API import.
max_retries = self._max_retries
retry_delay = self._retry_delay
for attempt in range(max_retries):
work_id: Optional[str] = None
Comment on lines
+1203
to
+1206
| max_retries = self._max_retries | ||
| retry_delay = self._retry_delay | ||
|
|
||
| for attempt in range(max_retries): |
Comment on lines
+2
to
+4
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "$id": "https://www.filigran.io/connectors/RST Threat Library_config.schema.json", | ||
| "type": "object", |
Comment on lines
+13
to
+30
| "OPENCTI_TOKEN": { | ||
| "description": "The API token to connect to OpenCTI.", | ||
| "type": "string" | ||
| }, | ||
| "CONNECTOR_NAME": { | ||
| "default": "RST Threat Library", | ||
| "description": "The name of the connector.", | ||
| "examples": [ | ||
| "RST Threat Library" | ||
| ], | ||
| "type": "string" | ||
| }, | ||
| "CONNECTOR_SCOPE": { | ||
| "description": "The scope of the connector, e.g. 'flashpoint'.", | ||
| "items": { | ||
| "type": "string" | ||
| }, | ||
| "type": "array" |
| @@ -0,0 +1,11 @@ | |||
| # Pin pycti to your OpenCTI platform version (OPENCTI_VERSION in root .env). | |||
| pycti==7.260706.0 | |||
Comment on lines
+104
to
+107
| try: | ||
| datetime.strptime(s, "%Y-%m-%d") | ||
| except ValueError: | ||
| return "" |
Comment on lines
+121
to
+126
| max_retries: int = Field( | ||
| description="Maximum retries when pushing bundles to OpenCTI.", | ||
| default=3, | ||
| ge=1, | ||
| examples=[3], | ||
| ) |
Comment on lines
+204
to
+205
| """Choose the OpenCTI entity that should survive a fusion merge. | ||
| """ |
Comment on lines
+1273
to
+1301
| def _batch_send_via_api( | ||
| self, stix_objects: List[Any], timestamp: int, obj_type: str | ||
| ) -> bool: | ||
| now = datetime.fromtimestamp(timestamp, tz=timezone.utc) | ||
| friendly_name = ( | ||
| f"RST Threat Library [{obj_type}] API @ " | ||
| f"{now.strftime('%Y-%m-%d %H:%M:%S')}" | ||
| ) | ||
| ordered = self._stix_objects_to_api_order(stix_objects) | ||
| self.helper.connector_logger.debug( | ||
| f"[{obj_type}] start API import of {len(ordered)} object(s) " | ||
| f"(update_existing={self.update_existing_data})" | ||
| ) | ||
|
|
||
| max_retries = max(1, int(self._max_retries)) | ||
| retry_delay = self._retry_delay | ||
|
|
||
| for attempt in range(max_retries): | ||
| work_id: Optional[str] = None | ||
| try: | ||
| work_id = self.helper.api.work.initiate_work( | ||
| self.helper.connect_id, friendly_name | ||
| ) | ||
| for obj in ordered: | ||
| data = json.loads(obj.serialize()) | ||
| self.helper.api.stix2.import_object( | ||
| data, update=self.update_existing_data | ||
| ) | ||
| self.helper.connector_logger.info( |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7032 +/- ##
===========================================
+ Coverage 35.64% 83.81% +48.16%
===========================================
Files 2048 3 -2045
Lines 125310 383 -124927
===========================================
- Hits 44672 321 -44351
+ Misses 80638 62 -80576 📢 Thoughts on this report? Let us know! 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Checklist
Fixes #7021