feat(cloudflare-rules-list): add Cloudflare Rules List stream connector (#7043)#7044
Open
Khidr6G wants to merge 1 commit into
Open
feat(cloudflare-rules-list): add Cloudflare Rules List stream connector (#7043)#7044Khidr6G wants to merge 1 commit into
Khidr6G wants to merge 1 commit into
Conversation
Filigran-Automation
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new STREAM connector under stream/cloudflare-rules-list/ that keeps a target Cloudflare Rules List (IP kind) synchronized with OpenCTI IPv4 indicators/observables using Cloudflare’s bulk snapshot/replace list-items API.
Changes:
- Introduces a new connector package (
cloudflare_rules_list) with a minimal Cloudflare API client, full-sync + live-stream processing, and sync throttling. - Adds runtime artifacts for deployment (Dockerfile, entrypoint, compose, sample config/env) and generated metadata (manifest, schema, config doc).
- Adds a unit test suite covering settings, client behavior, and connector stream/sync logic.
Reviewed changes
Copilot reviewed 21 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| stream/cloudflare-rules-list/tests/test-requirements.txt | Test dependencies for the connector test suite. |
| stream/cloudflare-rules-list/tests/test_settings.py | Validates pydantic-settings configuration parsing and defaults. |
| stream/cloudflare-rules-list/tests/test_main.py | Verifies construction of settings/helper/client/connector wiring. |
| stream/cloudflare-rules-list/tests/test_connector.py | Unit tests for stream message handling, full sync, and sync-to-Cloudflare behavior. |
| stream/cloudflare-rules-list/tests/test_client.py | Unit tests for the Cloudflare API client request/operation polling behavior. |
| stream/cloudflare-rules-list/tests/conftest.py | Test import path setup for src/ package layout. |
| stream/cloudflare-rules-list/tests/init.py | Marks tests as a package (empty). |
| stream/cloudflare-rules-list/src/requirements.txt | Runtime dependencies (pycti, connectors-sdk, pydantic-settings, requests). |
| stream/cloudflare-rules-list/src/main.py | Runtime entry point wiring settings + OpenCTI helper + Cloudflare client. |
| stream/cloudflare-rules-list/src/config.yml.sample | Sample YAML configuration for local/manual deployments. |
| stream/cloudflare-rules-list/src/cloudflare_rules_list/settings.py | Connector configuration models and env-var mapping via connectors-sdk bases. |
| stream/cloudflare-rules-list/src/cloudflare_rules_list/connector.py | Core connector logic: extraction, caching, full sync, throttled snapshot uploads. |
| stream/cloudflare-rules-list/src/cloudflare_rules_list/client.py | Minimal Cloudflare Rules Lists API client + bulk operation polling. |
| stream/cloudflare-rules-list/src/cloudflare_rules_list/init.py | Package exports for Connector and ConnectorSettings. |
| stream/cloudflare-rules-list/README.md | User/developer documentation and behavioral notes for the connector. |
| stream/cloudflare-rules-list/entrypoint.sh | Container entrypoint script. |
| stream/cloudflare-rules-list/Dockerfile | Alpine-based container build for the connector. |
| stream/cloudflare-rules-list/docker-compose.yml | Example compose service definition. |
| stream/cloudflare-rules-list/.env.sample | Example environment variable configuration. |
| stream/cloudflare-rules-list/metadata/connector_manifest.json | Connector catalog metadata (title/slug/image/etc.). |
| stream/cloudflare-rules-list/metadata/connector_config_schema.json | Generated JSON schema for supported env vars/config options. |
| stream/cloudflare-rules-list/metadata/CONNECTOR_CONFIG_DOC.md | Generated config documentation from the schema. |
Comment on lines
+166
to
+172
| def _sync_to_cloudflare(self) -> None: | ||
| """Push the full IPv4 snapshot to the Cloudflare Rules List.""" | ||
| self._last_sync_time = time.monotonic() | ||
|
|
||
| if not self._indicator_cache: | ||
| self.logger.info("No indicators to sync") | ||
| return |
Contributor
Author
There was a problem hiding this comment.
fixed in the amended commit
Comment on lines
+63
to
+72
| try: | ||
| response = self._session.request( | ||
| method=method, | ||
| url=url, | ||
| json=data, | ||
| timeout=request_timeout, | ||
| ) | ||
| response.raise_for_status() | ||
| return response.json() | ||
| except requests.exceptions.RequestException as exc: |
Contributor
Author
There was a problem hiding this comment.
fixed in the amended commit
Comment on lines
+1
to
+3
| -r ../src/requirements.txt | ||
| pytest==9.0.3 | ||
| pytest-mock==3.14.1 |
Contributor
Author
There was a problem hiding this comment.
fixed in the amended commit
Khidr6G
force-pushed
the
feat/cloudflare-rules-list-connector
branch
from
July 18, 2026 02:27
a090a76 to
48741d7
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 23 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
stream/cloudflare-rules-list/src/cloudflare_rules_list/connector.py:176
_sync_to_cloudflare()stamps_last_sync_timebefore attempting the Cloudflare upload. Ifreplace_list_items()(or the bulk operation) fails, the connector will still enter the throttle window and may not retry for up tosync_interval, delaying recovery from transient Cloudflare/API outages.
self._last_sync_time = time.monotonic()
Comment on lines
+251
to
+259
| list_info = self.client.get_list(self.list_id) | ||
| self.logger.info( | ||
| "Using Cloudflare list", | ||
| meta={ | ||
| "name": list_info.get("name"), | ||
| "id": self.list_id, | ||
| "kind": list_info.get("kind"), | ||
| }, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
stream/cloudflare-rules-listthat pushes IPv4threat-intelligence indicators and
IPv4-Addrobservables from OpenCTI into aCloudflare Rules List (IP kind) for use in WAF and firewall rules.
indicators/observables, then listen to the OpenCTI live stream and keep the
list in sync using Cloudflare's bulk list-items replace/snapshot API, with
the bulk operation polled to completion and uploads throttled to at most once
per
CONNECTOR_SYNC_INTERVAL. Deletes are honored whenCONNECTOR_LIVE_STREAM_LISTEN_DELETE=true.connectors-sdk+pydantic-settings,pycti==7.260715.0,python:3.12-alpine; includes__metadata__(manifest, config schema,config doc, logo),
README.md,docker-compose.yml,Dockerfile,entrypoint.sh,config.yml.sample,.env.sample.CLOUDFLARE_API_BASE_URLconfig (defaults to the public CloudflareAPI) to allow pointing at a compatible gateway or a mock for testing.
Related issues
Checklist
Further comments
Testing. The connector ships a unit-test suite (71 tests, 100% coverage of
the connector package, well above the 80% patch gate) that runs green under the
repository's
run_test.shflow (uv, Python 3.12,pyctifrom source,connectors-sdklocal).isort --profile black,black,flake8, and theSTIX-ID pylint checks all pass.
Beyond unit tests, it was validated end-to-end against a fresh OpenCTI
7.260715.0stack with a mock Cloudflare API standing in for the account(no real credentials required, via
CLOUDFLARE_API_BASE_URL):ip)Design notes. The Cloudflare list-items API is snapshot-based (each upload
replaces the entire list), so the connector owns the list and maintains the full
set in memory, rebuilt by a full sync on every restart.
_extract_ipv4handlesboth object shapes that reach a stream connector — the live-stream STIX shape
(
type) and the OpenCTI API shape from the full sync (entity_type). Entityselection (score, labels, markings) is intentionally delegated to the OpenCTI
live-stream definition rather than reimplemented in the connector; limitations
are documented in the README.