feat(email-cases-importer): add connector to import emails as incident-response cases (#7036)#7037
Open
Khidr6G wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new EXTERNAL_IMPORT connector (external-import/email-cases-importer) that polls email inboxes across multiple providers/protocols and creates/updates Incident Response Cases (Case-Incident) in OpenCTI, including attachment extraction/decryption and thread-based grouping.
Changes:
- Adds connector implementation + protocol clients (IMAP, Microsoft Graph, Gmail, EWS) and shared HTTP robustness helpers.
- Adds attachment handling pipeline (archives/documents/passthrough) with nested recursion and hashing.
- Adds extensive unit tests, containerization (Dockerfile/compose), and end-user documentation/config samples.
Reviewed changes
Copilot reviewed 48 out of 50 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| external-import/email-cases-importer/VERSION | Connector version pin. |
| external-import/email-cases-importer/tests/init.py | Marks test package. |
| external-import/email-cases-importer/tests/conftest.py | Test path setup and dependency stubs. |
| external-import/email-cases-importer/tests/test_attachment_handlers.py | Tests handler routing + passthrough behavior. |
| external-import/email-cases-importer/tests/test_connector.py | End-to-end orchestration tests with mocked OpenCTI helper. |
| external-import/email-cases-importer/tests/test_converter_to_stix.py | Tests identity and HTML/content formatting. |
| external-import/email-cases-importer/tests/test_email_client_factory.py | Tests protocol routing and unsupported protocols. |
| external-import/email-cases-importer/tests/test_ews_client.py | Tests EWS parsing + lifecycle (with exchangelib). |
| external-import/email-cases-importer/tests/test_ews_filter.py | Regression tests for correct exchangelib filter construction. |
| external-import/email-cases-importer/tests/test_gmail_client.py | Tests Gmail API client behavior. |
| external-import/email-cases-importer/tests/test_graph_client.py | Tests Microsoft Graph client behavior. |
| external-import/email-cases-importer/tests/test_handlers_extra.py | Tests archive/document handlers + nested recursion edges. |
| external-import/email-cases-importer/tests/test_http_helpers.py | Tests shared HTTP retry + JSON parsing robustness. |
| external-import/email-cases-importer/tests/test_imap_client.py | Tests IMAP client parsing and connection behavior. |
| external-import/email-cases-importer/tests/test_main.py | Tests entrypoint main execution paths. |
| external-import/email-cases-importer/tests/test_settings.py | Tests Pydantic validators/parsers for settings. |
| external-import/email-cases-importer/tests/test_utils.py | Tests string/html/hash utilities. |
| external-import/email-cases-importer/test-requirements.txt | Test dependencies for isolated connector test runs. |
| external-import/email-cases-importer/src/requirements.txt | Runtime dependencies for connector image. |
| external-import/email-cases-importer/src/main.py | Runtime entrypoint wiring settings → helper → connector. |
| external-import/email-cases-importer/src/email_client/init.py | Public email_client exports. |
| external-import/email-cases-importer/src/email_client/_http.py | Shared HTTP helpers (429 backoff + non-JSON 200 handling). |
| external-import/email-cases-importer/src/email_client/base.py | Email message/attachment dataclasses + base client interface. |
| external-import/email-cases-importer/src/email_client/ews_client.py | EWS client implementation. |
| external-import/email-cases-importer/src/email_client/factory.py | Protocol-based email client factory. |
| external-import/email-cases-importer/src/email_client/gmail_client.py | Gmail API client implementation. |
| external-import/email-cases-importer/src/email_client/graph_client.py | Microsoft Graph API client implementation. |
| external-import/email-cases-importer/src/email_client/imap_client.py | IMAP client implementation. |
| external-import/email-cases-importer/src/connector/init.py | Connector package exports. |
| external-import/email-cases-importer/src/connector/connector.py | Main connector orchestration + OpenCTI write paths. |
| external-import/email-cases-importer/src/connector/converter_to_stix.py | Identity + content/description formatting helpers. |
| external-import/email-cases-importer/src/connector/settings.py | Pydantic settings model and validators. |
| external-import/email-cases-importer/src/connector/utils.py | Utility helpers (subject normalization, HTML sanitization, hashing). |
| external-import/email-cases-importer/src/attachment_handler/init.py | Public attachment handler exports. |
| external-import/email-cases-importer/src/attachment_handler/archive_handler.py | ZIP/7z/RAR extraction logic. |
| external-import/email-cases-importer/src/attachment_handler/base.py | Attachment handler interface + ExtractedFile model. |
| external-import/email-cases-importer/src/attachment_handler/document_handler.py | XLSX/PDF handling + optional decryption. |
| external-import/email-cases-importer/src/attachment_handler/passthrough_handler.py | Passthrough handler for non-decrypted files. |
| external-import/email-cases-importer/src/attachment_handler/registry.py | Handler registry + recursion for nested archives. |
| external-import/email-cases-importer/scripts/test_mailbox_connection.py | Standalone mailbox connectivity tester script. |
| external-import/email-cases-importer/README.md | Connector documentation (features/config/behavior). |
| external-import/email-cases-importer/pyproject.toml | Formatter/linter/test config for this connector. |
| external-import/email-cases-importer/entrypoint.sh | Container entrypoint script. |
| external-import/email-cases-importer/Dockerfile | Alpine-based connector image build. |
| external-import/email-cases-importer/docker-compose.yml | Local deployment example. |
| external-import/email-cases-importer/config.yml.sample | Fully commented config sample. |
| external-import/email-cases-importer/.env.sample | Environment variable sample for compose. |
| external-import/email-cases-importer/.dockerignore | Image build context hygiene. |
| external-import/email-cases-importer/metadata/connector_manifest.json | Connector manifest metadata (image/version/support version). |
| content=content, | ||
| severity=severity or self.config.email_cases.default_severity, | ||
| priority=priority or self.config.email_cases.default_priority, | ||
| createdBy=author_id or self.converter.identity_id, |
Comment on lines
+23
to
+27
| def __init__(self, config, helper: OpenCTIConnectorHelper): | ||
| self.config = config | ||
| self.helper = helper | ||
| self.converter = ConverterToStix(helper, config) | ||
| self.handler_registry = HandlerRegistry() |
Comment on lines
+54
to
+65
| try: | ||
| zf.testzip() | ||
| if pwd: | ||
| zf.setpassword(pwd) | ||
| # Test extraction of first file | ||
| if zf.namelist(): | ||
| zf.read(zf.namelist()[0], pwd=pwd) | ||
| used_pwd = pwd | ||
| break | ||
| except (RuntimeError, zipfile.BadZipFile): | ||
| continue | ||
|
|
Comment on lines
+68
to
+70
| if since: | ||
| iso_since = since.strftime("%Y-%m-%dT%H:%M:%SZ") | ||
| filter_parts.append(f"receivedDateTime ge {iso_since}") |
Comment on lines
+163
to
+166
| if body_data and mime_type == "text/plain": | ||
| body_plain = base64.urlsafe_b64decode(body_data).decode( | ||
| "utf-8", errors="replace" | ||
| ) |
Comment on lines
+24
to
+36
| def connect(self) -> None: | ||
| from exchangelib import ( | ||
| DELEGATE, | ||
| Account, | ||
| Configuration, | ||
| Credentials, | ||
| ) | ||
| from exchangelib.protocol import BaseProtocol, NoVerifyHTTPAdapter | ||
|
|
||
| if not self._tls_verify: | ||
| BaseProtocol.HTTP_ADAPTER_CLS = NoVerifyHTTPAdapter | ||
|
|
||
| credentials = Credentials(self._username, self._password) |
Comment on lines
+176
to
+185
| def build_graph_client(args): | ||
| try: | ||
| from email_client.graph_client import GraphClient | ||
| except ImportError as exc: | ||
| print( | ||
| f"ERROR: Microsoft Graph client requires 'msal'. Install: " | ||
| f"pip install msal ({exc})", | ||
| file=sys.stderr, | ||
| ) | ||
| sys.exit(1) |
| |--------|-----------|-----------------|---------| | ||
| | ZIP archive | `.zip` | Yes | `zipfile` (stdlib) | | ||
| | 7-Zip archive | `.7z` | Yes | `py7zr` | | ||
| | RAR archive | `.rar` | Yes | `rarfile` + `unrar` | |
| | CSV | `.csv` | No | `csv` (stdlib) | | ||
| | Text | `.txt` | No | stdlib | | ||
| | Email | `.eml` | No | `email` (stdlib) | | ||
| | Outlook message | `.msg` | No | `extract-msg` | |
Comment on lines
+122
to
+125
| date_tuple = email.utils.parsedate_to_datetime(msg.get("Date", "")) | ||
| if date_tuple.tzinfo is None: | ||
| date_tuple = date_tuple.replace(tzinfo=timezone.utc) | ||
| in_reply_to = msg.get("In-Reply-To", "") |
Khidr6G
force-pushed
the
feature/email-cases-importer
branch
from
July 17, 2026 15:56
aab4060 to
d58346e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
external-import/email-cases-importer). It polls a mailbox and creates Incident Response Cases (Case-Incident) from matching emails, writing the email content to the case Content tab and uploading attachments.pycti.<Class>.generate_id(...); startup connectivity/credential validation withexit(1)on fatal failure; structured logging viahelper.connector_logger;Retry-After-aware 429 backoff and typed errors on non-JSON 200 responses; a fully-failed import cycle marks the OpenCTI Workin_errorand preserves state (no silent "success" with zero imports).config.yml.sampleand.env.sample; no hardcoded secrets.Related issues
Checklist
Further comments
>= 7.260715.0(matches__metadata__/connector_manifest.json).case_incident.create,stix_domain_object.add_file,update_field) rather than emitting STIX 2.1 bundles viasend_stix2_bundle. This is a deliberate, required choice: attachment files live on the case Files tab (binary attachments have no STIX representation), email content is appended to the case Content tab on each reply (bundles upsert, not append), and case templates / assignees / participants reference OpenCTI-internal objects with no STIX equivalent.send_stix2_bundleis also asynchronous and cannot return the case's internal ID needed for the immediately-following file and content writes. Idempotency is preserved via deterministicpycti.<Class>.generate_id(...). See the README Behavior section for the full rationale.bsdtar(libarchive-tools) is used for RAR extraction and cannot decrypt password-protected RAR archives — those are uploaded as-is.