test(all): add connector_config_schema.json validation tests (#6342)

[Powlinett]

Proposed changes

• Add test_connector_config_schema.py with 7 parametrized tests validating all connector_config_schema.json files against a meta-schema and connector-type-specific conventions (CONNECTOR_TYPE const, CONNECTOR_SCOPE array, CONNECTOR_AUTO boolean, CONNECTOR_LIVE_STREAM_* fields, CONNECTOR_DURATION_PERIOD with format duration)
• Fix 19 connectors using type: str instead of type: Literal["..."] in their pydantic models so CONNECTOR_TYPE is emitted as const in the JSON schema
• Fix config schema generation for google-ti-feeds, splunk-soar-push (json_schema_extra for CONNECTOR_SCOPE), ransomlook (extra="ignore"), and recorded-future (duration_period: timedelta)
• Remove hand-written/stale connector_config_schema.json files for non-manager-supported connectors (trukno, usta, ctm360-cyberblindspot-feed, ctm360-cyna-feed, osint-industries)
• Add given/when/then comments to test_connector_manifest.py for consistency

Related issues

• Close test(all): enable xfail tests for manager_supported connectors and connector_config_schema.json unit tests #6342

Checklist

• I consider the submitted work as finished
• I have signed my commits using GPG key.
• I tested the code for its functionality using different use cases
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

Test results: 451 passed, 2 skipped (internal-export-file/internal-import-file have no schemas yet), 7 xfailed (connectors not yet using connectors-sdk BaseSettings).

[codecov]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
11212	3	11209	203

View the top 3 failed test(s) by shortest run time

external-import.vulnerability-lookup.tests.test_settings.TestVulnerabilityLookupConnectorConfig::test_defaults

Stack Traces | 0.001s run time

self = <tests.test_settings.TestVulnerabilityLookupConnectorConfig object at 0x7f83483be7e0>

    def test_defaults(self):
        config = VulnerabilityLookupConnectorConfig(id="test-id")
        assert config.name == "Vulnerability Lookup"
>       assert config.scope == "vulnerability"
E       AssertionError: assert ['vulnerability'] == 'vulnerability'
E        +  where ['vulnerability'] = VulnerabilityLookupConnectorConfig(id='test-id', name='Vulnerability Lookup', scope=['vulnerability'], log_level='error', type='EXTERNAL_IMPORT', duration_period=datetime.timedelta(days=1)).scope

.../vulnerability-lookup/tests/test_settings.py:21: AssertionError

external-import.ransomlook.tests.test_config_contract::test_generated_configuration_schema_is_current

Stack Traces | 0.012s run time

def test_generated_configuration_schema_is_current():
        committed = json.loads(
            (ROOT / "__metadata__" / "connector_config_schema.json").read_text(
                encoding="utf-8"
            )
        )
        assert committed == ConnectorSettings.config_json_schema(
            connector_name="ransomlook"
        )
>       assert committed["additionalProperties"] is False
E       assert True is False

.../ransomlook/tests/test_config_contract.py:19: AssertionError

external-import.recorded-future.tests.test_settings::test_settings_should_accept_valid_input[full_valid_settings_dict]

Stack Traces | 0.014s run time

request = <FixtureRequest for <Function test_settings_should_accept_valid_input[full_valid_settings_dict]>>
settings_dict = 'full_settings_dict'

    @pytest.mark.parametrize(
        "settings_dict",
        [
            pytest.param(
                "full_settings_dict",
                id="full_valid_settings_dict",
            ),
            pytest.param(
                "minimal_settings_dict",
                id="minimal_valid_settings_dict",
            ),
        ],
    )
    def test_settings_should_accept_valid_input(request, settings_dict):
        """
        Test that `ConfigLoader` accepts valid input.
        For the test purpose, `ConfigLoader.settings_customise_sources` is overridden to return
        a fake but valid dict (instead of the env/config vars parsed from `config.yml`, `.env` or env vars).
    
        :param settings_dict: The dict to use as `ConfigLoader` input
        """
    
        class FakeConfigLoader(ConfigLoader):
            """
            Subclass of `ConfigLoader` for testing purpose.
            It overrides `ConfigLoader.settings_customise_sources`
            to return a fake but valid config dict.
            """
    
            @classmethod
            def settings_customise_sources(
                cls,
                settings_cls,
                init_settings,
                env_settings,
                dotenv_settings,
                file_secret_settings,
            ):
                def yml_settings() -> dict:
                    return request.getfixturevalue(settings_dict)
    
                return (yml_settings, env_settings, dotenv_settings, file_secret_settings)
    
>       settings = FakeConfigLoader()
                   ^^^^^^^^^^^^^^^^^^

.../recorded-future/tests/test_settings.py:55: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

__pydantic_self__ = FakeConfigLoader(), _case_sensitive = None
_nested_model_default_partial_update = None, _env_prefix = None
_env_prefix_target = None, _env_file = PosixPath('.'), _env_file_encoding = None
_env_ignore_empty = None, _env_nested_delimiter = None
_env_nested_max_split = None, _env_parse_none_str = None
_env_parse_enums = None, _cli_prog_name = None, _cli_parse_args = None
_cli_settings_source = None, _cli_parse_none_str = None
_cli_hide_none_type = None, _cli_avoid_json = None, _cli_enforce_required = None
_cli_use_class_docs_for_groups = None, _cli_exit_on_error = None
_cli_prefix = None, _cli_flag_prefix_char = None, _cli_implicit_flags = None
_cli_ignore_unknown_args = None, _cli_kebab_case = None, _cli_shortcuts = None
_secrets_dir = None, _build_sources = None, values = {}
sources = (<function test_settings_should_accept_valid_input.<locals>.FakeConfigLoader.settings_customise_sources.<locals>.yml_s...efix_len=0), SecretsSettingsSource(secrets_dir=None), DefaultSettingsSource(nested_model_default_partial_update=False))
init_kwargs = {}

    def __init__(
        __pydantic_self__,
        _case_sensitive: bool | None = None,
        _nested_model_default_partial_update: bool | None = None,
        _env_prefix: str | None = None,
        _env_prefix_target: EnvPrefixTarget | None = None,
        _env_file: DotenvType | None = ENV_FILE_SENTINEL,
        _env_file_encoding: str | None = None,
        _env_ignore_empty: bool | None = None,
        _env_nested_delimiter: str | None = None,
        _env_nested_max_split: int | None = None,
        _env_parse_none_str: str | None = None,
        _env_parse_enums: bool | None = None,
        _cli_prog_name: str | None = None,
        _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None,
        _cli_settings_source: CliSettingsSource[Any] | None = None,
        _cli_parse_none_str: str | None = None,
        _cli_hide_none_type: bool | None = None,
        _cli_avoid_json: bool | None = None,
        _cli_enforce_required: bool | None = None,
        _cli_use_class_docs_for_groups: bool | None = None,
        _cli_exit_on_error: bool | None = None,
        _cli_prefix: str | None = None,
        _cli_flag_prefix_char: str | None = None,
        _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None,
        _cli_ignore_unknown_args: bool | None = None,
        _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None,
        _cli_shortcuts: Mapping[str, str | list[str]] | None = None,
        _secrets_dir: PathType | None = None,
        _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None,
        **values: Any,
    ) -> None:
        sources, init_kwargs = (
            _build_sources
            if _build_sources is not None
            else __pydantic_self__.__class__._settings_init_sources(
                _case_sensitive=_case_sensitive,
                _nested_model_default_partial_update=_nested_model_default_partial_update,
                _env_prefix=_env_prefix,
                _env_prefix_target=_env_prefix_target,
                _env_file=_env_file,
                _env_file_encoding=_env_file_encoding,
                _env_ignore_empty=_env_ignore_empty,
                _env_nested_delimiter=_env_nested_delimiter,
                _env_nested_max_split=_env_nested_max_split,
                _env_parse_none_str=_env_parse_none_str,
                _env_parse_enums=_env_parse_enums,
                _cli_prog_name=_cli_prog_name,
                _cli_parse_args=_cli_parse_args,
                _cli_settings_source=_cli_settings_source,
                _cli_parse_none_str=_cli_parse_none_str,
                _cli_hide_none_type=_cli_hide_none_type,
                _cli_avoid_json=_cli_avoid_json,
                _cli_enforce_required=_cli_enforce_required,
                _cli_use_class_docs_for_groups=_cli_use_class_docs_for_groups,
                _cli_exit_on_error=_cli_exit_on_error,
                _cli_prefix=_cli_prefix,
                _cli_flag_prefix_char=_cli_flag_prefix_char,
                _cli_implicit_flags=_cli_implicit_flags,
                _cli_ignore_unknown_args=_cli_ignore_unknown_args,
                _cli_kebab_case=_cli_kebab_case,
                _cli_shortcuts=_cli_shortcuts,
                _secrets_dir=_secrets_dir,
                _init_kwargs=values,
            )
        )
    
>       super().__init__(**__pydantic_self__.__class__._settings_build_values(sources, init_kwargs))
E       pydantic_core._pydantic_core.ValidationError: 1 validation error for FakeConfigLoader
E       connector.duration_period
E         Input should be a valid timedelta, quantity invalid in time part of duration [type=time_delta_parsing, input_value='PT1D', input_type=str]
E           For further information visit https://errors.pydantic.dev/2.13/v/time_delta_parsing

.temp_venv/lib/python3.12.../site-packages/pydantic_settings/main.py:247: ValidationError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[Copilot]

Pull request overview

Adds repository-level validation for connector_config_schema.json files and aligns a set of connectors’ generated schemas with the expected conventions (notably CONNECTOR_TYPE as a JSON Schema const, CONNECTOR_SCOPE as an array, and duration formatting).

Changes:

• Add tests/tests_metadata/test_connector_config_schema.py to validate JSON Schema compliance and connector-type-specific invariants across all existing connector config schemas.
• Normalize multiple connectors’ Pydantic settings models so generated schemas emit CONNECTOR_TYPE as const and CONNECTOR_SCOPE as an array (and update generated schemas/docs accordingly).
• Remove stale hand-written schema/docs files for several non-manager-supported connectors and add jsonschema to test dependencies.

Reviewed changes

Copilot reviewed 63 out of 63 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/tests_metadata/test_connector_manifest.py	Refactor manifest test to use helper loader.
tests/tests_metadata/test_connector_config_schema.py	New test suite validating config schemas.
tests/test-requirements.txt	Add jsonschema dependency for tests.
stream/splunk-soar-push/src/models/configs/connector_configs.py	Make connector type a Literal for const schema emission.
stream/splunk-soar-push/src/models/configs/config_loader.py	Override schema for CONNECTOR_SCOPE to array convention.
stream/splunk-soar-push/metadata/connector_config_schema.json	Update scope to array and add CONNECTOR_TYPE.const.
stream/misp-intel/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
stream/misp-intel/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/virustotal/src/virustotal/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/virustotal/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/virustotal-downloader/src/connector/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/virustotal-downloader/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/osint-industries/metadata/connector_config_schema.json	Remove stale hand-written schema file.
internal-enrichment/osint-industries/metadata/CONNECTOR_CONFIG_DOC.md	Remove stale generated config doc.
internal-enrichment/ipinfo/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/ipinfo/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/hygiene/src/connector/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/hygiene/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/google-dns/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/google-dns/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
internal-enrichment/abuseipdb/src/connector/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
internal-enrichment/abuseipdb/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/vulnerability-lookup/src/connector/settings.py	Change scope to list-friendly type and default.
external-import/vulnerability-lookup/metadata/connector_config_schema.json	Emit scope as array with items/default list.
external-import/vulnerability-lookup/metadata/CONNECTOR_CONFIG_DOC.md	Update docs to reflect scope as array.
external-import/usta/metadata/connector_config_schema.json	Remove stale hand-written schema file.
external-import/usta/metadata/CONNECTOR_CONFIG_DOC.md	Remove stale generated config doc.
external-import/trukno/metadata/connector_config_schema.json	Remove stale hand-written schema file.
external-import/trukno/metadata/CONNECTOR_CONFIG_DOC.md	Remove stale generated config doc.
external-import/threatfox/src/models/configs/connector_configs.py	Make connector type a Literal.
external-import/threatfox/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/sparta/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/sparta/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/sekoia/src/connector/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/sekoia/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/recorded-future/src/models/configs/connector_configs.py	Switch duration to timedelta; make type a Literal.
external-import/recorded-future/metadata/connector_config_schema.json	Add duration format: duration, update default, add const.
external-import/ransomwarelive/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/ransomwarelive/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/ransomlook/src/connector/settings.py	Adjust pydantic extra handling to affect schema output.
external-import/ransomlook/metadata/connector_config_schema.json	Change additionalProperties to true.
external-import/mitre/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/mitre/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/mandiant/src/connector/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/mandiant/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/google-ti-feeds/connector/src/octi/configs/connector_config.py	Override schema for CONNECTOR_SCOPE to array convention.
external-import/google-ti-feeds/metadata/connector_config_schema.json	Emit scope as array with items/default list; adjust descriptions.
external-import/ft3/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/ft3/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/feedly/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/feedly/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/email-intel-microsoft/src/base_connector/enums.py	Remove critical level; add warn alias.
external-import/email-intel-microsoft/metadata/connector_config_schema.json	Update log level enum (drop critical, add warn).
external-import/email-intel-microsoft/metadata/CONNECTOR_CONFIG_DOC.md	Update docs to match log level enum.
external-import/email-intel-imap/src/base_connector/enums.py	Remove critical level; add warn alias.
external-import/email-intel-imap/metadata/connector_config_schema.json	Update log level enum (drop critical, add warn).
external-import/email-intel-imap/metadata/CONNECTOR_CONFIG_DOC.md	Update docs to match log level enum.
external-import/cve/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/cve/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.
external-import/ctm360-cyna-feed/metadata/connector_config_schema.json	Remove stale hand-written schema file.
external-import/ctm360-cyberblindspot-feed/metadata/connector_config_schema.json	Remove stale hand-written schema file.
external-import/cisa-known-exploited-vulnerabilities/src/models/configs/connector_configs.py	Make connector type a Literal; remove redundant validator.
external-import/cisa-known-exploited-vulnerabilities/metadata/connector_config_schema.json	Add CONNECTOR_TYPE.const to schema.