[Bug] ESQL Remote Validation Data Stream and Patch Version Validation by eric-forte-elastic · Pull Request #6251 · elastic/detection-rules

eric-forte-elastic · 2026-06-04T14:16:08Z

Pull Request

Issue link(s):

[Bug] ESQL Remote Validation for Unknown Indices #6213

Summary - What I changed

Fixes integration version resolution that ignored whether a referenced data stream actually exists in the resolved package, causing ES|QL remote validation and related_integrations to use the wrong package version. This surfaced on rules using logs-azure.aadgraphactivitylogs-* (added in azure 1.37.0), which resolved against older packages that predate the data stream. Please see the related issue for more details.

Changes by file:

detection_rules/integrations.py / find_latest_integration_patch_for_minor(): The stack-schema-map.yaml keys stacks at MAJOR.MINOR.0, but an integration may gate its latest package behind a later patch (e.g. azure ~8.19.10). Validating at the literal .0 resolves an older package that predates the data stream. This helper reads the integration manifests and returns the latest patch the rule's own integrations gate on for a given minor, so the inferred stack version reflects what a customer on that minor would actually receive. Only the rule's packages are inspected, not the full manifest.
detection_rules/rule_validators.py / ESQLValidator.remote_validate_rule(): The stack-version sweep now infers the patch per minor from the rule's integrations (e.g. 8.19.0 -> 8.19.10, 9.4.0 stays 9.4.0) instead of using the literal .0, so each minor resolves the up-to-date package based on whether or not this is needed for the integration versions.
detection_rules/integrations.py / find_least_compatible_version(): Previously this code decided compatibility from the package-level conditions.kibana.version only and never checked whether the requested integration/data stream existed in that version (the integration argument was used only in the error message). It now consults the integration schemas and skips versions that predate the data stream, so e.g. azure:aadgraphactivitylogs @ 8.19.10 resolves to ^1.37.0 instead of ^1.0.0. It falls back to kibana-compatibility alone when no schema data exists for a version.

How To Test

Checkout new-rule/azure-ad-graph-potential-roadrecon-enum
Run python -m detection_rules view-rule rules/integrations/azure/discovery_aad_graph_roadrecon_aiohttp_enumeration.toml --esql-remote-validation
Before the fix: the version sweep starts at the literal 8.19.0, resolves azure 1.34.1 (no aadgraphactivitylogs), and fails with:

Error (EsqlUnknownIndexError): Unknown index pattern(s): logs-azure.aadgraphactivitylogs-*. Known patterns: logs-azure.eventhub*, logs-azure.platformlogs*, logs-azure.activitylogs-*, logs-azure.platformlogs-* ...

After the fix: the related integrations populate correctly from the datastream, with the correct version of the package that is validated on the correct patch version of the stack.

  "related_integrations": [
    {
      "package": "azure",
      "version": "^1.22.0"
    },
    {
      "integration": "aadgraphactivitylogs",
      "package": "azure",
      "version": "^1.37.0"
    }
  ],

ES|QL Rule testing results

fr_esql_validation.txt

Backport tests

Note if you reproduce these manually, you will need to set an env GITHUB_EVENT_NAME=push to properly duplicate the results.

Note

When reviewing it will be necessary to have a remote stack setup for ES|QL validation. One must also verify that unit tests pass with this remote validation, as unit tests on the PR will not have the remote validation set since there are no ES|QL rule changes here.

Remote testing passed

Checklist

Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
Added the meta:rapid-merge label if planning to merge within 24 hours
Secret and sensitive material has been managed correctly
Automated testing was updated or added to match the most common scenarios
Documentation and comments were added for features that require explanation

Contributor checklist

Have you signed the contributor license agreement?
Have you followed the contributor guidelines?

github-actions · 2026-06-04T14:16:25Z

shashank-elastic · 2026-06-04T15:08:56Z

+    # latest patch a package gates on for the minor, i.e. the stack patch needed to receive the most
+    # up-to-date integration package on that minor. Scan each package once and track the newest
+    # matching package manifest.
+    manifests = load_integrations_manifests()


load_integrations_manifests() is called on every invocation, and it's called in a for version in get_stack_versions() loop in rule_validators.py.

Can we think of caching this somewhere. If the function isn't memoized/cached, this is operationally heavy and does the same load over and over again per version. Cache or optimised calls of manifest loads is a good idea.

Why this calls for optimised calls we have seen Manifest growing big with growing integration versions. Especially for AWS and Azure.

Unfortunately, no we cannot directly cache them given that each rule requirements (fields, versions, etc.) are different/potentially different per rule. Given this, we would need to evaluate them on a per rule level.

Granted we could build a hash map as an optimization so if a rule has the exact same integration info passed that we do not need to compute it again, but the goal of this PR was to go for a less complex approach first, de-duplicate with #6208 and then polish as needed.

++ Agreed. We should revisit this optimisations in future it will greatly enhance execution times.

shashank-elastic · 2026-06-04T15:11:14Z

+    # the data streams present per package version, so use them to skip versions that predate the
+    # integration. Only filter when schema data exists for a version, otherwise fall back to kibana
+    # compatibility alone (e.g. for synthetic manifests in tests).
+    package_schemas = load_integrations_schemas().get(package, {})


PR #6208 removed find_least_compatible_version from the public API and replaced it with find_compatible_version_range. This PR modifies find_least_compatible_version. If #6208 merges before this , this fix needs to be ported to the new function structure.

Yes, we are tracking this 👍

++ im going to update the other PR after this lands

terrancedejesus · 2026-06-04T15:45:31Z

            )
            mappings_lookup[version] = combined_mappings

        for version, mapping in mappings_lookup.items():


The AAD Graph rules (if not min-stacked) should still fail here somehow?

In pseudo it would be along the lines of:

If patch floor exists, get M.M

If M.M is in get_stack_versions() lookup

Find min-stack floor for rule

If no min-stack or min-stack floor < M.M

Error and suggest min-stack to earliest M.M.0 available.

We still don't want to push a rule to 8.19.4 stack who cannot pull an integration version 1.37.0 because it's gated at 8.19.10? We still have to min-stack the rule to the earliest M.M.0 just based on our release versioning.

I think we do want to push the rule to 8.19.4 even though the integration version required 8.19.10. 8.19.* is still supported (including lower patch versions). So customers will still get updates, and thereby their stack does support the rule if they choose to update.

In this way, if a customer is on 8.19.9 for example and they have the azure integration with version 1.34.1 installed, they will see the available update in the stack.

They will see a similar info/warning in the rule that the datastream/integration is either not installed (but available) or that it needs to be updated (only if it is installed), which is the desired flow for rules with integration updates.

As this warning shows there is no data (which is the intent)

Not shipping the rule, would be a change to how we ship rules, as we do ship rules that require customers to update their stack patch versions and/or integration versions.

Makes sense if the warning is that explicit.

Copilot

Pull request overview

This PR fixes ES|QL remote validation and related_integrations version resolution by ensuring integration package selection accounts for whether a referenced data stream actually exists in a given package version, and by inferring an appropriate stack patch per minor when integrations gate features behind later patches.

Changes:

Infer the correct stack patch per minor based on the rule’s referenced integrations during ES|QL stack-version sweeps.
Add find_latest_integration_patch_for_minor() to derive the patch floor implied by integration manifests’ Kibana version conditions.
Update find_least_compatible_version() to skip package versions that predate the requested integration/data stream when schema data is available.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
pyproject.toml	Bumps project version to `1.6.48`.
detection_rules/rule_validators.py	Updates ES
detection_rules/integrations.py	Adds patch-floor inference helper and enhances least-compatible version selection using per-version integration schema presence.

shashank-elastic

The fix is correct and well-motivated with good evidence. Concerns around PR #6208 is being addressed. This is good to go 🚀

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…detection-rules into patch_version_to_latest

…ranges Merge main (#6251) and port data-stream schema filtering into find_compatible_version_range instead of find_least_compatible_version. Keep find_latest_integration_patch_for_minor for ES|QL validation. Integration-specific export uses schema walk plus schema-floor fallback for streams gated behind newer package versions (e.g. aadgraphactivitylogs).

eric-forte-elastic added 2 commits June 3, 2026 18:07

Add ESQL patch version support and related integrations datastream fix

eb4069b

remove sort now O n log n

f67b60a

eric-forte-elastic requested review from shashank-elastic and terrancedejesus June 4, 2026 14:16

eric-forte-elastic self-assigned this Jun 4, 2026

eric-forte-elastic added bug Something isn't working python Internal python for the repository patch labels Jun 4, 2026

eric-forte-elastic marked this pull request as ready for review June 4, 2026 14:17

eric-forte-elastic requested a review from Mikaayenson as a code owner June 4, 2026 14:17

github-actions Bot added the backport: auto label Jun 4, 2026

Mikaayenson linked an issue Jun 4, 2026 that may be closed by this pull request

[Bug] ESQL Remote Validation for Unknown Indices #6213

Closed

shashank-elastic reviewed Jun 4, 2026

View reviewed changes

Comment thread detection_rules/integrations.py

Mikaayenson approved these changes Jun 4, 2026

View reviewed changes

Comment thread detection_rules/integrations.py

terrancedejesus reviewed Jun 4, 2026

View reviewed changes

Mikaayenson requested a review from Copilot June 4, 2026 16:12

Copilot started reviewing on behalf of Mikaayenson June 4, 2026 16:12 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread detection_rules/integrations.py Outdated

Comment thread detection_rules/integrations.py

Comment thread detection_rules/integrations.py

Comment thread detection_rules/integrations.py Outdated

Update unit test to not use remote route when available

d4ca0b0

shashank-elastic approved these changes Jun 4, 2026

View reviewed changes

eric-forte-elastic and others added 7 commits June 4, 2026 12:47

Add unit tests for new functionality

4d0a236

Potential fix for pull request finding

911c0b7

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Small optimization

512229a

Add type hint

e3fb131

Merge branch 'main' into patch_version_to_latest

44220e1

Update comment

d8bc779

Merge branch 'patch_version_to_latest' of https://github.com/elastic/…

3bc6a51

…detection-rules into patch_version_to_latest

eric-forte-elastic merged commit 671167f into main Jun 4, 2026
16 checks passed

eric-forte-elastic deleted the patch_version_to_latest branch June 4, 2026 17:30

Mikaayenson mentioned this pull request Jun 5, 2026

Fix stack-dependent related_integrations.version export #6208

Open

5 tasks

Conversation

eric-forte-elastic commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Summary - What I changed

How To Test

Checklist

Contributor checklist

Uh oh!

github-actions Bot commented Jun 4, 2026

Bug - Guidelines

Documentation and Context

Code Standards and Practices

Testing

Additional Checks

Uh oh!

shashank-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

eric-forte-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

shashank-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

shashank-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

eric-forte-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Mikaayenson Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shashank-elastic Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

terrancedejesus Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-forte-elastic Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terrancedejesus Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shashank-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eric-forte-elastic commented Jun 4, 2026 •

edited

Loading

Mikaayenson Jun 4, 2026 •

edited

Loading

terrancedejesus Jun 4, 2026 •

edited

Loading

eric-forte-elastic Jun 4, 2026 •

edited

Loading