refactor: detection rule sets as a registry (#478)

[n1ckl0sk0rtge]

Closes #478.

What this does

Every detection-rule class used to cache its own rule list with a Memoize.of(...) supplier and hand it out through a public static rules() method. Because that accessor was public and static — and because several classes had context-parameterized variants that skipped the memo — callers could rebuild whole rule subtrees. That is issue #476. The C# module had no caching at all and rebuilt everything on every call.

This replaces that with a registry in the engine module, package com.ibm.engine.rule:

• DetectionRuleSet<T> — a rule class extends it and implements only protected List<IDetectionRule<T>> buildRules(). protected is the point: nothing outside com.ibm.engine.rule can call it, and every rule class is final, so there is no way to widen access by subclassing.
• ContextualDetectionRuleSet<T> — for rule sets whose rules depend on one or more IDetectionContext. The context list is positional; a null means "use the default for that position"; read it with contextAt(list, index).
• RuleSets — the single door. rulesOf(Class) caches in a ClassValue; rulesOf(Class, IDetectionContext...) caches in a ConcurrentHashMap keyed on class plus contexts.
• Detection contexts gained value-based equals/hashCode so they can be part of that key, and DetectionContext now copies its property map defensively.
• RuleAccessorEnforcementTest in each of the four language modules scans the compiled classes and fails the build if any class under com.ibm.plugin.rules.detection exposes a public static method returning a rule list. The compiler cannot police a static method; this test can.

All four language modules are converted: java, python, go and csharp.

Result

Distinct reachable rule objects: 2,563 → 1,015 (a 60% reduction). The plan predicted only a small drop; most of the gain came from two places:

• Context-parameterized calls such as BcDigests.rules(ctx) used to rebuild a fresh subtree on every call. They now share one.
• RuleSets.rulesOf(X.class, someNullableContext) always binds the varargs overload, even when the value is null, so it landed on a separate cache entry from a bare rulesOf(X.class). RuleSets now trims trailing nulls and falls through to the default path — exact rather than heuristic, because contextAt already returns null past the end of the list. That one change alone took the count from 1,069 to 1,015.

Evidence the rules themselves did not change

The exported rule JSON was generated from the branch base in a separate worktree before any of this work and compared byte-for-byte after:

Module	md5	Nodes
java	a19a6ddf64e91bb156a39e4134677da8	515
python	d1c5ce3945ec370d15c5118372e0f043	114

Both identical before and after. Go and C# have no export test; their suites are the content check.

mvn clean package: BUILD SUCCESS, 433 tests, 0 failures, 0 errors. The base measured 413; the delta is +21 new engine tests, −2 net in java (two test classes deleted, one added) and +1 in csharp.

Also in this PR

• docs/LANGUAGE_SUPPORT.md and docs/DETECTION_RULE_STRUCTURE.md still taught the public static rules() pattern that the build now rejects. Both are updated — a contributor following them would otherwise have written a class that neither compiled nor passed the guard test.
• RuleGraphMemoizationTest asserted the object count was under 60,000 while the real value is 1,015. Tightened to 1,500 so the result is actually defended by CI. This test is also the only automated guard against the other route back to Reduce detection-rule memory footprint (BouncyCastle rule graph OOMs the Scanner Engine) #476: a buildRules() that builds a fresh context per call, which the accessor guard cannot see.
• Two rule sets were extracted so each class owns exactly one list: BcBlockCipherAndEngines (was BcBlockCipher.all()) and BcCipherParametersBases (was BcCipherParameters.bases(), which was never memoized and was called repeatedly).
• PycaHashWrapper in python, for the same reason (PycaHash.wrapperRules()).

Reviewing this

The bulk of the diff is the same mechanical conversion across ~180 rule classes. The design-bearing files are:

• engine/src/main/java/com/ibm/engine/rule/RuleSets.java, DetectionRuleSet.java, ContextualDetectionRuleSet.java
• engine/src/main/java/com/ibm/engine/model/context/DetectionContext.java and its four siblings
• java/src/test/java/com/ibm/plugin/rules/RuleAccessorEnforcementTest.java

Note the contextual cache deliberately uses get / build / putIfAbsent rather than computeIfAbsent: rule builds are recursive — a set asks the registry for another set while it is itself being built — and a nested update inside computeIfAbsent throws. There is a comment saying so at the call site.

Known limitation

RuleSets.instantiate reports "could not be instantiated; it needs an accessible no-argument constructor" for every reflective failure. That is accurate when the constructor is missing, but misleading for an abstract subclass or a constructor that throws — both of which do have one. It cannot fire today (every rule class is final and concrete), and the real cause is chained. Left as-is.

[n1ckl0sk0rtge]

Performance test results

I ran both levels of docs/PERFORMANCE_TESTING.md on this branch. Heap stays small — no regression.

A. JUnit harness

Run	Result	Gate
CallStackHeapPerfTest, 200 files	retainedWithTree=0 detached=367/367 ratio=1.000, 2.4 s	pass
CallStackHeapPerfTest, 3000-file soak	retainedWithTree=0 detached=5500/5500 ratio=1.000, 88 s	pass
RuleGraphMemoizationTest	1015 distinct rule objects	< 1500, pass

AST-detach is fully intact (ratio 1.000, nothing pinning a tree).

B. Keycloak end-to-end

Keycloak main (94 compiled modules), SonarQube 26.1, -Xmx6g, JDK 21.

• ANALYSIS SUCCESSFUL in 13 min 27 s, no OOM.
• CBOM generated: CycloneDX 1.6, 108 detected assets → 68 components. Same as the reference run in the doc, so detections really fired.
• Heap oscillates (4553 → 3425 → 4602 → 3624 MB …). G1 reclaims, no monotonic climb. That is the healthy shape the doc describes.
• Peak heap_used 5794 MB, peak RSS 6783 MB.

C. Attribution — the point of this PR

com.ibm.** bytes, sampled every 3 minutes after a forced GC:

Sample	engine.rule	callstack	mapper.model	plugin total	whole-heap floor
early	0.07 MB / 1880 inst	0.10 MB	0	0.39 MB	0.33 GB
t1	0.07 MB / 1880 inst	4.58 MB	0.01 MB	8.52 MB	1.50 GB
t2	0.07 MB / 1880 inst	7.98 MB	0.01 MB	14.60 MB	2.48 GB
t3	0.07 MB / 1880 inst	12.42 MB	0.01 MB	22.51 MB	3.30 GB
t4	0.07 MB / 1880 inst	15.58 MB	0.01 MB	28.18 MB	4.09 GB

The rule graph is flat at 1880 instances / 0.07 MB for the whole scan. The RuleSets registry builds each rule set once and shares it by reference. No duplication creeps back in over a long run.

Total plugin footprint is 28.18 MB, essentially the same as the 28.4 MB recorded for the AST-detach run.

One difference worth flagging

The post-GC floor ended at 4.09 GB here, against 2.90 GB in the doc's 2026-07-06 run. This is not the plugin. The top retained classes at t4 are all byte[] (542 MB), Object[] (348 MB), HashMap$Node (335 MB), InternalPosition (154 MB) and ECJ MethodBinding (145 MB) — SonarQube and ECJ baseline. No com.ibm.* class appears anywhere near the top. Keycloak main has grown since July, so its baseline analysis cost is higher. The plugin's own share is unchanged.

_{Note: the scan needs the sonar-java-crypto:Inventory rule activated in the quality profile, otherwise nothing is detected and the measurement is meaningless.}