Skip to content

fix: only glue prefilter literals that are adjacent in every match - #179

Merged
Mzack9999 merged 2 commits into
projectdiscovery:mainfrom
huuyafwww:fix-prefilter-glue
Oct 23, 2026
Merged

Mzack9999 merged 2 commits into
projectdiscovery:mainfrom
huuyafwww:fix-prefilter-glue

Conversation

@huuyafwww

@huuyafwww huuyafwww commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #180

Hi, and thanks for maintaining wappalyzergo. We run it through NewFromFile and FingerprintWithInfo in an attack surface scanner, and while checking v0.3.5 before upgrading I noticed that some technologies stopped being detected. For example, <meta name="generator" content="ShopSite Pro 14.0"> gives ShopSite:14.0 on v0.3.4 and nothing on v0.3.5.

The cause is in the literal prefilter from #135. coverConcat glues the literals of neighbouring parts into one needle, but it doesn't check that the left literal is at the end of its part or that the right literal is at the start of its part. In ^shopsite pro (\d+\.\d+) the capture's only literal is the . inside the version, so the needle becomes shopsite pro ., which the input never contains, and the regex never runs. This PR makes cover report, next to the groups, whether every match of a node starts with one of its literals (head), ends with one (tail), or is exactly one of them (exact). coverConcat now glues only when the text so far has tail and the next part has head. After a glue, the new needle keeps head only if the left side was exact, and tail only if the right side was. Everything else falls back to the existing AND of both sides, so each literal is still required.

Some background on how much this touches. Of the 6,957 distinct patterns compiled from the embedded fingerprints, this changes the literals of 13 and nothing else, and every pattern that had a prefilter still has one. Eleven of them are passive patterns:

  • ShopSite, WebSite X5 and AMP for WordPress (meta generator) and Wijmo (scriptSrc) never match on v0.3.5, because their needles are shopsite pro ., incomedia website x5 with two trailing spaces, amp plugin v. and /scripts/acts/wijmo/wijmo.min.js?v=.. Losing AMP for WordPress also loses the AMP it implies.
  • Cart.js, Criteo, Gambio, Offline.js, SQL Buddy and SaaSquatch miss one of their branches, for example //l.criteo.net/ or /bundles/app/offline.min.js.
  • eStat Streaming glues across w{1,2}, which happens to be safe because that branch always ends in w.

The other two are dom patterns. fonts\.(?:googleapis|google|gstatic)\.com shows why alternations need this as well: the parser factors the branches into g(?:oogle(?:apis|)|static), so the old needle fonts.google.com is not in fonts.googleapis.com.

The trade-off is a looser needle where a glue is no longer safe. Offline.js, for example, now needs both /bundles/ and /offline.min.js instead of the single (and wrong) /bundles//offline.min.js. BenchmarkFingerprintPassive and BenchmarkRealisticBody don't move: with benchstat over 18 runs each and the order alternated, the geomean is +0.23% with identical allocations on an M4 with Go 1.27.1. None of the benchmark pages contains text the changed needles look for, so the work per call is the same.

On correctness, I used v0.3.4, which has no prefilter, as the reference. I generated inputs that match each passive pattern (html, scriptSrc, meta, headers and cookies) from its parsed regex, 1,386,383 in total, and ran Evaluate on v0.3.4, v0.3.5 and this branch. v0.3.5 disagrees with v0.3.4 on 1,429 of them, all from the ten patterns above, and this branch disagrees on none; neither produces a match that v0.3.4 doesn't. I also fuzzed random patterns and inputs, over 400 million strings in total, checking that Evaluate never rejects a string the regex matches, and found no counterexample with this change, while the same generator finds them quickly on main.

Running 24,744 synthetic pages through FingerprintWithInfo gives the same picture, apart from three pages that hit a separate issue I left out of this PR. When a script src contains a character reference that decodes to an uppercase letter (&#74;query), or the long s (ſ) that (?i) matches as s, the scriptSrc index compares lowercase needles against text that isn't folded the way the regex folds it. Happy to send that separately.

TestLiteralPrefilterKeepsMatchingTargets covers each way the glue went wrong: a capture that starts with digits, an alternation branch that ends with a character class, a branch with an optional prefix, branches the parser factors, a repeat between two short literals, and a repeated literal between two literals. All six fail on main and pass here. go test ./... and golangci-lint run (v2.14.0) pass locally on darwin/arm64 with Go 1.27.1.

Happy to rename things or fold the new cases into TestLiteralPrefilter if you'd prefer.

huuyafwww and others added 2 commits October 14, 2026 14:15
coverConcat glued the literals of neighbouring parts into one needle
without checking that the left literal ends its part and the right
literal starts its part. For `^shopsite pro (\d+\.\d+)` the needle
became `shopsite pro .`, so the regex never ran and ShopSite, Wijmo,
WebSite X5, AMP for WordPress and some branches of six other
fingerprints stopped matching in v0.3.5.

Track whether every match of a node starts with, ends with or equals
one of its literals, and glue only when the text so far ends with a
literal and the next part starts with one. This changes the literals of
13 of the 6,957 distinct embedded patterns and keeps the prefilter on
all of them.
@Mzack9999
Mzack9999 merged commit df2dfd3 into projectdiscovery:main Oct 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Literal prefilter drops technologies detected before v0.3.5

2 participants