Repository navigation
fix: only glue prefilter literals that are adjacent in every match - #179
Merged
Merged
Conversation
coverConcat glued the literals of neighbouring parts into one needle without checking that the left literal ends its part and the right literal starts its part. For `^shopsite pro (\d+\.\d+)` the needle became `shopsite pro .`, so the regex never ran and ShopSite, Wijmo, WebSite X5, AMP for WordPress and some branches of six other fingerprints stopped matching in v0.3.5. Track whether every match of a node starts with, ends with or equals one of its literals, and glue only when the text so far ends with a literal and the next part starts with one. This changes the literals of 13 of the 6,957 distinct embedded patterns and keeps the prefilter on all of them.
Mzack9999
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #180
Hi, and thanks for maintaining wappalyzergo. We run it through
NewFromFileandFingerprintWithInfoin an attack surface scanner, and while checking v0.3.5 before upgrading I noticed that some technologies stopped being detected. For example,<meta name="generator" content="ShopSite Pro 14.0">givesShopSite:14.0on v0.3.4 and nothing on v0.3.5.The cause is in the literal prefilter from #135.
coverConcatglues the literals of neighbouring parts into one needle, but it doesn't check that the left literal is at the end of its part or that the right literal is at the start of its part. In^shopsite pro (\d+\.\d+)the capture's only literal is the.inside the version, so the needle becomesshopsite pro ., which the input never contains, and the regex never runs. This PR makescoverreport, next to the groups, whether every match of a node starts with one of its literals (head), ends with one (tail), or is exactly one of them (exact).coverConcatnow glues only when the text so far hastailand the next part hashead. After a glue, the new needle keepsheadonly if the left side wasexact, andtailonly if the right side was. Everything else falls back to the existing AND of both sides, so each literal is still required.Some background on how much this touches. Of the 6,957 distinct patterns compiled from the embedded fingerprints, this changes the literals of 13 and nothing else, and every pattern that had a prefilter still has one. Eleven of them are passive patterns:
metagenerator) and Wijmo (scriptSrc) never match on v0.3.5, because their needles areshopsite pro .,incomedia website x5with two trailing spaces,amp plugin v.and/scripts/acts/wijmo/wijmo.min.js?v=.. Losing AMP for WordPress also loses the AMP it implies.//l.criteo.net/or/bundles/app/offline.min.js.w{1,2}, which happens to be safe because that branch always ends inw.The other two are
dompatterns.fonts\.(?:googleapis|google|gstatic)\.comshows why alternations need this as well: the parser factors the branches intog(?:oogle(?:apis|)|static), so the old needlefonts.google.comis not infonts.googleapis.com.The trade-off is a looser needle where a glue is no longer safe. Offline.js, for example, now needs both
/bundles/and/offline.min.jsinstead of the single (and wrong)/bundles//offline.min.js.BenchmarkFingerprintPassiveandBenchmarkRealisticBodydon't move: with benchstat over 18 runs each and the order alternated, the geomean is +0.23% with identical allocations on an M4 with Go 1.27.1. None of the benchmark pages contains text the changed needles look for, so the work per call is the same.On correctness, I used v0.3.4, which has no prefilter, as the reference. I generated inputs that match each passive pattern (html, scriptSrc, meta, headers and cookies) from its parsed regex, 1,386,383 in total, and ran
Evaluateon v0.3.4, v0.3.5 and this branch. v0.3.5 disagrees with v0.3.4 on 1,429 of them, all from the ten patterns above, and this branch disagrees on none; neither produces a match that v0.3.4 doesn't. I also fuzzed random patterns and inputs, over 400 million strings in total, checking thatEvaluatenever rejects a string the regex matches, and found no counterexample with this change, while the same generator finds them quickly onmain.Running 24,744 synthetic pages through
FingerprintWithInfogives the same picture, apart from three pages that hit a separate issue I left out of this PR. When ascript srccontains a character reference that decodes to an uppercase letter (Jquery), or the long s (ſ) that(?i)matches ass, the scriptSrc index compares lowercase needles against text that isn't folded the way the regex folds it. Happy to send that separately.TestLiteralPrefilterKeepsMatchingTargetscovers each way the glue went wrong: a capture that starts with digits, an alternation branch that ends with a character class, a branch with an optional prefix, branches the parser factors, a repeat between two short literals, and a repeated literal between two literals. All six fail onmainand pass here.go test ./...andgolangci-lint run(v2.14.0) pass locally on darwin/arm64 with Go 1.27.1.Happy to rename things or fold the new cases into
TestLiteralPrefilterif you'd prefer.