Generated by
npx tsx bench/report.tsfrombench/results/*.json. Every number below was measured by the scripts named in each section; nothing is estimated or rounded into a claim. Re-run them to reproduce.
Command: npx tsx bench/accuracy.ts → bench/results/accuracy.json
- Ran at: 2026-10-21T05:07:45.202Z
- Machine: Darwin 27.0.0 arm64, Apple M1 (8 cores), 8 GiB RAM
- Node v24.21.0; git version 2.54.0 (Apple Git-157); git-leak-hunter commit
70f78a1
Corpus: the eight fixture repositories built by test/fixtures/buildRepos.ts (seed 1337). They contain 32 planted fabricated secrets and the false-positive bait listed in ground-truth.json. Each tool ran once per repository with its default rules, in a non-network mode, without live verification.
Matching: A tool finding is a true positive when its (repo, file path, line) equals a planted secret in ground-truth.json; several reports of the same planted secret count once. Each distinct unmatched (repo, path, line) is one false positive. Planted secrets never matched are false negatives.
| Tool | Version | Planted | TP | FP | FN | Precision | Recall | Raw reports |
|---|---|---|---|---|---|---|---|---|
| git-leak-hunter | 0.1.0 (commit 70f78a1) | 32 | 32 | 0 | 0 | 100.0% | 100.0% | 32 |
| gitleaks | 8.30.1 | 32 | 28 | 0 | 4 | 100.0% | 87.5% | 29 |
| trufflehog | trufflehog 3.99.3 | 32 | 14 | 0 | 18 | 100.0% | 43.8% | 15 |
Exact commands (<repo> is each fixture directory):
- git-leak-hunter:
node --import tsx src/cli.ts scan <repo> --format json --out <out> --quiet - gitleaks:
gitleaks git <repo> --report-format json --report-path <out>/gitleaks.json --redact --no-banner --exit-code 0 --log-level error - trufflehog:
trufflehog git file://<repo> --json --no-verification --no-update
| Repository | git-leak-hunter | gitleaks | trufflehog |
|---|---|---|---|
history-only |
3 / 0 / 0 | 3 / 0 / 0 | 2 / 0 / 1 |
head-secrets |
7 / 0 / 0 | 7 / 0 / 0 | 3 / 0 / 4 |
blob-reuse |
4 / 0 / 0 | 4 / 0 / 0 | 4 / 0 / 0 |
merge-conflict |
2 / 0 / 0 | 0 / 0 / 2 | 0 / 0 / 2 |
ci-configs |
12 / 0 / 0 | 10 / 0 / 2 | 1 / 0 / 11 |
false-positives |
0 / 0 / 0 | 0 / 0 / 0 | 0 / 0 / 0 |
hostile |
4 / 0 / 0 | 4 / 0 / 0 | 4 / 0 / 0 |
clean |
0 / 0 / 0 | 0 / 0 / 0 | 0 / 0 / 0 |
git-leak-hunter
- False negatives: none
- False positives: none
gitleaks
- False negatives:
merge-conflict: resolution-token (github-token);merge-conflict: evil-merge-slack (slack-token);ci-configs: docker-arg (dockerfile-literal);ci-configs: docker-env-space (dockerfile-literal) - False positives: none
trufflehog
- False negatives:
history-only: aws-secret (aws-secret-access-key);head-secrets: stripe-test (stripe-test-key);head-secrets: deploy-key (private-key);head-secrets: generic-app (generic-high-entropy);head-secrets: generic-docs (generic-high-entropy);merge-conflict: resolution-token (github-token);merge-conflict: evil-merge-slack (slack-token);ci-configs: gha-workflow-env (ci-literal);ci-configs: gha-job-env (ci-literal);ci-configs: gha-with-input (ci-literal);ci-configs: gitlab-global (ci-literal);ci-configs: gitlab-job (ci-literal);ci-configs: docker-arg (dockerfile-literal);ci-configs: docker-env-continuation (dockerfile-literal);ci-configs: docker-env-space (dockerfile-literal);ci-configs: dotenv (credential-file);ci-configs: id-rsa (private-key);ci-configs: credentials-json (credential-file) - False positives: none
- Home-field advantage. The fixtures were written by the same team, alongside Git Leak Hunter. Its thresholds were tuned against them (see
docs/RULES.md, "Tuning changes"). A perfect score here shows that the rules do what they were designed to do, not real-world accuracy. - Small, synthetic corpus. The fixtures hold a few dozen planted secrets with fabricated values. Real repositories have far more varied formats, much more ordinary code, and a different base rate.
- Location-based matching. A tool that reports a secret on a different line than the one planted, e.g. a multi-line match anchored elsewhere, counts as a miss plus a false positive.
- Default configurations only. No competitor was tuned. Custom rules would change their numbers.
- Category coverage differs by design. Some planted categories (Dockerfile
ARG/ENVliterals, CI variables) are scanned structurally by Git Leak Hunter. Other tools may not target them by default. - Fabricated keys are not structurally valid. Planted private keys are random bytes in PEM armor, not parseable keys. Tools that validate key structure skip them by design, which counts as a miss here. A manual check on 2026-10-21 with TruffleHog 3.99.3 (
trufflehog git file://<tmp-repo> --json --no-verification --no-update) on a throwaway repo committing one key freshly generated withopenssl genrsa 2048(never used anywhere) and one fabricated key: TruffleHog reported the generated key and not the fabricated one. Its private-key misses below therefore reflect the fixtures, not a detection gap. Gitleaks and Git Leak Hunter match on the PEM header alone, so they report both.
Command: npx tsx bench/memoization.ts 5 → bench/results/memoization.{json,csv}
- Ran at: 2026-10-20T17:31:35.032Z
- Machine: Darwin 27.0.0 arm64, Apple M1 (8 cores), 8 GiB RAM
- Node v24.21.0; git version 2.54.0 (Apple Git-157); git-leak-hunter commit
e4f3de8
Repositories: synthetic, generated with git fast-import by test/fixtures/synthRepo.ts. Each has 300 paths and 5 files rewritten per commit; every 10th file content carries a fabricated token (seed 2026).
- high-reuse: 150 distinct file contents reused across 300 paths (pool size 150).
- low-reuse: up to 50,000 distinct contents (almost every write is new) (pool size 50,000).
Each configuration: one unrecorded warm-up, then 5 measured runs alternating cache / no-cache. Time = the engine's measured stats.durationMs (git walk + blob reads + detection; report writing excluded). The scans run in-process.
"Snapshot file versions" is what a scanner that checks out every commit and reads every file would read. It is measured as the sum of git ls-tree -r entries over all commits. "File versions seen" is what Git Leak Hunter reads from the changed-files walk. "Unique blobs scanned" is what detection actually ran on.
| Profile | Commits | Snapshot file versions | File versions seen | Unique blobs scanned | Cache hits | Cache: median / min (ms) | No cache: median / min (ms) | Ratio of medians |
|---|---|---|---|---|---|---|---|---|
| high-reuse | 500 | 131,998 | 2,482 | 150 | 2,332 | 65.5 / 63 | 181.3 / 177.5 | 2.77× |
| high-reuse | 1,000 | 281,998 | 4,964 | 150 | 4,814 | 74.9 / 73.9 | 303.6 / 301.1 | 4.05× |
| high-reuse | 2,000 | 581,998 | 9,939 | 150 | 9,789 | 104.6 / 101.6 | 570.8 / 567.1 | 5.46× |
| high-reuse | 4,000 | 1,181,998 | 19,865 | 150 | 19,715 | 164 / 162.1 | 1084 / 1078.2 | 6.61× |
| low-reuse | 500 | 131,954 | 2,500 | 2,451 | 49 | 229.4 / 226.7 | 230 / 227.6 | 1.00× |
| low-reuse | 1,000 | 281,954 | 5,000 | 4,772 | 228 | 403.7 / 397.6 | 419.1 / 414.4 | 1.04× |
| low-reuse | 2,000 | 581,954 | 10,000 | 9,079 | 921 | 672.6 / 665.6 | 717.4 / 712.2 | 1.07× |
| low-reuse | 4,000 | 1,181,954 | 20,000 | 16,538 | 3,462 | 1131.3 / 1126.4 | 1314.8 / 1303.3 | 1.16× |
Raw per-run samples are in bench/results/memoization.json.
- In the high-reuse profile, the number of blobs detection runs on stays constant as history grows, while file versions seen grows linearly. This is the
O(unique blobs)vsO(commits × files)argument fromdocs/ARCHITECTURE.md, shown in measured counters. - In the low-reuse profile, almost every write is a new blob, so the cache has little to reuse and the two modes take similar time. Memoization pays off in proportion to blob reuse: reverts, copies, branches, generated files. It does nothing for histories where every change is unique.
- Even with
--no-cache, Git Leak Hunter reads only changed files per commit. The gap to a full-snapshot scan is the "Snapshot file versions" column; that column is a count, not a timing. - These are single-machine timings of synthetic repositories. They are not a prediction for any particular real repository.
The measured miss rates of the entropy layer, with and without length normalisation, are in docs/RULES.md ("Calibration"). They were generated by npx tsx bench/entropy-calibration.ts 10000.