Skip to content

Latest commit

 

History

History
108 lines (75 loc) · 9.78 KB

BENCHMARK.md

File metadata and controls

108 lines (75 loc) · 9.78 KB

Benchmarks

Generated by npx tsx bench/report.ts from bench/results/*.json. Every number below was measured by the scripts named in each section; nothing is estimated or rounded into a claim. Re-run them to reproduce.

1. Accuracy vs Gitleaks and TruffleHog

Command: npx tsx bench/accuracy.ts → bench/results/accuracy.json

  • Ran at: 2026-10-21T05:07:45.202Z
  • Machine: Darwin 27.0.0 arm64, Apple M1 (8 cores), 8 GiB RAM
  • Node v24.21.0; git version 2.54.0 (Apple Git-157); git-leak-hunter commit 70f78a1

Corpus: the eight fixture repositories built by test/fixtures/buildRepos.ts (seed 1337). They contain 32 planted fabricated secrets and the false-positive bait listed in ground-truth.json. Each tool ran once per repository with its default rules, in a non-network mode, without live verification.

Matching: A tool finding is a true positive when its (repo, file path, line) equals a planted secret in ground-truth.json; several reports of the same planted secret count once. Each distinct unmatched (repo, path, line) is one false positive. Planted secrets never matched are false negatives.

Tool Version Planted TP FP FN Precision Recall Raw reports
git-leak-hunter 0.1.0 (commit 70f78a1) 32 32 0 0 100.0% 100.0% 32
gitleaks 8.30.1 32 28 0 4 100.0% 87.5% 29
trufflehog trufflehog 3.99.3 32 14 0 18 100.0% 43.8% 15

Exact commands (<repo> is each fixture directory):

  • git-leak-hunter: node --import tsx src/cli.ts scan <repo> --format json --out <out> --quiet
  • gitleaks: gitleaks git <repo> --report-format json --report-path <out>/gitleaks.json --redact --no-banner --exit-code 0 --log-level error
  • trufflehog: trufflehog git file://<repo> --json --no-verification --no-update

Per repository (TP / FP / FN)

Repository git-leak-hunter gitleaks trufflehog
history-only 3 / 0 / 0 3 / 0 / 0 2 / 0 / 1
head-secrets 7 / 0 / 0 7 / 0 / 0 3 / 0 / 4
blob-reuse 4 / 0 / 0 4 / 0 / 0 4 / 0 / 0
merge-conflict 2 / 0 / 0 0 / 0 / 2 0 / 0 / 2
ci-configs 12 / 0 / 0 10 / 0 / 2 1 / 0 / 11
false-positives 0 / 0 / 0 0 / 0 / 0 0 / 0 / 0
hostile 4 / 0 / 0 4 / 0 / 0 4 / 0 / 0
clean 0 / 0 / 0 0 / 0 / 0 0 / 0 / 0

Misses and false positives, by tool

git-leak-hunter

  • False negatives: none
  • False positives: none

gitleaks

  • False negatives: merge-conflict: resolution-token (github-token); merge-conflict: evil-merge-slack (slack-token); ci-configs: docker-arg (dockerfile-literal); ci-configs: docker-env-space (dockerfile-literal)
  • False positives: none

trufflehog

  • False negatives: history-only: aws-secret (aws-secret-access-key); head-secrets: stripe-test (stripe-test-key); head-secrets: deploy-key (private-key); head-secrets: generic-app (generic-high-entropy); head-secrets: generic-docs (generic-high-entropy); merge-conflict: resolution-token (github-token); merge-conflict: evil-merge-slack (slack-token); ci-configs: gha-workflow-env (ci-literal); ci-configs: gha-job-env (ci-literal); ci-configs: gha-with-input (ci-literal); ci-configs: gitlab-global (ci-literal); ci-configs: gitlab-job (ci-literal); ci-configs: docker-arg (dockerfile-literal); ci-configs: docker-env-continuation (dockerfile-literal); ci-configs: docker-env-space (dockerfile-literal); ci-configs: dotenv (credential-file); ci-configs: id-rsa (private-key); ci-configs: credentials-json (credential-file)
  • False positives: none

Limitations of this comparison

  • Home-field advantage. The fixtures were written by the same team, alongside Git Leak Hunter. Its thresholds were tuned against them (see docs/RULES.md, "Tuning changes"). A perfect score here shows that the rules do what they were designed to do, not real-world accuracy.
  • Small, synthetic corpus. The fixtures hold a few dozen planted secrets with fabricated values. Real repositories have far more varied formats, much more ordinary code, and a different base rate.
  • Location-based matching. A tool that reports a secret on a different line than the one planted, e.g. a multi-line match anchored elsewhere, counts as a miss plus a false positive.
  • Default configurations only. No competitor was tuned. Custom rules would change their numbers.
  • Category coverage differs by design. Some planted categories (Dockerfile ARG/ENV literals, CI variables) are scanned structurally by Git Leak Hunter. Other tools may not target them by default.
  • Fabricated keys are not structurally valid. Planted private keys are random bytes in PEM armor, not parseable keys. Tools that validate key structure skip them by design, which counts as a miss here. A manual check on 2026-10-21 with TruffleHog 3.99.3 (trufflehog git file://<tmp-repo> --json --no-verification --no-update) on a throwaway repo committing one key freshly generated with openssl genrsa 2048 (never used anywhere) and one fabricated key: TruffleHog reported the generated key and not the fabricated one. Its private-key misses below therefore reflect the fixtures, not a detection gap. Gitleaks and Git Leak Hunter match on the PEM header alone, so they report both.

2. Blob memoization: history scan with cache on vs --no-cache

Command: npx tsx bench/memoization.ts 5 → bench/results/memoization.{json,csv}

  • Ran at: 2026-10-20T17:31:35.032Z
  • Machine: Darwin 27.0.0 arm64, Apple M1 (8 cores), 8 GiB RAM
  • Node v24.21.0; git version 2.54.0 (Apple Git-157); git-leak-hunter commit e4f3de8

Repositories: synthetic, generated with git fast-import by test/fixtures/synthRepo.ts. Each has 300 paths and 5 files rewritten per commit; every 10th file content carries a fabricated token (seed 2026).

  • high-reuse: 150 distinct file contents reused across 300 paths (pool size 150).
  • low-reuse: up to 50,000 distinct contents (almost every write is new) (pool size 50,000).

Each configuration: one unrecorded warm-up, then 5 measured runs alternating cache / no-cache. Time = the engine's measured stats.durationMs (git walk + blob reads + detection; report writing excluded). The scans run in-process.

"Snapshot file versions" is what a scanner that checks out every commit and reads every file would read. It is measured as the sum of git ls-tree -r entries over all commits. "File versions seen" is what Git Leak Hunter reads from the changed-files walk. "Unique blobs scanned" is what detection actually ran on.

Profile Commits Snapshot file versions File versions seen Unique blobs scanned Cache hits Cache: median / min (ms) No cache: median / min (ms) Ratio of medians
high-reuse 500 131,998 2,482 150 2,332 65.5 / 63 181.3 / 177.5 2.77×
high-reuse 1,000 281,998 4,964 150 4,814 74.9 / 73.9 303.6 / 301.1 4.05×
high-reuse 2,000 581,998 9,939 150 9,789 104.6 / 101.6 570.8 / 567.1 5.46×
high-reuse 4,000 1,181,998 19,865 150 19,715 164 / 162.1 1084 / 1078.2 6.61×
low-reuse 500 131,954 2,500 2,451 49 229.4 / 226.7 230 / 227.6 1.00×
low-reuse 1,000 281,954 5,000 4,772 228 403.7 / 397.6 419.1 / 414.4 1.04×
low-reuse 2,000 581,954 10,000 9,079 921 672.6 / 665.6 717.4 / 712.2 1.07×
low-reuse 4,000 1,181,954 20,000 16,538 3,462 1131.3 / 1126.4 1314.8 / 1303.3 1.16×

Raw per-run samples are in bench/results/memoization.json.

Reading the numbers

  • In the high-reuse profile, the number of blobs detection runs on stays constant as history grows, while file versions seen grows linearly. This is the O(unique blobs) vs O(commits × files) argument from docs/ARCHITECTURE.md, shown in measured counters.
  • In the low-reuse profile, almost every write is a new blob, so the cache has little to reuse and the two modes take similar time. Memoization pays off in proportion to blob reuse: reverts, copies, branches, generated files. It does nothing for histories where every change is unique.
  • Even with --no-cache, Git Leak Hunter reads only changed files per commit. The gap to a full-snapshot scan is the "Snapshot file versions" column; that column is a count, not a timing.
  • These are single-machine timings of synthetic repositories. They are not a prediction for any particular real repository.

3. Entropy calibration

The measured miss rates of the entropy layer, with and without length normalisation, are in docs/RULES.md ("Calibration"). They were generated by npx tsx bench/entropy-calibration.ts 10000.