AI Code Detector

A probabilistic forensics tool for detecting AI-generated code in GitHub repositories. Analyzes stylometric, structural, and git history patterns to estimate the likelihood that code was generated by an AI assistant.

Features

Phase 1 - Heuristic Detection (Complete)

Multi-language support: Python, JavaScript, TypeScript, Go, Rust, C/C++, Java
Stylometric analysis: Comment patterns, naming conventions, formatting consistency
Structural analysis: Code complexity, error handling patterns, dead code detection
Git history analysis: Commit patterns, author diversity, file creation bursts
Detailed reports: JSON and Markdown outputs with file-level breakdowns
CLI interface

Phase 2 - ML Enhancement (Complete)

Code embeddings: MLX-based embeddings for semantic analysis (with hash fallback)
Learned classifier: Trainable ML classifier using embeddings + features
Ensemble detection: Combines heuristics with ML for robust predictions
Flexible backends: Supports MLX (Apple Silicon) and fallback modes

Phase 3 - Natural Language Explanations (Complete)

AI-powered explanations: Qwen-based natural language explanations
Template fallback: Works without LLM using intelligent templates
Per-file reasoning: Explains why specific files were flagged
Enhanced reports: JSON and Markdown with integrated explanations

Installation

git clone https://github.com/BenjaminSRussell/Ai_code_detector.git
cd Ai_code_detector
pip install -r requirements.txt
pip install -e .
# One console script: aicd (or: python -m ai_code_detector ...)

Quick Start

# Repository report (Phase 1 heuristics only)
aicd scan ./repo --mode basic

# Enhanced: ML classifier + explanations (the default mode)
aicd scan ./repo --mode enhanced
aicd scan https://github.com/user/repo -f json -o reports/
aicd scan ./repo --embedder mlx          # Apple Silicon embeddings
aicd scan ./repo --no-ml                 # or --no-explanations

# Agent-readable findings (AI_SCAN_FINDINGS.md + ai_scan_findings.json)
aicd agent-scan ./repo

# Explain one file
aicd explain src/foo.py --json

# Train the Phase 2 classifier from labeled JSONL ({"path": ..., "label": 0|1})
aicd train labels.jsonl -o model.json
aicd explain src/foo.py --model model.json

aicd scan exits with 0 for low, 1 for moderate, and 2 for high AI probability.

Deprecated: ai-code-detector, ai-code-detector-enhanced, ai-code-detector-agent-scan, and python -m ai_code_detector <repo> still work for one release but print a notice. Use aicd scan --mode basic, aicd scan, and aicd agent-scan instead.

How It Works

The detector analyzes code across three dimensions:

1. Stylometric Features

AI-generated code often exhibits:

Boilerplate comments: Generic phrases like "This function does...", "Returns:", "Args:"
Tutorial-style language: "First, we...", "Then, we...", "Let's..."
Generic naming: Overuse of names like result, data, temp, helper
Perfect formatting: Unnaturally consistent indentation and spacing
High duplication: Similar code patterns repeated across files

2. Structural Features

AI code patterns include:

Over-explained simple functions: Low complexity with verbose docstrings
Generic error handling: Catch-all except Exception blocks
Print-based debugging: print(f"An error occurred: {e}")
Dead code: Unused functions, unreferenced imports
Missing cleanup: Resource allocation without proper cleanup

3. Git History Features

AI-generated repositories often show:

Commit bursts: Large amounts of code in single commits
Low author diversity: Single author, all code at once
Generic commit messages: "initial commit", "update", "fix"
File creation bursts: Most files created simultaneously
Dense timeline: Complete repo created in days/hours

Parsing backends (tree-sitter)

Structural metrics such as cyclomatic complexity, docstring ratio, try/except ratio, unused imports and unreachable code need a syntax tree. The backend is set by analysis.ast_backend in the config (configs/default.yaml):

`ast_backend`	Python	JS / TS / TSX / Go / Rust	Everything else
`auto` (default)	builtin `ast`	tree-sitter if installed, else regex	regex
`tree_sitter`	tree-sitter	tree-sitter if installed, else regex	regex
`python`	builtin `ast`	regex	regex

Install the grammars with pip install "ai-code-detector[treesitter]".

What changes for a TypeScript file (from tests/fixtures/tree_sitter/service.ts, asserted in tests/test_tree_sitter.py):

Metric	Regex fallback	Tree-sitter
avg cyclomatic complexity	0	2.8 (if/for/catch/case/ternary/`&&`/`\|\|`/`??`)
complexity-to-docstring ratio	0	> 0 (only JSDoc `/** */` counts as documentation)
try/except ratio	0	0.2
unused import ratio	0	0.2 (uses local names, so `{ get as httpGet }` counts as `httpGet`)
unreachable code	0	0.2 (a statement after `return`/`throw`/`break`/`continue` in the same block)
generic exceptions, print-on-error, missing cleanup	text patterns	same text patterns

Go uses // doc comments directly above a declaration, and methods count as functions. Rust uses /// doc comments, and impl blocks count as classes. Running Python through tree-sitter gives the same functions, classes and complexities as the builtin parser on examples/. The difference is imports: tree-sitter records alias names.

Without the grammars, tests marked treesitter are skipped with a reason. CI runs them in the test-treesitter job.

Configuration

Customize detection via YAML config:

ingestion:
  supported_extensions:
    - .py
    - .js
  excluded_dirs:
    - node_modules
    - dist
  max_file_size_mb: 1

scoring:
  weights:
    stylometry: 0.4
    structural: 0.4
    history: 0.2

Suppressions (`.aicdignore`)

Vendored, generated, or third-party code drowns out real signal. List it in a .aicdignore file at the repo root. Every scan picks it up automatically: aicd scan, aicd agent-scan, and the legacy commands.

# pattern                 # rationale (shown in reports)
third_party/              # vendored deps, reviewed upstream
**/*_pb2.py               # protoc output
build/generated/*.rs      # codegen
!build/generated/keep.rs  # re-include a hand-written file (last match wins)

Syntax is gitignore-like. A pattern containing / is anchored at the repo root; otherwise it matches a file or directory name at any depth. A trailing / matches directories only. * and ? stay within one path segment, ** crosses segments, and ! re-includes.
Suppressed files are not analyzed. They don't count toward the repository score, the scan exit code (the fail gate), top-suspicious files, or agent-scan findings.

Every JSON report carries the omission, so nothing disappears silently:

"suppressed": {
  "count": 2,
  "source": "/repo/.aicdignore",
  "rules": [{"pattern": "third_party/", "rationale": "vendored deps", "count": 1}],
  "paths": ["api_pb2.py", "third_party/lib/vendored.py"],
  "paths_truncated": false
}

Markdown reports show a "Suppressed" line when the count is non-zero.

--suppressions FILE uses a different file, for example a stricter CI-only list. --no-suppressions scans everything.
Repository-level git-history signals still cover the whole repo.

CI example (fails only on high AI probability; suppressed paths never trip the gate):

- run: pip install git+https://github.com/BenjaminSRussell/Ai_code_detector.git
- name: AI code gate
  run: |
    set +e
    aicd scan . --mode basic -f json -o aicd-report -q
    code=$?
    echo "suppressed: $(jq '.suppressed.count' aicd-report/ai_detection_report_enhanced.json)"
    [ "$code" -lt 2 ]   # 0 low, 1 moderate -> pass; 2 high -> fail

Project Structure

ai-code-detector/
├── src/ai_code_detector/    # the installable package
│   ├── ingest/              # Repository loading and file filtering
│   ├── analysis/            # Feature extraction
│   ├── model/               # Scoring, embeddings, classifier, explainer
│   ├── report/              # Report generation
│   ├── configs/default.yaml # Default config (shipped in the wheel)
│   ├── detector.py          # Phase 1 detector
│   ├── detector_enhanced.py # Phase 1+2+3 detector
│   ├── agent_scan.py        # Agent-prep scanner
│   ├── aicd.py              # aicd: scan / agent-scan / explain / train
│   ├── cli.py               # (deprecated) ai-code-detector
│   ├── cli_enhanced.py      # aicd scan (was ai-code-detector-enhanced)
│   └── cli_agent_scan.py    # aicd agent-scan (was ai-code-detector-agent-scan)
├── tests/                   # Test suite (pytest)
└── STATUS.txt           # Current project status

Development Status

See STATUS.txt for detailed information about what is implemented, tested, and ready to use.

Scan history (SQLite)

aicd scan ./repo --store ./scans.db      # bare `--store` uses ~/.cache/ai_code_detector/scans.db
aicd history --store ./scans.db --repo ./repo          # runs, newest first
aicd history --store ./scans.db --scan 3               # one run's files ([H]euristic/[M]L/[E]xplanation flags)
aicd history --store ./scans.db --repo ./repo --file src/app.py   # one path's probability across runs

Each run appends a row to scans. Earlier runs are never updated or deleted, so regressions can be compared over time.

Table	Row per	Columns
`scans`	run	repo path and source, start/finish time (UTC), mode, ML/explanation flags, detector version, config fingerprint, repo scores, files, lines, languages, suppressed count
`files`	analyzed path in a run	probability, stylometry/structural scores, `phase_heuristic`/`phase_ml`/`phase_explanation`, `parse_failed`, `content_sha256`
`features`	file	raw stylometry and structural feature values, feature contributions, indicators (all JSON)
`verdicts`	repo verdict, plus each file at or above the file threshold (`scoring.thresholds.file_threshold`, default 0.6)	verdict, with the explanation when there is one

Schema and migrations: every table is created with CREATE TABLE IF NOT EXISTS, and the schema version is kept in PRAGMA user_version (currently 1).

Opening a database runs any missing migrations in order (_MIGRATIONS in src/ai_code_detector/store.py).
A database written by a newer aicd is refused rather than downgraded.

There's no separate migration script: open the database with aicd, or apply _SCHEMA_V1 yourself.

-- latest probability of every file in the newest scan of a repo
SELECT f.path, f.ai_probability FROM files f
WHERE f.scan_id = (SELECT max(id) FROM scans WHERE repo_path = '/abs/path/to/repo')
ORDER BY f.ai_probability DESC;

HTML findings viewer

aicd scan ./examples --html ./out/report.html          # works with --mode basic or enhanced

This writes one self-contained file: inline CSS and JS, with the scan data embedded. It makes no network requests, so you can open it straight from disk or attach it to a ticket. It has three views:

Repo summary: probability, verdict, confidence, stylometry/structural/history scores, files and lines, languages, and the count of files suppressed by .aicdignore.
File table: sortable columns, a path filter, and a threshold slider. Set the slider's starting value with --html-threshold 0.6, or add ?threshold=0.6 or #threshold=0.6 to the URL.
File detail: click a row, or link to #file=path/to/file.py. It shows the top 5 features by their share of the file's AI probability, along with component scores, the indicators over threshold, and the natural-language explanation in enhanced mode.

Feature shares come from FileScore.feature_contributions, an additive breakdown of the heuristic score, so the shares sum to the probability. In enhanced mode the heuristic terms are scaled by 0.4 and the classifier appears as a single ml_classifier term (0.6 × p). JSON reports also carry top_features for each file. Every number in the viewer comes from the scan, and an empty scan shows an empty state.

Incremental scans

aicd scan ./repo --incremental                    # cache in ~/.cache/ai_code_detector/scans.db
aicd scan ./repo --incremental --store scans.db   # cache (and history) in a DB you choose
aicd scan ./repo --incremental --force-full       # recompute everything and refresh the cache

Each eligible file is hashed (SHA-256 of its content). If that content was already analyzed under the same analysis key, the stored result is reused. The key covers:

the detector version
the config fingerprint (weights, thresholds, feature settings)
the ML and explanation phases, and the embedder and explainer backends
the model file's hash

Changing any of these is a cache miss, and the old entries stay valid for the old key. Renamed or copied files are cache hits.

Every incremental run logs Incremental: N recomputed, M from cache to stderr. The JSON report's cache block lists the counts and recomputed_paths. The cache lives in the file_cache table (store schema v2), next to the scan history.

SARIF export (GitHub code scanning)

aicd scan . -f sarif -o reports/            # reports/ai_detection_report.sarif
aicd scan . -f all -o reports/              # json + markdown + sarif
aicd agent-scan . --sarif -o reports/       # reports/ai_scan_findings.sarif (+ the usual md/json)

Schema version: SARIF 2.1.0 (OASIS), with "$schema": "https://json.schemastore.org/sarif-2.1.0.json". This is the only version GitHub code scanning accepts. Tests validate the output against a vendored copy of the schema (tests/fixtures/sarif-schema-2.1.0.json).
Verdict → level (rule aicd/file/ai-generated, one result per file):

File AI probability SARIF level

≥ 0.80 (very likely AI) error

≥ 0.60 (likely AI) warning

≥ --sarif-min-probability (default 0.40) note

lower no result
Agent-scan findings use the rules aicd/agent/<type> (satd, duplication, performance_hotspot, ...) and map severity high → error, warning → warning, info → note.
- Findings that have a line number carry a region.
- Repository-level findings (for example commit-history attribution) have no file location, so they appear under runs[0].properties.repositoryFindings.
URIs are repo-relative under %SRCROOT%.
partialFingerprints are content-independent (rule + path), so alerts track files across commits instead of reopening.

Upload from a workflow:

permissions:
  contents: read
  security-events: write
steps:
  - uses: actions/checkout@v4
  - run: pip install git+https://github.com/BenjaminSRussell/Ai_code_detector.git
  - run: aicd scan . --mode basic -q -f sarif -o aicd || true   # exit code gates separately
  - uses: github/codeql-action/upload-sarif@v3
    with:
      sarif_file: aicd/ai_detection_report.sarif
      category: aicd

This repo's CI builds both SARIF files on every push and uploads them as the aicd-sarif artifact.

PR gate (GitHub Action + pre-commit)

aicd gate FILES... scores only the files you pass it, for example a PR's diff or staged files. It exits 1 if any file's AI probability is above the threshold.

It honors .aicdignore, skips deleted and non-source files, and lists every skipped file.
It writes --json and --sarif reports.
It appends a Markdown table to $GITHUB_STEP_SUMMARY.

git diff --name-only origin/main...HEAD | aicd gate --paths-from - --threshold 0.7

Default threshold: 0.7. Override it with --threshold, the AICD_THRESHOLD env var, or (in CI) the repository variable AICD_THRESHOLD under Settings → Secrets and variables → Actions → Variables.
Mode: basic by default (heuristics only: stable and fast). --mode enhanced adds the ML classifier.

This repo: .github/workflows/ai-scan.yml runs on pull_request when *.py|*.ts|*.tsx|*.js|*.jsx|*.go|*.rs change.

It diffs origin/<base>...HEAD, gates only those files, and lists flagged paths with scores in the job summary.
It uploads report.json, report.sarif, and changed.txt as the aicd-gate artifact.

Other repos: use the composite action:

on: pull_request
jobs:
  aicd:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - uses: BenjaminSRussell/Ai_code_detector@main
        env:
          AICD_THRESHOLD: ${{ vars.AICD_THRESHOLD }}   # unset → 0.7
        with:
          mode: basic            # inputs: threshold, mode, base-ref, extensions, report-dir, upload-artifact

pre-commit (optional): .pre-commit-config.yaml:

- repo: https://github.com/BenjaminSRussell/Ai_code_detector
  rev: main
  hooks:
    - id: aicd-gate
      args: [--threshold, "0.7"]

With --quiet (the hook default) it stays silent when the commit passes and prints the table when it fails.

Fixture PR: tests/fixtures/gate_pr/ is a ready-made "PR":

a generated-looking file that gets flagged at 0.4
a hand-written file that passes
a vendored file suppressed by .aicdignore
a README that is skipped

tests/test_gate.py commits it onto a base branch in a temp git repo, takes the same git diff the Action uses, and asserts the outcome. To reproduce by hand:

aicd gate --root tests/fixtures/gate_pr generated_helper.py hand_written.py third_party/vendored.py README.md -t 0.4

Evaluation (labeled set, ROC / calibration)

python scripts/eval_harness.py --manifest datasets/manifest.yaml --train   # same as: aicd eval --train

The harness writes these files to reports/eval/ (gitignored):

File	Contents
`metrics.json`	ROC AUC, Brier score, ECE, confusion matrix at `--threshold` (default 0.5), the full ROC curve, and the calibration table
`predictions.csv`	Per-file scores
`roc_<scorer>.csv`	ROC curve points, ready to plot
`calibration_<scorer>.csv`	Calibration table, ready to plot
`summary.md`	A Markdown summary

Scorers:

basic is the heuristic score used by aicd scan --mode basic and aicd gate. It isn't trained, so it is evaluated on every file.
trained (with --train) fits the Phase 2 classifier on the manifest's train split, using the same features as aicd train, then scores the held-out test split.
model (with --model model.json) scores an existing model on the test split.

--min-auc X exits 1 if any scorer drops below X. CI runs the harness on the fixture set with --min-auc 0.5, appends summary.md to the job summary, and uploads reports/eval/ as the aicd-eval artifact.

Manifest (datasets/manifest.yaml): entries are either a path or a glob, resolved relative to the manifest.

label: human or ai.
split: train or test. If omitted, the split is assigned deterministically from a hash of the path, using test_fraction.
source: free text recording provenance.

To train a classifier on the same split you evaluate, use aicd eval --train and keep reports/eval/model.json. You can then pass it to aicd explain --model or aicd eval --model. aicd train takes a JSONL file of {"path", "label"} rows instead.

Caveats (read before quoting numbers):

The bundled manifest is a 12-file synthetic smoke fixture. Its "ai" files were written to imitate typical LLM style, and its "human" files are terse hand-written utilities. Its numbers check that the pipeline works. They are not an accuracy claim.
On the fixture, basic ranks every ai file above every human file (AUC 1.0), but it is under-confident: ai files score 0.41–0.52 and human files 0.11–0.18, so at a 0.5 threshold it catches only 1 of 6 (ECE ≈ 0.35). Calibrate thresholds on your own data.
For a real evaluation, point glob entries at checked-out repos with known provenance (pin a commit SHA in source), with at least hundreds of files per class. Report AUC with that dataset's description.
Scores are probabilistic signals, not proof of authorship (see Limitations).

Limitations

This tool provides probabilistic analysis, not definitive proof:

False positives: Well-structured human code may score high
False negatives: Edited AI code may score low
Easy to fool: Simple refactoring reduces detection
Language bias: Works best on Python; limited for other languages

Do NOT use as sole evidence for policy/disciplinary decisions.

Best used as one signal among many:

Code review
Author interviews
Development timeline analysis
Project context

License

MIT License

Contact

Issues: https://github.com/BenjaminSRussell/Ai_code_detector/issues

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
datasets		datasets
docs/superpowers		docs/superpowers
examples		examples
scripts		scripts
src/ai_code_detector		src/ai_code_detector
tests		tests
.aicdignore		.aicdignore
.gitignore		.gitignore
.pre-commit-hooks.yaml		.pre-commit-hooks.yaml
README.md		README.md
STATUS.txt		STATUS.txt
action.yml		action.yml
pytest.ini		pytest.ini
requirements.txt		requirements.txt
setup.py		setup.py
test_comprehensive.py		test_comprehensive.py
test_enhanced.py		test_enhanced.py
test_quick.py		test_quick.py

Navigation Menu

Repository navigation

About

Repository files navigation

AI Code Detector

Features

Phase 1 - Heuristic Detection (Complete)

Phase 2 - ML Enhancement (Complete)

Phase 3 - Natural Language Explanations (Complete)

Installation

Quick Start

How It Works

1. Stylometric Features

2. Structural Features

3. Git History Features

Parsing backends (tree-sitter)

Configuration

Suppressions (`.aicdignore`)

Project Structure

Development Status

Scan history (SQLite)

HTML findings viewer

Incremental scans

SARIF export (GitHub code scanning)

PR gate (GitHub Action + pre-commit)

Evaluation (labeled set, ROC / calibration)

Limitations

License

Contact

About

Releases

Packages

Contributors

Languages

File AI probability	SARIF level
≥ 0.80 (very likely AI)	`error`
≥ 0.60 (likely AI)	`warning`
≥ `--sarif-min-probability` (default 0.40)	`note`
lower	no result

Latest commit

History

Folders and files

Repository files navigation

AI Code Detector

Features

Phase 1 - Heuristic Detection (Complete)

Phase 2 - ML Enhancement (Complete)

Phase 3 - Natural Language Explanations (Complete)

Installation

Quick Start

How It Works

1. Stylometric Features

2. Structural Features

3. Git History Features

Parsing backends (tree-sitter)

Configuration

Suppressions (.aicdignore)

Project Structure

Development Status

Scan history (SQLite)

HTML findings viewer

Incremental scans

SARIF export (GitHub code scanning)

PR gate (GitHub Action + pre-commit)

Evaluation (labeled set, ROC / calibration)

Limitations

License

Contact

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Suppressions (`.aicdignore`)