LLMEvalForge

LLMEvalForge is a local-first, dependency-light toolkit for evaluating LLM prompts, providers, and model changes with repeatable regression tests.

Version: 1.0.0

It is designed for developers building AI assistants, RAG systems, support bots, extraction pipelines, classifiers, and other LLM-powered applications.

Why LLMEvalForge?

LLM applications often break in ways normal unit tests do not catch:

• a prompt edit reduces answer quality
• a model upgrade changes output format
• a provider becomes slower
• JSON output stops being valid
• forbidden phrases start appearing
• a previously passing test case regresses
• latency increases even when quality stays the same

LLMEvalForge treats prompt/model changes like software changes: run a dataset, score every case, persist the run, compare it with a baseline, and fail CI when the regression is too large.

Core features

• JSONL evaluation datasets
• Prompt templates with {{variable}} placeholders
• Deterministic mock provider for offline development
• OpenAI-compatible /v1/chat/completions provider
• No SDK dependency required for OpenAI-compatible APIs
• Case-level assertions:
  • exact equality
  • contains
  • not contains
  • regex
  • valid JSON
  • required JSON keys
  • minimum output length
  • maximum output length
• Weighted case scoring
• Case tags
• Parallel evaluation workers
• Timeout handling
• Provider error capture
• SQLite run history
• Baseline vs candidate comparison
• Regression detection
• Pass-rate and average-score metrics
• Latency average / p50 / p95 metrics
• Approximate input/output token estimates
• JSON reports
• Markdown reports
• Standalone HTML reports
• CI-friendly exit codes
• Secret-safe API key handling
• Example project
• Automated unit tests
• GitHub Actions workflow
• Python 3.9–3.13

Architecture

Dataset (.jsonl)
      |
      v
Prompt Template
      |
      v
Provider Adapter
  |           |
 Mock     OpenAI-compatible
  |           |
      v
Generated Output
      |
      v
Assertion Engine
      |
      v
Case Score + Metrics
      |
      v
SQLite Run Store
      |
      +----> JSON Report
      +----> Markdown Report
      +----> HTML Report
      |
      v
Baseline Comparison

Requirements

• Python 3.9+
• No third-party runtime dependency is required

The OpenAI-compatible provider uses Python's standard-library HTTP client.

Installation

From source:

git clone https://github.com/bafnayeygian/LLMEvalForge.git
cd LLMEvalForge
python -m pip install .

Developer install:

python -m pip install -e .

Quick start

Create a starter evaluation project:

llm-eval-forge init demo-eval
cd demo-eval

The generated project contains:

demo-eval/
├── dataset.jsonl
├── prompt.txt
└── README.md

Run the sample with the deterministic mock provider:

llm-eval-forge run \
  --dataset dataset.jsonl \
  --template prompt.txt \
  --provider mock \
  --db runs.db \
  --run-name baseline

List stored runs:

llm-eval-forge runs --db runs.db

Generate reports:

llm-eval-forge report \
  --db runs.db \
  --run 1 \
  --html report.html \
  --markdown report.md \
  --json report.json

Dataset format

Each line in the JSONL file is one evaluation case.

Example:

{"id":"greeting","input":{"question":"Hello"},"mock_response":"Hello! How can I help?","expect":{"contains":["Hello"],"not_contains":["ERROR"]},"tags":["smoke"]}
{"id":"json-output","input":{"question":"Return JSON"},"mock_response":"{\"status\":\"ok\",\"count\":2}","expect":{"json_valid":true,"required_json_keys":["status","count"]},"tags":["json"]}

Required fields

id

A unique string for the case.

input

An object whose values are available to the prompt template.

Optional fields

mock_response

Used by the built-in mock provider.

expect

Assertion configuration.

tags

Free-form tags such as:

["smoke", "support", "json", "critical"]

weight

Optional case weight. Default is 1.0.

Prompt templates

Template:

You are a concise assistant.

User question:
{{question}}

Answer:

Dataset:

{"id":"hello","input":{"question":"Say hello"}}

Rendered prompt:

You are a concise assistant.

User question:
Say hello

Answer:

Missing variables fail the case before the provider is called.

Assertions

Exact output

{
  "expect": {
    "equals": "OK"
  }
}

Contains

All listed values must appear:

{
  "expect": {
    "contains": ["order", "status"]
  }
}

Must not contain

{
  "expect": {
    "not_contains": ["password", "secret"]
  }
}

Regex

A string or list is supported:

{
  "expect": {
    "regex": "^ORDER-[0-9]+$"
  }
}

Valid JSON

{
  "expect": {
    "json_valid": true
  }
}

Required JSON keys

{
  "expect": {
    "json_valid": true,
    "required_json_keys": ["status", "message"]
  }
}

Length checks

{
  "expect": {
    "min_chars": 10,
    "max_chars": 500
  }
}

Scoring

Every configured assertion contributes one assertion result.

Example:

contains "order"       PASS
contains "status"      PASS
not_contains "secret"  PASS
json_valid             FAIL

Score:

3 / 4 = 0.75

A case passes when all configured assertions pass.

If a case has no assertions, a successful provider response receives score 1.0.

Mock provider

The mock provider makes evaluation development fully offline.

It returns mock_response from the dataset.

Example:

llm-eval-forge run \
  --dataset examples/dataset.jsonl \
  --template examples/prompt.txt \
  --provider mock

This is useful for:

• testing datasets
• testing assertion logic
• testing CI
• developing reports
• reproducing expected failures

OpenAI-compatible provider

LLMEvalForge supports APIs implementing:

POST /v1/chat/completions

Example:

export LLM_API_KEY="your-key"

llm-eval-forge run \
  --dataset dataset.jsonl \
  --template prompt.txt \
  --provider openai-compatible \
  --base-url https://api.example.com \
  --api-key-env LLM_API_KEY \
  --model example-model \
  --temperature 0 \
  --timeout 60 \
  --db runs.db \
  --run-name candidate

The API key is read from the environment and is never written to the run database.

Custom API base URL

Both forms work:

https://api.example.com
https://api.example.com/v1

LLMEvalForge normalizes the endpoint to:

.../v1/chat/completions

Parallel workers

llm-eval-forge run \
  --dataset dataset.jsonl \
  --template prompt.txt \
  --provider openai-compatible \
  --model model-name \
  --workers 4

Default:

workers = 1

Use concurrency carefully because providers can enforce rate limits.

SQLite run history

By default:

llm_eval_forge.db

Tables:

runs
case_results

The run store includes:

• provider
• model
• run name
• start time
• template SHA-256
• dataset SHA-256
• aggregate score
• pass rate
• latency metrics
• case outputs
• assertion results
• errors

It does not store the API key.

Run listing

llm-eval-forge runs --db runs.db

Example:

ID  NAME        PROVIDER             MODEL          SCORE  PASS
1   baseline    mock                 mock           1.000  100.0%
2   candidate   openai-compatible    model-v2       0.920   90.0%

Compare two runs

llm-eval-forge compare \
  --db runs.db \
  --baseline 1 \
  --candidate 2

Output includes:

• average score delta
• pass-rate delta
• latency delta
• improved cases
• regressed cases
• missing cases

Example:

Baseline score:  0.940
Candidate score: 0.910
Delta:           -0.030

Regressed cases: 3
Improved cases:  1

CI regression gate

Allow at most a 2% average score regression:

llm-eval-forge compare \
  --db runs.db \
  --baseline 1 \
  --candidate 2 \
  --max-regression 0.02

The command exits with 1 if the regression exceeds the threshold.

You can also fail when any individual case regresses:

llm-eval-forge compare \
  --db runs.db \
  --baseline 1 \
  --candidate 2 \
  --fail-on-case-regression

Reports

JSON

llm-eval-forge report --db runs.db --run 1 --json report.json

Markdown

llm-eval-forge report --db runs.db --run 1 --markdown report.md

HTML

llm-eval-forge report --db runs.db --run 1 --html report.html

The HTML report is standalone and does not load external assets.

Metrics

Per run:

• total cases
• passed cases
• failed cases
• pass rate
• average score
• weighted average score
• average latency
• p50 latency
• p95 latency
• approximate prompt tokens
• approximate output tokens
• provider failures

Token counts are estimates based on text length and are not provider billing values.

Tag filtering

Run only selected cases:

llm-eval-forge run \
  --dataset dataset.jsonl \
  --template prompt.txt \
  --provider mock \
  --tag smoke

Multiple tags are OR-matched:

llm-eval-forge run ... --tag smoke --tag critical

Fail thresholds during a run

Fail when pass rate is lower than 95%:

llm-eval-forge run ... --min-pass-rate 0.95

Fail when average score is lower than 0.90:

llm-eval-forge run ... --min-score 0.90

Useful in CI:

- name: Run LLM regression suite
  run: |
    llm-eval-forge run \
      --dataset tests/evals.jsonl \
      --template prompts/support.txt \
      --provider mock \
      --min-pass-rate 1.0

GitHub Actions

The repository includes a test workflow for Python:

3.9
3.10
3.11
3.12
3.13

It runs:

python -m unittest discover -s tests -v
python -m compileall -q src

Security

LLMEvalForge is designed to avoid common credential mistakes.

The OpenAI-compatible provider:

• reads API keys from environment variables
• does not print API keys
• does not persist API keys
• does not include API keys in reports

Evaluation prompts and model outputs are stored in SQLite by default.

Do not evaluate sensitive production data unless storing those prompts and outputs is acceptable.

Privacy

The mock provider is fully local.

The OpenAI-compatible provider sends rendered prompts to the configured API endpoint.

LLMEvalForge itself does not send telemetry.

Exit codes

run

• 0 evaluation completed and thresholds passed
• 1 quality threshold failed
• 2 invalid configuration or execution error

compare

• 0 comparison passed configured regression gate
• 1 regression gate failed
• 2 invalid run IDs or local error

report

• 0 report written successfully
• 2 invalid run ID or local error

Python API

from llm_eval_forge.dataset import load_dataset
from llm_eval_forge.template import PromptTemplate
from llm_eval_forge.providers.mock import MockProvider
from llm_eval_forge.runner import EvaluationRunner

cases = load_dataset("dataset.jsonl")
template = PromptTemplate.from_file("prompt.txt")
provider = MockProvider()

runner = EvaluationRunner(
    provider=provider,
    template=template,
)

result = runner.run(cases)

print(result.metrics.average_score)
print(result.metrics.pass_rate)

Project structure

LLMEvalForge/
├── src/
│   └── llm_eval_forge/
│       ├── assertions.py
│       ├── cli.py
│       ├── compare.py
│       ├── dataset.py
│       ├── hashing.py
│       ├── models.py
│       ├── reports.py
│       ├── runner.py
│       ├── store.py
│       ├── template.py
│       └── providers/
│           ├── base.py
│           ├── mock.py
│           └── openai_compatible.py
├── examples/
│   ├── dataset.jsonl
│   └── prompt.txt
├── tests/
├── .github/workflows/tests.yml
├── CONTRIBUTING.md
├── SECURITY.md
├── CHANGELOG.md
├── LICENSE
└── pyproject.toml

Roadmap

Potential future features:

• embeddings-based semantic assertions
• pairwise LLM-as-a-judge
• multi-turn conversation datasets
• RAG retrieval evaluation
• tool-call assertions
• streaming provider support
• OpenAI Responses API adapter
• Anthropic adapter
• Ollama adapter
• vLLM adapter
• cost accounting
• experiment dashboards
• dataset sharding
• distributed evaluation
• GitHub PR annotations
• JUnit XML output
• prompt version registry
• redacted run storage
• human review queues

Contributing

See CONTRIBUTING.md.

License

MIT License. See LICENSE.