Skip to content

RUN.md

Latest commit

 

History

History
282 lines (224 loc) · 7.65 KB

RUN.md

File metadata and controls

282 lines (224 loc) · 7.65 KB
Raw

Running docmetry

This is the complete local workflow. Replace the angle-bracket placeholders with your own paths. Do not place PDFs, generated questions, datasets, or reports in the repository.

1. Install from this repository

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev,pypdf]"

Install a different provider instead of pypdf when needed:

python -m pip install -e ".[docling]"
python -m pip install -e ".[pdfplumber]"
python -m pip install -e ".[pdfminer]"
python -m pip install -e ".[pdf-inspector]"
python -m pip install -e ".[firecrawl]"
python -m pip install -e ".[all-providers]"

Confirm the CLI and provider registry:

docmetry --help
docmetry providers

2. Extract one PDF

Set the folders and filename for your local run:

CORPUS_DIR="<CORPUS_DIR>"
OUTPUT_DIR="<OUTPUT_DIR>"
PDF_NAME="<PDF_NAME>.pdf"
PROVIDER="pypdf"

Extract the PDF and generate its complete bundle:

docmetry extract \
  --input "$CORPUS_DIR/$PDF_NAME" \
  --output "$OUTPUT_DIR" \
  --provider "$PROVIDER" \
  --ocr auto

The output is:

<OUTPUT_DIR>/<document_id>/
  artifact.json
  chunks.jsonl
  canonical.txt
  manifest.json

Available OCR policies are auto, pages, always, and never. Use --ocr-page PAGE one or more times with --ocr pages to select pages.

Examples for other providers:

docmetry extract --input "$CORPUS_DIR/$PDF_NAME" --output "$OUTPUT_DIR" --provider docling --ocr auto
docmetry extract --input "$CORPUS_DIR/$PDF_NAME" --output "$OUTPUT_DIR" --provider pdfplumber
docmetry extract --input "$CORPUS_DIR/$PDF_NAME" --output "$OUTPUT_DIR" --provider pdfminer
docmetry extract --input "$CORPUS_DIR/$PDF_NAME" --output "$OUTPUT_DIR" --provider pdf-inspector
docmetry extract --input "$CORPUS_DIR/$PDF_NAME" --output "$OUTPUT_DIR" --provider firecrawl

Provider selection is strict. If the optional package is missing or the provider fails, the command exits with an error and does not silently use a different provider.

3. Run deterministic evaluations

Ground-truth cases are supplied at runtime. Set the dataset folder separately:

DATASET_DIR="<DATASET_DIR>"
RUNS_DIR="<RUNS_DIR>"

Run one stage:

docmetry run \
  --stage extraction \
  --dataset "$DATASET_DIR" \
  --input "$OUTPUT_DIR" \
  --output "$RUNS_DIR"

Available deterministic stages are:

extraction
ocr
visual-coverage
layout
classification
structured
chunks
provenance
entities
relationships
temporal
contradictions
arithmetic
retrieval
answer

Run all deterministic stages without semantic judging:

docmetry run \
  --all \
  --offline \
  --dataset "$DATASET_DIR" \
  --input "$OUTPUT_DIR" \
  --output "$RUNS_DIR"

Or run every deterministic stage explicitly:

for STAGE in extraction ocr visual-coverage layout classification structured \
  chunks provenance entities relationships temporal contradictions arithmetic \
  retrieval answer; do
  docmetry run \
    --stage "$STAGE" \
    --dataset "$DATASET_DIR" \
    --input "$OUTPUT_DIR" \
    --output "$RUNS_DIR"
done

Each run creates a timestamped directory containing summary.json and report.md. Empty stages are reported as SKIPPED, not as passing scores.

4. Have Codex or Claude Code generate questions

The installed Codex or Claude Code skill generates runtime-only questions and PDF-grounded reference answers. It must inspect both the PDF and canonical.txt, then write <QUESTIONS_DIR>/<DOCUMENT_ID>.jsonl. Do not use a hosted semantic adapter to author this file.

Install one skill using integrations/INSTALL.md, then ask it to generate a balanced set of answerable and unanswerable cases. Each JSONL line contains id, question, answerable, expected_answer, required_evidence, and one-based source_pages.

5. Evaluate generated questions against extracted text

The evaluator answers each generated question using the extracted artifact, then asks the explicitly selected semantic adapter to score that answer against the PDF-grounded reference. It also verifies that the supplied PDF matches the extraction manifest SHA-256.

docmetry evaluate-questions \
  --input "$CORPUS_DIR/$PDF_NAME" \
  --artifact "$OUTPUT_DIR/$DOCUMENT_ID" \
  --questions "$QUESTIONS_DIR/$DOCUMENT_ID.jsonl" \
  --output "$RUNS_DIR/$DOCUMENT_ID-judge.jsonl" \
  --judge-provider anthropic

Use gemini, openai, or anthropic as the judge provider and install its matching optional extra. The judge is optional; deterministic stages do not need an API key.

6. Compare two reports

BASELINE_RUN="<BASELINE_RUN_DIR>"
CURRENT_RUN="<CURRENT_RUN_DIR>"
docmetry compare "$BASELINE_RUN" "$CURRENT_RUN"

7. Benchmark every provider against every PDF

The benchmark script keeps the source corpus outside Git and writes all provider bundles, optional shared questions, deterministic results, and optional semantic results under the ignored runs/ folder. Questions are optional for the local artifact-integrity benchmark.

python scripts/benchmark_providers.py \
  --corpus "<CORPUS_DIR>" \
  --output "runs/provider-benchmark"

Add --questions "<QUESTIONS_DIR>" when files named <PDF_NAME>.questions.jsonl are available and evidence metrics are needed. The command above requires no model API key.

Add --evaluate to run an optional semantic answer judge with the same questions for every provider:

export ANTHROPIC_API_KEY="<ANTHROPIC_API_KEY>"
python scripts/benchmark_providers.py \
  --corpus "<CORPUS_DIR>" \
  --questions "<QUESTIONS_DIR>" \
  --output "runs/provider-benchmark" \
  --evaluate \
  --judge-provider anthropic

The same final semantic evaluation boundary supports Gemini, GPT, and Claude:

export OPENAI_API_KEY="<OPENAI_API_KEY>"
docmetry evaluate-questions \
  --input "$CORPUS_DIR/$PDF_NAME" \
  --artifact "$OUTPUT_DIR/$DOCUMENT_ID" \
  --questions "$QUESTIONS_DIR/$DOCUMENT_ID.jsonl" \
  --output "$RUNS_DIR/$DOCUMENT_ID-openai.jsonl" \
  --judge-provider openai

export ANTHROPIC_API_KEY="<ANTHROPIC_API_KEY>"
docmetry evaluate-questions \
  --input "$CORPUS_DIR/$PDF_NAME" \
  --artifact "$OUTPUT_DIR/$DOCUMENT_ID" \
  --questions "$QUESTIONS_DIR/$DOCUMENT_ID.jsonl" \
  --output "$RUNS_DIR/$DOCUMENT_ID-anthropic.jsonl" \
  --judge-provider anthropic

Install the selected adapter with pip install -e ".[openai]" or pip install -e ".[anthropic]". Hosted judges are optional and are never needed for deterministic local evaluation.

Agent skill installation instructions are in integrations/INSTALL.md.

The deterministic comparison is always written to runs/provider-benchmark/deterministic/results.jsonl; the aggregate metadata is in summary.json. Rerunning one provider preserves the other provider results and reuses completed bundles.

8. Python API

from pathlib import Path

from docmetry import extract, list_providers
from docmetry.io import write_bundle
from docmetry.models import ExtractionOptions

pdf = Path("<CORPUS_DIR>/<PDF_NAME>.pdf")
output = Path("<OUTPUT_DIR>")
options = ExtractionOptions(ocr="auto")
artifact = extract(pdf, provider="pypdf", options=options)
bundle = write_bundle(artifact, output, input_path=pdf, options=options)
print(bundle)
print(list_providers())

9. Contributor validation

python -m pip install -e ".[dev]"
python -m pip install build
python -m pytest -q
ruff check evaluator
ruff format --check evaluator
python -m build

The repository intentionally contains no PDF corpus, generated question file, runtime dataset, or evaluation output. Keep all such paths in ignored folders or outside the repository.