Optional mineru extra, pymupdf backend, and a Pages/Read the Docs site (issues #2, #3, #4)

.github/workflows/test.yml

@@ -30,7 +30,7 @@ jobs:
       - name: Install package and test dependencies
         run: |
           if [ "${{ matrix.package }}" = "document2md" ]; then
-            pip install -e ".[test]"
+            pip install -e ".[mineru,test]"
           else
             pip install -e "dof2md[test]"
           fi
@@ -42,12 +42,29 @@ jobs:
             pytest dof2md/tests
           fi
 
-  # Runs the documentation's worked examples for real, separate from the
-  # pytest matrix above. Read the Docs' own build stays HTML-only. The OCR
-  # examples are marked `# doctest: +SKIP` (entering a BatchConverter starts a
-  # real mineru-api server) and are verified instead by tests/test_batch.py
-  # and tests/test_cli.py — the exception is written down on the page itself.
-  docs-doctest:
+  # A light install with no mineru extra and no apt packages: proves the
+  # suite and the whole import graph need no mineru at all, i.e. that
+  # `pip install document2md` (no extra) is genuinely usable on its own.
+  document2md-light:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install package and test dependencies
+        run: pip install -e ".[test]"
+      - name: Run tests
+        run: pytest tests
+
+  # Sphinx gate for the Read the Docs developer site: a strict HTML build
+  # (any broken cross-reference or other warning fails it) plus the
+  # documentation's few real, executed examples, separate from the pytest
+  # matrix above. Usage examples that would need mineru or pymupdf4llm live
+  # on the Pages site (website/) instead, not here, so there is little left
+  # to mark `# doctest: +SKIP` — the exception, where one remains, is written
+  # down on the page itself.
+  docs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -59,9 +76,12 @@ jobs:
       - name: Install docs and package dependencies
         run: |
           pip install -r docs/requirements.txt
-          # --no-deps: keeps the heavy mineru[pipeline] install out of this
-          # job, which is also why .readthedocs.yaml installs it this way.
+          # --no-deps: keeps the heavy mineru[pipeline] install (and
+          # pymupdf4llm) out of this job, which is also why
+          # .readthedocs.yaml installs it this way.
           pip install --no-deps -e .
+      - name: Build the docs strictly
+        run: python -m sphinx -n -W --keep-going -b html docs/source docs/build/html
       - name: Run the documentation's doctest examples
         run: python -m sphinx -b doctest docs/source docs/build/doctest
 

.github/workflows/website.yml

@@ -0,0 +1,48 @@
+name: Publish website
+
+on:
+  pull_request:
+    paths:
+      - "website/**"
+      - ".github/workflows/website.yml"
+  push:
+    branches: [main]
+    paths:
+      - "website/**"
+      - ".github/workflows/website.yml"
+  workflow_dispatch:
+
+jobs:
+  render:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: quarto-dev/quarto-actions/setup@v2
+        with:
+          version: 1.9.38
+
+      - run: quarto render website
+
+  publish:
+    if: github.event_name != 'pull_request'
+    needs: render
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    concurrency:
+      group: website
+      cancel-in-progress: true
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: quarto-dev/quarto-actions/setup@v2
+        with:
+          version: 1.9.38
+
+      - uses: quarto-dev/quarto-actions/publish@v2
+        with:
+          target: gh-pages
+          path: website
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CLAUDE.md

@@ -6,12 +6,19 @@ code in this repository.
 ## What this is
 
 `document2md` converts a PDF, or an ordered set of scanned page images, into
-Markdown, via OCR and layout analysis. It is a wrapper around
-[mineru](https://github.com/opendatalab/MinerU); its own contribution is:
+Markdown, via one of two backends: [mineru](https://github.com/opendatalab/MinerU)
+(OCR and layout analysis, for scanned pages) or
+[pymupdf4llm](https://github.com/pymupdf/RAG) (reading a born-digital PDF's own
+embedded text layer, no OCR needed — see `document2md/pymupdf_backend.py`).
+Its own contribution is:
 
 - keeping mineru's `mineru-api` server warm across a batch of documents
   (`BatchConverter`), instead of paying its startup and model-loading cost
   once per document;
+- resolving which backend converts a given document (`BatchConverter`'s
+  `backend=`/the CLI's `--backend`; see the seam bullet below) and rejecting,
+  rather than silently OCR-falling-back on, input the resolved backend can't
+  handle;
 - stitching the OCR of several page images of the same document into one
   continuous Markdown document;
 - rewriting the raw HTML tables mineru falls back to (rowspan/colspan) into
@@ -20,6 +27,9 @@ Markdown, via OCR and layout analysis. It is a wrapper around
   one's title in the OCR'd text (`cutter`) — a scanned page usually holds the
   tail of one document and the head of the next.
 
+PyMuPDF and pymupdf4llm are AGPL-3.0 licensed; `document2md` itself stays
+Apache-2.0 (see the README's Install section).
+
 It was extracted from the [LegalIA](https://github.com/INGEOTEC/LegalIA)
 monorepo at commit `e1f258c` (issue
 [#233](https://github.com/INGEOTEC/LegalIA/issues/233)), where its whole
@@ -35,26 +45,41 @@ earlier history is still readable — as `packages/document2md`, and as
   what it does, not after the corpus that first needed it.*
 - **It downloads nothing.** It only ever converts a PDF or images already on
   disk. Getting a document is the caller's problem.
-- **There is no backend seam yet.** `mineru` is an implementation detail, and a
-  cloud OCR/layout service is a plausible second backend — that possibility is
-  the reason for the name, not something implemented here. No `backend=`
-  parameter, no registry, no second converter. It gets its own issue when it
-  happens.
+- **The backend seam has two backends behind it.** `BatchConverter(backend=...)`
+  and the CLI's `--backend` accept `auto` (default), `mineru` and `pymupdf`.
+  `auto` resolves to `mineru` when its CLI is on `PATH` (mineru remains the
+  reference backend where installed — it isn't itself replaced by `pymupdf`),
+  otherwise to `pymupdf`, which reads a PDF's own embedded text layer instead
+  of OCR-ing it. `mineru` is the `document2md[mineru]` extra rather than a
+  hard dependency; `pymupdf4llm` (the `pymupdf` backend's dependency) is a
+  core one, small enough that a bare `pip install document2md` still converts
+  something. Neither backend silently falls back to the other: a list of page
+  images, or a PDF without enough of an embedded text layer, under `pymupdf`
+  raises `RuntimeError` naming the `mineru` extra instead.
 
 ## Layout
 
 ```
 pyproject.toml            document2md itself; the repository *is* the package
 setup.py                  two-line setuptools shim
 document2md/              the package: cli, batch, mineru_server, converter,
-                          tables, cutter
+                          pymupdf_backend, tables, cutter
 tests/                    its pytest suite
 dof2md/                   the old PyPI name's tombstone (see below)
 scripts/check_package_versions.py
-docs/                     Sphinx site, published at document2md.readthedocs.io
-.github/workflows/        test.yml, publish-pypi.yml
+website/                  Quarto user site, published to GitHub Pages at
+                          ingeotec.github.io/document2md — install, CLI,
+                          Python, backends; no API reference
+docs/                     Sphinx developer site, published at
+                          document2md.readthedocs.io — architecture, backends,
+                          development, full API reference; no usage guides
+                          beyond a pointer to the Pages site
+.github/workflows/        test.yml, website.yml, publish-pypi.yml
 ```
 
+The two sites have two audiences and no overlapping page: the Pages site is
+for using the package, Read the Docs for extending it.
+
 Two packages are published from this repository, so the release tag convention
 is `<pkg>-v<version>` (`document2md-v0.3.0`, `dof2md-v0.3.0`) rather than a
 bare `v*`, which would not say which one.
@@ -86,8 +111,9 @@ bare `v*`, which would not say which one.
 pytest tests            # document2md
 pytest dof2md/tests     # the tombstone
 python scripts/check_package_versions.py
+python -m sphinx -n -W --keep-going -b html docs/source docs/build/html
 python -m sphinx -b doctest docs/source docs/build/doctest
-python -m sphinx -b html docs/source docs/build/html
+quarto render website   # the Pages user site; needs Quarto 1.9.38, no Python
 ```
 
 **The two pytest runs are two invocations on purpose, never a bare `pytest`.**
@@ -103,17 +129,28 @@ reason.
 The tests never import `mineru` — they mock the subprocess boundary
 (`document2md.converter.convert_to_markdown` /
 `convert_images_to_markdown`, `document2md.batch.MineruServer`) — so a local
-install with `pip install --no-deps -e .` plus `requests` and `pytest` runs
-the whole suite without the gigabytes of `mineru[pipeline]`. CI installs the
-package fully (`pip install -e ".[test]"`, plus `libgl1`/`libglib2.0-0` for
-mineru's opencv) on every supported Python, which is what proves the declared
-dependency actually installs.
-
-The docs' OCR examples are the one documented exception to "every public
-symbol has a verified example": entering a `BatchConverter` starts a real
-`mineru-api` server, so they are marked `# doctest: +SKIP` and verified
-instead by `tests/test_batch.py` and `tests/test_cli.py`. The exception is
-written on the docs page itself, not silently skipped.
+install with `pip install --no-deps -e .` plus `requests`, `pymupdf4llm` and
+`pytest` runs the whole suite without the gigabytes of `mineru[pipeline]`.
+`tests/test_pymupdf_backend.py` is the one exception: it exercises the real
+`pymupdf`/`pymupdf4llm`, building its own PDFs on the fly with PyMuPDF, since
+that backend is small and core rather than optional. CI's `document2md` job
+installs the package fully (`pip install -e ".[mineru,test]"`, plus
+`libgl1`/`libglib2.0-0` for mineru's opencv), proving the declared `mineru`
+extra actually installs; `document2md-light` installs `-e ".[test]"` with no
+apt step, proving the rest of the suite (including the `pymupdf` backend)
+needs no mineru at all.
+
+Usage examples that would need `mineru` or `pymupdf4llm` (entering a
+`BatchConverter`, running the CLI) live on the Pages site (`website/`) now,
+not on Read the Docs — that site's code blocks are illustrative and not
+executed at all, and say so once per page. Read the Docs itself currently
+has no `# doctest: +SKIP` example: `document2md.cutter`'s real,
+executed example on `architecture.rst` is the only one on the site, and it
+needs neither backend nor any file on disk. If a future page needs one
+(a new backend's own example, say), verify it for real instead in the
+matching test module (following `tests/test_batch.py`, `tests/test_cli.py`
+and `tests/test_pymupdf_backend.py`) and write the exception on the page
+itself, not silently.
 
 ## Publishing
 
@@ -126,6 +163,13 @@ set on this repository. `publish-pypi.yml` refuses a tag that disagrees with
 release whose `ocr` extra requires it** (`document2md>=0.3.0`), or
 `pip install nota2md[ocr]` breaks for everyone outside the LegalIA repository.
 
+The Pages site publishes itself: `website.yml` renders `website/` on every
+pull request that touches it, and publishes to the `gh-pages` branch on every
+push to `main` that does — no human action once the workflow exists. Read
+the Docs is the opposite: the project must be imported once, by a human, at
+readthedocs.org; nothing in this repository can do that, and
+`document2md.readthedocs.io` answers 404 until it happens.
+
 ## Language policy
 
 Everything written into this repository is in English: identifiers, comments,

README.md

@@ -5,10 +5,12 @@
 
 Converts a PDF, or a set of scanned page images, into Markdown — any
 document, such as an edition of Mexico's official gazette (DOF, *Diario
-Oficial de la Federación*) — optionally cropped down to a single note.
-It's a wrapper
-around [mineru](https://github.com/opendatalab/MinerU) for the OCR/layout
-analysis itself; `document2md`'s own contribution is:
+Oficial de la Federación*) — optionally cropped down to a single note. It
+has two backends: [mineru](https://github.com/opendatalab/MinerU) (OCR and
+layout analysis, for scanned pages) and
+[pymupdf4llm](https://github.com/pymupdf/RAG) (reading a born-digital PDF's
+own embedded text layer, no OCR needed); `document2md`'s own contribution
+is:
 
 - Keeping mineru's `mineru-api` server warm across a batch of documents,
   instead of paying its startup (and model-loading) cost once per document.
@@ -25,16 +27,38 @@ monorepo at commit `e1f258c`, where every commit of its earlier history (as
 `packages/document2md`, and as `packages/dof2md` before the rename) can still
 be read.
 
+## Documentation
+
+- **Using the package:** [ingeotec.github.io/document2md](https://ingeotec.github.io/document2md/)
+- **Extending the package:** [document2md.readthedocs.io](https://document2md.readthedocs.io/)
+
 ## Install
 
 ```bash
 pip install document2md
 ```
 
+converts born-digital PDFs (`--backend pymupdf`, via
+[pymupdf4llm](https://github.com/pymupdf/RAG)) out of the box — no OCR, no
+models. Note: PyMuPDF and pymupdf4llm are AGPL-3.0 licensed, unlike the rest
+of `document2md` (Apache-2.0); check that fits your project before
+redistributing. Scanned documents still need mineru's OCR/layout models —
+add the `mineru` backend for that:
+
+```bash
+pip install "document2md[mineru]"
+```
+
+`--backend`/`BatchConverter(backend=...)` name each backend explicitly so
+`auto`'s policy (prefer `mineru` when installed, fall back to `pymupdf`
+otherwise) is visible rather than implicit. `nota2md`'s `ocr` extra (in the
+LegalIA repository) must depend on `document2md[mineru]>=0.4.0`, not a bare
+`document2md>=0.3.0`, or `pip install nota2md[ocr]` stops installing mineru.
+
 For development, from a clone of this repository:
 
 ```bash
-pip install -e ".[test]"
+pip install -e ".[mineru,test]"
 ```
 
 ## Usage
@@ -82,6 +106,14 @@ flags:
   rendered PDFs...) in `<outdir>/<pdf stem>_mineru/` instead of discarding
   it; useful when a conversion looks wrong and mineru's own read of the page
   is the first thing worth inspecting.
+- `--backend {auto,mineru,pymupdf}` (default `auto`) — which backend
+  converts the document. `auto` resolves to `mineru` when it's on `PATH`,
+  otherwise `pymupdf`. `pymupdf` reads a PDF's own embedded text layer
+  instead of running OCR — far faster, at the cost of slightly worse
+  structure — and can't handle scanned page images or a PDF without enough
+  of a text layer; `document2md` exits with a message telling you to
+  `pip install "document2md[mineru]"` instead of a traceback when that
+  happens, or when `mineru` is requested but isn't installed.
 
 ### Python: batch conversion
 
@@ -107,7 +139,12 @@ spanning several scanned pages, and writes the result to `outdir/filename`.
 The same `titulo`/`titulo_siguiente`, `min_confidence`, `keep_pages` and
 `keep_mineru_output` options the CLI exposes are also its keyword
 arguments — see `BatchConverter.__call__`'s docstring for the full
-signature.
+signature. `BatchConverter(backend="auto")` (the default) picks which
+backend does the conversion: `"mineru"` when it's on `PATH`, otherwise
+`"pymupdf"`; the resolved name is available as `convert.backend` once
+entered. `"pymupdf"` raises `RuntimeError` (naming the `mineru` extra)
+rather than silently falling back to it on input it can't handle — a list
+of page images, or a PDF without enough of an embedded text layer.
 
 `nota2md.legal_provisions` accepts an already-`__enter__`'d `BatchConverter`
 as its own `converter` parameter, so a batch of DOF legal provisions can
@@ -116,5 +153,6 @@ share the same warm server too.
 ## Tests
 
 ```bash
-pytest -v
+pytest tests
+pytest dof2md/tests
 ```

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = document2md
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/source/api/batch.rst

@@ -0,0 +1,10 @@
+``document2md.batch``
+======================
+
+:py:class:`~document2md.batch.BatchConverter`, the package's public Python
+entry point, and the backend selector it resolves and dispatches on.
+
+.. automodule:: document2md.batch
+   :members:
+   :private-members:
+   :undoc-members:

docs/source/api/cli.rst

@@ -0,0 +1,9 @@
+``document2md.cli``
+====================
+
+The ``document2md`` console script's argument parser and entry point.
+
+.. automodule:: document2md.cli
+   :members:
+   :private-members:
+   :undoc-members: