Optional mineru extra, pymupdf backend, and a Pages/Read the Docs site (issues #2, #3, #4) - #5
Merged
Merged
Conversation
Moves mineru[pipeline] from a hard dependency to the document2md[mineru] extra, and gives BatchConverter/the CLI a backend= / --backend selector (auto, mineru) so the follow-up text-layer backend has a seam to plug into. Resolution and the missing-dependency check now happen in BatchConverter.__enter__, before any document is converted, and the CLI turns that RuntimeError into a clean exit message instead of a traceback. Bumps __version__ to 0.4.0 and adds a document2md-light CI job that installs with no mineru extra. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds document2md/pymupdf_backend.py, reading a born-digital PDF's embedded text layer with pymupdf4llm instead of running mineru's OCR. BatchConverter/CLI backend selector grows a "pymupdf" choice; "auto" now resolves to mineru when its CLI is on PATH, otherwise pymupdf, keeping mineru as the reference backend where installed. Neither backend silently falls back to the other: a list of page images or a PDF without enough of an embedded text layer under "pymupdf" raises RuntimeError naming the mineru extra. pymupdf4llm joins the core dependencies (its wheels are small, unlike mineru[pipeline]). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…cs (issue #4) Adds website/, a Quarto site in LegalIA's house style (litera theme, style.scss copied verbatim, same navbar/footer shape) covering install, CLI, Python and backend guidance for users, published to GitHub Pages by .github/workflows/website.yml (render on PRs, publish to gh-pages on push to main). Splits docs/source/index.rst into index, architecture, backends and development pages plus one API reference page per module under api/, so Read the Docs becomes the developer site: architecture walkthrough, how backend resolution/dispatch works and how to add a backend, repository conventions, and the full private+public API reference. Usage examples move to the Pages site, so Read the Docs doctest keeps only cutter's real example. Also: docs/Makefile, conf.py's dangling templates_path removed and html_title added, test.yml's docs-doctest job renamed to docs and gated by a strict Sphinx HTML build, tests/test_docs.py guarding every document2md/*.py module against zero or duplicate automodule pages, README's Documentation section, CLAUDE.md's Layout/Commands/Publishing updated for both sites, and the repository homepage set to the Pages URL. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements issues #2, #3 and #4 in that order (#3 builds on #2, #4 builds on
#2 and #3), as one run against
mainat9956c0b.#2 — Make mineru an optional extra and add the backend selector
Makes
mineruthedocument2md[mineru]extra instead of a hard dependency,and adds the
BatchConverter(backend=...)/ CLI--backendselector(
auto,mineru,pymupdf), withautoresolving tomineruwhen its CLIis on
PATHand topymupdfotherwise.__version__bumped to0.4.0.Commit: 470de5e
#3 — Add a pymupdf backend for PDFs with an embedded text layer
Adds
document2md/pymupdf_backend.py, reading a born-digital PDF's ownembedded text layer via
pymupdf4llminstead of OCR-ing it. Neither backendsilently falls back to the other: a list of page images, or a PDF without
enough of an embedded text layer, under
pymupdfraisesRuntimeErrornaming the
mineruextra. Imports are lazy inside functions so the docs job(installed with
--no-deps) can still autodoc the module.Commit: e612ad7
#4 — Publish a Quarto user site on Pages and developer docs on Read the Docs
Adds a Quarto user site under
website/(install, CLI, Python, backends;published to
ingeotec.github.io/document2mdviawebsite.ymlon push tomain) and a Sphinx developer site underdocs/(architecture, backends,development, full API reference; published to
document2md.readthedocs.io). The two sites have two audiences and nooverlapping page.
Commit: ace1751
Verification run on this branch before opening this PR
python -m pytest tests— 90 passedpython -m pytest dof2md/tests— 2 passedpython scripts/check_package_versions.py—document2md 0.3.0 (pypi) → 0.4.0 (local): OK;dof2md:OK: tombstone: final release, not maintainedpython -m sphinx -n -W --keep-going -b html docs/source docs/build/html— build succeeded, no warnings
python -m sphinx -b doctest docs/source docs/build/doctest— 4 tests,0 failures
quarto render website(Quarto 1.9.38) — all 5 pages rendered,_site/index.htmlcreatedFollow-ups that are not this PR's job
readthedocs.org — a human has to do that once. Until then
document2md.readthedocs.ioanswers 404.website.yml'srenderjob runs (and passes) on thispull request, but the
publishjob — and therefore thegh-pagesbranchand the live site at
ingeotec.github.io/document2md— only exists afterthe first push to
main, i.e. after this PR merges. Check the site aftermerging.
website/pages/document2md.ipynbtemplate still links to theold
legalia.readthedocs.ioAPI page rather thandocument2md.readthedocs.io; worth a follow-up fix in LegalIA once Readthe Docs is imported.
nota2md'socrextra needs to requiredocument2md[mineru]>=0.4.0(this release) before
nota2md's next release, orpip install nota2md[ocr]breaks for everyone outside the LegalIA repo.Closes #2
Closes #3
Closes #4
🤖 Generated with Claude Code