Repository navigation
[pull] main from firecrawl:main - #379
Merged
Merged
Conversation
* feat(api): expose fire-pdf typed blocks via parsers[].blocks Adds a `blocks: true` option to the PDF parser object (v2 scrape/parse/ crawl surfaces). When set, the request is pinned to fire-pdf with `include_blocks`, and the document carries a `blocks` array: per-page typed layout blocks with normalized bounding boxes, reading order, markdown char spans, and source/confidence metadata (camelCased from fire-pdf's wire shape). - sync /ocr + async /jobs both request and validate the payload; async falls back to sync like pageMarkdown does - content-cache: new block-capable variants (blocks-v1, ocr-blocks-v1, page-markdown-blocks-v1, ocr-page-markdown-blocks-v1); richer sidecars serve poorer requests with payload stripping; plain requests keep the historical 4-variant probe list - URL index: block-aware requests bypass reads and never write entries, mirroring the pageMarkdown policy - result parsing tolerates fire-pdf's legacy `pages` block-alias shape returned on block-only jobs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(api): validate cached block items before serving a sidecar hit A malformed GCS sidecar (e.g. items: [null]) previously passed isValidBlocks and threw later while mapping the public response; now it's skipped and the result regenerated (local cubic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(api): single wire contract for blocks; explicit legacy-pages union - infer FirePdfPageBlocks from the zod wire schema and reuse that schema as the GCS cache validator, so incomplete cached sidecars are skipped instead of served as invalid public block data - replace pages .catch(undefined) with a union that matches the legacy block-alias shape explicitly — genuine protocol corruption now fails the response parse instead of silently degrading (local cubic) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(api): extract fire-pdf block wire-to-public adapter (local cubic) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(api): address cubic PR review — index-only guard + strict legacy alias - agentIndexOnly requests demanding pageMarkdown/blocks now fail loud instead of being served an index document without the payload - legacy pages alias schema requires the full documented shape (width/height/status), keeping the union discriminating Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(api): agentIndexOnly capability guard throws AgentIndexOnlyError, not a bare 500 The blocks/pageMarkdown-vs-index-only rejection now uses the canonical typed error so it maps to a clean 4xx with the account-confirmation guidance, with a warn log preserving the capability-specific reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(api): rename PDF parser option pageMarkdown -> pages
Names the option after the response field it populates, matching the
blocks convention ({type: "pdf", pages: true} -> document.pages).
pageMarkdown stays accepted as a silent deprecated alias — it was never
documented, typed in any SDK, or in the OpenAPI spec, so only raw-JSON
early adopters could depend on it; pages wins when both are set.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(api): fold pageMarkdown alias into pages at the schema boundary (local cubic)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(api): read legacy pageMarkdown on serialized options; refresh schema tests (local cubic)
Serialized scrape options bypass schema re-parsing (queued jobs, crawl
option snapshots written pre-rename), so the getter defensively reads
the legacy key when the canonical pages is unset. Snips validation
tests updated to the normalized shape with alias-fold coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(api): exercise the canonical pages option in snips happy paths (local cubic)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(sdks): add parsers[].blocks for typed PDF layout blocks Expose the v2 PDF parser blocks option and Document.blocks across the JS, Python, Go, Java, Ruby, PHP, Rust, and .NET SDKs, and bump each package version so merge auto-publishes the new APIs. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * test(rust-sdk): cover PDF parser blocks serialization and hydration Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(java-sdk): type Document.blocks as List<PdfPageBlocks> Replace the raw List<Map<String, Object>> with typed page/block models so Java callers get compile-time access to pageNumber, items, bbox, and confidence, matching Menu/Product and the other SDKs. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
* feat(sdks): add parsers[].pages for per-page PDF markdown Expose the renamed v2 PDF parser pages option (formerly pageMarkdown) and Document.pages across the JS, Python, Go, Java, Ruby, PHP, Rust, and .NET SDKs. Bump each package version so merge auto-publishes. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(python-sdk): fold deprecated pageMarkdown into pages Pydantic dropped page_markdown/pageMarkdown on PDFParser before prepare_scrape_options could rewrite them. Accept both aliases on the model so they serialize as pages. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> * fix(python-sdk): drop unreachable pageMarkdown folds on model dump PDFParser already folds page_markdown/pageMarkdown into pages in a before-validator, so model_dump cannot emit those keys. Keep the alias rewrite only on the raw-dict parser branch. Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )