Repository navigation
[pull] main from firecrawl:main - #410
Merged
Merged
Conversation
* feat(api): OCR JPEG 2000 images Image OCR decides what FirePDF can open by sniffing magic bytes, and JPEG 2000 was missing from that sniff. `.jp2` files, the format digitized book scans and IIIF tile servers commonly serve (often labelled image/jpeg), were rejected as unsupported even though FirePDF opens them natively. Add image/jp2 (and the image/jpx alias) with the .jp2, .jpx and .j2k extensions to the image format tables, sniff both the 12-byte JP2 signature box and raw codestreams (FF 4F FF 51), and name JPEG 2000 in the unsupported-file error and the parser docs. The parse upload allowlist derives from the same table, so those uploads are accepted too. * fix(api): accept .j2c as a JPEG 2000 codestream extension Raw JPEG 2000 codestreams are written with either .j2k or .j2c, and the sniff already opens them, so .j2c uploads and URLs should route to image OCR the same way .j2k ones do. * fix(api): cover the remaining JPEG 2000 labels Add .jpf, the file suffix RFC 3745 recommends for JPX, and the content types servers use for raw codestreams (image/j2k, image/j2c, image/x-j2c), so extension-less uploads with those headers and .jpf URLs route to image OCR like the rest of the family.
…tes (#4502) * fix(api): route mislabeled PDF and Office downloads by their magic bytes A browser handoff whose Content-Type header names no parseable type (for example a PDF served as image/jp2 by a download endpoint) failed with "cannot process: image/jp2" even though the captured bytes were a normal PDF: only application/octet-stream responses were sniffed. specialtyScrapeCheck now checks the leading bytes of every handoff the header did not route. %PDF- goes to the pdf engine and the ZIP local-file header and the OLE2 header go to the document engine, the same handoff the octet-stream branch performed. The raster-image sniff stays first so real images keep reaching OCR, and a header that already declares a ZIP archive keeps the unsupported-file rejection. The ZIP check requires all four signature bytes, so a text body starting with "PK" is no longer mistaken for an archive, and header-less downloads are sniffed the same way. Test-site fixtures: mislabeled-pdf.jp2 (a PDF that static hosts serve as image/jp2) and tiny-image.jp2 (a real JPEG 2000 image). * test(test-site): give the mislabeled PDF fixture a full page of text The legacy pdf-parse fallback in the pdf engine rejects PDFs under about 4 KB when handed a Node Buffer ("bad XRef entry"), so a 700-byte fixture only parsed on the FirePDF and Rust paths. The fixture is now a 5.7 KB page with numbered body lines and still pure ASCII, so every parser path accepts it; the same bytes were verified through fire-engine end to end. * fix(api): drop the URL-derived file flag when a handoff names another parser A .pdf URL that serves a docx (or a .docx URL that serves a PDF) enters the waterfall with the file flag its extension implied. The byte- or header-based handoff then added the other parser's flag without removing the first one, leaving both set. File flags weigh 100 each in the support threshold, so the single-type parsers only just qualified with nothing else set, and any further flag (location, stealthProxy, waitFor) pushed the threshold past both, stranding a file that was already in hand. The AddFeatureError handler now merges flags through applyHandoffFeatureFlags, which replaces a conflicting file flag with the one the handoff names and leaves every other flag alone. Header-based handoffs get the same reconciliation.
Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
…#4506) The engine success check required non-empty markdown for a 2xx result, so an image with nothing to read (a blank scan, a photo, a tiny thumbnail) was deemed unsuccessful and, as the last engine in its waterfall, surfaced as SCRAPE_ALL_ENGINES_FAILED. A result from the image engine with a good status and no page error is now complete even when empty. The mislabeled-files JPEG 2000 snip is renamed to say what it covers (a team without the image OCR flag), and the flagged expectation is added to the image OCR suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )