Repository navigation
Transcribe scanned PDFs with readPdf(fileId, { ocr: true }) on Google Drive bindings - #1079
Maximo-Guk wants to merge 4 commits into
Conversation
convertToMarkdown() built its request body, a JSON document holding the file as one base64 string encoded chunk by chunk, in a private helper tied to the conversion's own JSON. The helper now takes the JSON around the string, so another streamed Workers AI request can use it. No behaviour change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
transcribeImage() posts what Ai.run() would, through the binding's fetch, so the image is base64-encoded into the request as it arrives instead of being held as one string, and reads the model's answer as server-sent events only as the text is asked for. It mirrors convertToMarkdown(): the same failure shape, logged by status and error code only, and the same release of both streams on failure or cancel. The model is @cf/google/gemma-4-26b-a4b-it at temperature 0 with thinking off. Measured on scanned pages, thinking tripled a page's tokens and time and held the text back until it was over, without reading a poor scan any better. Nothing calls it yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A scanner writes each page of a PDF as one image. PdfImageScan reads a PDF fed to it in chunks cut anywhere and reports where each page-sized image's data sits, how long it is and how it is encoded, holding only the object it is reading: stream data is counted past, never kept. It walks objects from the front, since the cross-reference table is at the end, takes lengths given directly, by a reference before or after the stream, or not at all, passes over masks and small images, and notes an encrypted file. A JPEG page is a JPEG file as it is stored. A losslessly compressed grey or RGB page is deflated rows of samples, which is what a PNG's image data is, so pngFromDeflated() wraps it as a PNG while it streams: the PDF's own bytes inside PNG's framing when the rows carry PNG filter bytes, re-deflated with one added per row when they do not. No pixel is decoded. Nothing calls either yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A scanned or image-only PDF has no text layer, so readPdf() returned hasText: false and nothing else. With ocr set, the PDF's page images are located as it streams into the conversion, and when the conversion finds no text the pages are transcribed by the vision model instead, under the same "### Page N" headings, with transcribed: true. Nothing is held and nothing is done early: a page's image is read from Drive by range and transcribed only as the text is read, one page at a time, and cancelling the text stops it. Each page read is a content read of its own, checked against the binding's scope, re-proved on a folder binding, reported as drive/content/<fileId>, and refused if the file's size, or on a folder binding its modification time, has moved since it was converted. A PDF with text of its own is returned as before and no model is called. A scanned PDF that is encrypted, or whose pages are fax, JBIG2 or JPEG 2000 images, is refused in our words before any model call. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| let { asked, readRange } = ranges(pdf); | ||
| await expect(pdfToText(ai, undefined, byteStream(pdf.bytes, [65_536]).stream, readRange)) | ||
| .rejects.toThrow(new RegExp(`^The PDF has no text of its own, and .*${ | ||
| why.replace(/[()]/g, "\\$&")}.*, so it cannot be transcribed\\.$`)); |
Preview:
|
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
| let parms = dict.get("/DecodeParms"); | ||
| if (Array.isArray(parms)) parms = parms[0]; | ||
| let predictor = parms instanceof Map ? parms.get("/Predictor") ?? 1 : 1; | ||
| let filtered = predictor !== 1; |
There was a problem hiding this comment.
[P2] Refuse unresolved decoding parameters instead of assuming no predictor
/DecodeParms can be an indirect reference, but every non-Map here defaults to Predictor = 1, and the scanner never retains dictionary objects to resolve it. A page using /DecodeParms 7 0 R, with object 7 containing << /Predictor 15 /Colors 1 /Columns 200 >>, is therefore accepted as lossless with filtered: false. pngFromDeflated() then inserts filter bytes into rows that already contain them. I reproduced valid 200×200 grayscale rows expanding from 40,200 to 40,401 bytes under the same PNG dimensions, corrupting the image sent for OCR. Resolve referenced parameter dictionaries, or classify unresolved parameters as unsupported and refuse before invoking the model.
| cancel: reason => { | ||
| cancelled = true; | ||
| return current?.cancel(reason); |
There was a problem hiding this comment.
[P2] Cancel the page request while its model response is still pending
When cancellation arrives after begin() starts transcribeImage() but before ai.fetch() returns, current is still undefined, so this handler never cancels the image reader or the model request. I reproduced this by holding the vision fetch pending: reader.cancel() resolved, but the image's cancellation callback ran only after I released the model response; the fetch also receives no abort signal. A slow or stalled request thus continues downloading/uploading and potentially generating billed output after the caller cancels. Keep a cancellation handle for the pending page before awaiting the response, and propagate cancellation to both its source and its fetch.
|
Review: 2 findings. Posted two actionable inline comments, both verified with targeted runtime probes. |
WIP - <Proposal/Not ready for review>
Adds an opt-in
readPdf(fileId, { ocr: true })to both Drive sessions. A scanned or image-only PDF has no text layer, soreadPdf()(#843) returnshasText: falseand nothing else. Withocrset, such a PDF's pages are transcribed by the Workers AI vision model@cf/google/gemma-4-26b-a4b-itand returned astextwithtranscribed: true. A PDF with any text of its own is returned as before, and no model is called.Like
readPdf(), nothing is held: not the PDF, not a page image, not decoded pixels. Onlypackages/gatekeeper-googlechanges; there is no kernel orworkshop-sharedchange.How it works
src/pdf-images.ts) notes where each page-sized image sits in the file, its length and its encoding. It walks objects from the front, takes lengths given directly, by reference or not at all, skips masks and small images, and notes an encrypted file. Stream data is counted past, never kept.downloadRange+exactRange, fromreadFile()), base64-encoded into the model's request as it arrives, and the model's SSE tokens are passed on under the same### Page Nheadings (src/workers-ai-vision.ts). The request is whatAi.run()posts, sent through the binding'sfetchso the body can stream, as the conversion already does.textis read, one at a time. Cancellingtextstops the page in progress and reads no more.nativeRead(..., { content: true }): checked against the binding's scope, re-proved on a folder binding, reported asdrive/content/<fileId>. It fails if the file is no longer the size it was converted at or, on a folder binding, if its modification time has moved.src/pdf-png.ts): the PDF's own bytes inside PNG framing when the rows carry PNG filter bytes, re-deflated with one added per row when they do not.Commits
What was measured (fixture pages with known text, 12 to 6 pt, a table; probes on a scratch Worker)
pdf.images.convertoption returns captions ("a heavily distorted image of a document"), not a transcription, so it is not used.HL/2291read asHL/229). 150 and 300 dpi read the same: the model sees about 280 image tokens either way. The types tell the agent a transcription is a model's reading and may misread.Checked
pdfToTextand the session layer (observations, folder re-proof per page, refused page reads, RPC validation).pnpm dev-server -- --use-workers-ai-bindingwith a Drive folder binding: an agent turn read a three-page scan with and withoutocr, a text PDF, a fax-encoded and an encrypted PDF, and cancelled a transcription at the second page heading. The gatekeeper logged one page transcribed for the cancelled read.Not in this PR
CF_AI_GATEWAYto this gatekeeper. Through a gateway that stores its logs, page images and transcriptions land in them.🤖 Generated with Claude Code