Skip to content

feat(enrichment): dispatch CSV enrichment through Cloudflare Queues and Workflows #310

Description

@niklhut

Problem

CSV enrichment currently processes multiple books within a web Worker invocation. Worker exceeded CPU time limit has been observed in the PR preview and previously in production. Accumulated CPU usage on Workers Free is the suspected cause; the failing endpoint and CPU hotspot still need verification.

Follow up on #280 after #309 is merged. Implement this in a separate PR and task.

Proposed approach

  • After persisting imported books and durable enrichment jobs, immediately publish dispatch messages to Cloudflare Queues. Do not wait for cron or depend on an open browser.
  • Keep the Queue consumer lightweight: start a Workflow using a stable job/instance identity, then acknowledge successful dispatch. Handle duplicate delivery and already-created instances safely.
  • Execute enrichment in a dedicated regular Worker with Workflows, using bounded per-book steps for metadata lookup, persistence and cover storage. Do not wrap the existing multi-book HTTP batch in one Workflow step.
  • Keep D1 as the durable source of progress and R2 for cover bytes; return references rather than large binary step results. Have the UI poll persisted progress.
  • Reuse existing ownership checks, leases, deduplication, canonical-book coordination, user-tag preservation and shared OpenLibrary rate limiting. Define one owner for processing retries so Workflow and existing job retries do not compete.
  • Cover the gap between database persistence and Queue publication with durable reconciliation. Retain scheduled recovery as a dispatcher/reconciler rather than heavy processing in the web Worker.
  • Preserve the self-hosted processing path behind the existing runtime abstraction and route/service/repository architecture.

Free-plan constraint

Workflows Free currently permits 10 ms of active CPU per step. Queue/Workflow dispatch does not itself remove CPU limits. Measure representative steps, keep the jobs entry point lightweight, and split further where needed. If a required operation cannot reliably fit, document the measured limitation and required plan/configuration rather than claiming Free compatibility.

References: Workflow limits, Queue limits.

Acceptance criteria

  • CSV import dispatches enrichment immediately after persistence; work continues after navigation, reload or closing the browser.
  • Duplicate messages, dispatch failures, retries and worker restarts neither lose jobs nor apply enrichment twice unsafely.
  • UI progress and terminal states remain accurate, and enrichment cannot overwrite user tag edits or introduce suggested tags on library cards.
  • Representative imports and individual processing steps are validated against the target plan's CPU limits. Logs identify import, dispatch and processing failures separately.
  • Production and PR previews have isolated Queue/Workflow bindings and appropriate provisioning, deployment and preview cleanup. No production data is processed by previews.
  • Self-hosted deployments retain working enrichment without Cloudflare bindings.
  • Add regression coverage for dispatch/reconciliation, idempotency, failure recovery and progress; run lint, typecheck and tests.

Scope boundary

This issue covers enrichment dispatch and execution. If profiling shows CSV parsing/persistence itself exceeds the request budget, capture that separately for chunked imports staged in R2. Broader email, storage reconciliation and image-product migrations are outside this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestlibraryLibrary inventory, books, metadata, import/export, or searchoptimization

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions