Idea (not a foregone conclusion)
Spun out of the retry-safe uploads work (#835) and the #829 discussion. Capturing it so it isn't lost — this needs a real design decision before it's built, and a chunk of it may be over-engineering.
The clear-cut observation
Today, uploading the same exact bytes twice is only deduplicated if the caller ties the two requests together with a shared Idempotency-Key (#835). Two content-identical uploads without a shared key are two separate objects — a bare f/<id>/… key mints a new id each attempt.
That default is defensible in the general case (a file host stores keyed resources: draft.pdf and final.pdf with identical bytes are legitimately two files). But there's one context where I can't think of a scenario where minting a duplicate is the preferred behavior:
Within the same workspace + same branch/PR context, uploading the same content hash twice. e.g. an agent or CI step re-runs a screenshot capture / uploads put / attach on the same branch, and the bytes are byte-identical to something already staged for that same branch/PR.
In that narrow case the second object is just noise: same bytes, same branch, no distinguishing context. Re-runs are common (agents retry, CI re-runs), and branch staging defaults to on (#403/#469), so this is the most likely place duplicates actually pile up.
Lightweight proposal
At upload time, when there is a branch/PR context (gh.* tags / staged-on-branch), look up whether the workspace already holds an object with this content-sha256 for the same branch/PR, and if so return that existing object (its key/url) instead of writing a new f/<id>. Effectively an implicit, content-derived idempotency scoped to the branch — no header required.
Why this is cheap and low-risk:
Open questions / where this could over-reach
Explicitly out of scope (for now)
- Full content-addressed storage / single-instancing across all keys and names.
- Cross-workspace dedup (privacy: leaks existence of identical files across tenants).
- Any refcounting/GC of shared blobs.
Prior art in-repo
Prior art / inspiration (to avoid re-deriving the edge cases)
Two camps in the wild; one is a trap for us.
- Content-addressed storage (Git blobs, IPFS CIDs, OCI/Docker layers, restic/borg): the hash is the key, identical bytes stored once globally, with a separate name→hash reference layer plus reachability GC to reclaim unreferenced blobs. Clean, but the reference layer + GC is exactly the refcount/reaper subsystem this issue is trying to avoid. Adopting it wholesale = over-engineering.
- Scope/key-based "return existing": dedup within a namespace by the key, not global content. Cloudinary dedups by
public_id (the name) with overwrite/unique_filename knobs rather than going content-addressed — a mature product deliberately choosing narrow + opt-out.
The pattern worth stealing — Docker Registry v2's two-step (spec):
- HEAD by digest (
HEAD /v2/<name>/blobs/<digest>) → skip the upload if the bytes already exist.
- Cross-repo blob mount (
POST …/blobs/uploads/?mount=<digest>&from=<repo>) → reference an existing blob in a new context without re-uploading.
The load-bearing lesson: Docker dedups the bytes but still writes a per-repository reference (the manifest) — dedup and context are separate layers. That is exactly why the same bytes on PR #5 vs #7 want one blob but two contexts. So the way to stay out of refcount-land is to scope dedup to the same context (same branch/PR) — then it's a write-skip, not a shared-blob system. That's the narrowing this issue already proposes; Docker is evidence it's the right seam.
Don't conflate with the retry axis. "Same request retried safely" is a different concern, already shipped via Stripe-style idempotency-key + request fingerprint (#835); the related storage primitive is S3/R2 conditional create (If-None-Match: *, "create only if absent"). Neither is content-dedup. Keep them out of this issue or the edge cases multiply.
Reality check: GitHub user-attachments does not dedup identical pastes (two pastes → two URLs). The ecosystem we mirror treats global dedup as non-default — another vote for keeping ours narrow.
Net: steal only the digest-precheck (we already have the file_content_hash reverse index, so "do we already hold these bytes for this branch/PR?" is a query we run today); keep dedup and context as separate concerns so no GC is needed; leave retry to #835. If the design starts talking about a shared blob across contexts + deletion, that's the Camp-A trap — stop there.
Suggested next step
Decide whether the narrow branch/PR-scoped version is worth it (I think it plausibly is, given how common re-runs are on staged branches), or whether it's over-engineering relative to just telling callers to pass an Idempotency-Key. If yes, design the exact scope key (workspace + branch/PR + content-sha256), the "return existing" response contract, and the interaction with --replace and explicit keys, before any code.
Refs #829, #835, #479.
Idea (not a foregone conclusion)
Spun out of the retry-safe uploads work (#835) and the #829 discussion. Capturing it so it isn't lost — this needs a real design decision before it's built, and a chunk of it may be over-engineering.
The clear-cut observation
Today, uploading the same exact bytes twice is only deduplicated if the caller ties the two requests together with a shared
Idempotency-Key(#835). Two content-identical uploads without a shared key are two separate objects — a baref/<id>/…key mints a new id each attempt.That default is defensible in the general case (a file host stores keyed resources:
draft.pdfandfinal.pdfwith identical bytes are legitimately two files). But there's one context where I can't think of a scenario where minting a duplicate is the preferred behavior:In that narrow case the second object is just noise: same bytes, same branch, no distinguishing context. Re-runs are common (agents retry, CI re-runs), and branch staging defaults to on (#403/#469), so this is the most likely place duplicates actually pile up.
Lightweight proposal
At upload time, when there is a branch/PR context (gh.* tags / staged-on-branch), look up whether the workspace already holds an object with this
content-sha256for the same branch/PR, and if so return that existing object (its key/url) instead of writing a newf/<id>. Effectively an implicit, content-derived idempotency scoped to the branch — no header required.Why this is cheap and low-risk:
file_content_hash (workspace, content_sha256) → object_key, oldest-first and indexed (apps/api/src/content-hash.ts,inheritableMetaForHashalready queries it for Content-identical re-uploads inherit the earlier object's derived metadata (server-side) #479 metadata inheritance). The "do we already have these bytes?" lookup is a query we already run.Open questions / where this could over-reach
gh.pr/gh.ref. This is exactly why content-hash inheritance already treatsgh.*as non-inheritable (Content-identical re-uploads inherit the earlier object's derived metadata (server-side) #479) — context is not a property of the bytes. So dedup must be scoped to same context, or it misattributes. (This is the main reason to keep it narrow rather than "dedup by hash globally.")--metatags → which wins? Narrow scope (same branch, re-run) mostly avoids this; broader scope reopens it.f/<id>) staging path.Explicitly out of scope (for now)
Prior art in-repo
Idempotency-KeyonPUT), part of Improve the public API contract, retry safety, and scalability #829.Prior art / inspiration (to avoid re-deriving the edge cases)
Two camps in the wild; one is a trap for us.
public_id(the name) withoverwrite/unique_filenameknobs rather than going content-addressed — a mature product deliberately choosing narrow + opt-out.The pattern worth stealing — Docker Registry v2's two-step (spec):
HEAD /v2/<name>/blobs/<digest>) → skip the upload if the bytes already exist.POST …/blobs/uploads/?mount=<digest>&from=<repo>) → reference an existing blob in a new context without re-uploading.The load-bearing lesson: Docker dedups the bytes but still writes a per-repository reference (the manifest) — dedup and context are separate layers. That is exactly why the same bytes on PR #5 vs #7 want one blob but two contexts. So the way to stay out of refcount-land is to scope dedup to the same context (same branch/PR) — then it's a write-skip, not a shared-blob system. That's the narrowing this issue already proposes; Docker is evidence it's the right seam.
Don't conflate with the retry axis. "Same request retried safely" is a different concern, already shipped via Stripe-style idempotency-key + request fingerprint (#835); the related storage primitive is S3/R2 conditional create (
If-None-Match: *, "create only if absent"). Neither is content-dedup. Keep them out of this issue or the edge cases multiply.Reality check: GitHub user-attachments does not dedup identical pastes (two pastes → two URLs). The ecosystem we mirror treats global dedup as non-default — another vote for keeping ours narrow.
Net: steal only the digest-precheck (we already have the
file_content_hashreverse index, so "do we already hold these bytes for this branch/PR?" is a query we run today); keep dedup and context as separate concerns so no GC is needed; leave retry to #835. If the design starts talking about a shared blob across contexts + deletion, that's the Camp-A trap — stop there.Suggested next step
Decide whether the narrow branch/PR-scoped version is worth it (I think it plausibly is, given how common re-runs are on staged branches), or whether it's over-engineering relative to just telling callers to pass an
Idempotency-Key. If yes, design the exact scope key (workspace + branch/PR + content-sha256), the "return existing" response contract, and the interaction with--replaceand explicit keys, before any code.Refs #829, #835, #479.