Skip to content

RFC: pubky-social-specs v1 (first stable, fully breaking release) - #142

Open
SHAcollision wants to merge 20 commits into
mainfrom
rfc/social-specs-v1
Open

RFC: pubky-social-specs v1 (first stable, fully breaking release)#142
SHAcollision wants to merge 20 commits into
mainfrom
rfc/social-specs-v1

Conversation

@SHAcollision

@SHAcollision SHAcollision commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Draft for design review, not for merge as-is.

This PR adds the complete v1 design and rollout as docs/rfc-v1-social-specs.md, so reviewers can comment inline on each decision. The doc is self-contained: it needs no other reading. The first stable and first breaking release renames the crate to pubky-social-specs (1.0.0) and moves all data from /pub/pubky.app/<res> to a versioned, app-neutral epoch /{pub|priv}/social/v1/<res>, with a permanent forward-compat contract so this is the last path break.

How to review

The design is written model by model (v0 shape, v1 shape, and the reason for each change), then migration, then the subtask rollout. Comment on the specific line you agree or disagree with.

The highest-value places to push back:

  • Per-edit path versioning (Post) and whether edit history earns its permanent cost.
  • The dual-root content family and the private-posts lifecycle.
  • The hand-written pure-JS second implementation gated by differential tests.
  • Content-addressed feed ids and the private-by-default feed placement.

Scope

Supersedes the v1 roadmap #12. Folds in and resolves #47, #48, #55 (deferred to v1.x with the shape pinned), #120, #141. No open design decisions remain; the destructive migration carve-out was considered and rejected in favor of strictly non-destructive migration.

Once the design is accepted, the subtasks in Part D become the tracked implementation issues.

@SHAcollision
SHAcollision force-pushed the rfc/social-specs-v1 branch 2 times, most recently from 6673054 to 4a4d408 Compare July 13, 2026 13:34
@SHAcollision SHAcollision changed the title RFC: pubky-social-specs v1 (first stable, first breaking release) RFC: pubky-social-specs v1 (first stable, fully breaking release) Jul 16, 2026
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment on lines +23 to +24
- No privacy tier, no file extensions, GET-per-file bookmarks, and `url::Url` validation that
normalizes junk into acceptance while rejecting valid short-form URIs.

@SHAcollision SHAcollision Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now have /priv/ data, the homserver also handles content-type so this enables new and better ways to build Pubky applications that we can only adapt via a breaking changes on our specs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we might not have private data. Requires re-planing.


- **Client-side, opt-in, resumable from the homeserver tree alone, permanent multi-epoch.** A
dormant user may migrate years later in one pass; the indexer dual-reads every epoch forever
(the permanent v0 parser stays, since the v1 parser classifies `pubky.app` as foreign).

@SHAcollision SHAcollision Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we could drop the v0 support. We never promised it to be stable and forever. But it's bad practice, and 1. any future Nexus reindexing Pubky App won't be able to index it from it's origin 2. our indexer already supports it and 3. we will need to support future versions anyway when they come without ditching v1.

@SHAcollision
SHAcollision force-pushed the rfc/social-specs-v1 branch from 4a4d408 to 0d50148 Compare July 16, 2026 07:58
@SHAcollision
SHAcollision marked this pull request as ready for review July 16, 2026 08:03
@SHAcollision SHAcollision self-assigned this Jul 16, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d50148b78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md Outdated
@SHAcollision
SHAcollision force-pushed the rfc/social-specs-v1 branch from 0d50148 to 6f8cf0f Compare July 16, 2026 08:16
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md
Comment thread docs/rfc-v1-social-specs.md Outdated
SHAcollision and others added 2 commits July 16, 2026 14:32
Co-authored-by: DZ <dzdidi@users.noreply.github.com>

@gillohner gillohner left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally like the direction of the proposed pubky-social-specs v1 changes. In particular, moving shared social data into an app-neutral, versioned namespace and treating foreign application data as valid rather than erroneous are important improvements.

One area that I think still needs a more explicit contract is how applications outside of pubky.app are expected to combine the shared social specs with their own application specs.

Folder ownership

This is the case already but should be clearly documented:
The specification that defines an object determines its storage namespace, not the application that writes it.

This should ideally also be reflected in the SDK. An app should be able to compose capabilities and path builders for both its application namespace and the shared social paths it uses.

I would recommend requiring an epoch in application namespaces as well. Otherwise the social schema gains a migration and compatibility mechanism while application schemas remain unversioned.

Universal tags

Universal tags should have one canonical write location:

/pub/social/v1/tags/{id}.json

The current behavior where universal tag objects may be written under a tags directory in multiple application namespaces can remain as a legacy read rule, but I do not think it should be the recommended v1 write model as it doesn't follow the same convention that other objects do.

A reusable post envelope

I would like to explore separating the reusable post envelope from the closed set of post kinds.

Conceptually:

PostEnvelope<Kind, Content>

would contain the shared post mechanics:

stable post and edit IDs;
versioned storage;
parent and embed references;
attachments;
locking;
unknown-field preservation;
common validation and path helpers.

PubkySocialPost would be the upstream specialization with the closed set of kinds accepted under /social/v1/posts.

Application specs could create their own specializations in their own namespaces.

A Mapky review could then be a review kind with typed JSON as its content.

The generic SDK could parse and preserve the common envelope, while the Eventky or Mapky package validates the custom kind and content. This could make it easier to build new applications from existing Pubky primitives.

It would also provide an incubation path for schemas used by real applications. A schema could prove itself in an application namespace before we decide to adopt it in Pubky Social Specs.

I do not think every application object needs to become a post. For example, an Eventky event is naturally post-like, while an attendee record is a relationship. The goal would be to make the post reusable where its semantics fit.

External resource

A related improvement in Nexus that was mentioned already but I want to write down so we don't forget it would be to extend the External Resource behavior of tags to all other Pubky objects. For example allowing PubkyAppPost on osm-urls.

## B11. Feed
v0 `pub/pubky.app/feeds/{HashId(serde-json config)}` (public) -> v1
`{priv|pub}/social/v1/feeds/{id}.json`.
- **Private by default, published by choice** (copy the same file to `/pub/`). Reader set is the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy or move the file to '/pub/'? Wouldn't moving make more sense so we don't have redundant data?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We keep it because private serves a functionality. Private is the master everywhere in the design: the client enumerates "my feeds" with one private LIST instead of a union of two directories, edit and rename flows operate on the private copy and re-publish, and unpublish is a pure delete of the /pub/ with nothing to restore. The redundancy cost is a few hundred bytes per published feed. Uniformity with the post publish model, plus simpler client reads seems worth it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Private by default, published by choice (copy the same file to /pub/)

Looks like the act of publishing implies a copy from /priv to /pub and all feed edits = edit /priv + re-publish.

Where is this publishing primitive defined? A method exposed by pubky-social-specs? Or is it the caller's (e.g. FE's) responsibility to implement?

@SHAcollision

Copy link
Copy Markdown
Collaborator Author

@gillohner thanks for this review, it moved the PR a lot. Quick rundown (commits d126745..7a7ec19):

Folder ownership + app epochs: you're right that we can't enforce either one, nothing stops an app from writing wherever it likes, but that's not a reason to leave it unwritten. So both are in the docs now: the ownership rule stated plainly ("the spec that defines an object determines its namespace, not the app that writes it"), your SDK composition point next to it, and epochs in app namespaces as a recommendation. Fun detail: the parser already surfaces the segment after the namespace from foreign paths, so apps that follow the convention get version-routing for free.

Universal tags: adopted. One canonical write location (pub/social/v1/tags/), any public target; reading tags from other app namespaces stays as a legacy rule only, which lets nexus drop the writing-app dimension for v1 data. Digging into "any target" also exposed that our draft had quietly narrowed targets to pubky+http/https, which would have rejected nostr/geo tags that were legal in v0. Fixed with a universal tier: any scheme-shaped URI through a small pinned gate, for tags, bookmarks, and embeds.

Post envelope: in for v1, not deferred. PostEnvelope<K: PostKind>, and the social post becomes its first specialization (type alias, wire bytes unchanged). The envelope owns storage + reference mechanics; the specialization owns kinds, content validation, and its namespace. One divergence from your sketch: a single generic param instead of <Kind, Content>, since content stays a JSON string on the wire and typing belongs in the kind's validation hook. Push back if Mapky/Eventky need the second param for something we missed.

External resources: also in v1. embed joins the universal tier (quote a nostr event or an OSM object) and nexus attaches those posts to the same External Resource nodes tags use. parent stays pubky-only, reply threads are post-to-post edges.

@SeverinAlexB SeverinAlexB left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand most of the spec but here are some comment of things I understood

so per-item metadata (alt text now; hash/blurhash later) is additive; ships two committed fields.
- **`lock`** kept: the value is the lock-FILE URI (`pub/locks.app/<lock_id>.json`), and presence
means "locked content" regardless of kind (matches the Locks feature's resolved design).
- **Dual-root:** posts may live under `/priv/` (drafts, private notes, private collections).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly discourage using /priv at this point. It's a hack that we had to deliver quickly for locks. We will redo this completely in the future so if you don't want to be forced to migrate away again then I suggest you do not use it yet.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to this one after the call. You said we can move off /priv in 6-12 months and that we should just build on it and figure it out, so I think we are good here.

Writing down why moving is cheap anyway, because that is the part that makes it safe: private data is closed world, nothing public points at it and the indexer never reads it. So moving later is just a sweep on your own files, not a network migration. Public is the opposite, that is where we are being careful.

We also said the homeserver could maybe do the priv migration itself on a version bump, if the final solution is not client side encrypted. This could work, better UX, but we can also make the migration happen even if it's client side encrypted, so no prob.

Can you dismiss the changes requested when you have a min? thanks

Comment thread docs/rfc-v1-social-specs.md Outdated
Comment on lines +20 to +21
The first stable and first breaking release of the shared social-data layer. Renames the crate
`pubky-app-specs` to `pubky-social-specs` (`1.0.0`) and moves all data from the single hard-coded

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "shared social-data layer"? Why "shared"?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shared = many apps read and write the same posts/follows/tags, so the schema can't belong to one app. Will add a sentence.

Comment thread docs/v0-vs-v1.md

These are listed once here; the per-model sections below only add what is specific to them.

- **Namespace and epoch: `pub/pubky.app/<res>` becomes `{pub|priv}/social/v1/<res>`.**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are sacrifying the permission audibility in Pubky Ring by renaming pubky.app to social. When a user gives permission to an app, the user somehow needs to determine if it gives permission to the right folders. So far, this was comparing the folder names. With this change, this is not possible anymore.

Not saying we shouldn't rename but if we go with this, we need to develop another way the user can verify that it gives the correct permission to the app. There was a reason we named it pubky.app initially.

@SHAcollision SHAcollision Jul 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean. However this rename is very much on purpose actually. /pubky.app only reads clear when pubky.app itself asks for it; for any other app (large part of what we are addressing in v1 by tackling shared models) the screen gets weird: why Eventky wants the "pubky.app" folder? What's "pubky.app" folder? There is a chance some users never even learn that pubky.app exists, the name is a brand. The old name tells you who created the folder, but /social tells you what it is actually.

We are not proposing every application should do this. Applications' own internal data should stay in their own namespace, /app.myapplication style (probably turning the domain around is best, avoids issues with Finder). Only real public specs for shared data models across apps should even think of /pub/<context>.

Unplanned, but to improve this further we could pin a human description for the social scope, so Ring and wallets all show the same friendly text.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why Eventky wants the "pubky.app" folder

This is the correct question and is one research that I want to get done in the next months:

  • How should one app interact with another? Should it access it's full folder structure? Is this dangerous? Should there be a specific folder structure for shared content like /share?
  • How should Pubky Ring Authorization requests look like? You mention well that we might need a human description. This could be a good solution IMO. We should properly brainstorm it though with others. There might be security implications around it like somebody faking such a description. TBD.

And again, I am not trying to prevent the rename here. Just trying to stimulate some deeper questions that are important for PubkyApp and other apps as well.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, that's also why I asked the above comment about what "shared" means.

@SHAcollision SHAcollision Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad the research is happening and happy to feed it with what we learned building the v0 spec and this bump. Some quick thoughts:

On /share: so current v1 kind of already does this. Just on another dir, specs live at /pub/<context> (social is just the first context) and app-internal stuff stays in the app's own namespace. I think we could go this way for v1 and then, once there is a better model coming from pubky-core, migrate on v2. Or maybe the research can start from that pattern and try to break it?

On apps accessing each other's folders: in our model they never do. An app asks for the shared contexts it uses + its own folder, that's it. Other apps' data you reach by reading public stuff and references, not by asking write permission to their folder.

On faking descriptions: the description should not come from the app. Ring has to keep its own trusted table (scope path -> friendly text, published together with the spec), the app only sends the scope string. Unknown scope = show raw path + a warning, no friendly text. So there is nothing to fake without owning Ring itself.

There is many unknowns on the pubky-core protocol side (shapes and timelines), but I think that's okay because we are designing this spec so we get upgradability + migration tooling. An option here could be to ship v1 on what exists, that builds the tooling needed for more seamless future migrations. Then we upgrade to v2 when the homeserver ships the new stuff. If core lands a different model later, we adapt on the next epoch, that's literally what the path versioning is designed for.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An app asks for the shared contexts it uses + its own folder, that's it. Other apps' data you reach by reading public stuff and references, not by asking write permission to their folder.

One new risk this new setup allows for, which is not there in v0, is:

  • AppA creates some tags (shared context)
  • AppB gets full access to shared context on auth
  • ether by malicious intent, or by mistake, or due to a complex UX flow gone wrong, it has the power to delete AppA tags

What defenses are there against this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Real risk, but I think it is the same one we have today. /pub/pubky.app/:rw is already total, so any client, or a bug in ours, can wipe your tags right now. What changes with v1 is that more apps will normally hold that scope, so more chances to hit it.

On delete-only-what-you-created: I don't think we can, and I am not sure we want to. Apps have no keys, everything is written under your key, so we cannot even tell who wrote what. And even if we could it gives us walled gardens again, your data only editable in the app that made it, which is the opposite of the point.

What we can do and I am adding it to the doc: capabilities are path prefixes, so a tagging app should ask for /pub/social/v1/tags/ and not the whole namespace. Does not stop a bad tagging app wiping your tags, does stop it touching your posts.

On the call Sev proposed something better for the general case, a manifest inside the folder that Ring reads, so any app gets a readable consent screen instead of just ours. Opened pubky/pubky-homeserver#544 for that plus some recovery (trash or undo).

Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/rfc-v1-social-specs.md Outdated
Comment thread docs/v0-vs-v1.md Outdated
Comment thread docs/v0-vs-v1.md
- **`items` accept any reference-tier pubky URI (any resource, any app).** Why: v0's item check
hard-restricted items to `posts/` under `pubky.app`, contradicting the interop goal; a curated
list may legitimately include files, profiles, or another app's resources. Items stay
pubky-only (a web link belongs in a post) and stay plain strings (per-item annotation is

@gillohner gillohner Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Items stay pubky-only (a web link belongs in a post)

If I want to make a collection of for example spotify songs or osm locations it could be useful to allow linking the external resource directly. Maybe discuss tomorrow?

@secondl1ght secondl1ght left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, and congrats on the progress towards the V1 milestone. I have a few comments/questions in no particular order:

  1. Replacing the collection follow functionality that is lost with bookmarks will have to be included from the beginning since it is a feature users already have. The spec says this is deferred, but until when? Want to make sure this doesn't get lost in the mix.

  2. Echoing @SeverinAlexB's point about the private data structure being reworked in the future. It's worth waiting for that to land before committing to it, but I don't think that blocks the rest of this migration.

  3. The post edit history, was this something the product team requested? Curious what the background on this is and if it was a feature request from users. Looks like it will add overhead, and I am not sure if it is worth it.

  4. Let's make sure to stay in sync when the spec is finalized and the backend starts development. We'll want plenty of time to plan the frontend migration and we should add this task into our roadmap to establish a timeline plan.

  5. We'll have to decide on a migration strategy for Franky, my initial thought is to not make this optional. I don't see any reason to keep users on the old spec. We can have a migration flow that the design team creates to guide them through the process.

  6. I assume the migration engine is going to be written in browser JS? This part isn't clear (M2 not included in J2).

  7. Downloading all of the media the user has on the homeserver and then uploading again could be quite a lot of data. Exposing the copy primitive on the HS instead would be much better to avoid this.

  8. We are going to double the HS used data storage for each user by retaining all V0 data, so we'll also need to double their storage limit at the same time.

  9. For the "deletion engine", ideally we have recursive and bulk delete capabilities exposed to authenticated users on the homeserver. Otherwise we will potentially have a large amount of requests to attempt to cleanup everything (could leave partial data remaining on failure and remain indexed since it is a non-atomic action from the client).

  10. Will V0 Nexus endpoints be deprecated eventually? I am assuming this spec bump will also come with a Nexus endpoint version bump to V1?

  11. Double-checking the normalization implementation--Nexus will convert V0 data into V1 ex. short > note?

  12. I think the publish/unpublish flow needs to be expanded on (but this will change once Sev finalizes the private data so it can probably wait). In its current form it will have similar fragility concerns as the delete flow if the client needs to make many requests to complete it (this type of action would normally be a transaction in the DB).

  13. We should come up with a communication plan to notify third-party developers of the breaking changes beyond just the release notes. Highlighting on social channels and the community call would be good as well.

Some of these will probably be answered on the call next week, but I wanted to ask as much as possible upfront so that someone from the frontend team has reviewed this proposal. Thanks!

@SHAcollision
SHAcollision requested a review from aldertnl July 25, 2026 13:07
@SHAcollision

Copy link
Copy Markdown
Collaborator Author

@secondl1ght thanks for going through it in detail. Going one by one:

1. Collections following goes private with bookmarks, but the feature does not change for the user. Same mechanic as bookmarks: the client lists them from the homeserver (the filename is the target, so listing is cheap) and then asks nexus for a stream by ids. This is already how we plan to rebuild the bookmarks feed. The public side (a follower count on a collection) we already accounted for, that is why we never implemented that count. If product wants it later we can add it as an optional "follow object" primitive.

2. Let's agree this on Thursday. It depends a lot on core timelines for the private research and on how urgent product thinks the privacy gap is (and it is a real gap today). My take: we do v1 as it is in the spec, and then v2 when the private rework lands, especially if that timeline is unclear or looks like more than 6 months.

3. Yes, product asked for it the same day we shipped edit on the old client. It stayed unaddressed until now precisely because it needed a breaking spec change, so this is the window.

4. Agree, let's put it in the roadmap and plan the frontend work with proper lead time.

5. That is the plan. Just to be clear on wording: opt-in in the spec means the network never forces anybody and un-migrated users keep working. It doesn't stop us from migrating all Pubky App users automatically with a nice flow.

6. We are aiming for no wasm in the migrator, reading v0 as plain JSON. We will see what ends up easier to implement.

7. True, and it would be nicer, but we are not requesting it.

8. Yes, we need to double to 2GB for existing Synonym HS users.

9. Remains to be seen. Core is quite KISS about endpoints and today there is no batch delete, so we should plan assuming it does not exist. If they add it later we just get faster.

10. Nexus API will break, yes, v0 endpoints go away because we don't want to keep two models in the DB. The API is already versioned as v0 so a v1 is expected anyway. Different thing: nexus keeps reading v0 data from homeservers forever, it just normalizes it into the same rows.

11. Yes, exactly. The v0 adapter maps the old wire values, short becomes note, so a note filter also matches un-migrated users.

12. The order is what protects it. Media first, then rewrite the references, and the post is PUT last. So if it dies halfway you get some orphan media in /pub/ and no post visible, instead of a published post with broken images. Retry is safe because copies skip if they already exist, and unpublish is only deletes so it is idempotent.

13. Agree on community call and channels. I think it is not so dramatic though: v0 was never promised forever (it says so at the top of the readme), v0 data survives, and third-party apps talking to the homeserver keep working. The nexus API bumps to v1, but that one is versioned on purpose so it is the expected path. So mostly we announce that v1 stable is out.

- **Client-side, opt-in, resumable from the homeserver tree alone, permanent multi-epoch.** A
dormant user may migrate years later in one pass; the indexer dual-reads every epoch forever
(the permanent v0 parser stays, since the v1 parser classifies `pubky.app` as foreign).
- **STRICTLY non-destructive.** Migration never deletes any legacy-epoch data. Privacy lost before

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once data has successfully been migrated, why not delete it from legacy? This would reduce the chance of hitting data storage limit during or soonafter migration.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see Matt had an overlapping point. We do already have a need to batch delete in pubky.app for wiping your data.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point on the storage, we had not written that down anywhere and it is real, you can hit the quota exactly when you migrate.

The reason we copy instead of moving is a few things: migrating should not break clients still on v0, we can recover if a migration goes bad, and we can fully verify a good one before anything is gone.

But I think you are right that keeping it forever is not the answer either. v0 was never promised stable, so it is fair game to remove, we just should not do it in the same breath as the migration. So the plan is a second pass, a cleanup that ships some weeks after the migrator, once we have seen it run clean on real data. It asks you first and only deletes what it can verify migrated correctly. That way a migrator bug is recoverable, and third parties still reading v0 get a window to move.

On the quota itself, secondl1ght raised the same thing in his review and we already said yes: we need to double existing Synonym HS users to 2GB, and that has to land before people start migrating, not after.

Going forward the rule is the opposite: from v1 on, migration never deletes. v0 is the exception because we never promised it, and I will write that down so it does not read as us being inconsistent.

applications read and write the same objects (posts, follows, tags), so the schema can belong to
no single app. Renames the crate
`pubky-app-specs` to `pubky-social-specs` (`1.0.0`) and moves all data from the single hard-coded
app path `/pub/pubky.app/<res>` to a versioned, app-neutral epoch `/{pub|priv}/social/v1/<res>`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is broadening permissions scope for all users' posts, attachements, tags (hard work and reputation). A single bad app that is granted permission just to tag could wipe it clean.

Is there no good way to guard data within somehow? Like apps only requesting permission to delete tags they created.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing ok300 raised on the other thread, answering there in full, short version here.

I think the risk is real but not new: /pub/pubky.app/:rw is already total today, so a bad client can wipe your tags right now. The rename does not widen it. What does change is that more apps will normally hold that scope, so more chances to hit it.

Delete-only-what-you-created I don't think we can do: apps have no keys, everything is written under your key, so the homeserver cannot tell who wrote what. And even if it could, it gives us walled gardens where your data is only editable in the app that made it.

What I am adding to the doc is that apps should ask for the narrowest prefix they need, /pub/social/v1/tags/ instead of the whole namespace. Does not stop a bad tagging app wiping your tags, does stop it touching your posts. And I opened pubky/pubky-homeserver#544 for the rest, recovery and readable consent screens.


## B0. Cross-cutting (applies to every model)

- **Namespace + epoch: `pub/pubky.app/<res>` becomes `{pub|priv}/social/v1/<res>`.** App-neutral

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the same file in different epochs seems complex, especially for the deletion process. It was simple to delete one file, now the app much know to delete every version of it if it exists else a different version of it persists (which could be wrong/obsolete). Could this versioning not be done within the file instead (metadata), and then only one exists at a time, only one to index or delete.

- **STRICTLY non-destructive.** Migration never deletes any legacy-epoch data. Privacy lost before
v1 is already lost; future activity is private. Private-tier legacy public copies are left inert
(the indexer stops surfacing them).
- **Deterministic and total** over real v0 data: each record sources from its highest present

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the non-destructive requirement, I sense the migration is optimising for safety. However, if an initial migration produces a bad output (e.g. v0 -> v1 with bad format) then the file exists and so would be skipped on all future migration runs. Is it the job of the app/migrator/user to track and delete bad format/corruptied etc. ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

each record sources from its highest present epoch

Doesn't this mean that, if a file exists in both v0 and v1, then reading that file should take v1?

And if so, doesn't this mean that moving instead of copying is safe, because reading will default to v0 if no v1 is there yet?

IMO moving instead of copying would eliminate a lot of migration edge-cases and headaches. This would also address the reply above (if migrate file flow = copy file, check if v1 is identical to v0, if yes then delete v0).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. Resume checks that the destination exists, which cannot tell a good file from a bad one, so anything a buggy run wrote gets skipped forever.

Changing it to compare source against destination. That also catches a v0-client edit after migrating, which is invisible today.

ok300, copy-verify-delete is exactly what we are doing, just not in the same breath as the migration. The delete is what breaks clients still on v0, so it becomes a later cleanup pass that asks first.

## B12. Settings (new)
v0: pubky-app's `pub/pubky.app/settings.json` was the only unspec'd homeserver artifact, world-readable
while exposing the user's privacy posture (`require_pin`, `sign_out_inactive`) to a reader set of
one. v1: `PubkySocialSettings` at `priv/social/v1/settings.json`, all sections optional, whole-file

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`priv/social/v1/settings.json`` seems odd to me. Either the app has its own settings and doesn't align with what we chose, or the generic settings (e.g. language) are not "social" but general across all types of apps.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. settings.json and last_read.json move to priv/app.pubky/v1/ and leave the library entirely, model definition included. Frontend owns them, they are the only reader and writer. The migrator still imports the v0 data so nobody loses anything.

social/v1 keeps social data only: posts, follows, tags, mutes, bookmarks, feeds, files, profile.

What we lose: a second social client will not know what you already read.

@ok300 ok300 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec proposal looks promising.

My biggest critique is that it appears to expand its scope from "these specs define what a valid model is" (as in v0) to also include "this is how a client should behave".

IMO the previous scope should be preserved. For the migration, it makes sense to have a "this is how a client must behave for migration", but all other parts outside of migration should remain exclusively focused on defining a valid model spec.

app path `/pub/pubky.app/<res>` to a versioned, app-neutral epoch `/{pub|priv}/social/v1/<res>`
(an epoch is the `vN` path segment; each epoch is a disjoint subtree holding one generation of
data).
The strategy has three parts. First, this one coordinated break makes every now-or-never

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, this one coordinated break makes every now-or-never change, those that cannot be added additively later. Second, a permanent forward-compat contract makes everything else additive, so v1.x grows without breaking.

This makes it sound like (A) this will be the only ever breaking change going forward.

The other places where the forward-compat is mentioned, make it seem like (B) the social models only are meant to be forever forward-compatible.

NIT: Either of them, but especially (A), seem like a big promise to make. Not sure if necessary.

Comment thread docs/v0-vs-v1.md
acceptance and rejects valid short forms, and two independent validation surfaces (parser vs
field validators) had already drifted; engine URL parsers cannot be version-pinned across
browsers.
- **The root rule.** A public-rooted object must never reference a priv-root pubky URI;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a public-rooted object must never reference a priv-root pubky URI; private objects may reference both roots

Since files can (presumably1) be moved from /priv to /pub and vice-versa, how are references handled? Shouldn't references ideally not include the /pub or /priv path segment? Otherwise, if such a file is referenced (e.g. in a Collection, or in another post), it won't be possible to update all references when the path changes.

Even if we define "publishing a private file" as copying it to /pub, wouldn't this then cause issues for timestampId-based files, because, as mentioned below:

TimestampId gains a per-session monotonic mint guard

I presume that means it's not possible to use any other TimestampIds in a session other than monotonically incrementing ones, which makes "publishing (as in copying)" such a file essentially impossible?

Footnotes

  1. I'm not aware of any spec denying or rejecting this. An app that publishes / un-publishes (e.g. changes visibility of a file from private to public) will instinctively attempt to move it (copy + delete). The "publish = copy to /pub" model I've only seen mentioned in reference to publishing private feeds.

Comment on lines +275 to +276
## B10. File (media), the v0 File + Blob pair collapsed
v0: two objects, `files/{id}` metadata + `blobs/{hash}` bytes -> v1: ONE content-addressed media

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

v0: two objects, files/{id} metadata + blobs/{hash} bytes -> v1: ONE content-addressed media object files/{hash}.{ext}

This means, if a file is referenced (post embeds, etc), then if the file author edits the file, this will result in a hew hash so a new file path, which will invalidate all references?

# Part C: Migration

- **Client-side, opt-in, resumable from the homeserver tree alone, permanent multi-epoch.** A
dormant user may migrate years later in one pass; the indexer dual-reads every epoch forever

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A dormant user may migrate years later in one pass; the indexer dual-reads every epoch forever

NIT: A very big promise to make.

Franky / Nexus in 2030 will have to support (detect, correctly migrate) random new HSs coming online with v0 data.

Perpetual technical debt: in every future update, we must test and ensure v0 is supported for migration. Why not promise this let's say for 1 year, or only for v0 -> v1, then make it best-effort? A multi-version upgrade path, years later, could involve several passes, or from a certain point forward, might include using a specialized tool ("your version is too old; please first migrate using this app, then try again to login").

- **Client-side, opt-in, resumable from the homeserver tree alone, permanent multi-epoch.** A
dormant user may migrate years later in one pass; the indexer dual-reads every epoch forever
(the permanent v0 parser stays, since the v1 parser classifies `pubky.app` as foreign).
- **STRICTLY non-destructive.** Migration never deletes any legacy-epoch data. Privacy lost before

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Migration: STRICTLY non-destructive. Migration never deletes any legacy-epoch data.

NIT: Doesn't this also mean legacy apps can continue writing to the legacy paths, at the same time that newer v1 apps write to the new path, leading to v0 / v1 drift? Treating migration as moving files would address this.

Btw "migration as moving" can even include an intermediary backup step:

  • copy v0 -> v0.bak
  • copy v0 -> v1
  • check if v1 == v0 (copied correctly, optional)
  • delete v0
  • later on, when all files migrated: delete all v0.bak files

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that seems to be a failure case, and I think it is what it is. Will write it down on the nexus side. Most likely we delete v0 a few weeks after migration anyway, which closes it for good.

It is silent too, which is the annoying part: the edit writes fine, the path exists, nobody sees it because the indexer keeps the newest epoch. Comparison on re-run picks it up, that is the change I just made in the other thread, but it narrows the window rather than closing it.

Your v0.bak flow is basically the cleanup pass, minus the backup step. We keep v0 itself as the backup instead of making a second copy.

target), and it accepts ANY external URI, the same universal tier as tags: http/https through
the strict web gate (the pinned regex validator for web URLs) (an OpenStreetMap object URL is an ordinary https reference), and any other
scheme-shaped identifier (`nostr:`, `geo:`, `ipfs:`, `did:`) through a pinned opaque gate
(lowercased scheme + rest verbatim, no engine parsing). The indexer attaches the post to the same External Resource

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lowercased scheme + rest verbatim, no engine parsing

NIT: URI parsing engines cover many specs, edge-cases, rules, etc. By not having the URL parser anymore, this exposes pubky-social-specs to the question: are all valid URIs supported here too? If yes, tests (Rust+JS) have to be added for every rule. If no, apps will know which kind of (otherwise valid) URIs to avoid. Regardless of the answer, IMO this should be made explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix: json file extensions

8 participants