UGC -> single bucket migration script - #317
Draft
bnolan wants to merge 3 commits into
Draft
Conversation
Pulls all parcel UGC (images, vox, audio/video, lightmaps, tilesets) into the voxels-ugc bucket (ugc.crvox.com), converts images to ETC1S .ktx2 + .webp, and rewrites parcel scene JSON in place. Bakes far-LOD placeholders into the JSON (8x8 jpeg feature.thumb for images, 4x4x4 fixed-palette feature.lod4 for vox) so distant parcels render instantly with zero fetches. Includes the discord cdn refresh workaround, 100MB cap, opensea skip, per-parcel dedup, idempotent uploads, and a --dry-run mode. Client-side consumption is phase 2 (see plans/ugc-single-bucket.md). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Require UGC_ACCESS_KEY_ID + UGC_SECRET env vars for live runs; harden idempotency and discord refresh handling. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Island-parameterised migration that consolidates all parcel UGC into the single
voxels-ugcbucket (served viaugc.crvox.com) and rewrites parcel scene JSON to point only at that host. Replaces the current sprawl of hosts (compressor, img, herring, media, files, cdn2 + external discord/reddit/dropbox/ipfs)..ktx2(512, viabasisu) +.webp(2k max side), stored side by side; url rewritten to the extensionless base key.feature.thumbfor images, 4x4x4 fixed-palettefeature.lod4for vox (shared palette incommon/vox-palette.ts)./attachments/refresh-urls(needsDISCORD_BOT_TOKEN), soft-fails to skip.--dry-runmode.Files:
scripts/migrate-island-assets.ts(cli),scripts/lib/ugc-migrate.ts(engine),common/vox-palette.ts(shared palette),plans/ugc-single-bucket.md(what/why notes).Not in this PR (phase 2, client)
The script writes the new bucket urls + placeholders but the client doesn't consume them yet: ktx2 transcoder registration, distance LOD swap (.ktx2 -> .webp), far-LOD render of
thumb/lod4, and makingvoxel-fieldtreat the migrated absolutecontent.tileset. See plan notes.Test plan
--dry-runon 5 Poneke parcels: 41 img / 80 vox / 3 copy planned, manifest mappings correct, dedup confirmed, 0 errors, 0 db writesUGC_SECRET+DISCORD_BOT_TOKEN+ network) -- holding for reviewMade with Cursor