Update Flatpak pnpm source preparation

[danyloburavlov]

Summary

This updates the Flatpak packaging flow to use pnpm instead of the old Yarn-based dependency setup, then adds locked DuckDB extension artifacts so the Flatpak build can keep working offline.

The repo now uses pnpm-lock.yaml, but the Flatpak source generation path still depended on Yarn-style generated sources and Yarn install commands. This PR switches the Flatpak JavaScript dependency sources to pnpm, installs with pnpm install --frozen-lockfile --offline, and regenerates flatpak/generated-sources.json from the current pnpm lockfile.

While testing that path, DuckDB extension downloads became the next offline-build gap. The downloader was still resolving extension URLs at build time, which does not fit Flatpak well because external files need to be declared before the sandboxed build starts. This PR adds a DuckDB extension lockfile with pinned URLs and SHA256 hashes, teaches the downloader to consume it, and generates Flatpak sources for those locked extension artifacts.

What I've changed, more specifically

Flatpak pnpm migration

• Updated the Flatpak manifest to use pnpm cache/store settings instead of Yarn cache settings
• Replaced Flatpak build commands from yarn ... to pnpm ...
• Switched flatpak-node-generator from Yarn mode to pnpm mode
• Updated the pinned flatpak-builder-tools commit to one that includes pnpm lockfile support
• Added flatpak/scripts/flatpak_prepare_pnpm.py to prepare the copied Flatpak build tree before offline install
• Normalized the pnpm lockfile used by the Flatpak generator/build so the Flatpak SDK-provided Node runtime is used instead of the repo-managed Node runtime package
• Removed the old Yarn hash helper and flatpak/yarnrc
• Regenerated flatpak/generated-sources.json and flatpak/generated-sources.hash from pnpm-lock.yaml
• Updated Flatpak documentation to describe the pnpm-based workflow

DuckDB extension locking

• Added src/main/duckdb-extensions.lock.json to pin DuckDB extension artifacts by DuckDB version, platform, URL, and SHA256
• Updated the DuckDB extension downloader to validate lock freshness, verify SHA256 before unpacking, support --update-duckdb-lock, and install from a prefetched source directory when VORTEX_DUCKDB_EXTENSION_SOURCE_DIR is set
• Added package scripts for downloading extensions and refreshing the DuckDB lockfile
• Updated the Nx target so the DuckDB lockfile is part of the cached extension download inputs
• Updated scripts/generate-query-types.ts to load the same locked local level_pivot extension instead of resolving current_release
• Added tests for lockfile validation, locked artifact lookup, SHA256 checking, platform mapping, and DuckDB extension path layout
• Extended flatpak/scripts/flatpak_sources.py to emit generated-duckdb-sources.json
• Added flatpak/generated-duckdb-sources.json and flatpak/generated-duckdb-sources.hash
• Updated the Flatpak manifest/workflow so DuckDB extension artifacts are prefetched by Flatpak and consumed from the offline build sandbox

Tech Notes

The generated JavaScript source diff is large, but that is expected. The output moved from Yarn-style generated sources to pnpm-style generated sources, including pnpm store population data based on pnpm-lock.yaml.

The Flatpak pnpm preparation step removes the managed Node runtime from the copied package metadata and normalizes the lockfile for the build sandbox. Flatpak already provides Node through org.freedesktop.Sdk.Extension.node24, so the offline pnpm install should use that SDK runtime instead of trying to resolve/download Node as a package dependency inside the sandbox.

DuckDB extension locking pins the compressed .duckdb_extension.gz artifacts, because those are the files Flatpak fetches before the build starts. The downloader verifies the compressed file hash first, then unpacks it into the normal build output directory.

Normal app builds still download the locked DuckDB URLs directly. Flatpak builds set VORTEX_DUCKDB_EXTENSION_SOURCE_DIR, so the downloader reads the already-prefetched artifacts from the Flatpak source directory instead of using the network.

scripts/generate-query-types.ts also uses the locked extension flow now. It validates the lockfile against the installed @duckdb/node-api, checks that the host platform artifact has been downloaded, sets DuckDB's extension_directory, and runs LOAD level_pivot from the local locked output. That keeps generated query types aligned with the extension artifact Vortex packages.

Refreshing the DuckDB lockfile is intentionally explicit:

pnpm run --filter @vortex/main duckdb:update-duckdb-lock

That keeps normal builds from silently accepting changed upstream DuckDB extension artifacts.

Open Questions

• DuckDB extension artifact URLs currently come from current_release. The lockfile pins SHA256, so changed upstream bytes fail builds instead of being accepted silently. However, previous builds are only reproducible while the old bytes remain available at that URL. How should the build process work if the actual bytes behind current_release change, but the previous bytes are still preferred?

Proposed answer: keep the SHA256 lock as the build gate, and do not auto-refresh it during normal builds. If current_release changes and the hash check fails, the build should fail with a clear message and require a maintainer decision. If the new bytes are intended, refresh the lockfile. If the previous bytes are still required, they need to come from an immutable source that still has them, such as a release asset, CI artifact, or project-owned mirror. If no such source exists, the previous build is not fully reproducible from current_release alone. Longer term, the extension publish flow should expose immutable versioned artifact URLs so the lockfile can pin both the URL and hash without depending on current_release.

---

• scripts/generate-query-types.ts now loads the unpacked DuckDB extension from src/main/build/duckdb-extensions/.... That file is hash-verified when produced by duckdb:download-duckdb-extensions, but the query type generator does not re-check SHA256 itself. Is that trust boundary OK, or should query type generation verify or recreate the locked extension artifact before loading it?

Proposed answer: keep SHA256 verification centralized in the downloader for now. The generator should require the local locked extension output to exist and fail with instructions to run pnpm run --filter @vortex/main duckdb:download-duckdb-extensions when it is missing. If maintainers want query generation to be independently verifiable later, the lockfile should also record the unpacked extension hash or the generator should call the downloader flow directly before loading DuckDB.

Validation Performed

Regenerated the pnpm Flatpak node sources:

python3 flatpak/scripts/flatpak_sources.py --only pnpm --force

Regenerated the Flatpak DuckDB sources from the DuckDB extension lockfile:

python3 flatpak/scripts/flatpak_sources.py --only duckdb --force

Ran the targeted DuckDB extension tests:

pnpm run --filter @vortex/main test -- download-duckdb-extensions.test.ts

Regenerated query types using the locked local DuckDB extension:

pnpm run generate:query-types

Checked that the generated Flatpak DuckDB source file is valid JSON:

python3 -c "import json; json.load(open('flatpak/generated-duckdb-sources.json'))"

What Is Still Needed To Be Done

This gets the Flatpak dependency/source-generation path onto pnpm and makes DuckDB extension artifacts available to the Flatpak build offline, but it does not prove the final Flatpak package is ready to publish.

1. Run a full Flatpak build
2. Smoke test the built Flatpak
3. Confirm install/bundle workflows still work end to end
4. Wire the generated pnpm, DuckDB source, and query type freshness checks into CI/Flatpak validation if needed

[MattSturgeon]

When experimenting with a nix package, I noticed that download-duckdb-extensions is downloading inputs from an unlocked manifest (duckdb-extensions.json).

Did you need to address this for the flatpak package too?¹ IIRC, flatpak has similar requirements for deterministic dependencies.

Footnotes

1. I don't see anything obvious in the diff ↩

[Sewer56]

Pretty interesting stuff; not sure at which time I'll be able to review this personally- as doing Linux support outside of dev-environment is more further down the road, but I'm all down for it.

[danyloburavlov]

Thank you!

I do agree with Matt's comment and I'll try my best to address it.

Linux support outside of dev-environment is much more further down the road

I hope it won't take too long 😅

[github-actions]

This PR has conflicts. You need to rebase the PR before it can be merged.

[github-actions]

This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed.

[danyloburavlov]

When experimenting with a nix package, I noticed that download-duckdb-extensions is downloading inputs from an unlocked manifest (duckdb-extensions.json).

Did you need to address this for the flatpak package too?1 IIRC, flatpak has similar requirements for deterministic dependencies.

Footnotes

1. I don't see anything obvious in the diff ↩

I've updated this PR with DuckDB sources SHA256 config & verification. There's a couple of documentation updates needed, but I will hold them off until the final approach is decided (so we avoid unnecessary changes and code review).

There's one open question though (see PR description).

[MattSturgeon]

I believe #23551 will also be an issue, as it is generating the API schema during postinstall. /cc @Aragas

Removing src/generated/ from packages/nexus-api-v3/.gitignore and having a scheduled "update schema" CI, instead of regenerating every build, is one way to address that.

Also, I wonder if it'd be easier to land this if the DuckDB stuff was split off into a dedicated PR?

[danyloburavlov]

@MattSturgeon I can split this into 2 PRs (Flatpak + DuckDB), or even 3 (Flatpak + DuckDB + workflow adjustments after proper review/verification of the DuckDB part). Happy to split it however you prefer.

Also, my bad if I’m missing something, but I don’t see how generate-query-types.ts is tied to postinstall. As far as I can tell, the new V3 API codegen target fetches the Nexus API OpenAPI schema, while generate-query-types.ts generates local DuckDB query types from src/queries. Is that the part you meant?

[MattSturgeon]

Happy to split it however you prefer.

I'm not part of the Nexus Mods team, so my opinion doesn't really matter, here. Just throwing suggestions out there 🙂

I don’t see how generate-query-types.ts is tied to postinstall.

I was going off of

Vortex/packages/nexus-api-v3/README.md

Lines 11 to 14 in 1c34e34

	`postinstall` runs `codegen`, so installing always regenerates the bindings
	from the latest spec — the same network you already need to install the
	dependency tree. `build` only runs `tsdown` against that generated output, so
	builds never hit the network and work offline.

(I didn't look too closely beyond that).

EDIT: to be clear, this is unrelated to DuckDB. It's just another impure/non-deterministic build step, that I believe would block reproducible packaging (like flatpak).

[danyloburavlov]

I'm not part of the Nexus Mods team, so my opinion doesn't really matter, here. Just throwing suggestions out there 🙂

Me neither, but any reasonable opinion is welcome 🙂

As for the doc, yes, it makes sense, but it seems to be targeting just V3 package + Nexus API OpenAPI schema, not DuckDB. IDK, that's what I see, let's hear from maintainers too, as soon as they have a chance to reply.

EDIT: Gotcha. Well, sounds like a separate PR for sure.

[github-actions]

This PR has been marked as stale due to inactivity.

[github-actions]

This PR has conflicts. You need to rebase the PR before it can be merged.