Embed the Dang runtime into generated modules - #24
Open
eunomie wants to merge 4 commits into
Open
Conversation
The runtime reads its own module source in exactly three places: the two Dockerfile image pins and the runtime.py entrypoint script. Gather them, plus the imageFrom helper they use, into one marker-delimited block, and write the entrypoint from its contents instead of mounting the source file. The SDK's generator can then embed this runtime into a generated module by replacing the block with literals, leaving a fully self-contained file. Signed-off-by: Yves Brissaud <yves@dagger.io>
…toml Generation now assembles the module runtime into a single file and adds it to every dagger-module.toml module's changeset: runtime/main.dang with its externals block replaced by literals — the image pins extracted from the runtime's Dockerfiles and the entrypoint script inlined as a string. Assembly fails loudly when the runtime source grows a module-source read outside the fenced block, so the emitted file stays self-contained by construction. The engine reads the file straight from the module root and evaluates it in-process, without resolving a runtime module. The authoring module's source needs runtime/ back in its include list; the exclusion predates the vendored library's move to sdk/, and the runtime directory is five small files today. Signed-off-by: Yves Brissaud <yves@dagger.io>
dagger module init now records embed:runtime.dang as the module's runtime source. The engine reads the named file from the directory holding dagger-module.toml — where generate emits it — and evaluates it in-process with its native Dang interpreter, the fastest runtime-loading path there is: no runtime module is resolved, fetched or loaded, and the unpinned git-ref exposure that pointing at this repository's runtime module would have carried never materializes. Engines that predate the embed runtime source reject it, so this flip must not ship before an engine release understands the form. Signed-off-by: Yves Brissaud <yves@dagger.io>
tomlGenerateCheck asserts what generation emits: runtime.dang lands next to dagger-module.toml, declares moduleRuntime, carries the generated marker, the pinned images and the inlined entrypoint script, and reads no module source. embedRuntimeCallCheck rebases the runtime fixture onto the embed form with a freshly generated runtime.dang and drives a call through a released CLI; it stays red until a released engine understands the embed runtime source, and its docstring says so. Signed-off-by: Yves Brissaud <yves@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modules created by this SDK no longer point at a separate runtime module:
dagger generatenow emits a self-containedruntime.dangnext todagger-module.toml, and new modules record[runtime] source = "embed:runtime.dang". An embed-aware engine reads that file straight from the module's own directory and evaluates it in-process with its native Dang interpreter — no runtime module is resolved, fetched, or loaded. This is the fastest runtime-loading path available, and it removes the unpinned-git-ref exposure that pointing modules at this repository's runtime module would have carried (seefuture/done/self-contained-python-sdk.md, which this supersedes as "PR 2").How it works
runtime/main.dangstays the single source of truth. Its only reads of its own module source — the two Dockerfile image pins and theruntime.pyentrypoint script — are now fenced between#<externals>/#</externals>markers (first commit, behavior-neutral: the module-ref runtime path still passes its e2e checks).Mod.generateassemblesruntime.dangby splicing that block with literals: the image refs extracted from the Dockerfiles, and the entrypoint script inlined as a string. Assembly raises if the result still containscurrentModule, so the emitted file is self-contained by construction — if the runtime ever grows a module-source read outside the fence, generation fails loudly instead of emitting a broken file.sdk/, for everydagger-module.tomlmodule. Legacydagger.jsonmodules are untouched.targetRuntimeflips from"python"to"embed:runtime.dang"in an isolated commit, so it can be reverted independently if sequencing requires.Tests
toml-generate-checknow asserts the emitted file: placed next todagger-module.toml, carries the generated-file marker, declarespub moduleRuntime(, contains the image pins and the inlined entrypoint, and contains nocurrentModule.embed-runtime-call-checkgrafts a freshly generatedruntime.dangplus the embed-form config onto the runtime fixture and drivesdagger callthrough a released CLI.module initwrites the embed source, the generatedruntime.dangloads, anddagger callsucceeds through the embed path with no external runtime-module resolution in the trace.Depends on engine PR: dagger/dagger#14018
Released engines (≤ v1.0.0-beta.11) reject the new runtime source with
invalid SDK: "embed:runtime.dang". Until an engine release understands the embed form, 18 checks are red, all with exactly that root cause:e-2-e:embed-runtime-call-check(fails at the released engine's SDK loader, proving the plumbing up to it)sdk-sdkchecks that init or load a scaffolded module through the released CLI:chain:*(3),generation:succeeds,generation:respects-cwd,init:*(4),module:loads,module:deps-list-succeeds,module:engine-required-reports-version,monorepo:*(5)The remaining 31 checks are green, including the full e2e suite and the legacy-path regression net (
generate-check,generate-all-check,runtime-call-check,runtime-requires-generated-files-check,sdk-test-check).