feat: add --job-context-dir to configure job coordination file locations#4090
Open
aeijdenberg-canva wants to merge 1 commit into
Open
feat: add --job-context-dir to configure job coordination file locations#4090aeijdenberg-canva wants to merge 1 commit into
aeijdenberg-canva wants to merge 1 commit into
Conversation
The agent creates several files to coordinate with the processes running a job: the job env files (BUILDKITE_ENV_FILE / BUILDKITE_ENV_JSON_FILE), the job timeout marker file (BUILDKITE_AGENT_JOB_TIMEOUT_FILE), and, in Kubernetes mode, the socket that kubernetes-bootstrap connects to. The directory for these was previously hardcoded: os.TempDir() normally, and /workspace in Kubernetes mode. This adds a --job-context-dir flag (BUILDKITE_JOB_CONTEXT_DIR) to both `agent start` and `kubernetes-bootstrap`. All coordination files are created in that directory, with the socket at a predictable name within it (buildkite.sock). Only the directory needs to be agreed upon between containers; the env/timeout file paths continue to flow to the other containers through the registration env. Defaults are unchanged (os.TempDir(), or /workspace under --kubernetes-exec), so existing stacks are unaffected. A stack can set BUILDKITE_JOB_CONTEXT_DIR pod-wide to a small dedicated shared volume, and drop the workspace mount from the agent container. The env var is included in kubernetes-bootstrap's existingEnvPriority so that each container's own value survives the registration-env overlay (the shared volume may be mounted at different paths per container), and in protectedEnv so job-level env cannot redirect it. Signed-off-by: Adam Eijdenberg <aeijdenberg@canva.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When running under k8s we would like more control over where the
startcommand writes it's socket and other job information which the other containers need to talk about to ti.Context
The agent creates several files to coordinate with the processes running a job: the job env files (
BUILDKITE_ENV_FILE/BUILDKITE_ENV_JSON_FILE), the job timeout marker file (BUILDKITE_AGENT_JOB_TIMEOUT_FILE), and, in Kubernetes mode, the socket thatkubernetes-bootstrapconnects to. The directory for these was previously hardcoded to/workspacein Kubernetes mode.Changes
This adds a
--job-context-dirflag (BUILDKITE_JOB_CONTEXT_DIR) to bothagent startandkubernetes-bootstrap. All coordination files are created in that directory, with the socket at a predictable name within it (buildkite.sock).Defaults are unchanged (os.TempDir(), or /workspace under --kubernetes-exec).
The env var is included in
kubernetes-bootstrap'sexistingEnvPriorityso that each container's own value survives theregistration-envoverlay (the shared volume may be mounted at different paths per container), and inprotectedEnvso job-level env cannot redirect it.Testing
go test ./...). Buildkite employees may check this if the pipeline has run automatically.go tool gofumpt -extra -w .)Also deployed into our k8s environment with the new var set to a different directory and jobs ran fine.
Affiliation (optional, external contributors)
Canva
Disclosures / Credits