[gjedlicska]

Why

The internal integration job gets viewer-.dat generation from speckle-server's Docker stand-in (CONVERSION_RUNNER_LOCAL_DEV): the server runs the converter image through the mounted Docker socket. The next speckle-server:latest (the job-system branch, feat/job-system) drops the stand-in. Its only off-cluster backend is a real cluster reached through JOB_SYSTEM_KUBECONFIG. Without a cluster, that server cannot generate .dat files under compose.

What

• scripts/integration-cluster.sh: one script for CI and local runs.
  • up (idempotent): creates a single-node kind cluster, speckle-sdk-integration by default (KIND_CLUSTER_NAME overrides it). The node is labelled purpose=conversion, the namespace is conversion, and the node image is pinned by digest to kind v0.31.0's default. It writes kind get kubeconfig --internal to the gitignored .kind/kubeconfig.
  • load-image: loads the converter image into the node.
  • env: prints the compose interpolation values.
  • check: a pod proves it can reach the server and garage.
  • status: dumps Jobs, pods and events.
  • down: deletes the cluster.
  • The script never touches ~/.kube/config.
• docker-compose-internal.yml:
  • speckle-server joins the external kind network. It mounts the internal kubeconfig with create_host_path: false, so a missing file fails loudly.
  • It sets JOB_SYSTEM_KUBECONFIG, CONVERSION_JOB_NAMESPACE, CONVERSION_NODE_POOL, CONVERSION_IMAGE_PULL_POLICY=IfNotPresent and FILEIMPORT_SERVICE_USE_PRIVATE_OBJECTS_SERVER_URL + PRIVATE_OBJECTS_SERVER_URL.
  • DATGEN_BASE_MEMORY_MIB drops to 1024. Requests are hard reservations on the one node that the stack and the test run share, and an OOM still climbs the ×2 ladder.
  • S3_PUBLIC_ENDPOINT and PRIVATE_OBJECTS_SERVER_URL use ${KIND_HOST_ADDRESS}. That is the kind network gateway on Linux, or host.docker.internal on Docker Desktop. Presigned upload URLs embed S3_PUBLIC_ENDPOINT, and Job pods and the host-side C# tests consume the same URLs, so one address has to work from both sides.
  • Garage is now published on ${KIND_PUBLISH_ADDRESS} (the gateway, or 127.0.0.1 on Docker Desktop). The server keeps 0.0.0.0:3000.
  • Both variables are :?-required, so compose fails with a pointer to the script instead of silently misrouting.
• Workflows (integration-test.yml for the internal image only, and the callable workflow):
  • install kind v0.31.0 with helm/kind-action@v1 (install_only);
  • run up and append env to $GITHUB_ENV before the background pulls;
  • after the build, pull the converter on the host as before and load-image it;
  • on failure, dump the cluster state and the server log.

Transition and when to drop it

Today's speckle-server:latest (server main) knows only CONVERSION_RUNNER_LOCAL_DEV and ignores JOB_SYSTEM_KUBECONFIG. The next one ignores CONVERSION_RUNNER_LOCAL_DEV. Both sets of settings stay in for now: the socket mount, group_add and CONVERSION_RUNNER_LOCAL_DEV, plus the new kind settings. That way .dat generation works with either image. Drop the socket mount, group_add, DOCKER_GID and CONVERSION_RUNNER_LOCAL_DEV once a speckle-server:latest without the stand-in is published. A comment in the compose file says the same.

The old stand-in still works with the new addresses:

• It runs the converter with host networking and rewrites only loopback SERVER_URL hosts. The gateway URLs pass through unchanged, and a host-networked container reaches a gateway-published port (smoke below).
• The host-networked bundle-migration-service reaches the gateway S3 URLs the same way.
• The converter is still pulled on the host, which the stand-in needs and kind load needs too.

Local flow

This is in the README:

./scripts/integration-cluster.sh up
./scripts/integration-cluster.sh load-image
./scripts/integration-cluster.sh env > .kind/compose.env
DOCKER_GID=$(stat -c %g /var/run/docker.sock) docker compose --env-file .kind/compose.env -f docker-compose-internal.yml up --wait
./scripts/integration-cluster.sh check

Obsolete branch

gergo/job-system-local-dev-env (adds JOB_SYSTEM_LOCAL_DEV) is obsolete: that switch never ships, because the job system's only off-cluster backend is the kind cluster. This PR does not touch that branch; it can be closed.

Server side after merge

speckle-server-internal's .github/workflows/manual-trigger-sharp-sdk-integration-tests.yml pins this callable workflow at b278c769… (Aug 2026), but it checks out the SDK at speckle-sharp-sdk-ref: 'main'. After this merges, the pinned workflow would run with the new compose file and no kind cluster, so compose fails on the missing KIND_* variables. The pin must be bumped to this PR's merge commit.

Smoke (local, throwaway cluster sdk-kind-smoke, deleted afterwards)

• KIND_CLUSTER_NAME=sdk-kind-smoke ./scripts/integration-cluster.sh up:
  • created the cluster, namespace/conversion created, exit 0;
  • a second run printed cluster sdk-kind-smoke already exists / namespace/conversion unchanged;
  • get nodes -L purpose showed PURPOSE conversion;
  • the kubeconfig server: is https://sdk-kind-smoke-control-plane:6443, mode -rw-r--r--;
  • ~/.kube/config was untouched.
• env printed KIND_HOST_ADDRESS=172.19.0.1 / KIND_PUBLISH_ADDRESS=172.19.0.1.
• docker compose -f docker-compose-internal.yml config:
  • exit 0 with those values. The output has garage host_ip: 172.19.0.1, S3_PUBLIC_ENDPOINT: http://172.19.0.1:9000, PRIVATE_OBJECTS_SERVER_URL: http://172.19.0.1:3000, the kubeconfig bind and kind: external: true.
  • Without the values it exits 1 with required variable KIND_PUBLISH_ADDRESS is missing a value: run scripts/integration-cluster.sh up, then export its env.
  • The --env-file .kind/compose.env form also validates.
• docker run --rm --network kind --user 65532:65532 -v .kind/kubeconfig:/kind/kubeconfig:ro registry.k8s.io/kubectl:v1.35.0 --kubeconfig /kind/kubeconfig get ns listed conversion, default, kube-* and local-path-storage. It ran under the server image's uid.
• A throwaway busybox httpd on its own docker network, published on 172.19.0.1:18090 (the same shape as garage), was reached from all three places, each printing hello-from-gateway-publish:
  • a pod pinned to purpose=conversion (wget);
  • the host (curl);
  • a --network host container (the old stand-in's mode).
• check correctly reported FAIL … server at 172.19.0.1:3000 (nothing listening) and ok … garage at 172.19.0.1:9000, then exited 1.
• load-image with a small image landed in the node's crictl images.
• down deleted the cluster.

Not run locally: the full compose stack and the C# suite. Ports 9000 and 6379 on this machine are held by another dev stack. This PR's CI is the end-to-end check.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.71%. Comparing base (9908ab9) to head (3c7096f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #594      +/-   ##
==========================================
- Coverage   75.71%   75.71%   -0.01%     
==========================================
  Files         483      483              
  Lines       27028    27028              
  Branches     2724     2724              
==========================================
- Hits        20464    20463       -1     
  Misses       5586     5586              
- Partials      978      979       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.