Repository navigation
ci(k3d): raise the serverlb nginx worker connections - #1807
Conversation
k3d's serverlb allows 1024 connections per nginx worker, and nearly all connections land on one worker. Each proxied API connection counts twice. The k3d Fixture and Configuration lane peaks at about 1,070 established connections on blacksmith 32vcpu (530-760 on ubuntu-22.04), and nginx drops the rest: kubectl reports 'Unable to connect to the server: EOF' and the Secret driver 'The Kubernetes Secret read failed.' (finding 682). A probe that polled /readyz at the host port, the serverlb and server-0 saw 5-11 serverlb drops in every 32vcpu run while server-0 always answered. Create clusters with --lb-config-override settings.workerConnections=8192.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. What this changesThis PR configures disposable CI Kubernetes clusters to allow 8192 nginx proxy connections per worker and updates the existing preparation fixture. Example: A contributor runs k3d Fixture and Configuration on the 32vcpu runner
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review This PR remains useful: current main lacks the capacity repair, and the linked diagnostic run supports its effectiveness. No actionable patch defect was found; Enterprise policy keeps this PR open for maintainer triage. Priority: P2 Before mergeNone. FindingsNone. Agent review detailsHow this fits togetherEnterprise CI creates disposable Kubernetes clusters through k3d, which places an nginx proxy between test clients and the Kubernetes API server. The shared preparation script configures that proxy before integration tests run. flowchart LR
A[CI integration lane] --> B[Shared cluster preparation]
B --> C[k3d cluster creation]
C --> D[nginx proxy configured for 8192 connections]
E[Test API requests] --> D
D --> F[Kubernetes API server]
Technical reviewBest possible solution: Keep the capacity setting in the existing shared CI preparation owner, without adding a competing cluster setup path or masking proxy saturation with retries. Do we have a high-confidence way to reproduce the issue? Yes, the linked three-hop probe provides a concrete reproduction path under the reported CI load, and current main retains the limiting configuration; this review did not execute it. Is this the best way to solve the issue? Yes, using k3d's supported setting in the shared preparation owner directly addresses the measured proxy capacity limit with minimal scope. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 0f102fb4a304. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Likely related people:
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReview history (1 earlier review cycle)
Reviewed October 9, 2026, 1:06 AM ET / 05:06 UTC (Revision 2). |
Review nit: the one-worker claim is inferred from the drop threshold. Also record the uncapped peak of about 2,000 seen in the probe.
Problem
Since the k3d lanes moved to
blacksmith-32vcpu-ubuntu-2404(#1794), k3d Fixture and Configuration failed on a dropped API connection in 2 of its 18 jobs (main run 37882490730: "The Kubernetes Secret read failed."; #1794 attempt 1:Unable to connect to the server: EOF). Onubuntu-22.04it happened once in 581 jobs over the two days before (finding 682).The API server is not the cause; k3d's serverlb is. A demo probe (#1805, closed unmerged) ran the lane while polling
/readyzevery 200 ms at three hops: the host port, the serverlb and server-0.ubuntu-22.04. The serverlb's nginx allowsworker_connections 1024per worker (k3d's default), and nearly all connections land on one worker. Each proxied connection counts twice. The drops start once a run goes over 1024. Hosted runs stay under it, mostly.Change
prepare.mjscreates every k3d cluster with--lb-config-override settings.workerConnections=8192. Theci-preparefake asserts the new arguments.Verification
node --test tests/integration/ci-prepare.test.mjs: 68/68 pass.Risks
Low. Only the serverlb's nginx connection limit changes (8192 per worker, under the container's 65536 open files). The topology, ports and NetworkPolicy behaviour stay the same.