Skip to content

ci(k3d): raise the serverlb nginx worker connections - #1807

Merged
freeqaz merged 2 commits into
mainfrom
flake-682b
Oct 9, 2026
Merged

freeqaz merged 2 commits into
mainfrom
flake-682b

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Since the k3d lanes moved to blacksmith-32vcpu-ubuntu-2404 (#1794), k3d Fixture and Configuration failed on a dropped API connection in 2 of its 18 jobs (main run 37882490730: "The Kubernetes Secret read failed."; #1794 attempt 1: Unable to connect to the server: EOF). On ubuntu-22.04 it happened once in 581 jobs over the two days before (finding 682).

The API server is not the cause; k3d's serverlb is. A demo probe (#1805, closed unmerged) ran the lane while polling /readyz every 200 ms at three hops: the host port, the serverlb and server-0.

  • server-0 answered every poll in every run.
  • The serverlb and the host port closed connections during the TLS handshake (curl exit 35) 5-11 times per 32vcpu run, all 13-24 s after the tests started. The lane's own retries usually hid them.
  • The serverlb's established connections peaked at 1,050-1,078 on 32vcpu and at 532-756 on ubuntu-22.04. The serverlb's nginx allows worker_connections 1024 per worker (k3d's default), and nearly all connections land on one worker. Each proxied connection counts twice. The drops start once a run goes over 1024. Hosted runs stay under it, mostly.
  • Open-file limits (65536), memory and load were not the limit.

Change

prepare.mjs creates every k3d cluster with --lb-config-override settings.workerConnections=8192. The ci-prepare fake asserts the new arguments.

Verification

  • node --test tests/integration/ci-prepare.test.mjs: 68/68 pass.
  • CI attempts on this head: see below.

Risks

Low. Only the serverlb's nginx connection limit changes (8192 per worker, under the container's 65536 open files). The topology, ports and NetworkPolicy behaviour stay the same.

k3d's serverlb allows 1024 connections per nginx worker, and nearly all
connections land on one worker. Each proxied API connection counts twice.
The k3d Fixture and Configuration lane peaks at about 1,070 established
connections on blacksmith 32vcpu (530-760 on ubuntu-22.04), and nginx
drops the rest: kubectl reports 'Unable to connect to the server: EOF'
and the Secret driver 'The Kubernetes Secret read failed.' (finding 682).
A probe that polled /readyz at the host port, the serverlb and server-0
saw 5-11 serverlb drops in every 32vcpu run while server-0 always answered.

Create clusters with --lb-config-override settings.workerConnections=8192.
@freeqaz
freeqaz requested a review from a team as a code owner October 9, 2026 04:49
@clawsweeper

clawsweeper Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Oct 9, 2026
@clawsweeper

clawsweeper Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

What this changes

This PR configures disposable CI Kubernetes clusters to allow 8192 nginx proxy connections per worker and updates the existing preparation fixture.

Example: A contributor runs k3d Fixture and Configuration on the 32vcpu runner

  • Before: The reported connection load can exceed the proxy limit and produce “Unable to connect to the server: EOF”.
  • After: The after-fix probe reports no mid-test connection drops across eight jobs, including connection peaks up to 1974.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, correct infrastructure repair supported by real before-and-after cluster observations.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The linked diagnostic workflow runs the real shared preparation CLI and k3d fixture lane on 32vcpu and hosted runners. Its after-fix report records nginx at 8192, connection peaks above the former limit, and no mid-test drops in eight jobs; the probe source contains the same runtime change as this head. Raw log access was blocked for the reviewer, but the readable observed-results report and verified execution resolve the prior confidence gap.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Contributors intermittently see Kubernetes integration checks fail when the test cluster closes API connections.
Reason: The reported failures have a measured infrastructure bottleneck, and the small repair preserves the existing CI resource owner. Prior merged area history establishes owner context for this direction.

Merge readiness

✅ Ready for maintainer review

This PR remains useful: current main lacks the capacity repair, and the linked diagnostic run supports its effectiveness. No actionable patch defect was found; Enterprise policy keeps this PR open for maintainer triage.

Priority: P2
Reviewed head: c34e6d261a358e3b254be5da0fb476752bbc0b32

Before merge

None.

Findings

None.

Agent review details

How this fits together

Enterprise CI creates disposable Kubernetes clusters through k3d, which places an nginx proxy between test clients and the Kubernetes API server. The shared preparation script configures that proxy before integration tests run.

flowchart LR
  A[CI integration lane] --> B[Shared cluster preparation]
  B --> C[k3d cluster creation]
  C --> D[nginx proxy configured for 8192 connections]
  E[Test API requests] --> D
  D --> F[Kubernetes API server]
Loading

Technical review

Best possible solution:

Keep the capacity setting in the existing shared CI preparation owner, without adding a competing cluster setup path or masking proxy saturation with retries.

Do we have a high-confidence way to reproduce the issue?

Yes, the linked three-hop probe provides a concrete reproduction path under the reported CI load, and current main retains the limiting configuration; this review did not execute it.

Is this the best way to solve the issue?

Yes, using k3d's supported setting in the shared preparation owner directly addresses the measured proxy capacity limit with minimal scope.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 0f102fb4a304.

Provenance checked

  • scripts/ci/prepare.mjs shared cluster creation keeps the original intent (d353d52: The introducing lane architecture centralizes disposable integration resources and enforces preparation, execution, and cleanup.)
  • tests/integration/ci-prepare.test.mjs cluster creation fixture keeps the original intent (Rename experimental Backends and add k3d startup diagnostics #341: Exercise startup diagnostics and preserve cluster ownership and failure cleanup through simulated external infrastructure.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

  • Introduced change: The pinned introduction delta adds the supported load-balancer override to the existing shared cluster creation command; topology, loopback binding, kubeconfig isolation, and cleanup remain intact. (scripts/ci/prepare.mjs:1117, c34e6d261a35)
  • Current main still needs the repair: Neither touched file changed between the merge base and fetched main, which still creates clusters without this override. No implementation closure or shipping release is established. (scripts/ci/prepare.mjs:1115, 0f102fb4a304)
  • Dependency configuration contract: The patch directly invokes k3d's configuration API. Enterprise pins version 5.8.3; that version accepts dotted configuration overrides, defines workerConnections as an integer with a baseline default of 1024, merges the override into proxy settings, and renders it into nginx worker_connections. (pkg/types/loadbalancer.go:76, 201b614ea679)
  • Observed before-and-after behavior: ci(demo): probe k3d api reachability on blacksmith 32vcpu #1805 (comment) reports 5–11 pre-fix proxy drops per 32vcpu run while server-0 stayed reachable. After applying this setting, run https://github.com/openclaw/openclaw-enterprise/actions/runs/37885672598 used worker_connections 8192, reached 1340–1974 established connections on 32vcpu, and had no mid-test drops in eight jobs. (194957bf1234)
  • Probe identity and execution: The completed probe revision contains the same override and 8192 constant. Its workflow exercises the real shared CI lane while polling host, proxy, and server /readyz endpoints; all eight jobs completed successfully. The current head changes only the explanatory comment from the earlier reviewed implementation. Raw job logs redirect to a host blocked by reviewer network policy; the readable result report and source identity remain available. (scripts/ci/prepare.mjs:1117, 194957bf1234)
  • Preparation ownership and intent: Feature-history search locates the shared preparation owner at d353d52. feat(ci): run explicit PR and protected integration test lanes #23 describes centrally owned disposable resources, explicit lane execution, and cleanup; this PR preserves that intent. (scripts/ci/prepare.mjs:1046, d353d527feb7)

Likely related people:

  • freeqaz: Raw commit 2ba1727 adds scripts/ci/prepare.mjs:1099 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 2ba172761637; files: scripts/ci/prepare.mjs)

Labels

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This addresses observed integration-lane connection failures with a bounded CI infrastructure change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only; repair and merge need explicit maintainer commands such as @clawsweeper autofix or @clawsweeper automerge.

History

Review history (1 earlier review cycle)
  • reviewed 2026-10-09T04:52:19.025Z sha 4533940 :: needs maintainer review before merge. :: none

Reviewed October 9, 2026, 1:06 AM ET / 05:06 UTC (Revision 2).

Review nit: the one-worker claim is inferred from the drop threshold.
Also record the uncapped peak of about 2,000 seen in the probe.
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Oct 9, 2026
@freeqaz
freeqaz merged commit b18e86c into main Oct 9, 2026
174 of 180 checks passed
@freeqaz
freeqaz deleted the flake-682b branch October 9, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant