Repository navigation
Conversation
Demo only, not for merge. Runs the k3d Fixture and Configuration lane twelve times and records host, serverlb and server-0 reachability with K3s and serverlb logs, to locate the API EOFs of finding 682.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs changes before merge. What this changesThis PR runs eight Kubernetes fixture jobs across Blacksmith and GitHub-hosted runners while collecting API reachability transitions and infrastructure diagnostics. Review scores
ProductKind: Maintenance · Worth it: Yes Merge readiness⛔ Needs changes before merge - 1 item remains Keep open for maintainer triage: this is a useful, temporary investigation of an unresolved CI infrastructure failure, rather than a root-cause repair. Priority: P3 Before merge
FindingsNone. Tests
Agent review detailsHow this fits togetherGitHub Actions runs platform integration tests against disposable Kubernetes clusters created with k3d. This diagnostic workflow observes connections through the host port, cluster load balancer, and Kubernetes server to help locate dropped connections. flowchart TD
A[Pull request update] --> B[Four jobs per runner type]
B --> C[Existing Kubernetes fixture lane]
C --> D[Disposable Kubernetes cluster]
D --> E[Watch three API paths]
D --> F[Collect infrastructure logs]
E --> G[Diagnostic job report]
F --> G
Technical reviewBest possible solution: Use the comparative diagnostics to locate the connection-drop cause, then address it in a separate focused change without retaining this temporary workflow. Do we have a high-confidence way to reproduce the issue? Not applicable to a diagnostic addition; related merged work documents actual EOF failures, but this review did not reproduce their cause. Is this the best way to solve the issue? Yes, as a temporary investigation: comparing the existing lane across runner types with observations at three network hops is focused and preserves test coverage. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against fc7eb2f7a93a. TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Review metrics
LabelsLabel changes: No label changes. Label justifications:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReview history (1 earlier review cycle)
Reviewed October 9, 2026, 12:45 AM ET / 04:45 UTC (Revision 2). |
Adds ubuntu-22.04 copies and records the serverlb's open-file limits, nginx worker settings, established connections and full log.
k3d's serverlb allows 1024 connections per nginx worker, and nearly all connections land on one worker. Each proxied API connection counts twice. The k3d Fixture and Configuration lane peaks at about 1,070 established connections on blacksmith 32vcpu (530-760 on ubuntu-22.04), and nginx drops the rest: kubectl reports 'Unable to connect to the server: EOF' and the Secret driver 'The Kubernetes Secret read failed.' (finding 682). A probe that polled /readyz at the host port, the serverlb and server-0 saw 5-11 serverlb drops in every 32vcpu run while server-0 always answered. Create clusters with --lb-config-override settings.workerConnections=8192.
|
Demo done, closing unmerged. Findings: without the fix, the k3d serverlb dropped connections 5-11 times per 32vcpu run while its established connections peaked at 1050-1078 (over nginx's 1024 per worker); hosted peaked at 532-756 with no drops; server-0 always answered. With #1807 merged in (run 37885672598), worker_connections was 8192, peaks were 1340-1974 on 32vcpu and 494-772 on hosted, and no mid-test drops occurred in 8/8 jobs. |
Problem
Finding 682: the k3d API server sometimes drops connections (
Unable to connect to the server: EOF, "The Kubernetes API was unreachable."). Since the k3d lanes moved to blacksmith 32vcpu, k3d Fixture and Configuration failed this way in 2 of 18 jobs, against 1 of 581 onubuntu-22.04over the two days before.Change
Demo only, do not merge. A separate workflow runs the k3d Fixture and Configuration lane twelve times on
blacksmith-32vcpu-ubuntu-2404with a watcher that records/readyzreachability at the host port, the serverlb and server-0, plus K3s, serverlb, Docker event and kernel logs.Verification
The probe jobs themselves.
Risks
None; closed unmerged after the data is collected.