Skip to content

ci(demo): probe k3d api reachability on blacksmith 32vcpu - #1805

Closed
freeqaz wants to merge 4 commits into
mainfrom
flake-682b-probe
Closed

freeqaz wants to merge 4 commits into
mainfrom
flake-682b-probe

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

Finding 682: the k3d API server sometimes drops connections (Unable to connect to the server: EOF, "The Kubernetes API was unreachable."). Since the k3d lanes moved to blacksmith 32vcpu, k3d Fixture and Configuration failed this way in 2 of 18 jobs, against 1 of 581 on ubuntu-22.04 over the two days before.

Change

Demo only, do not merge. A separate workflow runs the k3d Fixture and Configuration lane twelve times on blacksmith-32vcpu-ubuntu-2404 with a watcher that records /readyz reachability at the host port, the serverlb and server-0, plus K3s, serverlb, Docker event and kernel logs.

Verification

The probe jobs themselves.

Risks

None; closed unmerged after the data is collected.

Demo only, not for merge. Runs the k3d Fixture and Configuration lane
twelve times and records host, serverlb and server-0 reachability with
K3s and serverlb logs, to locate the API EOFs of finding 682.
@clawsweeper

clawsweeper Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Oct 9, 2026
@clawsweeper

clawsweeper Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

What this changes

This PR runs eight Kubernetes fixture jobs across Blacksmith and GitHub-hosted runners while collecting API reachability transitions and infrastructure diagnostics.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The focused experiment has real hosted execution, while confidence in the collected diagnostics remains limited.
Proof confidence 🦐 gold shrimp (3/6) Under the supplied MEMBER classification, the external-contributor gate does not apply. Exact-head GitHub Actions jobs exercised the workflow and completed its watcher-report step, but diagnostic contents remain unverified because this reviewer cannot access the redirected logs.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Maintenance · Worth it: Yes
User problem: Contributors see Kubernetes fixture checks fail intermittently when the cluster API drops connections.
Reason: A bounded experiment can distinguish runner-specific failures from Kubernetes infrastructure failures. The existing CI area contributor explicitly limits it to temporary data collection.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep open for maintainer triage: this is a useful, temporary investigation of an unresolved CI infrastructure failure, rather than a root-cause repair.

Priority: P3
Reviewed head: b9307e9d30f18cbea36df3531a5f8fed9212b64c

Before merge

  • Complete next step - Keep this diagnostic PR unmerged, as explicitly requested by its author.

Findings

None.

Tests

  • Missing end-to-end proof: The actual three-hop watcher output and infrastructure-log contents could not be inspected because hosted job-log downloads were blocked.
Agent review details

How this fits together

GitHub Actions runs platform integration tests against disposable Kubernetes clusters created with k3d. This diagnostic workflow observes connections through the host port, cluster load balancer, and Kubernetes server to help locate dropped connections.

flowchart TD
  A[Pull request update] --> B[Four jobs per runner type]
  B --> C[Existing Kubernetes fixture lane]
  C --> D[Disposable Kubernetes cluster]
  D --> E[Watch three API paths]
  D --> F[Collect infrastructure logs]
  E --> G[Diagnostic job report]
  F --> G
Loading

Technical review

Best possible solution:

Use the comparative diagnostics to locate the connection-drop cause, then address it in a separate focused change without retaining this temporary workflow.

Do we have a high-confidence way to reproduce the issue?

Not applicable to a diagnostic addition; related merged work documents actual EOF failures, but this review did not reproduce their cause.

Is this the best way to solve the issue?

Yes, as a temporary investigation: comparing the existing lane across runner types with observations at three network hops is focused and preserves test coverage.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against fc7eb2f7a93a.

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

Review metrics

Metric Value Why it matters
Diagnostic matrix 8 jobs: 4 Blacksmith, 4 GitHub-hosted The revised experiment compares runner types using the same fixture lane.

Labels

Label changes:

No label changes.

Label justifications:

  • P3: This is a bounded CI diagnostic experiment with no change to shipped product behavior.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Inspect completed reports to confirm capture of all three API paths and the infrastructure logs.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only; repair and merge need explicit maintainer commands such as @clawsweeper autofix or @clawsweeper automerge.

History

Review history (1 earlier review cycle)
  • reviewed 2026-10-09T04:39:15.027Z sha a1e96df :: needs changes before merge. :: none

Reviewed October 9, 2026, 12:45 AM ET / 04:45 UTC (Revision 2).

Adds ubuntu-22.04 copies and records the serverlb's open-file limits,
nginx worker settings, established connections and full log.
k3d's serverlb allows 1024 connections per nginx worker, and nearly all
connections land on one worker. Each proxied API connection counts twice.
The k3d Fixture and Configuration lane peaks at about 1,070 established
connections on blacksmith 32vcpu (530-760 on ubuntu-22.04), and nginx
drops the rest: kubectl reports 'Unable to connect to the server: EOF'
and the Secret driver 'The Kubernetes Secret read failed.' (finding 682).
A probe that polled /readyz at the host port, the serverlb and server-0
saw 5-11 serverlb drops in every 32vcpu run while server-0 always answered.

Create clusters with --lb-config-override settings.workerConnections=8192.
@freeqaz

freeqaz commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Demo done, closing unmerged. Findings: without the fix, the k3d serverlb dropped connections 5-11 times per 32vcpu run while its established connections peaked at 1050-1078 (over nginx's 1024 per worker); hosted peaked at 532-756 with no drops; server-0 always answered. With #1807 merged in (run 37885672598), worker_connections was 8192, peaks were 1340-1974 on 32vcpu and 494-772 on hosted, and no mid-test drops occurred in 8/8 jobs.

@freeqaz freeqaz closed this Oct 9, 2026
@freeqaz
freeqaz deleted the flake-682b-probe branch October 9, 2026 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant