perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops#2208
Conversation
The sandbox tunnel added 4-5ms of latency to every small request/response because no socket in the path disabled Nagle's algorithm, so sub-MSS writes waited on delayed ACKs at each hop. Set TCP_NODELAY on every latency-sensitive TCP socket: - gateway: accepted connections on the public listener (gRPC relay frames and WS tunnel writes) - CLI: edge tunnel local accept + underlying WebSocket TCP stream, insecure TLS connector (tonic's default connector already does this), and service-forward accepted sockets - supervisor: direct-tcpip connect into the sandbox netns, egress proxy accepted connections, and all upstream CONNECT/HTTP dials (via a new connect_upstream helper) The sandbox SSH transport rides a unix domain socket and gRPC client channels use tonic defaults (nodelay on), so no change is needed there. Fixes MEY-33 Signed-off-by: Jim Meyer <jim@meyer4hire.com>
purp
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
|
All contributors have signed the DCO ✍️ ✅ |
|
Converting to draft. Found a thing while benchmarking last night and left Fable working on it ... and it got a bit excited and sent up a PR. |
|
I have read the DCO document and I hereby sign the DCO. |
|
I'm going to kill this PR for now and open an issue instead. The underlying issue was revealed as I was doing some benchmarking yesterday as part of my learning and understanding process. I think it's a real issue and the fix doesn't look bad, but it feels like it should start as an issue for some discussion that might shape the approach differently. I also want to benchmark this fix and feel confident that it's a worthwhile improvement. Apologies for the noise. Fable has a couple of new memories now and is busily reviewing its approach as well as the code. 🤖🧠 |
1 participant
Summary
The sandbox tunnel added 4–5ms of latency to every small request/response because no socket in the path disabled Nagle's algorithm: each sub-MSS write waited on delayed ACKs at every TCP hop. This PR sets
TCP_NODELAYon all latency-sensitive sockets in the tunnel and egress-proxy paths.Changes
openshell-server): set nodelay on connections accepted by the public listener — covers gRPC relay frames and WS tunnel writesopenshell-cli): edge tunnel local accept loop and the WebSocket's underlying TCP stream;InsecureTlsConnector(matching tonic's default connector behavior);service_forward_tcpaccepted socketsopenshell-supervisor-process):connect_in_netns— the direct-tcpip final hop to services inside the sandbox — on both the netns and direct branchesopenshell-supervisor-network): accepted sandbox connections, plus all five upstream dial sites consolidated into a newconnect_upstreamhelper that sets nodelayTesting
mise run pre-commitpassesconnect_in_netns_sets_tcp_nodelay,connect_upstream_sets_tcp_nodelay)mise run e2e:dockersmoke passes against a standalone Docker-backed gatewayChecklist