Fix agent connection resets by lowering default ping interval to 30s

[subwaycookiecrunch]

Fixes #26338

Testing done

• Ran the existing ChannelPingerTest suite which covers basic verification that the constants are injected correctly into the Channel setup callbacks.
• Verified that decreasing PING_INTERVAL_SECONDS_DEFAULT to 30 automatically applies at runtime unless manually overridden by the deprecated hudson.slaves.ChannelPinger.pingInterval property.
• Since this change only adjusts a single static default integer responsible for keep-alive intervals, no logic has been altered so automated test coverage already handling channel lifecycle is sufficient.

Screenshots (UI changes only)

Before

After

Proposed changelog entries

• Decrease the default TCP agent connection ChannelPinger interval from 5 minutes to 30 seconds to prevent reverse proxies and load balancers from dropping idle connections.

Proposed changelog category

/label bug

Proposed upgrade guidelines

N/A

Submitter checklist

• The issue, if it exists, is well-described.
• The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
• There is automated testing or an explanation as to why this change has no tests.
• New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
• New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
• UI changes do not introduce regressions when enforcing the current default rules of Content Security Policy Plugin. In particular, new or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
• For dependency updates, there are links to external changelogs and, if possible, full differentials.
• For new APIs and extension points, there is a link to at least one consumer.

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

• There are at least two (2) approvals for the pull request and no outstanding requests for change.
• Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
• Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
• Proper changelog labels are set so that the changelog can be generated automatically.
• If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
• If it would make sense to backport the change to LTS, be a Bug or Improvement, and either the issue or pull request must be labeled as lts-candidate to be considered.

[MarkEWaite]

Please explain how you duplicated the issue described in issue #26338 and why this helps. Changing the timeout is unlikely to alter a connection error.

[subwaycookiecrunch]

Hi @MarkEWaite, thanks for reviewing!

To clarify right away: this PR does not change the connection timeout (PING_TIMEOUT_SECONDS_DEFAULT is intentionally left untouched at 4 minutes). Instead, this PR only changes the ping interval (PING_INTERVAL_SECONDS_DEFAULT) from 5 minutes (300 seconds) down to 30 seconds.

Why this helps and prevents the Connection Error: Issue #26338 is caused by network intermediaries (like AWS NLB or Nginx Ingress) silently dropping the TCP connection due to their own idle timeouts.

For example, Nginx has a default proxy-read-timeout of 60 seconds. If a Jenkins agent is idle, no log or build data is sent. With the old default

ChannelPinger
interval of 5 minutes (300 seconds), the agent remains completely silent on the wire for 5 minutes.

At the 60-second mark of silence, Nginx closes the connection due to inactivity.
At the 300-second mark, the agent finally attempts to send its PingThread$Ping.
Because Nginx already dropped the session, it immediately replies with a TCP RST.
The agent receives this and logs java.io.IOException: Connection reset and SEVERE: Connection error has occurred.
By reducing the ping interval from 300 seconds to 30 seconds, we guarantee that keep-alive traffic is sent frequently enough to prevent these proxies from purging the connection from their state tables.

(Note: this 30-second interval also perfectly aligns with the jenkins.websocket.pingInterval property added in recent Jenkins WebSocket transports, unifying the behavior across protocols).

How it was duplicated: While I relied on the highly detailed packet captures (pcaps) provided by the original reporter in #26338 to verify the exact TCP sequence, this behavior is completely reproducible by placing a Jenkins agent behind an Nginx reverse proxy with the default 60s idle timeout. If you let the agent sit completely idle (no jobs, no monitoring data), the agent will consistently drop with a Connection reset exactly when it tries to send its first packet after the proxy's silence limit.

Let me know if this makes sense or if you prefer this to be handled via a configurable annotation rather than changing the default!

[mawinter69]

I consider this as not needed. It is already possible to adjust the pinginterval seconds via a system property

[subwaycookiecrunch]

@mawinter69 Yeah, it's true you can override it with -Dhudson.slaves.ChannelPinger.pingIntervalSeconds, but I really think the default itself needs to drop.

The 5-minute default made sense back when a lot of Jenkins setups were just raw TCP sockets on a local network, but these days almost everyone is running behind an AWS NLB or a K8s Nginx ingress controller. Those proxies drop idle connections aggressively (nginx is 60s by default).

When that happens, the connection resets silently. So for most new Jenkins setups out-of-the-box, agents just randomly drop with Connection reset errors when they're idle, and the admin has no idea they need to dig up an undocumented system property just to keep them alive.

Also, it looks like jenkins.websocket.pingInterval was recently added and it defaults to 30s for this exact reason. Bumping the TCP pinger down to 30s just makes them consistent so people don't have to tune different properties depending on what transport they use.

[andreahlert]

The explanation for interval vs timeout and the proxy scenario is clear.

One open question: do we want to change the default for all installations, or keep 5 min and document the proxy/idle-timeout issue plus the hudson.slaves.ChannelPinger.pingIntervalSeconds property (e.g. in troubleshooting or release notes)?

Changing the default helps out-of-the-box behind NLB/ingress but increases ping traffic for on-prem/direct-TCP setups. Would be good to align with @MarkEWaite and others on that before merging.

[subwaycookiecrunch]

@andreahlert that's a fair point. I'd argue that changing the default still makes more sense than just documenting the property, mainly because:

• Most people hitting this won't know the system property exists, and the failure mode (silent connection drop after idle period) is really confusing to debug if you don't know what you're looking for.
• The extra ping traffic is minimal , one small keep-alive packet every 30s per agent. Even with hundreds of agents on a direct-TCP setup that's basically nothing.
• The WebSocket transport already defaults to 30s for jenkins.websocket.pingInterval, so having the TCP pinger at 5 minutes is just inconsistent.

That said, I'm totally fine waiting for @MarkEWaite to weigh in before this goes anywhere. If the preference is to keep 5 minutes and just document it better, I can close this and open a docs PR instead , no strong feelings either way, just think the default change is more practical for most setups.

[andreahlert]

@MarkEWaite please let me know if you need help evaluating this. Or if we should close as stale.

[MarkEWaite]

@MarkEWaite please let me know if you need help evaluating this. Or if we should close as stale.

I think that the analysis provided by @subwaycookiecrunch in #26353 (comment) makes sense, but would like to hear comments from @gbhat618, @jglick, or @jtnord to see if they have reasons we should not reduce the default ping interval to better suit the default settings of reverse proxies.

The websocket ping interval has been 30 seconds since it was introduced in pull request:

• [JEP-222] Add experimental WebSocket support + Jetty update in Winstone + Remoting 4.0 #4369

[gbhat618]

Read the details and analysis—I think it makes sense to reduce the interval, so no need of manually setting the pingIntervalSeconds when majority of the LBs have considerably lower idle timeout (such as default 60s) than default here 5m.

These two were made properties (hudson.slaves.ChannelPinger.pingIntervalSeconds and hudson.slaves.ChannelPinger.pingTimeoutSeconds) configurable in seconds in #2645 which were previously in minutes(2016). This shows people needed a way to lower the values.

As also mentioned by @MarkEWaite above — Websocket jenkins.websocket.pingInterval default 30s, and jenkins.websocket.idleTimeout default 60s (time to wait until declaring a single connection as bad).

While reading this, also verified that the timeout > interval won't cause overlapping ping calls when the previous ping call has not returned yet—it's single threaded.

Two points come to mind.

• I don't think there is any concern with increase in network traffic due to reduced default interval—this are just ping so. But a possible concern maybe load on controller that has hundreds of TCP agents - now needs to do 2X hundreds of pings per minute. Would this cause CPU utilization increase in the controller ? But given that websocket has 30s interval controller having large fleet of websocket agents (is that a real world scenario) must be all fine—or the admin must have increased the ping interval. If this seems like a concern we should a load test with say ~500 to 1000 agents concurrent builds perhaps using k8s agents for easy test simulation.
• Though in here setting the hudson.slaves.ChannelPinger.pingIntervalSeconds to 30s makes sense—solves the problem by keeping the connection alive. But should we also take this opportunity to decrease the hudson.slaves.ChannelPinger.pingTimeoutSeconds to 60s (similar to websocket, also nginx default idle timeout) or perhaps at least 90s ? so the dead connections are detected earlier than 4min.

[MarkEWaite]

@samrocketman are any of your large installations running behind a load balancer or other form of reverse proxy? Any opinion based on your experiences with large numbers of agents?

[samrocketman]

@MarkEWaite From a user perspective, my infra runs behind an ALB (aws application load balancer).

If I recall correctly the EC2 plugin I'm using uses SSH from controller to agent so the ALB isn't involved.

I'm not able to look more deeply today but I can take a look at more of the comments here and gain more understanding of this change.

[jtnord]

my memory is a little fuzzy and would need to refresh however I do believe there are differences between the websocket pings and the remoting pings.

Websocket pings are handled at the transport (websocket) layer before it gets to the Remoting layer, whereas the Remoting pings are handled inside the remoting layer (which is deeper) and involves all the intricacies of the remoting protocol stack interaction. The pings aren't sent only for idle connections, they are sent on highly active connections also.

All said and done, I can not see an immediate issue with this. If there are lots of channels that are highly active, adding a few extra remoting calls will probably disappear in the noise, if the channels are idle then if these pings can not be handled then there is little chance that the system could cope with activity when the channels become active.

But if the ALB / nginx is closing the connection it should be sending an RST (possibly a FIN?) so this should be noticed by the agent (and the controller), the Ping should be there only to catch deadlocked software, or zombie connections (closed without notification), so what is either swallowing these in the infrastructure, the description of the capture shows no such reset or closure of the connection - and thus I would say that is an infrastructure bug.

[jglick]

Websocket pings are handled at the transport (websocket) layer before it gets to the Remoting layer

Right, and that is set to 30s

jenkins/core/src/main/java/jenkins/websocket/WebSocketSession.java

Line 61 in 142d36e

private static Duration PING_INTERVAL = SystemProperties.getDuration("jenkins.websocket.pingInterval", ChronoUnit.SECONDS, Duration.ofSeconds(30));

precisely so that your connection is not considered idle by the likes of ingress-nginx. Even if there is no Remoting ping whatsoever.

In CloudBees CI HA controllers we found it necessary to adjust the Remoting ping interval down to 30s but only for the agent → controller pinger, not the controller → agent pinger. CloudBees-internal link and also note jenkinsci/remoting#1079 and #7580.

[jglick]

If I understand correctly your scenario is a non-WebSocket agent being routed by some L4 reverse proxy? That is not a generally recommended configuration, but is used sometimes. But ChannelPinger is designed to verify that the remote side is responsive; it is not intended as a keepalive for the transport. I would rather recommend having some ping Command at the Remoting level that is lighter-weight than callAsync, for example just an empty execute(). That would be awkward to roll out, though, since the agent would need to be running a current version of Remoting, which means a new capability. Pending that, changing the ChannelPinger default for a TCP agent to 30s would be reasonable but only if you knew it was being proxied somehow; CLI_HOST_NAME and/or CLI_PORT being set offer a hint but are not reliable. Really I think you just need to set the system property in this case.

[samrocketman]

In general, I agree with @jglick you should probably tune the property in your setup rather than changing the default here.

There's two competing thoughts:

• Jenkins should have sane defaults so that most people don't need to tune.
• For extreme setups I feel like tuning Jenkins is inevitable and sane defaults for extreme cases may not necessarily be good for everyone.

For example, the controller I operate is tuned with 16 JVM arguments and 5 Jenkins properties. A lot of the things in my configuration are not suitable as sane defaults for others (e.g. my configuration can be more expensive in cloud computing but the company would rather have Jenkins fast/expensive than slower/cheaper YMMV).