[Resume tracker] Add bounded worker recycling for RSS high-water control

[aivong-openhands]

Parent tracker

• [Tracker] Make sandbox resume idempotent and eliminate repeated allocation churn #244

Goal

Bound app-server worker lifetime so Python/glibc RSS high-water marks are released before a worker reaches the container memory limit.

The current workers are long-lived processes. Resource-limit changes can provide headroom and reduce CPU throttling, but they do not return retained allocator arenas to the operating system. Controlled recycling is therefore a separate durability mechanism.

Scope

• Select a production-safe worker manager or supervisor that supports request-bounded worker lifetime.
• Add a configurable maximum requests per worker.
• Add randomized request-count jitter so replicas and workers do not recycle simultaneously.
• Preserve the current worker-count and graceful-shutdown semantics unless load testing justifies a change.
• Ensure a replacement worker becomes ready before the retiring worker stops receiving traffic.
• Set termination grace periods and signal handling so in-flight webhook and API requests can drain.
• Prevent recycling from causing readiness flapping or a synchronized loss of capacity.
• Export per-worker request count, RSS, recycle reason, recycle duration, and graceful/forced termination metrics.
• Add a production-representative soak test covering sustained mixed traffic and repeated allocation bursts.
• Document rollback and emergency-disable controls.

Acceptance criteria

• Worker RSS is periodically reset without restarting the pod or dropping below safe serving capacity.
• Recycling is staggered across workers and replicas.
• In-flight requests drain within the configured grace period.
• A soak test demonstrates bounded long-term RSS under representative traffic.
• Operators can distinguish expected recycling from crashes, OOMKills, and readiness failures.
• The mechanism can be disabled quickly without rebuilding the application image.

This issue was created by an AI agent (OpenHands) on behalf of the user.