feat: add ALB sticky table support

.spelling

@@ -262,3 +262,4 @@ worstCase
 OpenMetrics
 backfill
 backfilled
+keep-alives

deploy/kube/configmap.yaml

@@ -1179,6 +1179,12 @@ data:
     #       #   - foo-01.example.com
     #       #   - name: foo-02.example.com
     #       #     backup: true
+    #       #
+    #       # a draining entry takes no new work, but keeps the sticky sessions it already has. e.g.:
+    #       # pool:
+    #       #   - foo-01.example.com
+    #       #   - name: foo-02.example.com
+    #       #     drain: true
 
     #       # healthy_floor is the minimum health status for a Backend to be considered healthy in the pool
     #       #  1 indicates only backends positively reporting as healthy are included
@@ -1224,6 +1230,43 @@ data:
     #         # first byte sent to the client), connect on tcp and tls (or first_byte), first_reply on udp
     #         signal: first_write
 
+    #       sticky: # keeps each client on the member it was first sent to; rr, p2c, lc, lt and hrw only
+    #         # mode is how a client's member is remembered: cookie (a signed token in a cookie),
+    #         # header (the same token in a header, for API clients) or table (a table kept by the ALB,
+    #         # by a key read from each request, connection or session). Unset, the default, is cookie
+    #         # on http listeners and table on tcp, tls, udp and native ones, and allows a cookie and a
+    #         # table block; a mode that is set allows only its own block, and cookie or header is
+    #         # refused on a listener that is not http
+    #         # mode: cookie
+    #         # ttl is a session's lifetime from its first request; 0 is no limit, and a browser-session
+    #         # cookie. default is 1h
+    #         ttl: 1h
+    #         # idle ends a session left unused. A token counts it from when it was last issued and
+    #         # is reissued after half of it, so an unused session ends between idle/2 and idle after its
+    #         # last request; a table entry counts it from its last use. default is 0: never
+    #         idle: 0
+    #         # on_unavailable is repick (default), to move a session whose member is unavailable, or
+    #         # reject, to answer 503, or refuse a connection, until its member is available again
+    #         on_unavailable: repick
+    #         # secret signs cookie and header tokens: at least 32 bytes, the same on every replica.
+    #         # secret_file reads it from a file instead. with neither, a random per-process key is used
+    #         # secret_file: /etc/trickster/sticky.key
+    #         cookie:
+    #           name: trickster_sticky  # default
+    #           path: /                 # default
+    #           secure: auto            # auto (default) marks it Secure over TLS; or true, false
+    #           http_only: true         # default
+    #           same_site: lax          # lax (default), strict or none (requires secure: true)
+    #           lifetime: permanent     # permanent (default) sets Max-Age from the token; session never does
+    #           mark_private: false     # true adds Cache-Control: private to a response that sets it
+    #         # header: # header mode only
+    #         #   name: X-Trickster-Session
+    #         # table: # table mode only
+    #         #   key: client_ip        # any hrw.key value the ALB's listeners can read
+    #         #   learn: request        # response also remembers a header:/cookie: value a member sets
+    #         #   ipv6_prefix: 64
+    #         #   max_entries: 100000
+
     #       stream: # only for an ALB that serves a tcp, tls or udp listener. See /docs/alb.md
     #         # connect_retries is how many other pool members a tcp or tls connection is offered when
     #         # it cannot connect to the one it was given. default is 0: the connection is refused
@@ -1751,3 +1794,8 @@ data:
     #     # access_log:
     #     #   filename: stdout
     #     #   format: combined
+    #     # sticky_secret_file keys the session tokens generated ALBs issue for a
+    #     # route's sessionPersistence or a sticky annotation, parameter or policy:
+    #     # at least 32 bytes, mounted from a Secret at the same path on every
+    #     # replica. Unset, each process uses its own random key (one replica only).
+    #     # sticky_secret_file: /etc/trickster/sticky/key

deploy/kube/crds/trickstercachepolicies.yaml

@@ -186,6 +186,36 @@ spec:
                     host, header:<name>, cookie:<name> or query:<name> for HTTP routes, and
                     client_ip or sni for TCP, TLS and UDP routes.
                   type: string
+                sticky:
+                  description: >-
+                    Keeps a client on the endpoint it was first sent to, in the endpoint
+                    routing mode: cookie or header (an issued token, HTTP routes only) or
+                    table (a pin kept by stickyKey). none turns off a less specific
+                    policy's. TCP, TLS and UDP routes always keep sessions in a table.
+                  type: string
+                  enum:
+                    - cookie
+                    - header
+                    - table
+                    - none
+                stickyKey:
+                  description: >-
+                    What table mode keeps a client's endpoint by: client_ip (the default),
+                    host, header:<name>, cookie:<name> or query:<name> for HTTP routes, and
+                    client_ip or sni for TCP, TLS and UDP routes.
+                  type: string
+                stickyTTL:
+                  description: >-
+                    Ends a session this long after it began; a duration with a unit of at
+                    least 1s. 1h unless set.
+                  type: string
+                  pattern: '^[0-9]+(ns|us|µs|ms|s|m|h|d|w|y)([0-9]+(ns|us|µs|ms|s|m|h|d|w|y))*$'
+                stickyIdle:
+                  description: >-
+                    Ends a session unused for this long; a duration with a unit of at least
+                    1s. Never unless set.
+                  type: string
+                  pattern: '^[0-9]+(ns|us|µs|ms|s|m|h|d|w|y)([0-9]+(ns|us|µs|ms|s|m|h|d|w|y))*$'
                 resultHeader:
                   description: >-
                     Whether the X-Trickster-Result response header reaches the client;

deploy/kube/gateway-rbac.yaml

@@ -23,6 +23,11 @@ rules:
   - apiGroups: ["gateway.networking.k8s.io"]
     resources: ["referencegrants", "backendtlspolicies"]
     verbs: ["list", "watch"]
+  # the experimental group's XBackendTrafficPolicy, read for sessionPersistence
+  # where the cluster serves it; without the grant the kind is skipped
+  - apiGroups: ["gateway.networking.x-k8s.io"]
+    resources: ["xbackendtrafficpolicies"]
+    verbs: ["list", "watch"]
   - apiGroups: ["gateway.networking.k8s.io"]
     resources:
       - gatewayclasses/status
@@ -43,7 +48,8 @@ rules:
   - apiGroups: ["networking.k8s.io"]
     resources: ["ingresses/status"]
     verbs: ["update"]
-  # backends, TLS certificates (narrowed server-side to kubernetes.io/tls),
+  # backends, TLS certificates (narrowed server-side to kubernetes.io/tls) and
+  # session token keys (to the trickstercache.org/sticky-key label),
   # GatewayClass parameters and BackendTLSPolicy CA bundles, and the
   # Namespace labels a Gateway's allowedRoutes selector reads
   - apiGroups: [""]

docs/alb-autodiscovery.md

@@ -188,17 +188,26 @@ one whose readiness the provider does not convey, waits for its first
 passing probe as before.
 
 In `provider` mode, readiness reported by the discoverer maps onto member
-health: ready members enter as passing (1), not-ready and terminating
+health: ready and terminating members enter as passing (1), not-ready
 members as failing (-1), readiness-unknown members as unchecked (0) — so a
 `healthy_floor` of 1 excludes members until the provider reports them
 ready.
 
-Members a provider reports as shutting down (terminating Kubernetes
-endpoints, deletion-stamped pods) are removed from the snapshot entirely,
-so they drain from the pool *before* the workload is killed — enabling
-zero-error rolling deploys. On removal, a member's in-flight requests
-complete, its health check stops, its metrics series are deleted, and its
-idle upstream connections close after the configured drain timeout.
+Members a provider reports as shutting down but still serving (terminating
+Kubernetes endpoints whose `serving` condition holds, deletion-stamped pods
+that are still ready) stay in the pool as
+[draining](./alb.md#draining-pool-members) members: from the moment the
+change is observed they take no new requests, connections or sessions, while
+[sticky sessions](./alb.md#sticky-sessions) already pinned to them keep
+reaching them for as long as they are available. Once the provider reports
+that a member no longer serves, it is removed from the snapshot, *before*
+the workload is killed — enabling zero-error rolling deploys. On removal, a
+member's in-flight requests complete, its health check stops, its metrics
+series are deleted, and its idle upstream connections close after the
+configured drain timeout. Draining members count toward `min_members` and
+`trickster_alb_discovery_members`, and are shown on the
+[health status page](./alb.md#all-backends-health-status-page) and by the
+`trickster_alb_member_draining` gauge.
 
 ### TSM Replica Groups
 
@@ -360,9 +369,10 @@ the discoverer keeps watching for the API server to come back.
 
 ### Query Kinds
 
-**`endpointslices`** (default) discovers the ready endpoint addresses of a
-named Service — the pod IPs behind it — and is the right choice for
-routing around the Service's own load balancing:
+**`endpointslices`** (default) discovers the endpoint addresses of a named
+Service — the pod IPs behind it — and is the right choice for routing
+around the Service's own load balancing. Terminating endpoints that still
+serve join as draining members:
 
 ```yaml
 query:
@@ -409,8 +419,9 @@ annotation on the watched Service/Pod, selects `https`; the default is
 
 ### Zero-Error Rolling Deploys
 
-Terminating endpoints are removed from the discovered membership as soon
-as the change is observed, so members drain out ahead of pod deletion.
+Terminating endpoints stop taking new work as soon as the change is
+observed: they drain, keeping only their sticky sessions, and leave the
+discovered membership once they stop serving, ahead of pod deletion.
 One piece belongs to the workload, though: Kubernetes marks an endpoint
 `terminating` and signals the container at the same moment, so — as with
 any EndpointSlice consumer, kube-proxy included — the pod must keep