Replace lib-data-stream-redis with lib-data-workqueue

[pditommaso]

Moves the job queues from the message stream API onto the work queue API of libseqera: lib-data-stream-redis:1.2.0 → lib-data-workqueue:2.0.0 + lib-data-workqueue-redis:2.0.0.

lib-data-workqueue-redis is lib-data-stream-redis 1.5.0 re-landed on lease semantics — identical Redis stream layout (xadd/xreadGroup/xautoclaim, xack+xdel on settle, same data field). Queue keys (jobs-pending/v2, jobs-queue/v1) and the consumer group are left untouched, so no migration is needed and messages queued before the upgrade are still delivered afterwards.

Changes

build.gradle	swap the stream artifact for the two work-queue artifacts
service/data/workqueue/BaseWorkQueue	was BaseMessageStream, now on AbstractWorkQueue; same Moshi encoding strategy
service/data/workqueue/RedisWorkQueueConfigBean	was RedisStreamConfigBean, now implements RedisWorkQueueConfig
JobPendingQueue / JobProcessingQueue	extend BaseWorkQueue; queue ids unchanged
JobManager	(job) -> boolean becomes (job, lease) -> Decision.ACK/RETRY
docs/configuration.md	"Message stream" section becomes "Work queue"

launchJob/processJob are unchanged and still return a boolean, mapped onto ACK/RETRY at the addConsumer site — so JobManager keeps its own admission control and the maxRunningJobs cap is unaffected.

Why: work queue vs message stream

	message stream (before)	work queue (after)
Slow consumer	An entry's idle clock only resets on delivery/claim, so a job outlasting claim-timeout can be claimed and run concurrently by another replica. The timeout had to exceed worst-case processing time — hence 45s in prod.	A lease is renewed at visibility-timeout/4 while the owner is alive (one XCLAIM JUSTID per queue per tick), holding the idle clock near zero. The timeout now only governs recovery from a dead owner.
Shutdown	thread.interrupt() then join(1s) — severs a consumer mid-processing, and an interrupt can hand a RESP-desynced connection back to the Jedis pool.	Cooperative: a closing flag checked at loop head, waits (default 10s) for the current cycle, never interrupts. A processJob cycle completes its notifyJobCompletion/cleanup before teardown. awaitQuiescent(timeout) available for a bounded drain.
Retry pacing	Any delivery counted as progress, so a cycle that only returned false skipped the poll-interval sleep.	Only ACK/DEFERRED count as progress, so a queue that is merely retrying (job still running, pending queue full) paces at its poll interval.
Ownership safety	—	Renewal does an XPENDING ownership check first: an entry that drifted to another consumer is dropped and logged rather than seized back — the residual duplicate window is observable, not silent.

Not used by this PR, but now available: MessageConsumer.ready() (an admission gate checked before claiming — the natural home for the maxRunningJobs check, which currently claims a message and then refuses it), MessageLease/DEFERRED with retryAfter(delay), and lease/renewal/saturation metrics. Still at-least-once: renewal narrows the duplicate window, it does not close it.

Rollout

No Redis migration. Entries queued by the old code are read as-is (the integration test logs consume group=wave-message-stream already exists), and entries left pending by a terminating pod are reclaimed after the visibility timeout — in-flight jobs resume. Mixed-version replicas are safe both ways: an entry held by a new replica is renewed so an old replica's 45s-min-idle claim never sees it; an entry held by an old replica may be claimed after 45s idle, exactly as today.

Config keys are renamed wave.message-stream.* → wave.work-queue.* (claim-timeout → visibility-timeout), so the matching rename in platform-deployment (prod + stage, 45s) must land with this. That change is prepared but not yet pushed.

⚠️ The one hazard is rollback: revert the image with the new ConfigMap in place and the old code finds no message-stream.claim-timeout, falling back to a 5s claim timeout with no renewal. Keeping both key sets in the ConfigMap for one release makes it order-independent in both directions. Running pods are unaffected either way — the ConfigMap is mounted via subPath (never updated in place) and the Deployment has no config checksum annotation, so config is only read at container start.

Also note the cooperative close() can take up to 10s per queue, sequentially across two beans, against the default 30s terminationGracePeriodSeconds. In practice the dispatcher exits at its next loop-head check unless mid-consume; raising the grace period would add headroom.

The consumer group keeps defaulting to wave-message-stream deliberately — it is a wire identifier now, and renaming it would strand the pending-entry list.

Testing

• compileGroovy clean
• io.seqera.wave.service.job.* — 39 tests, 0 failures
• BuildStoreRedisTest (7) and RegistryControllerRedisTest (2) green against real Redis; logs confirm RedisWorkQueue reusing the existing consumer group and picking up the config bean (lease renewal period=1250ms)

Full suite not run locally — left to CI.

🤖 Generated with Claude Code