Expose circuit breaker configuration for restore workloads

[obacak]

Is your feature request related to a problem?

Environment

• OSO Kafka backup version: v0.22.0
• Kafka: Strimzi-managed Kafka 3.8.0
• Target cluster: 4 brokers
• Restore type: Bulk disaster-recovery restore
• Concurrent partitions: 8
• Approximate records: 1.35 billion
• Checkpointing: Enabled

Summary

The Kafka backup/restore tool contains an internal circuit breaker in kafka-backup-core. Its configuration includes:

failure_threshold
reset_timeout
success_threshold

These values are currently hardcoded and cannot be configured through YAML, CLI flags, or environment variables.

Problem

During a large Kafka restore with version v0.22.0, we repeatedly observed:

Component kafka became Degraded
Component kafka recovered

Packet captures showed approximately 45-60 seconds of complete bidirectional silence across all broker connections during these events. Kafka broker logs remained clean, indicating that the pause occurred inside the client rather than in Kafka or the network.

This behavior is consistent with the global Kafka circuit breaker entering the Open state and waiting for its reset timeout.

Because the breaker appears to cover the entire Kafka operation, a transient failure affecting one partition can pause all concurrently restored partitions.

Our restore contained approximately 1.35 billion records and used:

restore:
  max_concurrent_partitions: 8
  produce_batch_size: 500
  rate_limit_records_per_sec: 100000
  checkpoint_state: /backup/restore-checkpoint.json

The repeated circuit-breaker pauses significantly increased the total restore duration.

Related work

PR #34, released as v0.8.2, exposed several previously hardcoded networking and resilience settings, including:

• connections_per_broker
• produce_acks
• produce_timeout_ms
• Connection retry handling
• Socket read/write timeouts

Circuit-breaker behavior appears to be a related area that is still not configurable.

Describe the solution you'd like

Requested feature

Please expose circuit-breaker settings in the configuration. For example:

restore:
  circuit_breaker:
    enabled: true
    failure_threshold: 20
    reset_timeout_ms: 2000
    success_threshold: 1

At minimum, please provide an option to disable the circuit breaker for restore operations:

restore:
  circuit_breaker:
    enabled: false

Restore operations already support checkpointing and retry behavior. For large, resumable, one-shot restores, operators may reasonably prefer to tolerate isolated transient failures without globally pausing the entire restore.

A further improvement would be to scope circuit breakers per broker or partition instead of globally. A failure affecting one partition should not suspend unrelated restore work.

Suggested configuration

For a bulk restore, reasonable starting values might be:

failure_threshold: 15
reset_timeout_ms: 2000
success_threshold: 1

Existing production defaults could remain unchanged when no override is provided.

Describe alternatives you've considered

No response

Use case

No response

Additional context

No response

[aywengo]

Hi @sionsmith ,

we've been running kafka-backup as part of our Kafka backup/DR setup. In non-prod it has gone very well and looks really promising — thank you for building it.

In production we ran into trouble on large topics (> 500 GB): restores stall repeatedly, with "Component kafka became Degraded / recovered" and 45–60 s of complete silence on all broker connections. We wrote it up in #197 and opened two PRs:

#198 — config only: exposes the circuit-breaker settings (enabled, thresholds) in YAML, as requested in the issue.
#199 — the same config plus the actual fix. While digging in we found the breaker is advisory (it never gated requests); the cross-partition stalls come from the connection cache being cleared for all brokers on any error and from per-record locking of the offset mapping. Includes repro tests, docs and CHANGELOG.

Would you have a moment to take a look? Happy to adjust anything to fit how you'd like the project to evolve. The CI workflows on #199 are also waiting for a maintainer to approve them.

Thanks a lot, Roman

[sionsmith]

hi @aywengo we will take a look at this on Monday - thank you so much for your support and feedback!

[sionsmith]

Update after digging into this properly — thanks again @obacak and @aywengo for the report and the PRs.

What shipped in v0.23.0 (#199, released today): the circuit_breaker settings you asked for, plus the real router fix. To be clear about the breaker: it was advisory — the engines only ever recorded successes/failures into it and never consulted it before a request — so its 30 s reset_timeout could not have paused anything. What was wrong was that any connection error or NOT_LEADER on one partition cleared the connection pools of every broker, forcing all concurrent partitions to rebuild TCP/TLS/SASL connections. That is fixed (eviction is now scoped to the broker the failed request went to, and NOT_LEADER no longer drops any pool).

Your Degraded / recovered log lines turned out to be a different, pre-existing bug I hit while verifying #199 end-to-end: a partition with no leader for a moment (metadata leader_id = -1, e.g. during a broker restart or an election) was treated as terminal — Unknown broker ID: -1 — killing the partition task while the others kept going, which is exactly what produces that pair of lines. Tracked as #201, fix in #202 (waits for a leader / a refused reconnect for up to 120 s instead of failing).

The 45–60 s of complete silence across all brokers: I could not reproduce it with connection faults alone — with the fixes above, partitions on healthy brokers never paused. I could reproduce it by running your restore config (max_concurrent_partitions: 8, produce_batch_size: 500, rate_limit_records_per_sec: 100000, checkpointing) on a 20M-record backup inside a container with a memory limit: all 8 partitions stopped for ~20 s and again for ~9 s with memory pinned at the limit and no Kafka-side errors at all, and the process was eventually OOM-killed. The cause is the restore's offset-mapping table, which kept one 24-byte entry per restored record in a Vec behind a single mutex — about 32 GB for your 1.35 billion records, with every Vec growth copying gigabytes while all partition tasks waited on that lock. At your scale that is either constant memory pressure (if the pod limit is near it) or a very large working set.

Measurements (your restore config, 20M records / 8 partitions, 3-broker KRaft cluster, restore running in a Linux container):

	0.23.0	run-length mapping (#203)
No memory limit — peak RSS / growth during the run	4.28 GB / +2.0 GB	3.15 GB / +0.8 GB
2.8 GB limit	completes in 86 s, all 8 partitions silent for 4 s at the limit	completes in 63 s, no stalls
900 MB limit	all 8 partitions silent for 20 s and 9 s, then OOM-killed at 133 s	—
Offset-mapping table for 20M records	20,000,000 entries (~480 MB, plus Vec doubling headroom)	8 runs (one per partition), 4 KB report

The ~2.2 GB baseline in both columns is segment/record buffers (8 partitions × 128 MB segments materialised at once), which is a separate improvement. Scaled to 1.35 billion records the per-record table alone is ~32 GB (up to twice that while the Vec doubles); the run-length table stays at a few entries per partition.

A fix is in progress: the mapping is stored as runs ({source_start, target_start, len} — one entry per contiguous stretch instead of per record), collected per partition task with no shared lock, so memory becomes proportional to the number of discontinuities rather than to the record count.

To confirm this matches what you saw, could you share:

1. the pod's memory limit / request, and the container's RSS over the run (or whether it was ever OOM-killed / restarted — a checkpoint_state resume would hide a restart);
2. whether restore.offset_report, reset_consumer_offsets / auto_consumer_groups or a non-skip consumer_group_strategy were set (they are the only consumers of that table);
3. the full restore YAML and security_protocol, and a log excerpt around one of the silent windows (with timestamps) — in particular whether there were Connection error … retrying lines or none at all.