Add OpenTelemetry tracing across the backbeat pipeline#2733
Draft
delthas wants to merge 3 commits into
Draft
Conversation
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
Codecov Report❌ Patch coverage is Additional details and impacted files
... and 6 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2733 +/- ##
===================================================
- Coverage 74.73% 74.66% -0.07%
===================================================
Files 199 201 +2
Lines 13650 13741 +91
===================================================
+ Hits 10201 10260 +59
- Misses 3439 3471 +32
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
9d08f7b to
2f7afb0
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
2f7afb0 to
d562a0a
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
d562a0a to
51a9f61
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
51a9f61 to
b9d3528
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
970a811 to
849d6b0
Compare
delthas
added
the
claude-review-retro
DarkIsDude
requested changes
Jun 1, 2026
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
6357120 to
a54ec82
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
a54ec82 to
54399e3
Compare
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
54399e3 to
3189d3a
Compare
Pin arsenal at the ARSN-586 branch (shared tracing module + W3C trace-context stamping on MongoDB metadata writes). Drop the SDK-core packages now that arsenal carries them as optionalDependencies, and keep the four instrumentation packages (http, ioredis, mongodb, aws-sdk) here — the consumer owns and configures them. Issue: BB-764
lib/tracing/index.js becomes a thin shim over arsenal's shared module: it carries backbeat's config (serviceName, the http/ioredis/mongodb/ aws-sdk instrumentations, outbound-only HTTP via makeHttpInstrumentationConfig + disableIncomingRequestInstrumentation) so the 8 entry points keep calling init() with no args. kafkaTraceContext.js re-exports arsenal's kafka helpers so the existing require sites are unchanged. The trust-boundary filter and SDK bootstrap now live in arsenal. Issue: BB-764
delthas
force-pushed
the
improvement/BB-764/otel-replication-tracing
branch
from
3189d3a to
cedc91c
Compare
Wire arsenal's tracing into the replication, lifecycle, GC, notification, and oplog-populator pods: init() at each entry point, per-pod spans, and trace-context propagation across the Kafka pipeline (producers stamp traceparent via the kafka helpers; consumers start linked spans from it). Out-of-process Kafka hops use span links, not parent/child, so traces stay bounded. Issue: BB-764
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add OpenTelemetry tracing across the backbeat pipeline, gated behind
ENABLE_OTEL=true. When the flag is unset, no@opentelemetry/*package is loaded — zero overhead off the OTEL path.The SDK bootstrap, trust-boundary host filter, and Kafka trace-context helpers now live in arsenal's shared
lib/tracingmodule (scality/Arsenal#2632, ARSN-586); backbeat consumes it through a thin shim instead of carrying its own copy. Companion to the cloudserver (#6140, CLDSRV-884) and vault (#203, VAULT-708) PRs, so all four services share one implementation.Commits
chore: depend on arsenal OTEL tracing module— pin arsenal at the ARSN-586 branch (shared tracing module + the W3C trace-context stamping on MongoDB metadata writes, ARSN-572, which the Kafka pipeline relies on to continue traces across the oplog boundary). Drop the SDK-core packages now that arsenal carries them asoptionalDependencies, and keep the four instrumentation packages backbeat configures itself:instrumentation-http/-ioredis/-mongodb/-aws-sdk.feat: replace in-tree tracing with arsenal shim—lib/tracing/index.jsbecomes a thin shim overrequire('arsenal/build/lib/tracing')carrying backbeat's config in one place:serviceName: 'backbeat', the four instrumentations, and outbound-only HTTP (...makeHttpInstrumentationConfig()for the trust-boundaryrequestHook, plusdisableIncomingRequestInstrumentation: truesince backbeat pods serve no application HTTP).lib/tracing/kafkaTraceContext.jsre-exports arsenal's kafka helpers so the existing require sites are unchanged. The trust-boundary filter and SDK bootstrap that used to live here are now arsenal's.feat: instrument backbeat pods and the Kafka pipeline— wire tracing into the replication, lifecycle, GC, notification, and oplog-populator pods:init()at each of the 8 entry points, per-pod spans, and trace-context propagation across the Kafka pipeline. Producers stamptraceparentonto message headers via the kafka helpers; consumers start a span linked to (not a child of) the upstream span — out-of-process Kafka hops can fire long after the original request, so links keep traces bounded.Why a shim (vs cloudserver/vault's direct calls)
backbeat has 8
init()entry points and 6 Kafka-helper require sites. The shim keeps all 14 call sites untouched and the backbeat-specific config in one file; cloudserver and vault have a single entry point each, so they deep-require arsenal directly.Configuration
OpenTelemetry environment variables are documented in the arsenal module.
Out of scope (follow-ups)
The trust-boundary enforcement (read
OTEL_TRUSTED_HOSTS, striptraceparenton untrusted outbound) lives entirely in arsenal and is wired automatically. Populating that env var per deployment is routine operator config — no backbeat-side work.Related tickets
Issue: BB-764