[da-vinci][server] Global RT DIV: Add OTel metrics for end-to-end feature observability

[KaiSernLim]

Problem Statement

The Global RT DIV feature (leader→follower RT DIV state propagation) has zero metrics today. Several of its phases are best-effort — they catch and log rather than fail ingestion — so operators have no way to verify:

• That leaders are actively producing RT DIV snapshots to the version topic
• That followers are receiving and persisting them
• That VT position checkpointing is firing correctly
• That a promoted leader actually restored RT DIV state from disk
• Whether any phase is silently failing

Solution

Adds 7 OTel metrics covering the full lifecycle, with all call sites wired in, so the feature is observable end to end the moment server.global.rt.div.enabled is turned on.

Metric	Type	What it measures
ingestion.global_rt_div.send_size	histogram	Byte size of each payload produced by the leader. The count aggregation doubles as the send count
ingestion.global_rt_div.send_rt_producer_count	histogram	RT producers tracked per broker in each payload sent
ingestion.global_rt_div.persist_count	counter	States persisted to metadata storage by the drainer
ingestion.global_rt_div.vt_sync_producer_count	histogram	VT producers in the snapshot checkpointed into the OffsetRecord. The count aggregation doubles as the sync count
ingestion.global_rt_div.error_count	counter	Errors, dimensioned by SEND/PERSIST/VT_SYNC/DELETE/LOAD
ingestion.global_rt_div.load_count	counter	Load attempts on F→L promotion, dimensioned by FOUND/NOT_FOUND
ingestion.global_rt_div.load_rt_producer_count	histogram	RT producers restored from disk on leader promotion

All metrics carry VENICE_STORE_NAME, VENICE_CLUSTER_NAME, and VENICE_VERSION_ROLE.

How to use them: the send_size count vs persist_count ratio across leader and follower replicas detects drops. A zero vt_sync_producer_count count alongside a non-zero persist_count means the offset checkpointing path is broken. load_count{NOT_FOUND} is expected for the first leader ever elected for a partition, but a sustained rate is not. Any non-zero error_count is directly actionable.

New dimensions:

• venice.global_rt_div.error.type → VeniceGlobalRtDivErrorType (SEND, PERSIST, VT_SYNC, DELETE, LOAD)
• venice.global_rt_div.load.outcome → VeniceGlobalRtDivLoadOutcome (FOUND, NOT_FOUND)

LOAD exists as an error type because readGlobalRtDivState and deserializeGlobalRtDivState both return null on failure, which is indistinguishable from "genuinely absent" at the loadGlobalRtDiv call site. load_count{NOT_FOUND} therefore covers both, and error_count{LOAD} is the signal that disambiguates them.

Call-site error handling: phases that currently propagate exceptions (PERSIST, DELETE, VT_SYNC) record the error and rethrow, so behavior is preserved exactly while failures become observable — storage failures are deliberately not swallowed. Phases that already catch-and-log (send compression, load read/deserialize) simply record alongside the existing log.

Infrastructure added:

• VeniceGlobalRtDivErrorType and VeniceGlobalRtDivLoadOutcome enums + their keys in VeniceMetricsDimensions
• 7 IngestionOtelMetricEntity constants
• 6 recording methods in IngestionOtelStats + no-op overrides in NoOpIngestionOtelStats
• 6 router methods in AggVersionedIngestionStats
• Call sites in LeaderFollowerStoreIngestionTask (send, load) and StoreIngestionTask (persist, delete, vt_sync)

Code changes

• Added new code behind a config. All recording is gated by the existing server.global.rt.div.enabled config (default false), since the code paths themselves only execute when the feature is on.
• Introduced new log lines. No new log lines — only metric recording calls.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues. OTel counters and histograms are thread-safe by design.
• Proper synchronization mechanisms are used where needed. N/A — no shared mutable state introduced.
• No blocking calls inside critical sections. Metric calls are non-blocking.
• Verified thread-safe collections are used. N/A.
• Validated proper exception handling in multi-threaded code. The record-and-rethrow pattern preserves the existing propagation semantics on every path it touches.

How was this PR tested?

• New unit tests added. VeniceGlobalRtDivErrorTypeTest and VeniceGlobalRtDivLoadOutcomeTest validate the new enums. New test methods in IngestionOtelStatsTest cover all 7 metrics end-to-end using in-memory OTel readers.
• New integration tests added.
• Modified or extended existing tests. IngestionOtelMetricEntityTest (7 new entities), ServerMetricEntityTest (count 185→192), VeniceMetricsDimensionsTest (2 new dimensions), LeaderFollowerStoreIngestionTaskTest (asserts the persist and load-error metrics actually fire).
• Verified backward compatibility. Metrics are purely additive; no behavior changes.

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.

[Copilot]

Pull request overview

Adds end-to-end OpenTelemetry (OTel) observability for the Global RT DIV best-effort propagation lifecycle, enabling operators to measure leader send/payload size, follower persist, VT position sync, and categorized errors without changing ingestion behavior.

Changes:

• Introduces a new metrics dimension (VENICE_GLOBAL_RT_DIV_ERROR_TYPE) and enum (VeniceGlobalRtDivErrorType: SEND, PERSIST, VT_SYNC, DELETE).
• Registers 5 new ingestion OTel metric entities and adds corresponding recording APIs in IngestionOtelStats (+ no-op overrides, + AggVersioned router methods).
• Extends unit tests to validate the new dimension naming, metric entity registration, and end-to-end metric recording via in-memory OTel readers.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensions.java	Adds VENICE_GLOBAL_RT_DIV_ERROR_TYPE dimension key.
internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceGlobalRtDivErrorType.java	New enum implementing VeniceDimensionInterface for Global RT DIV error categorization.
internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensionsTest.java	Verifies naming transforms (snake/camel/pascal) for the new dimension.
internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceGlobalRtDivErrorTypeTest.java	New unit test validating enum-to-dimension wiring and expected values.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/ingestion/IngestionOtelMetricEntity.java	Registers 5 new Global RT DIV ingestion OTel metrics.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/ingestion/IngestionOtelStats.java	Adds metric state initialization + 4 recording methods for Global RT DIV.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/ingestion/NoOpIngestionOtelStats.java	Adds no-op overrides for the new Global RT DIV recording methods.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/AggVersionedIngestionStats.java	Adds routing methods so call sites can record the new metrics per store/version.
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/ingestion/IngestionOtelMetricEntityTest.java	Updates expected metric entity registry with the 5 new entities + dimensions.
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/ingestion/IngestionOtelStatsTest.java	Adds tests validating all Global RT DIV metrics (counter + histogram + error dimension).
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/ServerMetricEntityTest.java	Updates expected total metric entity count to include the 5 new metrics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Hi there. This pull request has been inactive for 30 days. To keep our review queue healthy, we plan to close it in 7 days unless there is new activity. If you are still working on this, please push a commit, leave a comment, or convert it to draft to signal intent. Thank you for your time and contributions.

[github-actions]

Closing this pull request due to 37 days of inactivity. This is not a judgment on the value of the work. If you would like to continue, please reopen or open a new PR and we will be happy to take another look. Thank you again for contributing.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceGlobalRtDivLoadOutcome.java:19

• The NOT_FOUND enum value’s Javadoc says the leader “starts from EARLIEST”, but the leader-start behavior when no Global RT DIV state is loaded is determined by the normal leader start logic (and can fall back to a rewind-based start), not necessarily EARLIEST. This comment is misleading for operators interpreting the metric dimension.

  /** No usable state read from disk; leader starts from EARLIEST */

internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensions.java:126

• This dimension’s Javadoc says it’s for “best-effort operations”, but the error type is used for all Global RT DIV operation failures (including phases that record-and-rethrow like PERSIST/DELETE/VT_SYNC). The current wording is inaccurate and could confuse readers about which failures are expected to be swallowed vs propagated.

  /** {@link VeniceGlobalRtDivErrorType} Error type for Global RT DIV best-effort operations */
  VENICE_GLOBAL_RT_DIV_ERROR_TYPE("venice.global_rt_div.error.type"),

[KaiSernLim]

🤖 Responding to github-actions bot notices 4317549461 and 4362649401: declined as code feedback, since both are automated PR lifecycle notices and contain no code review findings.
The requested lifecycle actions have been completed: PR 2658 was reopened/continued, rebased onto current main (it was 218 commits behind with conflicts), and is now mergeable.
I squashed the original work to a single commit, force-pushed the branch, rewrote the PR description, marked it ready for review, and removed the stale / closed-by-stale labels.
Substantive cleanup was included too: metrics reduced 9 -> 7, counter types corrected, LOAD was added to VeniceGlobalRtDivErrorType, and all call sites were wired in.
The original revision defined metrics but never invoked them, so this update makes the feature observable rather than dead code.
Targeted unit tests for the touched modules plus spotlessCheck are green.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.