Skip to content

[TASK] eSignet Go Performance: Confirm OTLP observability config for load runs (disabled or sampled, reachable collector) #2497

Description

@rachik-hue

Overview

Observability/tracing is hard-enabled by default, which is fine for normal operation but risky for a load test: if the OTLP collector endpoint isn't reachable from the perf environment, or sampling isn't turned down, the exporter will retry in the background during the run — adding overhead and noise that skews the very performance numbers the test is trying to measure.

Description

Files: perf-env deployment.yaml (observability: block)

Actual:

Observability.Enabled is hard-set true in applyDefaults, with no check on whether the perf environment actually has a reachable OTLP collector or a sane sample rate configured.

Expected:

  • Confirm the OTLP sink for the perf environment is either explicitly disabled, or pointed at a real, reachable collector.
  • If enabled, set sample_rate ≤ 0.01 for load runs, so tracing overhead stays negligible at 50 TPS.
  • Verify that an unreachable OTLP endpoint does not cause background exporter retries during the test — this silently adds load and can distort latency/throughput results.

Acceptance Criteria

  • Perf environment's observability: block is confirmed to have either Enabled: false, or Enabled: true with a reachable collector endpoint and sample_rate ≤ 0.01.
  • During a load run, no OTLP exporter connection errors or retry log spam appear.
  • Load test results (latency, throughput) are unaffected by tracing overhead — confirmed by comparing a run with observability on vs. off, if feasible.

Definition of Done

  • Perf-env deployment.yaml observability: block reviewed and set correctly (disabled or sampled + reachable collector)
  • sample_rate set to ≤ 0.01 if observability is enabled for load runs
  • Verified no exporter retry/connection-error noise during a load run
  • Documented the chosen configuration and rationale for future perf runs
  • Code/config reviewed and merged into develop-go

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Fields

Priority

Low

End Date

Sep 13, 2026

Complexity

LOW

Start Date

Aug 31, 2026

Original Estimate

1

Time Tracking

None yet

Resolved By

None yet

Story Points

1

Severity

MINOR

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions