Overview
Observability/tracing is hard-enabled by default, which is fine for normal operation but risky for a load test: if the OTLP collector endpoint isn't reachable from the perf environment, or sampling isn't turned down, the exporter will retry in the background during the run — adding overhead and noise that skews the very performance numbers the test is trying to measure.
Description
Files: perf-env deployment.yaml (observability: block)
Actual:
Observability.Enabled is hard-set true in applyDefaults, with no check on whether the perf environment actually has a reachable OTLP collector or a sane sample rate configured.
Expected:
- Confirm the OTLP sink for the perf environment is either explicitly disabled, or pointed at a real, reachable collector.
- If enabled, set
sample_rate ≤ 0.01 for load runs, so tracing overhead stays negligible at 50 TPS.
- Verify that an unreachable OTLP endpoint does not cause background exporter retries during the test — this silently adds load and can distort latency/throughput results.
Acceptance Criteria
- Perf environment's
observability: block is confirmed to have either Enabled: false, or Enabled: true with a reachable collector endpoint and sample_rate ≤ 0.01.
- During a load run, no OTLP exporter connection errors or retry log spam appear.
- Load test results (latency, throughput) are unaffected by tracing overhead — confirmed by comparing a run with observability on vs. off, if feasible.
Definition of Done
Overview
Observability/tracing is hard-enabled by default, which is fine for normal operation but risky for a load test: if the OTLP collector endpoint isn't reachable from the perf environment, or sampling isn't turned down, the exporter will retry in the background during the run — adding overhead and noise that skews the very performance numbers the test is trying to measure.
Description
Files: perf-env
deployment.yaml(observability:block)Actual:
Observability.Enabledis hard-settrueinapplyDefaults, with no check on whether the perf environment actually has a reachable OTLP collector or a sane sample rate configured.Expected:
sample_rate≤ 0.01 for load runs, so tracing overhead stays negligible at 50 TPS.Acceptance Criteria
observability:block is confirmed to have eitherEnabled: false, orEnabled: truewith a reachable collector endpoint andsample_rate≤ 0.01.Definition of Done
deployment.yamlobservability:block reviewed and set correctly (disabled or sampled + reachable collector)sample_rateset to ≤ 0.01 if observability is enabled for load runsdevelop-go