Skip to content

v2: log when bootstrap is blocked (#1746) - #1775

Draft
hamza-syndica wants to merge 4 commits into
mainfrom
hamza/bootstrap-blocked-logging
Draft

v2: log when bootstrap is blocked (#1746)#1775
hamza-syndica wants to merge 4 commits into
mainfrom
hamza/bootstrap-blocked-logging

Conversation

@hamza-syndica

@hamza-syndica hamza-syndica commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Add periodic observability logging to gossip, snapshot downloader, accounts_db, and replay services while they are blocked waiting on upstream bootstrap data.

Service instrumentation

Service What it logs Warn after
gossip No packets received from cluster 60s
snapshot No usable peers from gossip (distinguishes 'none received' vs 'received but unusable') 60s
accounts_db Awaiting snapshot bytes from snapshot service 5 min
replay Awaiting runtime metadata from accounts_db 15 min

Each service also emits a one-shot info log when the awaited data arrives ( pattern).

Closes #1746

\nAdd periodic observability logging to gossip, snapshot downloader, accounts_db, and replay services while they are blocked waiting on upstream bootstrap data.\n\nEach service:\n- Logs at info level every 10s while waiting\n- Escalates to warn after a service-specific timeout\n  (gossip: 60s, snapshot: 60s, accounts_db: 5min, replay: 15min)\n- Emits a one-shot info log when data arrives\n\nIntroduces a shared ThrottledLogger utility in v2/lib/telemetry.zig\nthat gates log frequency and tracks escalation state.\n\nCloses #1746
@github-project-automation github-project-automation Bot moved this to 🏗 In progress in Sig Jul 30, 2026
@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.32203% with 37 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
v2/components/snapshot/download.zig 0.00% 16 Missing ⚠️
v2/services/gossip.zig 0.00% 11 Missing ⚠️
v2/services/accounts_db.zig 0.00% 9 Missing ⚠️
v2/lib/telemetry.zig 99.48% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (84.32%) is below the target coverage (85.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines Coverage Δ
v2/services/replay.zig 38.98% <100.00%> (+0.40%) ⬆️
v2/lib/telemetry.zig 77.28% <99.48%> (+16.54%) ⬆️
v2/services/accounts_db.zig 0.00% <0.00%> (ø)
v2/services/gossip.zig 0.00% <0.00%> (ø)
v2/components/snapshot/download.zig 10.31% <0.00%> (-0.13%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🏗 In progress

Development

Successfully merging this pull request may close these issues.

Log when bootstrap is blocked

1 participant