Skip to content

fix(logging): rotate exo's log file by size, not only when exo starts - #2390

Open
AlexCheema wants to merge 1 commit into
mainfrom
fix/rotate-log-by-size
Open

AlexCheema wants to merge 1 commit into
mainfrom
fix/rotate-log-by-size

Conversation

@AlexCheema

Copy link
Copy Markdown
Contributor

Problem

exo starts a new log file (~/.exo/exo_log/exo.log) each time it starts, and never during a run. A node that runs for days or weeks keeps one ever-growing file.

At default verbosity each request logs a few lines: the worker's plan, the task it starts (with its parameters), and KV cache activity. On a 2-node soak test, two Mac Studios (M3 Ultra) serving steady load of about 1 request a second, exo.log grew 8.5 MB an hour on each node, about 200 MB a day, with no limit.

Fix

Rotate the file when exo starts, as before, and also whenever it passes 50 MB. Rotated files are compressed with zstd, and the last 5 are kept, as before. So a node's logs are bounded at about 50 MB plus a few MB of archives.

Tests

test_log_rotation.py sets exo's logger up with a 20 KB limit:

  • writing past the limit rotates the file more than once, and keeps at most 5 compressed archives;
  • the live file stays near the limit, and the last line written is in it;
  • a small log is only rotated at start.

A 24-hour soak with this change follows; I'll add its disk measurements here.

🤖 Generated with Claude Code

exo started a new log file on each start and never again, so a node that ran
for a long time kept one ever-growing file. At default verbosity each request
logs a few lines: on a 2-node soak with steady load (~1 request/s) exo.log
grew 8.5 MB an hour on each node, about 200 MB a day, without bound.

Also rotate when the file passes 50 MB. Rotated logs are compressed and the
last 5 kept, as before, so a node's logs stay bounded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant