fix(logging): rotate exo's log file by size, not only when exo starts - #2390
Open
AlexCheema wants to merge 1 commit into
Open
AlexCheema wants to merge 1 commit into
AlexCheema wants to merge 1 commit into
Conversation
exo started a new log file on each start and never again, so a node that ran for a long time kept one ever-growing file. At default verbosity each request logs a few lines: on a 2-node soak with steady load (~1 request/s) exo.log grew 8.5 MB an hour on each node, about 200 MB a day, without bound. Also rotate when the file passes 50 MB. Rotated logs are compressed and the last 5 kept, as before, so a node's logs stay bounded. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
exo starts a new log file (
~/.exo/exo_log/exo.log) each time it starts, and never during a run. A node that runs for days or weeks keeps one ever-growing file.At default verbosity each request logs a few lines: the worker's plan, the task it starts (with its parameters), and KV cache activity. On a 2-node soak test, two Mac Studios (M3 Ultra) serving steady load of about 1 request a second,
exo.loggrew 8.5 MB an hour on each node, about 200 MB a day, with no limit.Fix
Rotate the file when exo starts, as before, and also whenever it passes 50 MB. Rotated files are compressed with zstd, and the last 5 are kept, as before. So a node's logs are bounded at about 50 MB plus a few MB of archives.
Tests
test_log_rotation.pysets exo's logger up with a 20 KB limit:A 24-hour soak with this change follows; I'll add its disk measurements here.
🤖 Generated with Claude Code