Skip to content

Reliability sprint b - #50

Merged
primoco merged 6 commits into
mainfrom
reliability-sprint-b
Oct 10, 2026
Merged

primoco merged 6 commits into
mainfrom
reliability-sprint-b

Conversation

@primoco

@primoco primoco commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

What changes

How it was tested

A decision was delivered only to a run waiting in the same process: with
a second Core, or a decision written to the database, the run never woke
and expired, and a decision landing just before the expiry left the
database saying approved while the tool never ran. A waiting run now
also re-reads its approval from the store, and after expiring it honours
whatever the store holds. At startup, runs a previous process of the
same kind (api: or telegram:) left queued, running or waiting fail as
interrupted and their approvals expire. The API stops gracefully on
Ctrl-C or SIGTERM.
The daily limits were checked before the work and counted after it, so
requests arriving together (fetches waiting for their host slot, above
all) all passed the same check. Admitted runs and fetches now hold a
reservation counted against the limit until the store records them, and
the check-then-reserve step is serialised. The monthly budget is also
re-checked before every model call of a run, so one long run cannot
overshoot it. The per-host pacing map drops expired entries instead of
growing without bound.
- Analyses, discoveries and drafts started from the API, and every
  scheduled step, run as jobs: a failure is recorded with its error and
  shown on the site page (GET /api/jobs too) instead of 'started' forever;
  jobs with no result after two hours are reported as interrupted.
- The hourly tick runs each step on its own, so a failing collection no
  longer skips maintenance or the briefing. Maintenance runs once a local
  day from 03:00 and the review once a week from Monday 04:00, based on
  their last successful run: a late or missed tick no longer skips them.
- A briefing built but not delivered is sent again within its window
  (channels already sent are skipped). An address the mail library
  rejects no longer aborts the tick, recipients get the briefing in Bcc,
  and recipient addresses with control characters are refused.
- A publication is marked 'sending' before anything leaves: two workers
  cannot both send it, and one cut short stays 'sending' until a person
  checks the CMS and marks it failed, instead of being sent twice.
Editor Mode had an EDITOR_GITHUB_TOKEN it never used, and the Core
forwards only Accept and conditional headers, so source discovery ran on
GitHub's unauthenticated rate limit. api.fetch.credentials lets the
operator give the Core a header value (from an environment variable) for
an exact host: added over https only, on the first request only, never
after a redirect. Callers never hold the token. The unused Editor Mode
setting is removed, and the README documents both it and the publishing
secret variables.
@primoco
primoco merged commit 45f43c3 into main Oct 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant