Skip to content

docs(ops): record the staging cutover and correct two rite-calendar runbook steps (#955) #966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JohnRDOrazio merged 3 commits into from
Sep 2, 2026
Merged

docs(ops): record the staging cutover and correct two rite-calendar runbook steps (#955) #966

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3d6ae69
docs(ops): record the staging cutover and correct two Step 1/4 instru…
JohnRDOrazio Sep 1, 2026
0e45334
docs(ops): address review on the rite-calendar runbook (#955)
JohnRDOrazio Sep 2, 2026
cbae84a
docs(ops): harden the token read and anchor the PR number (#955)
JohnRDOrazio Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 92 additions & 10 deletions docs/ops/rite-calendar-migration-runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,39 @@ itself is left in the model unchanged — it is only removed at the prune milest
the latest authorization model already in that store (it does not create or upload one), and —
with `--update-env` — writes the resulting `OPENFGA_STORE_ID` and `OPENFGA_MODEL_ID` into the
`.env.*` file(s) and Docker Compose `.env` files in this repo and its siblings.

**That command is for a checkout, not for the deployed vhost, and running it in the wrong place
re-pins the wrong store.** The deployed API's pin against the PRODUCTION store lives in exactly
one file — `api/dev/.env.staging` on the VPS, the only `.env*` that deployment has, so there is
no Dotenv precedence to reason about. `api/v4` and `api/v5` carry no `OPENFGA_*` keys at all.
Run from a local checkout the command above resolves whatever store that checkout points at
(a local dev store, not production) and writes those values, so edit the one VPS file instead.

Edit it as its owner, and never point a shell redirect at the live file: a redirect truncates
its destination BEFORE the command on its left runs, so `sed ... > .env.staging` or
`cat tmp > .env.staging` leaves the file EMPTY if that command then fails. A zero-byte
`.env.staging` takes the API down immediately, since phpdotenv reads it per request (observed
2026-09-01).

Write a temp copy in the SAME directory instead, validate it, and `mv` it into place. `cp -p`
seeds the temp with the live file's owner, group and mode, so the replacement keeps them; the
redirect then truncates only that temp; and `mv` within one directory is an atomic rename, so
no in-flight request can read a half-written file. Build the temp as the same user that
performs the write — a `mktemp` file made by `ubuntu` is mode 600 and unreadable to the user
performing the write.

```bash
sudo -u <owner> bash -c 'cd <deploy-dir> \
&& cp -p .env.staging .env.staging.bak.<label> \
&& cp -p .env.staging .env.staging.new \
&& sed "s/^OPENFGA_MODEL_ID=<old>$/OPENFGA_MODEL_ID=<new>/" .env.staging.bak.<label> > .env.staging.new \
&& grep -q "^OPENFGA_MODEL_ID=<new>$" .env.staging.new \
&& mv .env.staging.new .env.staging \
|| rm -f .env.staging.new'
```

Then confirm owner/group/mode are unchanged (`stat -c %U:%G,%a .env.staging`) and that the API
still answers (`curl -o /dev/null -w '%{http_code}' <base>/calendars`). No restart is needed.
3. Re-run the pre-flight model check above and confirm `"rite_calendar"` now appears in the output.

**Nothing else in this runbook can start until this step is confirmed done.** Steps 2 and 3 both
Expand All @@ -100,7 +133,7 @@ outright rather than silently doing nothing.

## Step 2 — Deploy this API version

Deploy the #955 build to the target environment. Both `OpenFgaAuthorizationMiddleware::forRiteCalendar()`
Deploy the #955 build — PR #965, the change this runbook accompanies — to the target environment. Both `OpenFgaAuthorizationMiddleware::forRiteCalendar()`
and `::forMissals()` check the object `rite_calendar:{rite}/{subResource}` first, with a legacy
fallback — but the two do **not** fall back the same way, and conflating them is a real
operational hazard (an operator could wrongly conclude Ambrosian missal grants stop being honoured
Expand Down Expand Up @@ -136,10 +169,16 @@ they preserve *whether a caller may write*, and nothing else. In particular they
`resource_id` stored on the change-request row. After Step 4 rewrites those to the new type, a
user still holding only the legacy tuple stops seeing those batches in their review queue.

Both resume as soon as Step 3 has run. This is deliberate and fail-closed: that path decides
governance rather than access, and silently auto-approving off a legacy tuple during the migration
window would be worse than queueing for a human. Nothing is lost — queued requests are approved
normally once the tuples are migrated.
Both resume as soon as Step 3 has run, but not in the same way, and the difference decides what
an operator has to chase afterwards. Reviewer-queue visibility is evaluated per query, so it
returns in full on its own. Auto-approval is evaluated ONCE, on the way in:
`ChangeRequestSourceDataWriter` calls `ChangeRequestReview::administers()` at submit time and
records the batch as `submitted` when it is false, and nothing re-evaluates it later. So a request
that queued during the window stays queued — it needs an explicit reviewer approval; only new
submissions auto-approve again. This is deliberate and fail-closed: that path decides governance
rather than access, and silently auto-approving off a legacy tuple during the migration window
would be worse than queueing for a human. Nothing is lost, but "approved normally" means by a
human, not retroactively.

**Therefore: run Step 3 immediately after Step 2.** Do not leave the two separated by a maintenance
window or a working day; every minute in between is a minute in which legitimate change requests
Expand Down Expand Up @@ -205,10 +244,53 @@ skipped (same meaning as in the dry run).

## Step 4 — Apply the Doctrine migration

**On a deployed vhost this step runs itself**, and cannot be run the way the command below
suggests: `bin/doctrine-migrations` is excluded from the rsync payload
(`.github/deploy/rsync-exclude.txt`) precisely because migrations are applied in-process, and
`deploy.yaml` POSTs `/_ops/migrate` after every rsync. So on `api/dev` the migration lands at
deploy time, as part of Step 2. Check the state rather than assuming.

`DEPLOY_TOKEN` is a shared secret and is not set in an operator shell by default — read it from
that host's env file rather than assuming it is exported, or the request goes out with an empty
header and `DeployTokenMiddleware` fails closed on it. Pass it through a curl config on stdin
rather than in `-H`, which would put it in the process argument list where any other user on the
VPS can read it with `ps`. `BASE` must be an `https://` URL; `--proto '=https'` makes curl refuse
to send the token over anything else rather than trusting a route-level redirect.

Read it by absolute path, not from whatever the current directory happens to be, and check that
it is non-empty before calling curl — a wrong path yields an EMPTY token, and the request then
goes out with an empty header and comes back a fail-closed 401/403 that looks like a token
problem rather than a path problem. No `sudo` is needed for this read: `.env.staging` is mode 640
and group `psacln`, which the operator account is in.

```bash
D=/var/www/vhosts/johnromanodorazio.com/httpdocs/LiturgicalCalendar/api/dev
DEPLOY_TOKEN=$(sed -n 's/^DEPLOY_TOKEN=//p' "$D/.env.staging")
if [ -z "$DEPLOY_TOKEN" ]; then
echo "no DEPLOY_TOKEN in $D/.env.staging" >&2
else
printf 'header = "X-Deploy-Token: %s"\n' "$DEPLOY_TOKEN" \
| curl -fsS --proto '=https' -K - "${BASE}/${SUBDIR}/_ops/migrate/status"
fi
unset DEPLOY_TOKEN
```

Only where you have a CLI checkout (local, CI) is the command:

```bash
composer db:migrate
```

**This inverts the Step 3 → Step 4 order on any auto-migrating deployment**, and the sequence
cannot be rearranged: the deploy applies the migration, so Step 4 completes at Step 2 time,
before the tuples move. The effect is to widen the Step 2 window described above rather than to
create a new hazard — change-request rows name `rite_calendar` while their tuples are still
legacy, so those requests queue for a reviewer instead of auto-approving. New submissions
auto-approve again as soon as Step 3 runs, but the ones that queued during the window stay
queued and need an explicit reviewer approval: auto-approval is decided once, at submit time.
Nothing is corrupted and no row is lost. It is one more reason to run Step 3 immediately after
Step 2.

`Version20260901130000` rewrites two Postgres tables from `general_roman_calendar[_test]` onto
`rite_calendar[_test]`: `sourcedata_change_requests.resource_type`/`resource_id`, and the JSONB
`permissions` array on `access_requests` (element-wise, order-preserving). Both statements are
Expand All @@ -221,11 +303,11 @@ written.

**Record the cutover date here, at the moment this step is run in each environment:**

| Environment | Cutover date (this step run) | Operator |
|-------------|------------------------------|----------|
| Dev | | |
| Staging | | |
| Production | | |
| Environment | Cutover date (this step run) | Operator |
|-------------|------------------------------|-----------------------------------------------|
| Dev | | |
| Staging | 2026-09-01T20:23:37Z | `deploy.yaml` (automatic, at the #965 deploy) |
| Production | | |

An `audit_log` row timestamped before its environment's date above names `general_roman_calendar` /
`general_roman_calendar_test`; a row timestamped after names `rite_calendar` / `rite_calendar_test`.
Expand Down
Loading