Skip to content

Latest commit

 

History

History
407 lines (352 loc) · 24.2 KB

production-operations.md

File metadata and controls

407 lines (352 loc) · 24.2 KB

Production operations

Use this template for a deployed website after validating its installation, backup/restore and notification configuration. Locked checks and update canaries describe repository CI; website monitoring is configured for each installation.

This recipe targets PHP 8.5 FPM, nginx, MariaDB and Composer-managed WordPress. It is executable, but hostname, Unix identities, SSH host keys, backup encryption recipient and notification recipient must be selected for the actual deployment. Do not apply it to a running site without the site's change approval.

Build before deployment credentials

The reusable deploy-deployer.yml workflow fetches locked dependencies without scripts or plugins, then builds in a separate job with no installation or deployment secrets. In that isolated build job, the installed Composer plugins repair the WordPress layout; Composer scripts stay disabled until the explicit Runtime command. A run-scoped artifact is verified before the deployment job loads GITHUB_USER_SSH_KEY. Its tool project is deployment/, with committed composer.json and composer.lock; dependencies are installed with --no-dev --no-scripts --no-plugins. The default ./vendor/bin/dep deploy works inside deployment/: its thin deployment/deploy.php loads the project recipe. The locked root npm build script calls dev-ops/build.php before SSH credentials are loaded; no custom deploy override is needed. The recipe accepts stages production and staging and reads DEPLOY_HOSTNAME, DEPLOY_PORT, DEPLOY_USER and optional DEPLOY_PATH (default /srv/sympress). Set SYMPRESS_FPM_SERVICE to the remote systemd service (default php8.5-fpm). The deploy identity needs a narrow passwordless sudo rule for /usr/bin/systemctl reload php8.5-fpm using the selected service. The recipe checks the active service and sudo permission before deploy or rollback. Doctor also needs permission to execute as the configured PHP identity after the release permissions are applied. With the default paths and identity, add this exact command to the host's narrow passwordless sudo policy (replace the deploy user and base path as needed). This example uses sudo 1.9.10 or newer and matches one release directory name:

deploy ALL=(www-data) NOPASSWD: /usr/bin/php ^/srv/sympress/releases/[A-Za-z0-9][A-Za-z0-9._-]*/vendor/bin/runtime doctor --production --database-health --php-user=www-data --json --no-interaction$

This checks actual filesystem access, including ACLs, under the PHP identity; passing only --php-user from the deployment identity cannot establish it. Install acl for setfacl, and libfcgi-bin for /usr/bin/cgi-fcgi; give the deploy identity access to the private pool socket. SYMPRESS_FPM_SOCKET defaults to /run/php/php8.5-fpm.sock. The nginx template uses the same socket; update its fastcgi_pass as well when selecting a custom pool. Published build verification uses the private FPM socket to inspect the active release's actual WordPress configuration, database and build ID. It requires a healthy response with the exact current build ID and rejects malformed, oversized and stale reports. No public REST endpoint or private package is required. Test the canonical HTTPS site separately after deploying. Supply independently verified SSH_KNOWN_HOSTS; dependency install credentials must be a separate read-only key and must end before any lifecycle/build code.

Build the artifact without production credentials:

composer install --no-dev --no-interaction --prefer-dist --no-scripts --no-plugins
# In the separate build environment, after installation credentials are removed:
composer install --no-dev --no-interaction --prefer-dist --no-scripts
php vendor/bin/runtime --no-interaction
# Demo only: composer compile-assets --mode production
composer dump-autoload --no-dev --optimize --classmap-authoritative
composer audit --locked
composer --working-dir=deployment install --no-dev --no-scripts --no-plugins

Do the install in an isolated build environment; runtime:setup runs the explicit reviewed orchestration provider. The demo asset compiler requires the npm lock; production must not use --ignore-lock. Run PHP/asset QA in a separate development install before the final --no-dev install; dropping dev autoload alone leaves dev bundle metadata behind. Authoritative autoload is for the completed deploy artifact; development keeps dynamic class discovery, and WordPress/plugin classes outside Composer remain WordPress-owned. Do not inject a production .env into a build.

Make the deploy identity a member of the selected PHP group (www-data by default), provision deployment-root group traversal, and select SYMPRESS_PHP_USER and SYMPRESS_PHP_GROUP when using a different FPM pool.

Provision /srv/sympress/shared/.env privately (0600, readable by the required runtime identity) and the persistent paths below before first deployment. DB, mail, salts and any API credentials belong here or in a private secret manager. A first production site installation is an explicit operator action; routine deployment expects an existing database and never seeds/resets its content.

WORDPRESS_ENV=production
WP_HOME=https://www.example.com
WP_SITEURL=${WP_HOME}
DISALLOW_FILE_EDIT=true
DISALLOW_FILE_MODS=true
DISALLOW_UNFILTERED_HTML=true
ALLOW_UNFILTERED_UPLOADS=false
FORCE_SSL_ADMIN=true
WP_DEBUG=false
WP_DEBUG_DISPLAY=false
WP_DEBUG_LOG=/srv/sympress/shared/var/log/wordpress-debug.log
DISABLE_WP_CRON=true
# Only when TLS terminates at a controlled reverse proxy:
# WP_FORCE_SSL_FORWARDED_PROTO=true
# SYMPRESS_RUNTIME_TRUSTED_PROXIES=10.10.0.0/24

Add the private DB connection and WordPress salts. Task-6 Runtime requires canonical WP_HOME for staging/production and trusts forwarded HTTPS only when the immediate REMOTE_ADDR matches SYMPRESS_RUNTIME_TRUSTED_PROXIES and the header is exactly https. Do not trust X-Forwarded-For as the peer address. The committed stable Runtime requirement is recorded in composer.json; use its actual published lock revision and a current successful update canary instead of a historical candidate harness.

Deployer uploads the built artifact, copies a private regular .env snapshot from shared storage, symlinks public/wp-content/uploads and var/log, creates release-specific var/cache, generates release-local Runtime configuration with installer commands skipped, checks that WordPress is already installed, then runs Runtime and bin/console lint:container to warm/validate the kernel, writes the selected environment dump, then runs runtime doctor --production --database-health and a real WP configuration/DB gate before atomically changing current. The recipe selects SYMPRESS_KERNEL_IMMUTABLE_CACHE=true with a unique nonempty SYMPRESS_KERNEL_BUILD_ID=release-<source_commit>-<release_name> in GitHub Actions (or release-<release_name> for a manual deployment) for every release; default content/listing checks remain active outside this explicit deployment policy. The source recipe never clones or builds on production. Failed pre-switch checks leave the current release active; deploy:failed releases only the lock acquired by this deployment. A failure before acquiring the lock preserves any existing deployment lock. The database is not automatically migrated or rolled back.

The deploy identity owns release files. The permission step assigns the selected PHP group, makes directories 0750 and code/config 0640 (executables 0750), makes shared uploads/logs 2770/0660, then runs doctor for the selected PHP identity. PHP-FPM receives read/execute access to code/config and write access only to uploads/logs/runtime cache as required. Warm and seal release-specific compiled kernel caches where supported; use Runtime's --php-user=www-data --webroot=.../public doctor to verify that identity after permission provisioning. Never make the project root the HTTP docroot.

./deployment/vendor/bin/dep -f deploy.php deploy production -v
# If a code-only rollback is appropriate:
./deployment/vendor/bin/dep -f deploy.php rollback production -v

After switching the release symlink and after an existing rollback, the recipe gracefully reloads FPM, resets OPcache through its private Unix socket, and checks the private FPM health response against the current release's SYMPRESS_KERNEL_BUILD_ID. A failed reset, reload or Build-ID check fails the operation before deploy cleanup. Candidate helpers are installed privately in shared/sympress-tools, so retained releases need no helper files. A retained release without a Build-ID still reloads FPM, then explicitly fails verification. A CLI opcache_reset() does not reset FPM or its preload state. Do not automatically restore the database for a code rollback; inspect migration compatibility and use an approved paired backup when restoration is needed. Five old releases are retained. Shared media/logs/.env survive rollbacks.

The isolated regression below runs native Deployer 8 worker processes and its actual symlink and failure hooks against a temporary deployment and an owned PHP-FPM pool. It checks foreign lock preservation before locking and on a lock conflict, then injects failures before publication and during reload and OPcache reset. It verifies the old release through FastCGI and proves releasing the owned lock even when recovery itself fails. The WP-CLI HTTPS transport is a fixture double; the FPM execution and Deployer state transfer are real.

python3 dev-ops/tests/failed_deploy_probe.py --deployer deployment/vendor/bin/dep
# qa:operations includes this fixture when that binary and PHP-FPM 8.5 exist;
# SYMPRESS_DEPLOYER_PROBE selects an installed Deployer binary elsewhere.

Projects using database migrations require sympress/migration:^1.0.3 or newer. Its WP-CLI commands reject rollback, execute --down and a backward migrate target outside local/development, before changing data or applied history. Missing or unknown WordPress environment also rejects these directions. Recovery uses an explicitly reviewed backup procedure; there is no CLI force bypass.

nginx and cache policy

Render the server recipe before installation:

python3 dev-ops/render-nginx.py --hostname www.example.com \
  --root /srv/sympress --output /tmp/sympress-nginx

Install the rendered files in the corresponding release dev-ops/nginx paths and include cache-http.conf once inside nginx http {} and production-server.conf in the site's enabled virtual hosts. Set the actual FPM socket/certificate paths, provision /var/cache/nginx/wordpress for nginx, run nginx -t, then reload only through the approved deployment mechanism. HTTP redirects to a literal canonical HTTPS hostname. Enable HSTS only after certificate/chain, canonical redirect and HTTPS login/health acceptance using the renderer's explicit --hsts flag. Rendered configurations omit HSTS by default. HSTS applies only to the accepted TLS site and deliberately omits includeSubDomains/preload until every subdomain is ready. Headers are repeated inside locations that declare their own add_header because nginx inheritance otherwise drops them. The baseline CSP limits frame-ancestors; extend it only with a tested application script/style policy.

Uploads/cache/upgrade paths deny nested PHP, phpN, phtml, pht, phar and inc including PATH_INFO, before asset/PHP locations. Index listings are disabled. XML-RPC is intentionally unavailable at nginx and WordPress. The native MU policy works without any unpublished sibling security package. File changes belong to Composer deployments; WordPress file editing/modification are disabled in production.

Only filenames with 8–64 hex fingerprints get immutable for one year. Mutable assets and JSON manifests use five minutes plus revalidation. Pure sequences of utm_*, fbclid and gclid parameters use the clean original request path for page-cache keys. Unknown/semantic/search/auth parameters bypass. Cookie bypass includes login/password/comment, commerce, membership and language/currency variants. SymPress Consent gates scripts in cookie-invariant HTML, so its cookie shares the anonymous page cache. Add an explicit cookie bypass only for integrations that render different HTML on the server. Authorization and unsafe HTTP methods bypass. Upstream Set-Cookie prevents storage. REST/admin/login endpoints bypass. Consent HTML must still remain cookie-invariant as defined by the consent package.

OPcache and object cache

Install dev-ops/php-production.ini for the FPM SAPI; validate with php-fpm -tt and inspect the loaded settings before reload. opcache.validate_timestamps=0 requires immutable release paths and FPM reloads. Optional opcache.preload points to the release's dev-ops/preload.php, with opcache.preload_user=www-data; it compiles a small reviewed dependency core from the authoritative class map without executing WordPress hooks. Whole-vendor preloading includes optional integrations whose parent classes may be absent; extend the selected list only with measured, validated production dependencies. Enable only after testing with the final package set and budget memory from actual measurements. The kernel cache stays release-specific to avoid stale containers.

The supported WordPress profile explicitly sets opcache.jit=disable and opcache.jit_buffer_size=0 while retaining ordinary OPcache. In the native review fixture, PHP 8.5.9 with OPcache enabled, JIT 1235 and a 256 MiB JIT buffer exited with SIGSEGV (11); the same pinned WordPress/guard fixture passed with JIT disabled. This is evidence for that tested combination, not a finding that every PHP 8.5 release or application fails with JIT. The PHP default disable prevents enabling JIT at request time; a zero buffer also reserves no JIT memory. Install this file after inherited PHP fragments and check the effective FPM settings. The native profile test verifies that a previously enabling fragment is overridden, OPcache still works, and a request cannot enable JIT. See the PHP configuration reference.

The optional sympress/framework-bundle supplies a WordPress object-cache adapter. The pilot default is Redis with that adapter from ^1.0.3, exactly one drop-in owner and an independent site/environment secret. Keep the generic template optional; select this profile in the generated pilot project. In dev-ops/runtime.json, add:

"dropins": {"object-cache.php": "vendor/sympress/framework-bundle/dropin/object-cache.php"},
"dropins-op": "copy"

Runtime publishes the drop-in during the credential-free build. Re-run Runtime before deployment; production's file-modification policy intentionally prevents request-time installation. Keep any existing foreign drop-in until its owner has been explicitly removed. The Redis PHP extension is required. Add the following private values to the deployment .env, then regenerate its selected environment dump (do not print the secret or DSN):

SYMPRESS_CACHE_DRIVER=redis
SYMPRESS_CACHE_DSN=redis://127.0.0.1:6379/0
SYMPRESS_CACHE_PREFIX=example.production
SYMPRESS_CACHE_SECRET=<independent-long-random-secret>

Use a protected Redis endpoint and private ACL credentials in the DSN when needed; deny public Redis access and use verified TLS for network connections. Allocate a unique prefix for every site/environment. A prefix is not tenant isolation: use a separate customer Redis instance or an ACL that demonstrably restricts that customer's keys and commands. The fixed framework adapter reads native Runtime dotenv values without exporting them into child process environments and rejects legacy unsigned cache entries. Optional cache installation is not selected by this template without operator choice.

Verify persistence in two separate WordPress processes, since a request-local fallback can still report an external drop-in:

probe=$(php -r 'echo bin2hex(random_bytes(16));')
php wp-cli.phar eval-file dev-ops/object-cache-check.php write "$probe"
php wp-cli.phar eval-file dev-ops/object-cache-check.php read "$probe"

Run the same probe as the deployed PHP identity after deployment and perform an actual HTTP request through FPM. The check uses one unique 60-second key and never flushes a shared backend. Backend failure must fail the second process's check; inspect the private application log for the generic fallback warning.

Encrypted backups and controlled restore

Install Python 3.12+, age, PHP and the database CLI. Copy dev-ops/operations.example.json to /etc/sympress/operations.json, mode 0600, owned by the backup/deploy identity. Set the canonical project/shared uploads path and the off-host age public recipient. The private decryption identity never belongs on the web host or in Git. Backups enable WordPress maintenance, export the DB with a consistent transaction, package uploads plus a checksum manifest, encrypt with age and leave only a 0600 encrypted archive. Maintenance enabled by the backup is cleared after failure; pre-existing maintenance remains active. Set the shared lock_file to serialize DB/media operations across releases. Other systems writing media/DB must be quiesced for a paired snapshot. Retention, off-host replication and restore drills are operator responsibilities.

python3 dev-ops/operations.py --config /etc/sympress/operations.json \
  backup /srv/backups/sympress/20261001T030000Z.tar.gz.age
# Take a new pre-restore backup FIRST; the target DB is overwritten.
python3 dev-ops/operations.py --config /etc/sympress/operations.json restore \
  /trusted/backups/snapshot.tar.gz.age --identity /private/backup-key.txt \
  --confirm https://www.example.com --allow-destructive

Restore requires exact installed target home URL confirmation; production also requires allow_production_restore=true in private config. It validates age, rejects archive links/device/path traversal, checks the database digest, imports, performs serialized-aware search-replace (skipping GUIDs), preserves the old uploads directory, installs restored media and checks DB/cache. A partial restore stays in maintenance and reports failure; it never claims transactionally restored media/DB or silently rolls back. Use only backups from trusted storage. The archive contains production personal data and integration secrets even while encrypted.

For staging, provision a separate DB, private staging .env with WORDPRESS_ENV=staging, DISABLE_WP_CRON=true, canonical staging WP_HOME and outbound network restrictions before importing. Activate the private Security MU loader or a project-owned staging mail guard first. Sync checks that pre_wp_mail blocks mail and refuses before decryption or database changes if no guard is active. Select/write a reviewed private scrub PHP script that rotates/removes copied users' passwords/tokens, API/webhook/payment credentials and personal data for that site. There is no universal safe scrub for arbitrary plugins; sync refuses without it.

python3 dev-ops/operations.py --config /etc/sympress/staging.json sync-staging \
  /trusted/backups/snapshot.tar.gz.age --identity /private/backup-key.txt \
  --confirm https://staging.example.com --allow-destructive \
  --scrub-script /private/staging-scrub.php

Install the supplied backup service/timer only after filling private config and validating the commands on a disposable restore target. % in systemd date formats is intentionally escaped as %%. Run systemd-analyze verify before enabling.

Uptime/error and canary notifications

The starter does not ship a public health endpoint. The optional private natterer-schaeffner/bundle-security MU loader provides /wp-json/sympress/v1/health, returning status and an optional opaque build_id, with 503 for DB failure. Alternatively supply your own application health endpoint and update health_url in the private monitor configuration. The supplied monitor checks canonical HTTPS, the status and newly appended fatal/uncaught/error log lines. Set log_glob to production-????-??-??.log for the supplied Monolog rotating handler. Each file has its own private byte cursor, so late records in yesterday's file are still read. A quiet day before the first warning needs no log file; a missing/unreadable log directory is a failure. A fixed log_file remains supported. Rotation/truncation reset the corresponding cursor. Failed health/log checks return nonzero and send a generic message through local sendmail to the explicit alert_recipient. They never include URLs/log content/provider secrets. Select a real recipient and configure/test local MTA delivery before enabling the monitor timer. Use a separate /etc/sympress/monitor.json owned by the monitor identity (0600), from dev-ops/monitor.example.json, with only health/log/cursor/recipient settings. Monitoring does not load the site environment or require access to DB credentials. Install the reviewed monitor script separately so its identity needs no access to release code or private .env snapshots. Give sympress-monitor read access to the selected PHP/application log, and write access only to /var/lib/sympress-monitor.

systemd-analyze verify dev-ops/sympress-monitor.service dev-ops/sympress-monitor.timer
install -d -m 0755 /usr/local/lib/sympress
# Provision the dedicated sympress-monitor Unix identity first.
install -d -o sympress-monitor -g sympress-monitor -m 0700 /var/lib/sympress-monitor
install -m 0755 dev-ops/operations.py /usr/local/lib/sympress/operations.py
# Provision /etc/sympress/monitor.json from the monitor example, mode0600.
python3 /usr/local/lib/sympress/operations.py --config /etc/sympress/monitor.json monitor
# Approved operator installs units/config and enables both reviewed timers.

Canary runs verify the locked install and scheduled dependency updates. They do not use heartbeat services or create alert issues. Their failures remain visible in GitHub Actions.

The first log-monitor run establishes a cursor at the end of existing logs. Subsequent runs report new error severity/PHP locations once per 24 hours while repeated occurrences continue. Only hashes of severity and PHP location are kept, without error messages or credentials. A failed notification remains pending.

Deployment tools and this recipe are loaded from a fresh trusted checkout. SYMPRESS_RELEASE_DIRECTORY points to the separately verified payload, which is uploaded as data. The private shared environment is copied into the release; SYMPRESS_PROJECT_DIR is set to the stable deployment base. Build IDs include the source commit when GitHub supplies it. A failure after the symlink switch restores the previous release and refreshes FPM; first deployments have no previous release to restore.

Primary references

Both Runtime configs pin WP-CLI v2.12.0 with SHA256 ce34ddd838f7351d6759068d09793f26755463b4a4610a5a5c0a97b68220d85c, verified against the official release PHAR. Retain sympress-runtime.lock; a GitHub metadata outage does not require floating latest or an integrity bypass.

The optional Security bundle's public health response exposes only generic status and the opaque Build-ID (null in development without an ID), never configuration or credentials. The monitor account needs the configured PHP log group; its systemd example uses SupplementaryGroups=sympress-log. Daily logs use production-????-??-??.log; new log errors send one alert, a persistent HTTP outage is deduplicated until recovery, and failed deliveries remain pending for retry.

The log group is separate from the PHP code/configuration group. Create the selected SYMPRESS_LOG_GROUP (default sympress-log) and add the FPM and deploy identities to it; the monitor receives only this group. Code and .env remain under SYMPRESS_PHP_GROUP. Health monitoring rejects redirects and requires a bounded JSON object with top-level status: ok from the canonical HTTPS endpoint. The recipe grants the log group only directory traversal (--x) on the deploy, shared and shared/var ancestors with POSIX ACLs; it never grants that group access to .env or code. Keep the monitor log path under shared/var/log.