migration to another DB backend #3336 #3343
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| # `bw_db_migrate.py` (BunkerWeb DB backend migration) | ||
|
|
||
| `bw_db_migrate.py` migrates BunkerWeb database data between supported backends by: | ||
| 1. Creating the BunkerWeb schema on the target backend (Alembic). | ||
| 2. Dumping all BunkerWeb tables from the source backend into intermediate JSONL files. | ||
| 3. Importing those JSONL files into the target backend. | ||
| 4. Verifying per-table row counts between source and target. | ||
|
|
||
| It can also run in a “no changes” mode (`--dry-run` / `--test-target`) or launch an interactive TUI (`--tui`). | ||
|
|
||
| ## When to use it | ||
|
|
||
| Use this script when you want to move your BunkerWeb runtime database backend, for example: | ||
|
|
||
| - MySQL/MariaDB -> PostgreSQL | ||
| - PostgreSQL -> MySQL/MariaDB | ||
| - Any supported backend -> SQLite (or vice-versa) | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python dependencies as shipped with the BunkerWeb installation. | ||
| - Network connectivity (when using remote source/target backends). | ||
| - Sufficient privileges on the source/target to read tables and write/import the BunkerWeb schema. | ||
|
|
||
| ## Configuration inputs | ||
|
|
||
| The script reads connection URIs from an environment file, or from CLI arguments. | ||
|
|
||
| ### Environment file | ||
|
|
||
| By default, use: | ||
|
|
||
| `/etc/bunkerweb/variables.env` | ||
|
|
||
| It should contain at least: | ||
|
|
||
| - `DATABASE_URI`: source database SQLAlchemy URI | ||
| - `DB_MIGRATION_TARGET_URI`: target database SQLAlchemy URI | ||
|
|
||
| ### CLI overrides | ||
|
|
||
| You can override either value with: | ||
|
|
||
| - `--source-uri <URI>` | ||
| - `--target-uri <URI>` | ||
|
|
||
| ## CLI options | ||
|
|
||
| Run: | ||
|
|
||
| `python3 bw_db_migrate.py [options]` | ||
|
Comment on lines
+49
to
+51
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Locate migration script and any wrapper/entrypoint mentions =="
fd -i 'bw_db_migrate.py'
rg -n -C2 --type=py 'bw_db_migrate|argparse|ArgumentParser|add_argument\('
echo
echo "== Check README command examples for invocation style =="
rg -n -C2 'python3 bw_db_migrate.py|--auto-switch|--test-target|--dry-run|--tui' src/common/utils/README-bw_db_migrate.md || true
echo
echo "== Cross-check same command style in other docs =="
rg -n -C2 'bw_db_migrate.py|--auto-switch|--test-target|--dry-run|--tui' --type=mdRepository: bunkerity/bunkerweb Length of output: 50375 🏁 Script executed: # Check for setup.py, pyproject.toml, or installation entry points
find . -maxdepth 3 -type f \( -name "setup.py" -o -name "pyproject.toml" -o -name "setup.cfg" \) 2>/dev/null | head -20
# Look for entry points or console_scripts references to bw_db_migrate
rg -n 'bw_db_migrate|entry.points|console_scripts' --type=py,toml,txt 2>/dev/null | grep -i 'entry\|console\|script' | head -30
# Check if there's a wrapper script or symlink mechanism
fd -type f -o -type l 'bw.?db.?migrate|bw-db-migrate' 2>/dev/null
# Check Docker or packaging files that might install the script
fd -type f \( -name "Dockerfile*" -o -name "*.sh" \) -exec rg -l 'bw_db_migrate' {} \; 2>/dev/null | head -10Repository: bunkerity/bunkerweb Length of output: 79 🏁 Script executed: # Examine pyproject.toml for entry points and installation configuration
cat -n pyproject.toml | head -100
# Search specifically for bw_db_migrate in pyproject.toml
rg -n 'bw_db_migrate|console.scripts|entry' pyproject.tomlRepository: bunkerity/bunkerweb Length of output: 522 🏁 Script executed: # Get the full pyproject.toml file
wc -l pyproject.toml
# Look for any [project.scripts] or similar entry point sections
rg -n '\[.*script' pyproject.toml
# Check if bw_db_migrate is referenced anywhere in packaging config
rg -n 'bw_db_migrate' . --type=toml --type=py --type=txt 2>/dev/null | grep -E '(setup|script|entry|install|package)' | head -20
# Check for install/deployment files that might show how the script is deployed
fd -type f -name '*.mk' -o -name 'Makefile*' | xargs grep -l 'bw_db_migrate' 2>/dev/null
# Look at Dockerfile entries
fd -type f -name 'Dockerfile*' | xargs grep -A2 -B2 'bw_db_migrate' 2>/dev/null | head -50Repository: bunkerity/bunkerweb Length of output: 439 🏁 Script executed: # Get full pyproject.toml content
cat pyproject.toml
# Search for script installation references
rg -n 'bw_db_migrate' . 2>/dev/null | grep -E '(Dockerfile|install|script|setup|entry)' | head -20Repository: bunkerity/bunkerweb Length of output: 389 🏁 Script executed: # Search for Dockerfile references to bw_db_migrate
find . -name 'Dockerfile*' 2>/dev/null | xargs grep -B3 -A3 'bw_db_migrate' 2>/dev/null | head -60
# Search for install scripts or deployment docs that mention bw_db_migrate
rg -n 'bw_db_migrate' --type-add 'docker:Dockerfile*' -g '*.md' -g 'Dockerfile*' 2>/dev/null | head -40
# Look at the actual script invocations in tests or CI
rg -n 'python.*bw_db_migrate' . 2>/dev/null | head -20Repository: bunkerity/bunkerweb Length of output: 1554 Update command invocation paths to match installed packaging behaviour. Lines 51, 79–97 show Align README examples with the correct invocation path or document that users must change to 🤖 Prompt for AI Agents |
||
|
|
||
| Common options: | ||
|
|
||
| - `--env-file <path>` | ||
| - Environment file path (default: `/etc/bunkerweb/variables.env`) | ||
| - `--source-uri <URI>` | ||
| - Source SQLAlchemy URI override | ||
| - `--target-uri <URI>` | ||
| - Target SQLAlchemy URI override | ||
| - `--test-target` | ||
| - Only tests that the target is reachable and appears empty (no migrations/copy) | ||
| - `--dry-run` | ||
| - Summarizes what would be migrated by counting rows per BunkerWeb table | ||
| - Does not modify the target | ||
| - `--tui` | ||
| - Launches the interactive terminal UI to collect inputs | ||
| - `--auto-switch` | ||
| - After a successful migration: | ||
| 1. Updates `DATABASE_URI` in the `--env-file` to point to the migrated target. | ||
| 2. Restarts BunkerWeb services via a best-effort init-system restart: | ||
| - systemd if available (restarts `bunkerweb`, `bunkerweb-scheduler`, `bunkerweb-ui`) | ||
| - otherwise falls back to init scripts / `service` | ||
|
|
||
| ## Examples | ||
|
|
||
| ### 1) Quick connectivity + empty target check | ||
|
|
||
| `python3 bw_db_migrate.py --test-target` | ||
|
|
||
| ### 2) Dry-run (counts rows) | ||
|
|
||
| `python3 bw_db_migrate.py --dry-run` | ||
|
|
||
| Note: dry-run runs a `COUNT` query per known BunkerWeb table, so it can be slow on large datasets. | ||
|
|
||
| ### 3) Full migration | ||
|
|
||
| `python3 bw_db_migrate.py` | ||
|
|
||
| ### 4) Full migration with auto-switch | ||
|
|
||
| `python3 bw_db_migrate.py --auto-switch` | ||
|
|
||
| ### 5) Interactive TUI | ||
|
|
||
| `python3 bw_db_migrate.py --tui` | ||
|
|
||
| ## SQLite target notes (important) | ||
|
|
||
| If the target backend is SQLite (e.g. `sqlite:////var/lib/bunkerweb/db.sqlite3`): | ||
|
|
||
| - The target file must be empty (the script checks that BunkerWeb tables contain 0 rows). | ||
| - If the file already exists and has data, the script will instruct you to move/rename it. | ||
| - Example: | ||
| - `mv /var/lib/bunkerweb/db.sqlite3 /var/lib/bunkerweb/db.sqlite3.bak-<timestamp>` | ||
| - After a successful migration, the script attempts to `chown` the SQLite DB file to match the parent directory owner/group. | ||
| - This helps avoid scheduler failures caused by SQLite journal/WAL permission issues. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - Target not empty: | ||
| - For SQL backends: recreate the target database. | ||
| - For SQLite: move/rename the DB file, then re-run. | ||
| - Unencrypted connection warnings: | ||
| - When SSL is not enabled in the target URI, the script emits warnings and performs best-effort connectivity checks. | ||
| - Scheduler retry loops after auto-switch: | ||
| - If using SQLite, ensure the SQLite file is writable by the runtime service user (commonly `nginx`). | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use “vice versa” (no hyphen).
At Line 17, “vice-versa” should be “vice versa” in standard British English.
As per coding guidelines,
**/*.md: Documentation should be concise, accurate, and written in British English.🧰 Tools
🪛 LanguageTool
[grammar] ~17-~17: The expression “vice versa” is spelled without hyphens.
Context: ...B - Any supported backend -> SQLite (or vice-versa) ## Requirements - Python dependencie...
(VICE_VERSA)
🤖 Prompt for AI Agents