migration to another DB backend #3336#3343
Conversation
migration to another DB backend bunkerity#3336
Quoted env file values cause database connection failure
|
Hi @mkf-sysangels, great idea. I’d prefer each P/R to be small, atomic, and independent, without depending on other open P/Rs, since GitHub recommends “small, focused pull requests that fulfill a single purpose” because they’re easier to review and merge. |
|
thank you for the information. As the initial version is still alpha i need first fix issues to make it working on all scenarios. First migration SQLLite > mysql-cluster was successful Currently fixing mysql > postgresql patroni cluster As soon as i have full cycle i will call it beta so others can test it too. |
- fix postgres connection on patroni configuration (postgresql cluster) - fix SSL connect error on wrong selected root CA PEM - fetch root CA when cert is issued by trusted CA - add timeouts - add PID file - add debug logging - compare TLS conections for differences - abort on critical certificate error
- fix TUI scrolling and size changes - fix ssl connection issue - add --auto-switch for instant migration
fix errors on migration to SQLLite
|
I successfully migrated: SQLLite > mysql > postgrsql > mysql > SQLLite Ready for public review Status: |
…ces are not re-synced afterward in the migration utility. data rows are imported with explicit id values, but sequences are not re-synced afterward in the migration utility. adding an automatic sequence reset step right after import so next inserts use MAX(id)+1
…odel Post-copy step: ensure primary key constraints match the SQLAlchemy model
…on the target Post-copy step: recreate missing indexes and update index statistics on the target
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
Component: src/commonDatabase Migration Feature – New database migration utility ( ChangesDocumentation added (
Security & Configuration Impact
User-Visible Changes
Testing & Documentation
Status: Marked BETA pending comprehensive test case coverage. Feature has undergone iterative validation across multiple database backend combinations. WalkthroughA new documentation file is added describing the Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/common/utils/README-bw_db_migrate.md`:
- Line 17: Replace the hyphenated phrase "vice-versa" with "vice versa" in the
README-bw_db_migrate.md documentation (the string "Any supported backend ->
SQLite (or vice-versa)") so it uses standard British English; update the text to
"Any supported backend -> SQLite (or vice versa)" and ensure other occurrences
of "vice-versa" in the same file follow the same change.
- Around line 49-51: Update the README examples that call bw_db_migrate.py so
they use the repository-relative path used in the script's docstring—e.g.,
replace occurrences of "python3 bw_db_migrate.py [options]" with "python3
src/common/utils/bw_db_migrate.py [options]" (or alternatively document the
required directory change like "cd src/common/utils && python3 bw_db_migrate.py
[options]"); adjust all instances mentioned in the comment (the examples around
lines showing invocation) so they match the actual invocation behavior until a
console script/entry point is added to pyproject.toml.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: df56929f-d5c6-4fdd-b65e-6b79cc3db181
📒 Files selected for processing (2)
src/common/utils/README-bw_db_migrate.mdsrc/common/utils/bw_db_migrate.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
**/*.md
⚙️ CodeRabbit configuration file
**/*.md: Documentation should be concise, accurate, and written in British English:
- Keep the structure clear with a sensible heading hierarchy.
- Prefer concrete instructions, accurate examples, and explicit prerequisites.
- Check that commands, paths, and references still match the codebase and supported deployment modes.
- When a PR changes behaviour, defaults, packaging, or security posture, ask for the corresponding documentation update.
Files:
src/common/utils/README-bw_db_migrate.md
🪛 LanguageTool
src/common/utils/README-bw_db_migrate.md
[duplication] ~15-~15: Possible typo: you repeated a word.
Context: ...ckend, for example: - MySQL/MariaDB -> PostgreSQL - PostgreSQL -> MySQL/MariaDB - Any supported backen...
(ENGLISH_WORD_REPEAT_RULE)
[grammar] ~17-~17: The expression “vice versa” is spelled without hyphens.
Context: ...B - Any supported backend -> SQLite (or vice-versa) ## Requirements - Python dependencie...
(VICE_VERSA)
[uncategorized] ~37-~37: Loose punctuation mark.
Context: ...ould contain at least: - DATABASE_URI: source database SQLAlchemy URI - `DB_MI...
(UNLIKELY_OPENING_PUNCTUATION)
|
|
||
| - MySQL/MariaDB -> PostgreSQL | ||
| - PostgreSQL -> MySQL/MariaDB | ||
| - Any supported backend -> SQLite (or vice-versa) |
There was a problem hiding this comment.
Use “vice versa” (no hyphen).
At Line 17, “vice-versa” should be “vice versa” in standard British English.
As per coding guidelines, **/*.md: Documentation should be concise, accurate, and written in British English.
🧰 Tools
🪛 LanguageTool
[grammar] ~17-~17: The expression “vice versa” is spelled without hyphens.
Context: ...B - Any supported backend -> SQLite (or vice-versa) ## Requirements - Python dependencie...
(VICE_VERSA)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/common/utils/README-bw_db_migrate.md` at line 17, Replace the hyphenated
phrase "vice-versa" with "vice versa" in the README-bw_db_migrate.md
documentation (the string "Any supported backend -> SQLite (or vice-versa)") so
it uses standard British English; update the text to "Any supported backend ->
SQLite (or vice versa)" and ensure other occurrences of "vice-versa" in the same
file follow the same change.
| Run: | ||
|
|
||
| `python3 bw_db_migrate.py [options]` |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate migration script and any wrapper/entrypoint mentions =="
fd -i 'bw_db_migrate.py'
rg -n -C2 --type=py 'bw_db_migrate|argparse|ArgumentParser|add_argument\('
echo
echo "== Check README command examples for invocation style =="
rg -n -C2 'python3 bw_db_migrate.py|--auto-switch|--test-target|--dry-run|--tui' src/common/utils/README-bw_db_migrate.md || true
echo
echo "== Cross-check same command style in other docs =="
rg -n -C2 'bw_db_migrate.py|--auto-switch|--test-target|--dry-run|--tui' --type=mdRepository: bunkerity/bunkerweb
Length of output: 50375
🏁 Script executed:
# Check for setup.py, pyproject.toml, or installation entry points
find . -maxdepth 3 -type f \( -name "setup.py" -o -name "pyproject.toml" -o -name "setup.cfg" \) 2>/dev/null | head -20
# Look for entry points or console_scripts references to bw_db_migrate
rg -n 'bw_db_migrate|entry.points|console_scripts' --type=py,toml,txt 2>/dev/null | grep -i 'entry\|console\|script' | head -30
# Check if there's a wrapper script or symlink mechanism
fd -type f -o -type l 'bw.?db.?migrate|bw-db-migrate' 2>/dev/null
# Check Docker or packaging files that might install the script
fd -type f \( -name "Dockerfile*" -o -name "*.sh" \) -exec rg -l 'bw_db_migrate' {} \; 2>/dev/null | head -10Repository: bunkerity/bunkerweb
Length of output: 79
🏁 Script executed:
# Examine pyproject.toml for entry points and installation configuration
cat -n pyproject.toml | head -100
# Search specifically for bw_db_migrate in pyproject.toml
rg -n 'bw_db_migrate|console.scripts|entry' pyproject.tomlRepository: bunkerity/bunkerweb
Length of output: 522
🏁 Script executed:
# Get the full pyproject.toml file
wc -l pyproject.toml
# Look for any [project.scripts] or similar entry point sections
rg -n '\[.*script' pyproject.toml
# Check if bw_db_migrate is referenced anywhere in packaging config
rg -n 'bw_db_migrate' . --type=toml --type=py --type=txt 2>/dev/null | grep -E '(setup|script|entry|install|package)' | head -20
# Check for install/deployment files that might show how the script is deployed
fd -type f -name '*.mk' -o -name 'Makefile*' | xargs grep -l 'bw_db_migrate' 2>/dev/null
# Look at Dockerfile entries
fd -type f -name 'Dockerfile*' | xargs grep -A2 -B2 'bw_db_migrate' 2>/dev/null | head -50Repository: bunkerity/bunkerweb
Length of output: 439
🏁 Script executed:
# Get full pyproject.toml content
cat pyproject.toml
# Search for script installation references
rg -n 'bw_db_migrate' . 2>/dev/null | grep -E '(Dockerfile|install|script|setup|entry)' | head -20Repository: bunkerity/bunkerweb
Length of output: 389
🏁 Script executed:
# Search for Dockerfile references to bw_db_migrate
find . -name 'Dockerfile*' 2>/dev/null | xargs grep -B3 -A3 'bw_db_migrate' 2>/dev/null | head -60
# Search for install scripts or deployment docs that mention bw_db_migrate
rg -n 'bw_db_migrate' --type-add 'docker:Dockerfile*' -g '*.md' -g 'Dockerfile*' 2>/dev/null | head -40
# Look at the actual script invocations in tests or CI
rg -n 'python.*bw_db_migrate' . 2>/dev/null | head -20Repository: bunkerity/bunkerweb
Length of output: 1554
Update command invocation paths to match installed packaging behaviour.
Lines 51, 79–97 show python3 bw_db_migrate.py, but the script's own docstring (lines 9, 14, 19 of the source file) documents invocations as python3 src/common/utils/bw_db_migrate.py. Since no entry point or console script installation is configured in pyproject.toml, the bare script name will fail with "file not found" for developers working from the repository root.
Align README examples with the correct invocation path or document that users must change to src/common/utils/ first.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/common/utils/README-bw_db_migrate.md` around lines 49 - 51, Update the
README examples that call bw_db_migrate.py so they use the repository-relative
path used in the script's docstring—e.g., replace occurrences of "python3
bw_db_migrate.py [options]" with "python3 src/common/utils/bw_db_migrate.py
[options]" (or alternatively document the required directory change like "cd
src/common/utils && python3 bw_db_migrate.py [options]"); adjust all instances
mentioned in the comment (the examples around lines showing invocation) so they
match the actual invocation behavior until a console script/entry point is added
to pyproject.toml.
2 participants
migration to another DB backend #3336
this PR has several other dependencies that will be published in separate PRs. Current state is ALPHA until the dependencies will be fixed.