mdbx: expose Env.Copy* with MDBX_CP_OVERWRITE and add Env.Defrag - #223
mdbx: expose Env.Copy* with MDBX_CP_OVERWRITE and add Env.Defrag#223JkLondon wants to merge 13 commits into
Conversation
- Uncomment Env.Copy/CopyFlag/CopyFD/CopyFDFlag and rewire them through the unified mdbx_env_copy / mdbx_env_copy2fd entry points. - Expose MDBX_CP_* flags (CopyDefaults, CopyForceDynamicSize, CopyDontFlush, CopyThrottleMVCC and the new CopyOverwrite for clobbering an existing target file). - Bind mdbx_env_defrag via a thin cgo helper (mdbxgo_env_defrag), expose DefragOptions / DefragResult and the MDBX_defrag_* stopping reasons. - Replace the long-commented Copy tests with working ones plus dedicated tests for the new CopyOverwrite flag and Env.Defrag.
On Windows mdbx_filehandle_t is HANDLE (void*), so the previous C.mdbx_filehandle_t(fd) where fd is uintptr would not compile under GOOS=windows. Route the call through a thin C helper that takes a uintptr_t and performs the platform-specific cast in C, so the Go side stays identical on every target. The CopyFD test now runs on Windows as well.
r.spent_time_dot16 is already typed C.uint64_t, so the explicit conversion is flagged by unconvert.
mdbx_env_defrag opens its own write transaction via txn_basal_start, and on Windows it trips ERROR_LOCK_VIOLATION against the env's own LockFileEx region when run on a handle that has just committed writes. The defrag API binding itself is fine — only the test scenario hits the libmdbx Windows locking interaction.
Keep the failure visible in CI until libmdbx clarifies the expected mdbx_env_defrag calling sequence on Windows / fixes the LockFileEx region conflict that ERROR_LOCK_VIOLATIONs on a freshly written env.
Copy: a non-compacting copy silently produces an EMPTY database. libmdbx's copy_asis (v0.14.2) builds the destination meta-pages from a pristine model — trees.main.root = P_INVALID, first_unallocated = NUM_METAS — and never writes the source txn's roots or geometry into them, unlike copy_with_compacting. The data pages are copied byte-for-byte, so the target opens without complaint and reads as an empty db. That is what the "does not reproduce the committed data" note in the tests was describing. Reject copies without CopyCompact (ErrCopyNotCompacting) instead of exposing a backup API that loses data, and let Copy/CopyFD compact on their own; TestEnv_Copy and TestEnv_CopyFD now cover the flagless entry points that had no test. Defrag: open the environment Exclusive in the test, like libmdbx's mdbx_defrag tool does (MDBX_ENV_DEFAULTS|MDBX_EXCLUSIVE with an MDBX_ACCEDE fallback). Cutting off trailing pages needs the whole-file lock; taking it per transaction is what failed the win-2025 job with ERROR_LOCK_VIOLATION. The requirement is now documented on Env.Defrag. Lint (7 golangci-lint findings): intrange, fmtappendf, thelper, and the "shrinked" misspelling — DefragResult.PagesShrinked is now PagesShrunk, with the wrapper's C field renamed to match.
|
Review pass on 210e400 — three things, one of them a data-loss footgun. 1. Non-compacting copies produce an empty database (fixed by refusing them). The The data pages are copied faithfully, so the result opens without error and reads as an empty db. Verified on darwin/arm64 across Shipping that as a backup API is worse than not shipping it, so a copy requested without 2. Cutting off the trailing pages needs the whole-file lock, and taking it per transaction is what Windows rejected. libmdbx's own 3. Lint (7 findings). intrange, fmtappendf, thelper, and the Full suite green on darwin/arm64. |
Summary
Env.Copy/CopyFlag/CopyFD/CopyFDFlagmethods and rewires them onto the modernmdbx_env_copy(env, dest, flags)/mdbx_env_copy2fd(env, fd, flags)entry points (the oldmdbx_env_copy2/mdbx_env_copyfd2API the bindings used no longer exists).CopyDefaults,CopyForceDynamicSize,CopyDontFlush,CopyThrottleMVCC, andCopyOverwrite(=MDBX_CP_OVERWRITE) so callers can clobber an existing target file.mdbx_env_defragfor explicit in-place defragmentation (no full copy needed): introducesEnv.Defrag(DefragOptions) (*DefragResult, error), aDefragResultstruct mirroringMDBX_defrag_result_t, and constants forMDBX_defrag_stopping_reasons_t(DefragStepSize,DefragLargeChunk,DefragLaggardReader,DefragEnoughThreshold,DefragTimeLimit,DefragAborted,DefragError, ...). The libmdbx-side progress callback is intentionally left out for now (alwaysNULL); the final metrics are still returned.TestEnv_Copy*skeletons with working tests and adds dedicated coverage forCopyOverwriteandEnv.Defrag.Windows portability
mdbx_filehandle_tisHANDLE(i.e.void*) on Windows andinton POSIX, so writingC.mdbx_filehandle_t(fd)over a Gouintptrdoes not compile underGOOS=windows. The fix is a tiny cgo wrappermdbxgo_env_copy2fd(MDBX_env*, uintptr_t, MDBX_copy_flags_t)that performs the platform-specific cast in C, so the Go side stays identical on every target. Cross-checked withGOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc go buildandgo test -c.Test plan
go build ./...go test -count=1 ./mdbx/...(darwin/arm64) — all green, including the newTestEnv_Copy,TestEnv_CopyFD,TestEnv_CopyFlag_Overwrite,TestEnv_DefragGOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc go build ./mdbx/...— cross-compiles cleanlygo test -cfor the same Windows target — produces a valid PE32+ test binary