database crash #110

Activity

1. linear commented on Dec 30, 2025

   

   linearBot

   on Dec 30, 2025 – with Linear

   More actions

   NDB-99
2. alltheseas

   marked database crash on profile notedeck#1228 as a duplicate of this issue on Dec 30, 2025
3. alltheseas commented on Dec 30, 2025

   

   alltheseas

   on Dec 30, 2025

   Author

   More actions

   Root Cause Analysis
   
     The crash occurs at metadata.c:244:
     assert(((uintptr_t)entry % 8) == 0);
   
     The issue: LMDB does not guarantee 8-byte alignment for stored values. When values are written to LMDB, they're stored contiguously within B-tree pages. While each page is aligned, individual key-value pairs within the page may start at any offset.
   
     Why it affects only certain databases:
     - Fresh databases happen to have metadata stored at aligned addresses (by chance)
     - Older databases with more data have metadata at various offsets, some of which are not 8-byte aligned
   
     The assertion was added in commit e90d933c ("metadata: add online metadata counts") as a defensive check, but it's incompatible with how LMDB stores data.
   
     Technical Details
   
     1. ndb_get_note_meta() returns a pointer directly into LMDB's memory-mapped file
     2. The ndb_note_meta and ndb_note_meta_entry structs are packed (#pragma pack(push, 1))
     3. The assertion expects all entry pointers to be 8-byte aligned
     4. LMDB doesn't guarantee this alignment
   
   
4. alltheseas commented on Dec 30, 2025

   

   alltheseas

   on Dec 30, 2025

   Author

   More actions

   Option	Read Performance	Memory Overhead	Code Complexity	Portability	Best For
   1. Align at storage	★★★★★ Fastest	High (padding waste)	High (storage format changes)	Good	New designs where you control format
   2. Copy to aligned buffer	★★★★☆ Fast after copy	Stack/heap allocation	Low (single memcpy)	Excellent	Hot paths with multiple field accesses
   3. memcpy per field	★★★★☆ Compiler-optimized	None	Medium (wrapper functions)	Excellent	Sparse field access, max portability
   4. Accept hardware penalty	★★★☆☆ ~5-15% slower	None	None (current code)	Good (modern CPUs)	Cold paths, simple code

   Summary:

   Criteria	Winner
   Simplest implementation	Option 4 (do nothing)
   Best for hot loops	Option 2 (copy once, access many)
   Most portable	Option 3 (memcpy per field)
   Best raw performance	Option 1 (but rarely practical with LMDB)
   Best for nostrdb's case	Option 4 or 2 depending on access patterns
5. alltheseas commented on Dec 30, 2025

   

   alltheseas

   on Dec 30, 2025

   Last edited by alltheseas

   Author

   More actions

   Root Cause Analysis

   The padding strategy doesn't work because LMDB adds per-node overhead:

     Per key-value pair in LMDB:
     ┌─────────────────┬─────────────┬─────────────┐
     │ MDB_node header │     Key     │    Value    │
     │   (4-8 bytes)   │ (32 bytes)  │ (N bytes)   │
     └─────────────────┴─────────────┴─────────────┘
   

   The MDB_node header size varies:

   • 64-bit systems: 8 bytes (alignment may coincidentally work)
   • 32-bit systems: 4-6 bytes (alignment breaks)

   Even on 64-bit, it's not guaranteed - it depends on LMDB compile options and version.

   ---

   The Math (64-bit case)

   Node header: 8 bytes
   Key: 32 bytes
   ────────────────────
   Total: 40 bytes (before value)
   40 % 8 = 0 ✓ (happens to align)

   The Math (32-bit case)

   Node header: 4 bytes
   Key: 32 bytes
   ────────────────────
   Total: 36 bytes (before value)
   36 % 8 = 4 ✗ (misaligned!)

   ---

   Why iOS crashes

   Even though modern ARM64 handles most misaligned access, crashes can occur from:

   1. SIMD/NEON instructions (require alignment)
   2. Atomic operations
   3. Compiler optimizations assuming alignment

   ---

   Solutions

   If alignment must be global, the options are:

   Option	Tradeoff
   1. Pad keys to align values	Add 4 bytes to 32-byte keys → 36 bytes, so 8 + 36 = 44, still misaligned on 32-bit
   2. Use MDB_INTEGERKEY for META	Keys become 8 bytes, need separate id→key mapping
   3. Store data differently	Store offset into a separate aligned arena, not direct values
   4. Accept copy-on-read	The dev rejected this

   The fundamental issue: LMDB cannot guarantee 8-byte value alignment without controlling the node header size, which is internal to LMDB.

   Would the dev consider switching META db to MDB_INTEGERKEY like NOTE and PROFILE dbs use?

   Sources:

   • http://www.lmdb.tech/doc/group__internal.html
   • Question: alignment inside lmdb AltSysrq/lmdb-zero#8