feat: 0.7 serialization, scalar indexes, and server functions

[jamals86]

Summary

0.7 is not three independent projects. It is one release whose product bar is: typed nested data that persists and executes correctly, equality lookups that stay prefix scans as tables grow, and SQL-defined server procedures that use those same types.

Source of truth: docs/plans/2026-09-01-kalamdb-0.7.md. File-level steps: docs/plans/2026-09-05-kalamdb-0.7-implementation.md.

Track	Plan	0.7 product outcome
Serialization	serialization plan	One kalamdb-serialization crate. Nested STRUCT / CREATE TYPE / List persist through that crate only. Live USER/SHARED/STREAM row values are ordinal KOBJ (identity is not repeated in the payload).
Scalar secondary indexes	index plan	USER/SHARED scalar CREATE INDEX, hot prefix seek, cold bloom/min-max prune, CLI schema-diff, RLS membership bind.
Functions V1	functions plan (ADR-021)	SQL procedures, generated SDKs, nested typed CALL, transactional topics, durable triggers.

Do not ship 0.7 with only one of these. Nested procedure types without a central codec fork persistence. Indexes that keep JSON PK lists fight the serialization migration. Functions that scan WHERE conversation_id = ? stay too slow for the chat/RLS workloads 0.7 is supposed to carry.

SQL contract (tables, CREATE TYPE, CREATE INDEX, CREATE PROCEDURE)
        │
        ▼
ContractSnapshot / TableDefinition / TypeId
        │
        ├─► kalamdb-serialization     persist STRUCT/List/rows/catalog/index values
        ├─► indexing core             prefix seek + flush prune for equality
        └─► kalamdb-functions         CALL / nested ctx.functions / triggers
                │
                └── uses the same types, same row codec, same indexes

Not in 0.7: Vortex-as-Parquet, schedules/CREATE EVENT, extra function runtimes, unique-by-default scalar indexes, ANN answering scalar equality, changing RocksDB ordered key encoding.

Status now (2026-09-06)

Wave 1 is met for dialect + nested codec tests. Wave 1 is not met for leftover non-indexed store paths.

Track	Done	Live gap
Serialization	Live USER/SHARED/STREAM values are ordinal KOBJ (identity on the storekey). StorageSchema from TableDefinition. SharedTableRow in commons. Indexed stores use EntityCodec; other entities use encode_object. Commons FlatBuffers codecs and server decode_*_compat are gone. Codec benches live in kalamdb-serialization.	Non-indexed leftover UserTableStore/SharedTableStore still use the object envelope.
Indexes	Scalar CREATE INDEX catalog + hot prefix seek + cold prune + CLI schema-diff + RLS membership bind. Chat bench and example schemas create conversation/membership indexes.	Re-run chat realtime bench vs 2026-08-30 historic_select p90.
Functions	Tasks 1–12: dialect + catalog + V8 + CAS artifacts + typed CALL / REST / PGWire; nested CALL; staged topic publish; durable topic triggers; kalam deploy/kalam dev; EXECUTE ACL, metrics, SQL/CLI/skill docs. Checkpoint C: SQL/REST CALL → nested INSERT + typed topic + GRANT/REVOKE + rollback.	Full cli/run-tests.sh against a running 0.7 server (wipe-required on-disk format).

On-disk / upgrade policy

0.7 is a breaking on-disk format. The server will not dual-decode old RocksDB row/object bytes on the hot path.

• No in-process migration of existing cluster data.
• No forever decode_*_compat for production reads.
• Dev/test clusters: wipe the data directory. Old KENV / name-keyed rows / unenveloped payloads are not readable by 0.7.
• A separate offline upgrade tool (follow-on, not the 0.7 server) will read old codecs and rewrite to the 0.7 layout.
• HTTP JSON, WebSocket, PGWire, and SDK payloads stay out of KOBJ.
• RocksDB keys stay today’s storekey / SeqId encoding. kalamdb-serialization owns values only.

Target USER/SHARED/STREAM row value:

RocksDB key   = existing storekey / SeqId bytes (do not change)
RocksDB value = KOBJ row envelope + compact payload:

  schema_version u16
  commit_seq     u64
  deleted        u8
  ordinal values[]   // schema slots; STRUCT/List recurse; no column names

Drop user_id / _seq from the value; reconstruct them from the key. Keep _commit_seq and _deleted for MVCC.

0.7 release gate

Ship 0.7 only when all of the following are true.

Serialization

• kalamdb-serialization is the only internal persisted-object codec crate.
• Nested STRUCT / named CREATE TYPE / List round-trip without JSON intermediates or ScalarTag::Fallback.
• Direct FlatBuffers/FlexBuffers usage outside that crate is CI-blocked.
• Row values are ordinal KOBJ and do not repeat StorageKey fields (user_id / _seq).
• RocksDB keys remain today’s storekey / SeqId encoding.
• Index values are no longer stored as JSON.

Scalar secondary indexes

• USER/SHARED tables honor catalog scalar indexes on hot seek and on flush/compaction prune.
• CREATE INDEX / DROP INDEX parse, persist on system.schemas, and round-trip through kalam-schema-diff.
• RLS membership bind prefix-scans an indexed principal column.
• Chat realtime bench no longer full-scans messages / conversation_members for the documented equality filters.

Functions V1

• Local ContractSnapshot, typed CALL, nested calls/context, transactional topics, durable triggers, CLI deploy/dev.
• Procedure and topic payloads use the shared codec; there is no functions-only serializer.

Together

• A representative chat schema with CREATE TYPE, scalar indexes, and CREATE PROCEDURE compiles locally, deploys, serves indexed SQL, and executes typed CALL / nested procedures on those types.
• Backend/CLI smoke tests and the production-like server build pass.
• No track is deferred to “right after 0.7” while still claiming 0.7 complete.

Test plan

• Nested STRUCT/List round-trip through kalamdb-serialization with no string fallback (cargo nextest run -p kalamdb-serialization)
• Wipe the data directory, start a 0.7 server, then run cli/run-tests.sh (old KENV / name-keyed rows are unreadable)
• CREATE TYPE, CREATE INDEX, CREATE PROCEDURE, typed CALL / REST / PGWire, GRANT/REVOKE EXECUTE
• kalam-schema-diff emits CREATE TYPE / CREATE INDEX / CREATE PROCEDURE; local generate does not need a live server
• WHERE conversation_id = ? seeks the hot scalar index and cold scan does not visit every flushed segment
• Re-run chat realtime bench vs 2026-08-30 historic_select p90
• Production-like server build and backend/CLI smoke pass