RHIDP-14967: add durable conversation handling for unified/synthesized configs

docs/devel_doc/conversations_api.md

@@ -158,15 +158,15 @@ response = await client.responses.create(
 
 ### Conversation Storage
 
-Conversations are stored in **two databases**:
+Conversations are stored across **LCORE and Llama Stack / OGX** layers:
 
-#### 1. Llama Stack Database (PostgreSQL `public` schema)
+#### 1. Llama Stack / OGX conversations store
 
 **Tables:**
 - `openai_conversations`: Stores conversation metadata
-- `conversation_items`: Stores individual messages/turns in conversations
+- `conversation_items`: Stores individual messages/turns (durable source of truth for continue-chat)
 
-**Configuration (in `config/llama_stack_client_config.yaml`):**
+**Baseline configuration** (shipped `default_run.yaml` / typical profile):
 ```yaml
 storage:
   stores:
@@ -175,7 +175,13 @@ storage:
       backend: sql_default
 ```
 
-#### 2. Lightspeed Stack Database (PostgreSQL `lightspeed-stack` schema)
+In **unified mode**, when `conversation_cache` is `postgres` or `sqlite`,
+synthesis upserts `storage.backends.conversations_default` from that cache and
+sets `stores.conversations.backend: conversations_default` (unless
+`native_override` retargets the store afterward). See
+[Conversation persistence (unified mode)](../user_doc/deployment_guide.md#conversation-persistence-unified-mode).
+
+#### 2. Lightspeed Stack database
 
 **Table:** `user_conversation`
 
@@ -187,6 +193,11 @@ Stores user-specific metadata:
 - Message count
 - Topic summary
 
+#### 3. Lightspeed conversation cache (optional)
+
+When configured, `conversation_cache` holds V2 Q&A history and topic summaries.
+It does **not** replace the OGX conversations store used to continue chats.
+
 ---
 
 ## API Endpoints

docs/user_doc/deployment_guide.md

@@ -147,6 +147,44 @@ The reference profiles are sanity-checked by the unit suite
 (`tests/unit/test_llama_stack_synthesize.py`), so they stay loadable as the
 synthesizer evolves.
 
+### Conversation persistence (unified mode)
+
+When `conversation_cache` is `postgres` or `sqlite`, unified synthesis upserts an
+OGX backend named `conversations_default` from that cache and points
+`storage.stores.conversations` at it **before** applying
+`llama_stack.config.native_override`. Inference/agents SQL on `sql_default` is
+left alone. Continuing a chat after a restart (such as a Kubernetes Pod redeploy) needs this OGX store; listing and
+ownership also need a durable matching-type LCORE `database`.
+
+**Happy path.** Use unified library mode (`llama_stack.config.baseline` or a
+profile), set durable `conversation_cache`, set `database` to the same backend
+*type* (postgres or sqlite) on a non-ephemeral path, and do **not** set
+`storage.stores.conversations` under `native_override`. See
+[`examples/lightspeed-stack-unified-conversation-persistence-pg.yaml`](../../examples/lightspeed-stack-unified-conversation-persistence-pg.yaml).
+
+**`native_override` wins.** Dumb migration lifts a full `run.yaml` into
+`native_override`, which usually still has `stores.conversations.backend:
+sql_default`. That undoes enrichment; LCORE logs a warning and chats will not
+survive restart until you remove that key, point it at `conversations_default`,
+or accept a deliberate split.
+
+**`database` must be durable too.** If `database` is omitted (default
+`/tmp/lightspeed-stack.db`), is under `/tmp/`, or is a different type than the
+cache, synthesis warns. Same-type different hosts/paths do not warn.
+
+**Secrets.** Prefer `${env.*}` for postgres
+passwords. Literals are copied into `.generated/run.yaml` (mode 0600) and
+trigger a warning.
+
+**SQLite.** Sharing one `db_path` between cache and OGX conversations is fine
+for single-worker. Prefer Postgres for multi-worker / production.
+
+**Legacy two-file mode** is unchanged. Migrate to unified or edit `run.yaml`
+manually. OpenAI-compatible `previous_response_id` continuation still uses an
+ephemeral responses store and does not survive restart; normal continue-chat
+via the `conversation` id does.
+
+More detail: [Conversations API Guide](../devel_doc/conversations_api.md).
 
 
 ### Llama Stack as a server

examples/lightspeed-stack-unified-conversation-persistence-pg.yaml

@@ -0,0 +1,49 @@
+# Example: unified mode with durable conversation persistence (Postgres).
+#
+# Synthesis upserts OGX storage.backends.conversations_default from
+# conversation_cache and points stores.conversations at it. Do not set
+# storage.stores.conversations under native_override unless you intend to
+# override that wiring (native_override wins; LCORE will warn).
+#
+# Keep database and conversation_cache on the same backend type so ownership
+# metadata and chat continuation both survive restart.
+name: Lightspeed Core Service (LCS) — unified conversation persistence
+service:
+  host: localhost
+  port: 8080
+  auth_enabled: false
+  workers: 1
+  color_log: true
+  access_log: true
+llama_stack:
+  use_as_library_client: true
+  config:
+    baseline: default
+authentication:
+  module: "noop"
+user_data_collection:
+  feedback_enabled: false
+  transcripts_enabled: false
+database:
+  postgres:
+    host: ${env.POSTGRES_HOST:=127.0.0.1}
+    port: 5432
+    db: ${env.POSTGRES_DB:=lightspeed}
+    user: ${env.POSTGRES_USER:=lightspeed}
+    password: ${env.POSTGRES_PASSWORD}
+    ssl_mode: disable
+    gss_encmode: disable
+conversation_cache:
+  type: postgres
+  postgres:
+    host: ${env.POSTGRES_HOST:=127.0.0.1}
+    port: 5432
+    db: ${env.POSTGRES_DB:=lightspeed}
+    user: ${env.POSTGRES_USER:=lightspeed}
+    password: ${env.POSTGRES_PASSWORD}
+    ssl_mode: disable
+    gss_encmode: disable
+inference:
+  providers:
+    - type: openai
+      api_key_env: OPENAI_API_KEY

src/app/main.py

@@ -9,9 +9,8 @@
 from fastapi import FastAPI, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
-from ogx_client import APIConnectionError, AsyncOgxClient
 from fastapi.routing import iter_route_contexts
-
+from ogx_client import APIConnectionError, AsyncOgxClient
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 
 import version

src/constants.py

@@ -198,6 +198,17 @@
 CACHE_TYPE_POSTGRES: Final[str] = "postgres"
 CACHE_TYPE_NOOP: Final[str] = "noop"
 
+# Default sqlite path when DatabaseConfiguration has no backend configured.
+# Ephemeral (typically tmpfs); conversation-persistence warnings treat this as non-durable.
+DEFAULT_SQLITE_DATABASE_PATH: Final[str] = "/tmp/lightspeed-stack.db"
+
+# Dedicated OGX SQL backend for stores.conversations when conversation_cache is
+# durable (RHIDP-14967). Injected only during unified synthesis; not seeded in
+# default_run.yaml.
+CONVERSATIONS_BACKEND_NAME: Final[str] = "conversations_default"
+# Default OGX stores.conversations.table_name when the baseline/store omits one.
+DEFAULT_CONVERSATIONS_TABLE_NAME: Final[str] = "openai_conversations"
+
 # BYOK RAG
 # Default RAG type for bring-your-own-knowledge RAG configurations, that type
 # needs to be supported by Llama Stack