fix(embed): fail the run when embed rows arrive with no embedding

docs/docs/extraction/troubleshoot.md

@@ -38,7 +38,45 @@ configured invoke URL: Page Elements, OCR, Table Structure, Nemotron Parse,
 and embedding. It does not automatically raise for:
 
 - Local-only pipelines (`pdfium` without remote URLs), even when rows contain
-  `metadata.error` or column-level error payloads.
+  `metadata.error` or column-level error payloads. One exception applies to the
+  default local embedding runtime: an in-process vLLM embedding engine that is
+  refused at startup, dies after a crash, or returns no vectors aborts the ingest
+  under every error policy. Continuing on that route would publish an index that
+  covers only part of the corpus while the run reports success. The error message
+  names the settings to change.
+
+  The Designer **Text Embedder** component uses a different handler. It converts
+  an embedding failure to `None`, which can still be dropped silently as described
+  below.
+
+  Alternate handlers that convert row-level failures to `None` still populate
+  the error column. The default local runtime is stricter: if its embedder
+  returns no vector for any submitted input, the embedder raises and aborts the
+  ingest. A plain CUDA out-of-memory is not classified as an engine failure,
+  because on the HuggingFace embedding backend a smaller next batch can succeed.
+  That is about the embed stage, not the run: if the retry succeeds nothing
+  changes. If the batch is lost, whether the ingest then fails depends on the
+  shape those rows carry. The dense writer refuses `[]` but not `None`, as
+  described below. If you see `Refusing to build an incomplete index` with no
+  engine-startup message in the embed actor logs, look for an out-of-memory
+  instead.
+
+  `LanceDB(on_bad_vectors=...)` does not suppress that error. On a dense write, a
+  row that arrives with an empty list or tuple embedding used to be counted as a
+  wrong-length vector and silently excluded; it now fails the run, and no policy
+  value restores the old behavior. Fix the embed stage.
+
+  This covers empty list and tuple values only. A failed embedding that arrives
+  as `None` keeps its pre-existing handling. For direct nested LanceDB records,
+  the writer silently drops the row and counts it as `dropped_no_embedding`.
+  Graph rows are converted before the writer: a mixed batch filters out `None`
+  rows without incrementing that writer counter, while a batch with no
+  uploadable rows raises `VdbUploadError`. Some embed operators produce `None`
+  for a lost batch as well as for a row they chose not to embed. The two carry
+  different `error` payloads upstream, but the writer does not see that key. If
+  recall is low and the run reported success, inspect the embed-stage row errors
+  and writer logs; do not rely on the empty-vector guard alone.
+
 - Caption or remote VLM stages. Missing credentials fail at actor setup;
   inference failures can abort the entire ingest.
 - Audio or video ASR over gRPC or HTTP. Failures can drop individual rows and
@@ -81,7 +119,7 @@ troubleshooting path. A single document can pass through several stages.
 | `ExtractParams(method="ocr")` | Page rendering, Page Elements, and the local or remote OCR backend | Missing local model dependencies, invalid image payload, authentication/transport status, or OCR row-level failure |
 | `ExtractParams(method="nemotron_parse")` | PDF rendering and local Nemotron Parse model or configured Nemotron Parse NIM | Missing `open_clip`, missing local model configuration, unsupported image input, or Nemotron Parse row-level/HTTP failure |
 | `.caption(...)` | Local caption model or remote VLM endpoint | `ValueError` at setup when credentials or endpoint/protocol are invalid; remote inference failures can abort the whole ingest rather than populate a row error column |
-| `.embed(...)` | Local embedding model or configured embedding NIM | Model/dependency error, input-size or schema rejection, authentication/transport status, or embedding row-level failure; `GraphIngestionError` when a remote embed URL is configured |
+| `.embed(...)` | Local embedding model or configured embedding NIM | Model/dependency error, input-size or schema rejection, authentication/transport status, or embedding row-level failure; `GraphIngestionError` when a remote embed URL is configured. On the default local embedding runtime, an in-process vLLM engine that has stopped serving aborts the ingest regardless of the error policy. The Designer **Text Embedder** component retains the `None` gap described above |
 | Audio or video extraction | `ffmpeg`/`ffprobe`, media decoding, frame/chunk creation, and local or remote ASR | Missing executable, malformed media, codec failure, gRPC status, or credential error; ASR failures may omit rows and log warnings instead of raising, so verify logs when output is unexpectedly empty |
 
 `pdfium` itself is primarily a local parser, so a Page Elements, Table

nemo_retriever/src/nemo_retriever/common/vdb/README.md

@@ -93,6 +93,12 @@ When `vdb_op="lancedb"` (or `vdb=LanceDB(...)` is passed explicitly), `_construc
 1. **`create_index`** — connects with `lancedb.connect(self.uri)`, transforms ingestion batches into Arrow rows (`vector`, `text`, `metadata`, `source`), and **`db.create_table(...)`** with schema and `on_bad_vectors` policy.
 2. **`write_to_index`** — builds the **vector index** (e.g. IVF/HNSW) and optionally an **FTS/BM25** index over the ingested `text` column when `hybrid=True`.
 
+During a dense-vector step 1, a row that arrives with an **empty list or tuple** embedding raises `RuntimeError`, and no table rows are written. The embed failure path writes `[]` when it produces no vector for a row. Such a row used to be counted as a wrong-length vector and silently excluded, letting a run publish a short index and still report `success: true`.
+
+The new dense-writer guard treats only an empty list or tuple as fatal. A row whose embedding is absent or `None` keeps its pre-existing handling. For direct nested LanceDB records, the writer silently drops that row and counts it as `dropped_no_embedding`. Graph rows are converted first: a mixed batch can omit failed `None` rows while indexing healthy rows, but an all-rejected batch raises `VdbUploadError`. For this policy, `None` has two relevant meanings: a deliberate skip and a genuine embedding failure. For example, `operators/embed/text_embed.py` writes `error: None` alongside a blank-text skip, while its failure handler writes a populated `error` dict with stage, type, message, and traceback. Other embedding paths can produce the same two meanings. `common/vdb/records.py` reads that payload during conversion, so the discriminator is available there, but it is not forwarded to the writer. Making every `None` fatal would fail deliberate skips; source-aware handling is follow-up work. `on_bad_vectors` is unaffected: it governs malformed vectors, and an empty embedding is the absence of a vector rather than a short one. The returned `counts` dict gains an `empty_embedding` key.
+
+The other half of the guard is upstream: a local embedder that fails to embed some rows raises `LocalEmbedderRowsLostError` from `_finalize_vectors` instead of zero-padding them, which this writer could not otherwise detect.
+
 Common constructor arguments include:
 
 | Parameter        | Purpose |

nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py

@@ -366,12 +366,22 @@ def _create_lancedb_results(
         ``(rows, counts)`` where ``rows`` is the list of dicts shaped for
         LanceDB ingestion (``vector``, ``text``, ``metadata``, ``source``)
         and ``counts`` is a dict containing ``accepted``,
-        ``dropped_no_embedding``, ``dropped_bad_length``, and
-        ``dropped_no_text`` keys.
+        ``dropped_no_embedding``, ``empty_embedding``,
+        ``dropped_bad_length``, and ``dropped_no_text`` keys.
+
+    An empty list or tuple embedding means the embed stage produced no vector for
+    that row. Such rows are counted as ``empty_embedding`` and make this function
+    raise :class:`RuntimeError` after the loop, so no table rows are written.
+
+    :meth:`LanceDB.create_index` calls this function twice when ``vector_dim`` is
+    ``None``: pass 1 infers the dimension and its rows are discarded, pass 2
+    filters. The check ignores ``expected_dim``, so it raises in pass 1 and pass 2
+    never runs - which is why the error reports ``expected_dim=None``.
     """
     lancedb_rows: list = []
     accepted = 0
     dropped_no_embedding = 0
+    empty_embedding = 0
     dropped_bad_length = 0
     dropped_no_text = 0
 
@@ -388,6 +398,13 @@ def _create_lancedb_results(
                 dropped_no_embedding += 1
                 continue
 
+            # ``[]`` is not ``None``, so the branch above misses it. Explicit
+            # isinstance/len, never ``not embedding``, which raises on an array.
+            if isinstance(embedding, (list, tuple)) and len(embedding) == 0:
+                empty_embedding += 1
+                logger.debug("Dropping row with an empty embedding (doc_type=%s)", doc_type)
+                continue
+
             if enforce_length and (not isinstance(embedding, (list, tuple)) or len(embedding) != expected_dim_int):
                 dropped_bad_length += 1
                 got_len: Any = len(embedding) if hasattr(embedding, "__len__") else "n/a"
@@ -434,6 +451,7 @@ def _create_lancedb_results(
     counts: dict[str, int] = {
         "accepted": accepted,
         "dropped_no_embedding": dropped_no_embedding,
+        "empty_embedding": empty_embedding,
         "dropped_bad_length": dropped_bad_length,
         "dropped_no_text": dropped_no_text,
     }
@@ -450,6 +468,26 @@ def _create_lancedb_results(
             expected_dim_repr,
         )
 
+    if empty_embedding:
+        logger.warning(
+            "_create_lancedb_results: empty_embedding=%d",
+            empty_embedding,
+        )
+        total = accepted + dropped_no_embedding + empty_embedding + dropped_bad_length + dropped_no_text
+        raise RuntimeError(
+            "Refusing to build an incomplete index: "
+            f"{empty_embedding} of {total} rows had no embedding. No table rows are written and "
+            "this run fails; the alternative is an index that is silently short by those "
+            "rows while the run reports success. "
+            f"Counters: empty_embedding={empty_embedding}, no_embedding={dropped_no_embedding}, "
+            f"bad_length={dropped_bad_length}, no_text={dropped_no_text}, "
+            f"expected_dim={expected_dim_int if enforce_length else 'None'}. "
+            "Only empty_embedding caused this failure - the other three are rows filtered "
+            "under the configured policy. This normally means the embed stage failed for "
+            "whole batches: check the embed actor logs for engine initialization or "
+            "out-of-memory errors."
+        )
+
     return lancedb_rows, counts
 
 

nemo_retriever/src/nemo_retriever/common/vdb/lancedb_collections.py

@@ -168,22 +168,59 @@ def _collection_rows(
     *,
     context: CollectionWriteContext,
 ) -> list[dict[str, Any]]:
-    """Convert canonical NRL record batches into collection-managed LanceDB rows."""
+    """Convert canonical NRL record batches into collection-managed LanceDB rows.
+
+    An empty list or tuple embedding means the embed stage produced no vector for
+    that row. Such rows are counted as ``empty_embedding`` and make this function
+    raise :class:`RuntimeError` after the loop, so no rows are written for the
+    document.
+    This is the same guarantee :func:`nemo_retriever.common.vdb.lancedb._create_lancedb_results`
+    carries on the pipeline write path for this representation: an empty list or
+    tuple must not reach the index, and its presence fails the write rather than
+    publishing a short document while reporting success.
+
+    Every other row-level skip remains non-fatal and is
+    counted only as ``skipped_other``: a malformed record, metadata or embedding
+    value, and a text-free non-image row, are all still dropped without failing
+    the write. A malformed outer batch is ignored but is not included in the row
+    counters. Only an empty list or tuple
+    is fatal. The embed stage writes ``[]`` to mean "this output carried no
+    embedding". ``None`` can represent either a row the embed stage chose not to
+    embed or a chunk whose embedding failed. Those meanings are distinguishable
+    by the ``error`` key that travels with the payload -
+    ``None`` for the deliberate skip, a populated dict for the failure - but that
+    key is not visible at this layer, so ``None`` keeps its silent drop until the
+    discriminator is threaded down. That is follow-up work, not an impossibility.
+    """
     rows: list[dict[str, Any]] = []
     created_at = _now()
     row_index = 0
+    empty_embedding = 0
+    skipped_other = 0
 
     for batch in records or []:
         if not isinstance(batch, list):
             continue
         for record in batch:
             if not isinstance(record, dict):
+                skipped_other += 1
                 continue
             metadata = record.get("metadata")
             if not isinstance(metadata, dict):
+                skipped_other += 1
                 continue
             vector = metadata.get("embedding")
+            # Empty lists and tuples are folded into the skip below by ``not vector``,
+            # which drops them silently. Check them first with ``isinstance``/``len``;
+            # never ``not vector``: the latter raises ValueError on a
+            # multi-element array. The skip below is left exactly as it was, so all
+            # other values keep their current route.
+            if isinstance(vector, (list, tuple)) and len(vector) == 0:
+                empty_embedding += 1
+                logger.debug("Row has an empty embedding (document_id=%s)", context.document_id)
+                continue
             if not isinstance(vector, (list, tuple)) or not vector:
+                skipped_other += 1
                 continue
             content_metadata = metadata.get("content_metadata")
             if not isinstance(content_metadata, dict):
@@ -200,6 +237,7 @@ def _collection_rows(
                 content_metadata["type"] = content_type
                 content_metadata["_content_type"] = content_type
             if not text.strip() and content_type != "image":
+                skipped_other += 1
                 continue
 
             source_id = str(
@@ -238,6 +276,24 @@ def _collection_rows(
                 }
             )
             row_index += 1
+
+    if empty_embedding:
+        total = len(rows) + empty_embedding + skipped_other
+        logger.warning("_collection_rows: empty_embedding=%d", empty_embedding)
+        raise RuntimeError(
+            "Refusing to write an incomplete document: "
+            f"{empty_embedding} of {total} rows had no embedding. No rows are written and "
+            "this write fails; the alternative is a document that is silently short by those "
+            "rows while the ingest reports success. "
+            f"Counters: empty_embedding={empty_embedding}, skipped_other={skipped_other}, "
+            f"accepted={len(rows)}, document_id={context.document_id}, "
+            f"document_version={context.document_version}. "
+            "Only empty_embedding caused this failure - skipped_other counts rows filtered "
+            "under the non-fatal row rules. This normally means the embed stage failed for "
+            "whole batches: check the embed actor logs for engine initialization or "
+            "out-of-memory errors."
+        )
+
     return rows
 
 

nemo_retriever/src/nemo_retriever/models/embed_errors.py

@@ -0,0 +1,80 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
+# All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Embed failures that mean rows were lost.
+
+Kept in their own module, with no third-party imports, so the local embedders
+and the embed runtime can both import them on the endpoint-only path.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Sequence
+
+from nemo_retriever.models.nim.error_reporter import report_error
+
+logger = logging.getLogger(__name__)
+
+
+class LocalEmbedderReturnedNothingError(RuntimeError):
+    """An in-process embedder returned no vectors for a non-empty batch."""
+
+
+class LocalEmbedderRowsLostError(RuntimeError):
+    """An in-process embedder produced no vector for some rows of a batch.
+
+    Raised by :func:`report_lost_rows`. ``models/inference/runtime.py``
+    classifies both errors in this module as fatal by name.
+
+    Args:
+        lost: Number of rows with no vector.
+        total: Size of the batch.
+        embedder: Class name of the embedder that lost them.
+    """
+
+    def __init__(self, *, lost: int, total: int, embedder: str) -> None:
+        self.lost = int(lost)
+        self.total = int(total)
+        self.embedder = str(embedder)
+        super().__init__(
+            f"{embedder} returned no vector for {lost} of {total} row(s) in this batch. "
+            "Continuing would pad or drop those rows, hide the loss, and allow an invalid or "
+            "incomplete index to be published. This normally means the in-process engine failed "
+            "for the batch - check the embed actor logs for engine initialization or out-of-memory "
+            "errors."
+        )
+
+
+def _has_no_vector(vector: object) -> bool:
+    """Return whether ``vector`` is empty or absent.
+
+    Explicit rather than truthy: ``not vector`` raises on a multi-element numpy
+    array.
+    """
+    if vector is None:
+        return True
+    if hasattr(vector, "__len__"):
+        return len(vector) == 0
+    return False
+
+
+def report_lost_rows(vectors: Sequence[Sequence[float]], *, embedder: str) -> int:
+    """Raise :class:`LocalEmbedderRowsLostError` if any row came back empty.
+
+    Returns ``0`` when nothing was lost; never returns a non-zero count.
+
+    Called from the local embedders' ``_finalize_vectors`` before they discard
+    empty placeholders. Raising before padding is required because a padded row
+    has the right width, so ``has_embedding`` would report ``True`` for a row
+    carrying nothing.
+    """
+    lost = sum(1 for vector in vectors if _has_no_vector(vector))
+    if not lost:
+        return 0
+
+    exc = LocalEmbedderRowsLostError(lost=lost, total=len(vectors), embedder=embedder)
+    logger.error("%s", exc)
+    report_error("embed", exc)
+    raise exc