feat(gooddata-eval): record why an agentic simulated-user loop stopped

Every agentic evaluator drives a loop that can exit several ways, but the result
only ever recorded *whether* the agent produced its output. A run that ran out of
turns while doing the right thing was reported identically to one that refused,
and identically to one that answered wrongly.

It is worse than a missing field, because the downstream checks are all of the
form `produced_output and <check>`. An exhausted alert run reports
operator_correct, threshold_correct, metric_correct and recipients_correct as
False -- four specific-sounding content failures for work the agent was never
given the chance to attempt.

Adds `LoopExit` (core/models.py) and threads it through all five loops, plus
`turns_used`/`max_iterations` in `detail`:

    success                 the agent produced its output
    agent_silent            neither text nor a tool call -- genuinely stuck
    budget_exhausted        hit max_iterations; says nothing about being on track
    simulated_user_failed   OUR simulated-user model failed, not the agent
    chat_error              the chat call raised mid-conversation (kda partial path)
    not_run                 the loop never started (conversation $ref skip)

The default is BUDGET_EXHAUSTED and every other exit assigns explicitly, so a loop
that simply runs out of range() is labelled correctly with no trailing else.

Two exits were previously invisible and are the reason this is worth doing:

- metric_skill catches SimulatedResponseError and breaks. A harness-side outage
  was scored against the product as metric_created=False/maql_correct=False.
- kda_skill breaks on a chat error with a partial result.

Deliberately NOT included:

- No verdict changes. An exhausted run still fails. The point is that the cases
  become countable, not that any of them start passing.
- No change to any max_iterations default (4-7, already tuned per kind). Whether
  a budget is too tight becomes answerable from data instead of argued -- which is
  the actual ask behind GDAI-2200, where ~13% of alert runs are estimated to need
  7 turns against a ceiling of 6. Raising the ceiling first would have hidden the
  interaction with GDAI-2199's MANDATORY STOPs, which make prescribed
  end-turn-without-a-tool-call behaviour consume budget.
- No try/except added to alert_skill's simulated-user call: there a failure
  already propagates as a hard error rather than being swallowed, which is the
  behaviour we want. Only metric_skill needed the label.

Tests: existing detail assertions extended across all five kinds, plus dedicated
coverage for budget_exhausted vs agent_silent vs success (including the turn the
tool landed on), simulated_user_failed, and a regression guard asserting that two
runs with identical scored booleans differ only in exit_reason -- the exact
ambiguity this removes.

739 passed; ruff check clean.

Author: Tomkess

packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py

@@ -27,6 +27,7 @@
 from gooddata_eval.core.models import (
     AgenticAssertionError,
     AgenticEvalOutcome,
+    LoopExit,
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
@@ -473,6 +474,12 @@ class AlertRunResult:
     response_id: str | None = None
     tool_call_events: list[ToolCallEvent] = field(default_factory=list)
     reasoning_step_events: list[ReasoningStepEvent] = field(default_factory=list)
+    # Why the simulated-user loop stopped, and how many turns it took. Without these a run
+    # that ran out of turns is indistinguishable from one that refused: both land on
+    # alert_created=False, and every downstream check is `alert_created and ...`, so both
+    # also report operator/threshold/metric/recipients as False.
+    exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
+    turns_used: int = 0
 
 
 @dataclass
@@ -669,7 +676,12 @@ def _run_once(conv_id: str) -> AlertRunResult:
             conversation_history: list = []
             current_question = question
 
+            # Defaults to BUDGET_EXHAUSTED: every other exit sets it explicitly, so a loop
+            # that simply runs out of range() is correctly labelled without a trailing else.
+            exit_reason = LoopExit.BUDGET_EXHAUSTED
+            turns_used = 0
             for _iteration in range(max_iterations):
+                turns_used = _iteration + 1
                 chat_result = client.send_message(conv_id, current_question)
                 reasoning_steps.extend(chat_result.reasoning_steps or [])
                 response_id = chat_result.response_id or response_id
@@ -684,6 +696,7 @@ def _run_once(conv_id: str) -> AlertRunResult:
                 alert_id, actual_args, tool_called = _extract_alert_call(chat_result.tool_call_events or [])
                 if tool_called:
                     alert_id_to_delete = alert_id
+                    exit_reason = LoopExit.SUCCESS
                     break
                 response_text = (chat_result.text_response or "").strip()
                 if not response_text and chat_result.alert_proposals:
@@ -692,10 +705,16 @@ def _run_once(conv_id: str) -> AlertRunResult:
                     response_text = render_answer_text(chat_result)
                 # Stop if agent gave a completely empty response (stuck)
                 if not response_text and not chat_result.tool_call_events:
+                    exit_reason = LoopExit.AGENT_SILENT
                     break
                 # Stop before generating a follow-up for the last iteration
                 if _iteration >= max_iterations - 1:
                     break
+                # No try/except here on purpose: a simulated-user failure in this evaluator
+                # already propagates as a hard error rather than being swallowed into a
+                # content failure, which is the behaviour we want. Contrast metric_skill,
+                # which catches SimulatedResponseError and breaks -- that one needs
+                # LoopExit.SIMULATED_USER_FAILED to stay distinguishable.
                 follow_up = generate_simulated_alert_response(
                     response_text, expected, conversation_history, question=question
                 )
@@ -724,6 +743,8 @@ def _run_once(conv_id: str) -> AlertRunResult:
                 response_id=response_id,
                 tool_call_events=all_tool_call_events,
                 reasoning_step_events=all_reasoning_step_events,
+                exit_reason=exit_reason,
+                turns_used=turns_used,
             )
         finally:
             if alert_id_to_delete:
@@ -886,6 +907,11 @@ def _write_scores(ctx: RunTraceContext) -> None:
         "attributes_correct": ev.attributes_correct,
         "granularity_correct": ev.granularity_correct,
         "actual_alert_arguments": best.actual_alert_arguments,
+        # Why the loop stopped. alert_created=False alone cannot tell a refusal from a run
+        # that hit max_iterations while still on track -- see LoopExit.
+        "exit_reason": best.exit_reason.value,
+        "turns_used": best.turns_used,
+        "max_iterations": max_iterations,
         "latency_breakdown": build_latency_breakdown(best.tool_call_events, best.reasoning_step_events),
     }
 

packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py

@@ -29,6 +29,7 @@
     AgenticAssertionError,
     AgenticEvalOutcome,
     ChatResult,
+    LoopExit,
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
@@ -93,6 +94,10 @@ class TurnResult(BaseModel):
     active_skills: list[str] = Field(default_factory=list)
     clarification_turns_used: int = 0
     output_correct: bool | None = None
+    # Why this turn's clarification loop stopped -- see LoopExit. output_present=False alone
+    # cannot separate a turn that ran out of clarification budget from one where the agent
+    # went silent, and skill_success folds both into the same failure.
+    exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
 
     @property
     def skill_success(self) -> bool:
@@ -112,6 +117,8 @@ def skill_success(self) -> bool:
         # What skill_routing was judged against -- without it, a turn showing
         # skill_routing=True and activated_skills=[] looks like a scoring bug.
         "active_skills",
+        # Why the clarification loop ended on this turn.
+        "exit_reason",
     }
 
     def detail(self) -> dict:
@@ -421,6 +428,10 @@ def run_agentic_conversation(
                         # read as "nothing was active", which is a different claim.
                         active_skills=sorted(active_skills),
                         output_correct=False,
+                        # This turn's loop never ran at all -- a $ref pointing at an earlier
+                        # turn's output could not be resolved. Labelling it BUDGET_EXHAUSTED
+                        # (the field default) would claim it ran out of clarification turns.
+                        exit_reason=LoopExit.NOT_RUN,
                     )
                 )
                 continue
@@ -431,6 +442,9 @@ def run_agentic_conversation(
             current_message = turn.message
             final_result: ChatResult | None = None
 
+            # Defaults to BUDGET_EXHAUSTED: every other exit assigns explicitly, so a loop
+            # that simply runs out of range() is labelled correctly with no trailing else.
+            turn_exit = LoopExit.BUDGET_EXHAUSTED
             for _iter in range(max_clarification_turns + 1):
                 chat_result = client.send_message(conversation_id, current_message)
                 final_result = chat_result
@@ -447,6 +461,7 @@ def run_agentic_conversation(
                 response_id = chat_result.response_id or response_id
 
                 if _check_output_present(resolved_turn, chat_result):
+                    turn_exit = LoopExit.SUCCESS
                     break
 
                 response_text = (chat_result.text_response or "").strip()
@@ -455,6 +470,7 @@ def run_agentic_conversation(
                 if not response_text:
                     response_text = render_answer_text(chat_result)
                 if not response_text and not chat_result.tool_call_events:
+                    turn_exit = LoopExit.AGENT_SILENT
                     break
                 if clarification_turns >= max_clarification_turns:
                     break
@@ -500,6 +516,7 @@ def run_agentic_conversation(
                     active_skills=sorted(active_skills),
                     clarification_turns_used=clarification_turns,
                     output_correct=output_correct,
+                    exit_reason=turn_exit,
                 )
             )
 

packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py

@@ -23,6 +23,7 @@
     AgenticAssertionError,
     AgenticEvalOutcome,
     ChatResult,
+    LoopExit,
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
@@ -181,6 +182,10 @@ class KdaRunResult:
     response_id: str | None = None
     tool_call_events: list[ToolCallEvent] = field(default_factory=list)
     reasoning_step_events: list[ReasoningStepEvent] = field(default_factory=list)
+    # Why the simulated-user loop stopped -- see LoopExit. `triggered=False` alone cannot
+    # separate a refusal from a run that hit max_iterations while still on track.
+    exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
+    turns_used: int = 0
 
 
 @dataclass
@@ -268,7 +273,12 @@ def _accumulate(result: ChatResult) -> None:
             all_tool_call_events.extend(result.tool_call_events or [])
             all_reasoning_step_events.extend(result.reasoning_step_events or [])
 
+        # Defaults to BUDGET_EXHAUSTED: every other exit assigns explicitly, so a loop that
+        # simply runs out of range() is labelled correctly with no trailing else.
+        exit_reason = LoopExit.BUDGET_EXHAUSTED
+        turns_used = 0
         for iteration in range(max_iterations):
+            turns_used = iteration + 1
             try:
                 chat_result = client.send_message(conv_id, current_question)
             except Exception as exc:  # noqa: BLE001 -- end this run, not the whole assertion
@@ -282,6 +292,7 @@ def _accumulate(result: ChatResult) -> None:
                     if create_args is not None:
                         turn_wall_clock_sec = partial.turn_wall_clock_sec
                 turn_completed = False
+                exit_reason = LoopExit.CHAT_ERROR
                 break
             reasoning_steps.extend(chat_result.reasoning_steps or [])
             response_id = chat_result.response_id or response_id
@@ -296,8 +307,10 @@ def _accumulate(result: ChatResult) -> None:
                 # final either way -- execute_result may still be None (e.g. the skill's
                 # execute tool isn't available at all when data-sharing is off for the org).
                 turn_wall_clock_sec = chat_result.turn_wall_clock_sec
+                exit_reason = LoopExit.SUCCESS
                 break
             if not response_text:
+                exit_reason = LoopExit.AGENT_SILENT
                 break
             if iteration >= max_iterations - 1:
                 break
@@ -313,6 +326,7 @@ def _accumulate(result: ChatResult) -> None:
                 disambiguated = True
             except Exception as exc:  # noqa: BLE001 -- safety net, not the assertion; end only this run
                 _log.warning("Simulated KDA user reply failed for conversation %s: %s", conv_id, exc)
+                exit_reason = LoopExit.SIMULATED_USER_FAILED
                 break
 
         ev = _evaluate_run(create_args, execute_result, turn_completed, disambiguated)
@@ -326,6 +340,8 @@ def _accumulate(result: ChatResult) -> None:
             response_id=response_id,
             tool_call_events=all_tool_call_events,
             reasoning_step_events=all_reasoning_step_events,
+            exit_reason=exit_reason,
+            turns_used=turns_used,
         )
 
     try:
@@ -482,6 +498,10 @@ def _write_scores(ctx: RunTraceContext) -> None:
         "disambiguated": ev.disambiguated,
         "actual_create_args": best.actual_create_args,
         "actual_execute_result": best.actual_execute_result,
+        # Why the loop stopped -- see LoopExit.
+        "exit_reason": best.exit_reason.value,
+        "turns_used": best.turns_used,
+        "max_iterations": max_iterations,
         "latency_breakdown": build_latency_breakdown(best.tool_call_events, best.reasoning_step_events),
     }
 

packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py

@@ -27,6 +27,7 @@
 from gooddata_eval.core.models import (
     AgenticAssertionError,
     AgenticEvalOutcome,
+    LoopExit,
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
@@ -159,6 +160,11 @@ class MetricRunResult:
     response_id: str | None = None
     tool_call_events: list[ToolCallEvent] = field(default_factory=list)
     reasoning_step_events: list[ReasoningStepEvent] = field(default_factory=list)
+    # Why the simulated-user loop stopped. metric_created=False alone cannot separate a run
+    # that ran out of turns, one where the agent went silent, and one where the harness's
+    # own simulated user failed -- see LoopExit.
+    exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
+    turns_used: int = 0
     timings: PhaseTimings = field(default_factory=PhaseTimings)
 
 
@@ -256,6 +262,9 @@ def _execute_single_metric_run(
     reasoning_index_offset = 0
 
     try:
+        # Defaults to BUDGET_EXHAUSTED: every other exit assigns explicitly, so a loop that
+        # simply runs out of range() is labelled correctly with no trailing else.
+        exit_reason = LoopExit.BUDGET_EXHAUSTED
         for _iteration in range(max_iterations):
             turns += 1
             agent_started = time.monotonic()
@@ -282,11 +291,13 @@ def _execute_single_metric_run(
                     f"{agent_elapsed:.2f}s; metric result received"
                 )
                 metric_result = candidate
+                exit_reason = LoopExit.SUCCESS
                 break
             response_text = (chat_result.text_response or "").strip()
             if not response_text:
                 response_text = render_answer_text(chat_result)
             if not response_text and not chat_result.tool_call_events:
+                exit_reason = LoopExit.AGENT_SILENT
                 break
             if _iteration >= max_iterations - 1:
                 break
@@ -305,6 +316,9 @@ def _execute_single_metric_run(
                     f"[timer] metric_skill {conversation_id} gpt-4o-mini simulated user failed after "
                     f"{simulated_elapsed:.2f}s"
                 )
+                # A harness-side fault, not an agent one. Recorded so reporting can treat it
+                # as an error instead of scoring the agent down for it.
+                exit_reason = LoopExit.SIMULATED_USER_FAILED
                 break
             simulated_elapsed = time.monotonic() - simulated_started
             timings.simulated_user_s += simulated_elapsed
@@ -328,6 +342,8 @@ def _execute_single_metric_run(
             tool_call_events=all_tool_call_events,
             reasoning_step_events=all_reasoning_step_events,
             timings=timings,
+            exit_reason=exit_reason,
+            turns_used=turns,
         )
     finally:
         for metric_id in created_metric_ids:
@@ -492,6 +508,11 @@ def _write_scores(ctx: RunTraceContext) -> None:
         "maql_correct": best.maql_correct,
         "expected_maql_candidates": [c.get("maql", "") for c in expected_outputs_list],
         "actual_maql": best.actual_maql,
+        # Why the loop stopped. metric_created=False alone cannot tell a refusal from a run
+        # that hit max_iterations while still on track -- see LoopExit.
+        "exit_reason": best.exit_reason.value,
+        "turns_used": best.turns_used,
+        "max_iterations": max_iterations,
         "latency_breakdown": build_latency_breakdown(best.tool_call_events, best.reasoning_step_events),
     }
 

packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py

@@ -32,6 +32,7 @@
     AgenticAssertionError,
     AgenticEvalOutcome,
     CreatedVisualization,
+    LoopExit,
     ReasoningStepEvent,
     ToolCallEvent,
     build_latency_breakdown,
@@ -57,6 +58,9 @@ class RunResult:
     response_id: str | None = None
     tool_call_events: list[ToolCallEvent] = field(default_factory=list)
     reasoning_step_events: list[ReasoningStepEvent] = field(default_factory=list)
+    # Why the simulated-user loop stopped. visualization_created=False alone cannot separate
+    # a refusal from a run that hit max_iterations while still on track -- see LoopExit.
+    exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
 
 
 @dataclass
@@ -191,6 +195,9 @@ def _execute_single_run(
 
     current_result = client.send_message(conversation_id, question)
 
+    # Defaults to BUDGET_EXHAUSTED: every other exit assigns explicitly, so a loop that
+    # simply runs out of range() is labelled correctly with no trailing else.
+    exit_reason = LoopExit.BUDGET_EXHAUSTED
     for iteration in range(max_iterations):
         total_turns += 1.0
         total_steps += float(current_result.reasoning_step_count)
@@ -207,9 +214,14 @@ def _execute_single_run(
 
         viz_produced = bool(current_result.created_visualizations and current_result.created_visualizations.objects)
         if viz_produced:
+            exit_reason = LoopExit.SUCCESS
             break
+        # render_answer_text (QA-29230) is the broader emptiness test: a turn carrying
+        # search results but no plain text is not silent. Keeping it means AGENT_SILENT
+        # labels only genuinely empty turns.
         response_text = render_answer_text(current_result)
         if not response_text:
+            exit_reason = LoopExit.AGENT_SILENT
             break
         if iteration >= max_iterations - 1:
             break
@@ -235,6 +247,7 @@ def _execute_single_run(
         response_id=response_id,
         tool_call_events=all_tool_call_events,
         reasoning_step_events=all_reasoning_step_events,
+        exit_reason=exit_reason,
     )
 
 
@@ -422,6 +435,11 @@ def _write_scores(ctx: RunTraceContext) -> None:
     ev = best.eval_result
     detail = {
         **evaluation_result_detail(ev),
+        # Why the loop stopped -- see LoopExit. total_turns is already the turn count for
+        # this run, so it doubles as turns_used.
+        "exit_reason": best.exit_reason.value,
+        "turns_used": int(best.total_turns),
+        "max_iterations": max_iterations,
         "latency_breakdown": build_latency_breakdown(best.tool_call_events, best.reasoning_step_events),
     }