Skip to content

fix(sdk): strip leading 思考 block from LLM-generated title - #4534

Closed
santhiprakash wants to merge 1 commit into
OpenHands:mainfrom
santhiprakash:fix/4530-think-tag-title-leak
Closed

fix(sdk): strip leading 思考 block from LLM-generated title#4534
santhiprakash wants to merge 1 commit into
OpenHands:mainfrom
santhiprakash:fix/4530-think-tag-title-leak

Conversation

@santhiprakash

@santhiprakash santhiprakash commented Aug 18, 2026

Copy link
Copy Markdown

HUMAN:

Santhi Prakash — small correctness fix for the SDK title-leak bug class cross-referenced from #4541; see AGENT section for evidence.


AGENT:

Why

Some providers (notably Qwen3 behind Nebius) return reasoning inline
in content as 思考...思考 instead of the normalized
reasoning_content field. The transparent leak surfaced in Agent
Canvas 1.14.0 (software-agent-sdk#4541)
as the autogenerated conversation title containing the raw 思考
reasoning markup — the SDK ships the reasoning into the title, and Agent
Canvas just renders it.

This is the SDK-side root-cause fix: peeling the leading 思考 block in
the SDK means every consumer of Conversation.generate_title() and the
agent-server auto-title path benefits, not just Agent Canvas.

Summary

  • generate_title_with_llm() now peels a leading 思考 block from the
    LLM title response before using it as the title.
  • A response that is only a 思考 block (closed or unclosed) yields
    no LLM title and falls back to the existing user-message truncation.
  • Mid-text literal 思考 occurrences in a legitimate title are
    preserved.

Issue Number

Fixes #4530

How to Test

uv run pytest tests/sdk/conversation/test_generate_title.py -q

Expected: 12 passed (4 new regression tests + 8 existing).

The 4 new tests cover:

  1. test_generate_title_strips_leading_think_block — closed
    思考…思考 block is peeled, the real title after the block is returned.
  2. test_generate_title_unclosed_think_block_falls_back — unterminated
    leading 思考… returns no LLM title; truncation fallback supplies the
    title.
  3. test_generate_title_think_only_response_falls_back — response that
    is only a 思考 block yields no LLM title; truncation fallback
    supplies the title.
  4. test_generate_title_preserves_mid_text_think_literal — a mid-text
    literal 思考 occurrence in a legitimate title is preserved.

I verified the regression coverage by reverting
generate_title_with_llm() to its pre-fix behavior and re-running the
new tests: 3/4 fail on the unfixed code (the literal-preservation test
correctly passes either way), proving the tests do catch the bug class.

$ uv run pytest tests/sdk/conversation/test_generate_title.py -q
collected 12 items
tests/sdk/conversation/test_generate_title.py ............               [100%]
============================== 12 passed in 0.38s ==============================

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • The fix is intentionally scoped to the SDK-side title generation path;
    it does not touch the Agent Canvas frontend rendering issues (plaintext
    thoughts / duplicate thought block) covered separately in
    [Bug]: Weird <think> behavior with Qwen3-32B #4541.
  • The peel is a leading-block conservative peel — only the first leading
    block is stripped, so a legitimate title that happens to contain the
    characters 思考 mid-text is preserved.
  • The unterminated-leading case consumes the entire response rather than
    returning the raw reasoning string, so the LLM path surfaces no title
    for the truncation fallback to take over.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

🚦 CI is currently failing on this PR's latest commit.

Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request @all-hands-bot as a reviewer to have it reviewed regardless of CI status.)

This is an automated check - no AI was used to generate this comment.

Some providers (notably Qwen3 behind Nebius) return reasoning inline
in  content as `思考...思考` instead of the normalized
`reasoning_content` field. The transparent leak surfaced in Agent
Canvas 1.14.0 (openhands/OpenHands#16687) as the autogenerated
conversation title containing the raw 思考 reasoning markup.

`generate_title_with_llm()` now peels a leading 思考 block from the
LLM response text before using it as the title. The peel is
conservative:
- A leading closed block `思考…思考` is stripped and the remainder is
  returned.
- An unterminated leading block (response begins with 思考 but never
  closes) consumes the entire response, and the LLM title path
  returns  None so the existing user-message truncation fallback
  supplies the title.
- Mid-text literal 思考 occurrences in a legitimate title are
  preserved.

Fixes OpenHands#4530.
@santhiprakash
santhiprakash force-pushed the fix/4530-think-tag-title-leak branch from e7133f1 to 44c81c7 Compare August 20, 2026 12:02
@github-actions

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. Because this is a fork PR, the directory will be automatically removed from main immediately after merge.

@santhiprakash

Copy link
Copy Markdown
Author

Rebased onto current main (1de2e6d, which restructured generate_title_with_llm with the new on_error callback in #4535) — the conflict was only in the import block; the think-block peel now sits inside the updated signature alongside the on_error plumbing.

Verified locally on the rebased head:

Heads-up: every workflow run on this PR (including the original Aug 18 head) has been sitting in action_required — "awaiting workflow approval" — so CI has never actually executed. The bot's "CI failing" notice reflects that gate, not a test failure. Could a maintainer approve the workflow runs so the suite can run?

@neubig

neubig commented Aug 24, 2026

Copy link
Copy Markdown
Member

I think this is probably a fix on the litellm side. The "think" blocks should be parsed out into the reasoning_content response. I don't think this is something that we should be making heuristic changes to here.

@neubig neubig closed this Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: LLM-generated conversation titles leak raw <think> reasoning blocks

3 participants