[Bugfix] Clear conflicting structured outputs in strict tool calling - #44134
Closed
alexeldeib wants to merge 1 commit into
Closed
[Bugfix] Clear conflicting structured outputs in strict tool calling#44134alexeldeib wants to merge 1 commit into
alexeldeib wants to merge 1 commit into
Conversation
ToolParser.adjust_request's strict structural-tag path (added in vllm-project#40894, gated by VLLM_ENFORCE_STRICT_TOOL_CALLING) installs structural_tag on a pre-existing StructuredOutputsParams via in-place attribute assignment and returns without nulling response_format. The in-place set bypasses StructuredOutputsParams.__post_init__, so the params keep a prior mutually-exclusive constraint (json/regex/choice/grammar/json_object, or one lowered from response_format) next to the new structural_tag. On the next re-validation this trips the one-constraint invariant, so a strict-mode request that also carries a structured-output constraint or a response_format fails with: ValueError: You can only use one kind of structured outputs constraint but multiple are specified This affects any parser that installs a structural tag -- currently DeepSeek-V4 and Qwen3-Coder via get_structural_tag. The env var is off by default, and a request with no pre-existing constraint is unaffected. Fix: rebuild structured_outputs with only the structural tag (preserving the whitespace / additional-properties knobs) and null response_format, mirroring Step 2 of the same method. This "tool constraint wins, response_format dropped" resolution already exists in Step 2 and the DeepSeek-V3.2 override (vllm-project#41178), and is the intent of the open auto-path fix vllm-project#39969; the in-place-vs-rebuild trade-off was discussed on vllm-project#40894 and vllm-project#43155 (whose Kimi path already rebuilds). Repro / regression test (CPU, no model required): pytest tests/tool_use/test_strict_tool_calling_adjust_request.py The added tests enable strict mode, give a parser a structural tag, and send tools together with a response_format or a structured_outputs.json constraint (tool_choice auto and required). On the pre-fix code adjust_request leaves two constraints, and to_sampling_params raises the ValueError above; with this change structured_outputs holds only the structural tag, response_format is None, and the user's whitespace knobs are preserved. The conflict tests fail without this patch and pass with it; the no-pre-existing-constraint case passes either way. Equivalently over HTTP: with strict mode on, a tool_choice="auto" request that also sets response_format returns HTTP 400 (the error above) before this change and a normal tool call after; a required-tool request is unaffected because that path already rebuilds. Signed-off-by: Ace Eldeib <aeldeib@coreweave.com> Co-authored-by: OpenAI Codex <codex@openai.com>
alexeldeib
force-pushed
the
alex/fix-strict-tool-calling-structured-outputs
branch
from
June 10, 2026 03:18
e58f771 to
147136c
Compare
alexeldeib
marked this pull request as ready for review
June 10, 2026 03:18
alexeldeib
requested review from
aarnphm,
bbrowning,
chaunceyjiang and
sfeng33
as code owners
June 10, 2026 03:18
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Contributor
Author
|
Superseded by #45003: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix strict tool-calling request adjustment when a structural tag is installed over an existing structured-output constraint or
response_format.With
VLLM_ENFORCE_STRICT_TOOL_CALLING,ToolParser.adjust_requestcould mutatestructured_outputs.structural_tagin place and leave an existingjson/regex/choice/grammar/json_objectconstraint or aresponse_format. A later validation step then rejects the request with the one-constraint error. This change rebuildsstructured_outputswith only the structural tag, preserves whitespace/additional-property knobs, and clearsresponse_format.Duplicate-work check: open PR searches for
VLLM_ENFORCE_STRICT_TOOL_CALLING structural_tag response_format,strict tool calling structured_outputs structural_tag,ToolParser adjust_request response_format structural outputs,You can only use one kind structured outputs constraint structural_tag, andstrict structural tag conflicting jsonfound no duplicate genericToolParser.adjust_requestfix. #41178 and #35904 are related but target model/API-specific paths.No docs update is needed. AI assistance was used; I reviewed the changed code and test results.
Test Plan
Test Result
Local validation after rebasing on
origin/main:pre-commit run --files \ tests/tool_use/test_strict_tool_calling_adjust_request.py \ vllm/tool_parsers/abstract_tool_parser.py # PassedEssential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model. Not applicable.