Skip to content

Let callers set how many tool rounds a response takes - #331

Merged
mattt merged 2 commits into
huggingface:mainfrom
akbashev:fix/maximum-tool-rounds
Oct 17, 2026
Merged

mattt merged 2 commits into
huggingface:mainfrom
akbashev:fix/maximum-tool-rounds

Conversation

@akbashev

Copy link
Copy Markdown
Contributor

GenerationOptions gains maximumToolRounds, 8 by default as before. A caller that runs longer agent turns can then allow more rounds of tool calls without changing what anyone else gets.

Every provider honours it:

  • the remote providers and MLX through ToolRoundLimit;
  • llama.cpp in its own tool loop, which had its own hard-coded 8.

@mattt

mattt commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks, @akbashev. Now that #343 is in, I'd like to settle what happens at the cap before we make it configurable. Right now hitting it throws decodingFailure and drops the round's work. I'm thinking of a separate error type that carries the entries completed so far, so a caller can continue from there. It shouldn't be a new GenerationError case, since that would break code that switches over it, and I'd like this to go out in a 1.x release. What do you think?

When you rebase, a few things to pick up:

  • The MLX tests don't compile — MLXLanguageModelTests.swift:589 and :598 are missing the options argument.
  • TranscriptCoding doesn't encode maximumToolRounds, so a decoded prompt goes back to 8.
  • maximumToolRounds accepts 0 or a negative number. With either, the first round of tool calls already goes over the limit, so every response that calls a tool fails right away. Could values below 1 be rejected?
  • SystemLanguageModel ignores it, which the documentation should say.

GenerationOptions gains maximumToolRounds, 8 by default as before, so a
caller that runs longer agent turns can allow more rounds of tool calls
without changing what anyone else gets.
A response past its tool-round limit threw a decoding failure, and the
rounds that had run were lost. It now throws ToolRoundLimitExceeded, a
separate error type rather than a new GenerationError case, which carries
the rounds that ran, each tool call followed by its output, so a caller
can add them to the transcript and continue. The round past the limit
doesn't run and isn't included. Every provider stops the same way,
llama.cpp included, through ToolRoundLimit.

maximumToolRounds must be at least 1, as below that every response that
calls a tool would fail at once; coded options carry it and refuse a value
below 1 as a decoding error. SystemLanguageModel ignores it, as its
documentation now says.
@akbashev

Copy link
Copy Markdown
Contributor Author

Thanks, @mattt. I agree with a separate error type. It's in now, rebased on main:

  • At the limit, every provider throws LanguageModelSession.ToolRoundLimitExceeded. It's not a GenerationError case, so nothing that switches over it breaks. It carries limit, provider and transcriptEntries: the rounds that ran, each tool call followed by its output. The round past the limit doesn't run and isn't included, since most providers reject a tool call without its output, so a caller can add the entries to the transcript and continue. llama.cpp's own loop now uses the same ToolRoundLimit.
  • maximumToolRounds must be at least 1, enforced with a precondition and covered by an exit test. Coded options carry it, write only non-default values so existing transcripts encode as before, and throw a decoding error for a value below 1, so coded data can't stop the program.
  • The documentation says SystemLanguageModel ignores it.
  • The MLX test that didn't compile was removed by Run a repeated tool-call round instead of stopping #343. Everything builds and passes with the MLX and Llama traits.

@akbashev
akbashev force-pushed the fix/maximum-tool-rounds branch from f03d847 to 59d106e Compare October 17, 2026 19:24
@mattt
mattt merged commit 0d5aa96 into huggingface:main Oct 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants