Needle / Diff Idea

[tomelgato]

Im not sure if it is just Claude Desktop or my environment. In my current project (python) the LLM very often does symbolic edits / regex needle operations, reads the file after for verification and needs to redo the regex till finally it decides to rewrite the whole file.
Im excluding here the "why it missed" part and focus just on token efficiency:
The first part (regex / verification) could be done with just one tool call instead of two. The *set tools with needle function returns a diff with what was changed (+/- X lines for orientation), that gives the LLM a proper verification without needed further tool calls (if it suceeded). I did this already on a MCP im working on and it seems to be quite sucessful. I integrated also a bool to switch off this function on demand. For pure insert or replacement updates no diff is required

[opcode81]

A verification is actually never required, because Serena's replacement tool is extremely robust: In particular, it will fail if the regex match is ambiguous (overlapping matches) or an unexpected number of matches exists. We spent a lot of time optimising the tool to make it LLM/fool-proof, also eliminating the need to apply escapings in the replacement text.

We explicitly instruct the LLM not to check the results of replacements in the editing mode (which is enabled by default):

You are extremely good at regex, so you never need to check whether the replacement produced the correct result.

So the only thing I would recommend trying is to more strongly discourage the checking of replacements. It is literally always unnecessary for the replace_content tool. Not so, however, with the replace_symbol_body tool (because there is no agreement among language servers as to what actually constitutes the body of a symbol).

[tomelgato]

Happens less often in my C++ projects....

[tomelgato]

Hmm, is Claude Sonnet 4.6 subpar?

[opcode81]

I concluded "subpar" from your statement that your LLM is bad at regex.
While I have specifically tested only earlier versions of Sonnet, I don't expect Sonnet to have problems that could be rectified by returning a diff. If the tool returns success, then the result should almost always be fine.

[tomelgato]

If you like i post some back and forth next time i encounter that problem. Maybe i should try with Serena JetBrains...

[opcode81]

Sure, post some examples here. Then we'll have a better basis for deciding what the right measure to take might be.

[tomelgato]

Btw. while we are here. I see often a higher token usage because the LLM is doing many symbol updates / regex in the same quite small file. Instructing the LLM to do a full rewrite if a. the file is small (< ~ X lines) and b. there are more then Y replacements needed in the same file seems to improve the behaviour.
P.S. this is based on my understanding that every tool call needs a full token session window (except there is a provider cache)

[opcode81]

I wouldn't be surprised if Anthropic nerfed Sonnet to save compute resources. Model performance has been all over the place in recent months.

[tomelgato]

Will try with Gemini CLI to see if there is a difference. In fact Sonnet did like 5 symbol based replacements (code blocks, not just a name) in a 500 lines code file. And it agreed that doing it like that including verification is a token waste :)

[tomelgato]

But to Claudes rescue, it did a targeted verify with lines from-to. Which still uses more tokens if you calculate the session window in

[opcode81]

5 symbol based replacements ...
doing it like that including verification is a token waste

The token waste is purely a result of unnecessary verifications.
Doing symbol-based replacements is more efficient otherwise.

calculate the session window in

The compute cost is a function of the number of tokens generated and the context window size when the generation starts.
For a given context window size before the operations, the question whether
a) one full rewrite of the entire file or
b) 5 symbol-based replacements
is more token-efficient depends only on the number of tokens used by
(a), i.e. the full file, and
(b), i.e. the 5 replacement texts plus minor overhead for the tool calls.

Therefore, option (b) will definitely be more efficient if the 5 replacement texts (plus overhead) are smaller than the full file, assuming no unnecessary verifications.
Doing the 5 replacements without verification would be the optimal strategy, but it hinges on the LLM knowing precisely what the tools do. This is not a problem for the replace_content tool, while it is not entirely clear for the replace_symbol tool (as stated above), as the behaviour can depend on the language.

[tomelgato]

Therefore, option (b) will definitely be more efficient if the 5 replacement texts (plus overhead) are smaller than the full file, assuming no unnecessary verifications.

Lets say our file has 100 lines and our current window size is 10k and the LLM needs to do 3 different symbol replacements. This would be 3x10k + overhead compared to 1 write of 100 lines. Or am i misleaded with that assumption?

Anyhow, right now Claude is quite wacky. This morning i had a very long session without hitting limits and with many tool calls. The next 5 hour windows i had new sessions (with same startup prompt) which did hit the limit after just a few tool calls.