Anthropic prompt caching: runtime context leads the single system block, so the cached prefix misses on nearly every Copilot turn

[Ron-Brouwer]

Which component is this issue related to?

Umbraco.AI (Core) with Umbraco.AI.Agent and Umbraco.AI.Anthropic (feature request / performance)

Which Umbraco AI version are you using?

Umbraco.AI 17.4.0-rc.3, Umbraco.AI.Agent 17.2.0-rc.3, Umbraco.AI.Anthropic 17.1.0, Umbraco.AI.Agent.Copilot 17.1.0-rc.3, Umbraco CMS 17.6.2.

Summary

With prompt caching enabled on an Anthropic profile, the runtime context (Entity Context, page details, anything an IAIRuntimeContextContributor adds) is folded into the front of the single system block. That block changes as soon as the editor or a write tool touches the document, which in a Copilot session is almost every turn. Because it sits at the very start of the prompt, the whole cached prefix (system prompt, context resources, agent instructions and the conversation so far) misses, and the request pays full input price again. The stable part of the prompt, which is most of it, never gets to be cached.

Specifics

How the prompt is composed today (rc.3):

• Umbraco.AI.Agent.Core/Chat/ScopedAIAgent.cs StageSystemMessageParts (line 200) joins every SystemMessageParts entry with \n\n into one string under PendingSystemMessage.
• Umbraco.AI.Agent.Core/Chat/AIAgentSystemMessageChatClient.cs Inject (from line 77) puts that string at index 0. When the first message is already a system message it is folded into it as $"{systemPrompt}\n\n{existingContent}", i.e. runtime context first, then the agent's instructions and whatever the context injector adds. The remark says so explicitly: "runtime context first, then anything a later middleware — notably the context injector — adds to the same block."
• Umbraco.AI.Anthropic/AnthropicChatCapability.cs ApplyCapabilitySettings translates the profile's prompt-caching TTL into a single top-level cache_control (ResolveCacheControl, line 292), which marks the last block of the request. The remark documents why a block-level marker was not used: the M.E.AI adapter appends the caller's instructions after any raw System blocks the representation supplies, so a marker on one of those would sit ahead of the content worth caching. AnthropicPromptCachingWireTests.BlockLevelCacheControl_IsNotReachable_… pins that down.

Why that defeats the cache:

• Anthropic caches by exact prefix. Order on the wire is tools → system → messages. The system block is one text block; the first bytes of it are the runtime context.
• The Entity Context contains the document's current property values. Any set_value / add_item, or the editor typing, changes it, so the system block differs from the previous turn from its first line on. Every later block (context resources, instructions, the whole conversation) is then a cache miss as well.
• With the single top-level marker, cache lookups walk back from the end of the request over the last ~20 block boundaries. In a longer conversation with tool calls those 20 blocks do not reach the system prompt, so even when the system block is unchanged, the stable system + tools prefix has no breakpoint of its own.

Net effect: the "prompt caching" profile setting yields little on the Copilot, which is exactly the surface with the largest, most repetitive prompts (Entity Context with full block JSON, knowledge items, tool definitions).

Steps to reproduce

1. Anthropic profile, prompt caching set to 1h.
2. Copilot sidebar on a document with a block list. Ask a question that only reads (no writes). Repeat once: the second turn should show cache_read_input_tokens close to the first turn's input size.
3. Now ask for a change ("set the title to X", which runs set_value), then ask another read-only question.
4. Compare usage: the turn after the write shows cache_creation_input_tokens ≈ the full prompt and cache_read_input_tokens ≈ 0, although the agent instructions, context resources, tools and the earlier conversation did not change.

Expected result / actual result

Expected: the stable part of the prompt (tool definitions, agent instructions, context resources) is a cached prefix that survives changes to the runtime context and to the conversation; only the volatile context and the new messages are billed as fresh input.

Actual: the volatile runtime context leads the single system block, so a document change invalidates the entire prompt.

Suggested change

1. Order (Agent.Core): append the runtime context after the stable content instead of prepending it, or keep it as a separate system message: stable instructions and context resources first, PendingSystemMessage last. AIAgentSystemMessageChatClient.Inject is the one place that decides this. An option to emit the runtime context as a separate ChatRole.System message (rather than folding into messages[0]) would let providers treat it as its own block.
2. Breakpoint (Anthropic): with the stable content first, place a block-level cache_control on the last stable system block (and rely on Anthropic's automatic prefix matching for tools), keeping the top-level marker for the conversation tail. The wire test shows raw System blocks precede the adapter's appended instructions, so putting the stable prompt into the raw representation (and only the volatile part into Instructions) makes the block-level marker reachable after all.
3. Optionally expose cache_creation_input_tokens / cache_read_input_tokens per conversation in the usage dashboard so the effect is visible to the people who enable the setting.

Dependencies

Umbraco CMS 17.6.2; Anthropic prompt caching (ephemeral, 5m/1h).

[mattbrailsford]

Fixed — this is out now on NuGet as of RC 4:

• Umbraco.AI 17.4.0-rc.4 / 18.4.0-rc.4
• Umbraco.AI.Agent 17.2.0-rc.4 / 18.2.0-rc.4

If you get a chance, please pull the RC and confirm the cached prompt prefix now holds across turns. Reopen if you're still seeing cache misses.