feat(providers): log prompt cache tokens in LLM response debug output - #3317
Open
vmuliadi-astro wants to merge 3 commits into
Open
feat(providers): log prompt cache tokens in LLM response debug output#3317vmuliadi-astro wants to merge 3 commits into
vmuliadi-astro wants to merge 3 commits into
Conversation
Adds prompt_cache_hit_tokens / prompt_cache_miss_tokens (DeepSeek) and prompt_tokens_details.cached_tokens (OpenAI-compatible) to UsageInfo and surfaces them in the "LLM response" debug log line so token-usage collection can report real cache split instead of treating it as unknown.
Surfaces prompt_cache_hit_tokens / prompt_cache_miss_tokens on the "LLM response" debug line when the provider reports them, so token collection can capture the real cache split.
Reapplies the prompt cache token logging additions on top of the current upstream main version of pipeline_llm.go, preserving the recent SetLastFinishReason/SetLastUsage turn-state fix (ts-based, not turnStateFromContext).
This was referenced Aug 4, 2026
|
This PR has had no activity for 7 days and has been marked as stale. If you are still working on it, please push an update or leave a comment; otherwise it will be closed automatically in 7 days. |
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gateway logs only
prompt_tokens/completion_tokens/total_tokenson its "LLM response" debug line. Providers like DeepSeek (via Cloudflare AI Gateway) report cache metadata in the sameusageobject:This metadata is parsed but then discarded:
UsageInfohas no fields for it, and the debug log line never emits it. Any token-usage collector consuming the log therefore cannot report the real cache hit/miss split — it can only estimate (or treat cache as unknown).Fix
pkg/providers/protocoltypes/types.go— extendUsageInfowith:PromptCacheHitTokens *int(prompt_cache_hit_tokens, DeepSeek-style)PromptCacheMissTokens *int(prompt_cache_miss_tokens)PromptTokensDetails *PromptTokensDetails(prompt_tokens_details.cached_tokens, OpenAI-compatible style)Pointers distinguish "provider did not report" (nil) from a real zero (cold cache).
pkg/agent/pipeline_llm.go— surface those fields on theLLM responsedebug log line when present.No provider behavior changes; purely additive logging. Verified against a live DeepSeek-via-AIG call and a full rebuild on linux/arm64.