Skip to content

feat(providers): log prompt cache tokens in LLM response debug output - #3317

Open
vmuliadi-astro wants to merge 3 commits into
sipeed:mainfrom
vmuliadi-astro:feat/log-prompt-cache-tokens
Open

feat(providers): log prompt cache tokens in LLM response debug output#3317
vmuliadi-astro wants to merge 3 commits into
sipeed:mainfrom
vmuliadi-astro:feat/log-prompt-cache-tokens

Conversation

@vmuliadi-astro

Copy link
Copy Markdown

Problem

The gateway logs only prompt_tokens / completion_tokens / total_tokens on its "LLM response" debug line. Providers like DeepSeek (via Cloudflare AI Gateway) report cache metadata in the same usage object:

"usage": {
  "prompt_tokens": 85,
  "completion_tokens": 10,
  "total_tokens": 95,
  "prompt_tokens_details": {"cached_tokens": 0},
  "prompt_cache_hit_tokens": 0,
  "prompt_cache_miss_tokens": 85
}

This metadata is parsed but then discarded: UsageInfo has no fields for it, and the debug log line never emits it. Any token-usage collector consuming the log therefore cannot report the real cache hit/miss split — it can only estimate (or treat cache as unknown).

Fix

  1. pkg/providers/protocoltypes/types.go — extend UsageInfo with:

    • PromptCacheHitTokens *int (prompt_cache_hit_tokens, DeepSeek-style)
    • PromptCacheMissTokens *int (prompt_cache_miss_tokens)
    • PromptTokensDetails *PromptTokensDetails (prompt_tokens_details.cached_tokens, OpenAI-compatible style)

    Pointers distinguish "provider did not report" (nil) from a real zero (cold cache).

  2. pkg/agent/pipeline_llm.go — surface those fields on the LLM response debug log line when present.

No provider behavior changes; purely additive logging. Verified against a live DeepSeek-via-AIG call and a full rebuild on linux/arm64.

Adds prompt_cache_hit_tokens / prompt_cache_miss_tokens (DeepSeek) and
prompt_tokens_details.cached_tokens (OpenAI-compatible) to UsageInfo and
surfaces them in the "LLM response" debug log line so token-usage
collection can report real cache split instead of treating it as unknown.
Surfaces prompt_cache_hit_tokens / prompt_cache_miss_tokens on the
"LLM response" debug line when the provider reports them, so token
collection can capture the real cache split.
Reapplies the prompt cache token logging additions on top of the current
upstream main version of pipeline_llm.go, preserving the recent
SetLastFinishReason/SetLastUsage turn-state fix (ts-based, not
turnStateFromContext).
@github-actions

Copy link
Copy Markdown

This PR has had no activity for 7 days and has been marked as stale. If you are still working on it, please push an update or leave a comment; otherwise it will be closed automatically in 7 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant