Grob includes an in-memory response cache backed by moka, a concurrent cache with TTL-based expiration. Caching eliminates redundant LLM calls for identical deterministic requests.
A response is cached only when all of the following conditions are met:
[cache] enabled = truein configuration.- The request temperature is
0or absent (unset). Any non-zero temperature makes the request non-deterministic and uncacheable. - The request is non-streaming (
stream = falseor absent). - The response body size does not exceed
max_entry_bytes.
Streaming requests bypass the cache entirely -- both lookup and storage.
The cache key is a SHA-256 hex digest (64 characters) computed from these fields, separated by | delimiters:
- Tenant ID (from JWT claims;
"anon"if unauthenticated) - Model name (after routing / auto-mapping)
- Messages array (JSON-serialized)
- System prompt (JSON-serialized, if present)
- Tools (JSON-serialized, if present)
- max_tokens value
The serialization streams JSON bytes directly into the SHA-256 hasher via a Sha256Writer adapter, avoiding intermediate string allocations.
The tenant ID is the first component of the cache key. This means tenant A and tenant B never share cached responses, even for identical requests. Unauthenticated requests all share the "anon" tenant namespace.
On a cache hit, the response includes:
- HTTP status
200 content-typeheader from the original cached responsex-grob-cache: hitheader- Transparency headers (provider, model, request ID) if transparency is enabled
For streaming clients that receive a cache hit, Grob synthesizes a valid SSE stream from the cached non-streaming response body. Both Anthropic-native and OpenAI-compatible SSE formats are supported.
[cache]
enabled = true
max_capacity = 2000 # Maximum number of cached entries (default: 2000)
ttl_secs = 3600 # Time-to-live per entry in seconds (default: 3600 / 1 hour)
max_entry_bytes = 2097152 # Maximum single response size in bytes (default: 2 MiB)| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable or disable the response cache |
max_capacity |
integer | 2000 |
Maximum number of entries the cache holds before eviction |
ttl_secs |
integer | 3600 |
Seconds before an entry expires regardless of access |
max_entry_bytes |
integer | 2097152 (2 MiB) |
Responses larger than this are not cached |
Moka handles eviction automatically:
- TTL expiration: entries expire after
ttl_secsregardless of access frequency. - Capacity eviction: when the cache reaches
max_capacity, the least-recently-used entry is evicted. - Size skip: responses exceeding
max_entry_bytesare never inserted. Theskipped_too_largecounter tracks these occurrences.
All evictions are counted in the evictions statistic via an eviction listener.
The cache emits Prometheus-compatible metrics:
| Metric | Description |
|---|---|
grob_cache_hits_total |
Number of cache hits |
grob_cache_misses_total |
Number of cache misses |
grob_cache_skipped_too_large_total |
Responses skipped due to size limit |
Internal statistics are also available programmatically via cache.stats():
hits-- total cache hitsmisses-- total cache missesevictions-- entries evicted by TTL or capacityskipped_too_large-- entries skipped due to sizeentry_count-- current number of entries
The invalidate_all() method clears all entries. This is a lazy operation in moka; entries are removed as pending tasks are processed.
Cache lookup runs as Step 5 of the dispatch pipeline, after DLP input scanning and routing but before the provider loop:
- DLP input scan
- MCP tool calibration (if enabled)
- Cache key computation (returns
Nonefor non-cacheable requests) - Route classification
- Provider mapping resolution
- Cache lookup (non-streaming only)
- Fan-out or provider loop with fallback/retry
On a cache hit, steps 7+ are skipped entirely. On a cache miss after a successful provider response, the response is stored in the cache before being returned to the client.
[cache]
enabled = true
max_capacity = 5000
ttl_secs = 7200 # 2 hours
max_entry_bytes = 1048576 # 1 MiB -- skip caching very long responses
[router]
default = "smart"
[[models]]
name = "smart"
[[models.mappings]]
priority = 1
provider = "anthropic"
actual_model = "claude-sonnet-4-20250514"With this configuration, identical temperature=0 requests to the "smart" model are served from cache for up to 2 hours, avoiding redundant API calls. Responses larger than 1 MiB are passed through without caching.