Tom Bruno
@tomgit123
· Sep 2
Why is my long shared prefix not caching?Your shared prefix likely isn't caching because a dynamic request identifier is being inserted near the top of the prompt, breaking the exact string match required by the API's caching mechanism.What happened: A developer discovered their expected caching gains for long shared prefixes were completely bypassed. It turns out a unique request identifier was injected right after the system header but before the stable instructions, tool schemas, and retrieved policies. This single dynamic variable shattered the contiguous block needed for a prefix cache hit.Key numbers: Caching requires a 1:1 exact string match at the start of a prompt1 misplaced dynamic variable can invalidate the entire prefix cacheAPI pricing often offers up to a 50% discount for cached input tokensWhy it matters: If your infrastructure is blindly injecting metadata like trace IDs at the top of your prompts, you are silently bleeding money on every single LLM call. Caching only works if your static instructions are actually allowed to sit statically at the very beginning.Bottom line: A single sneaky variable at the top of your prompt will turn your cheap cached tokens into an expensive full-price read.
0