Problem 124 · LLM Systems

Prompt-Cache Cost Simulator

MediumAI engineering

01Problem

Simulate prompt caching over a time-ordered request log. Each request has a timestamp, a `prefix_id` (identity of the exact cacheable prefix), `prefix_tokens`, `suffix_tokens` and `output_tokens`. Pricing (illustrative, per million tokens): input, output, cache-write multiplier (e.g. 1.25), cache-read multiplier (e.g. 0.1), TTL in seconds (refreshed on every hit) and a minimum cacheable prefix length. Return total cost, cost without caching, hits and writes. Also write `break_even_reads(write_mult, read_mult)`: the fewest cache reads after one write for caching to cost no more than not caching.

02Key insight

Prompt caching is a bet: you pay a premium to write the prefix and win only if it is read again before the TTL expires. The simulator makes the bet explicit for your real traffic shape, because the answer depends on inter-arrival time per prefix, not average volume: a prefix hit every 6 minutes against a 5-minute TTL pays the write premium every time and loses money. The key must be the exact prefix (model, tools, system prompt, documents in order); a timestamp or user name injected early in the prompt silently makes every request a miss.

03Solution

solution.pypython50 lines

import math
from dataclasses import dataclass


@dataclass(frozen=True)
class Pricing:
    input_per_mtok: float
    output_per_mtok: float
    write_mult: float = 1.25
    read_mult: float = 0.1
    ttl_s: float = 300.0
    min_cacheable: int = 1024


@dataclass(frozen=True)
class Request:
    t: float
    prefix_id: str
    prefix_tokens: int
    suffix_tokens: int
    output_tokens: int


def simulate(requests: list[Request], p: Pricing) -> dict[str, float]:
    inp, out = p.input_per_mtok / 1e6, p.output_per_mtok / 1e6
    expires: dict[str, float] = {}
    cost = base = 0.0
    hits = writes = 0
    for r in sorted(requests, key=lambda r: r.t):
        rest = r.suffix_tokens * inp + r.output_tokens * out
        base += r.prefix_tokens * inp + rest
        if r.prefix_tokens < p.min_cacheable:
            cost += r.prefix_tokens * inp + rest
            continue
        if expires.get(r.prefix_id, -math.inf) > r.t:
            cost += r.prefix_tokens * inp * p.read_mult
            hits += 1
        else:
            cost += r.prefix_tokens * inp * p.write_mult
            writes += 1
        expires[r.prefix_id] = r.t + p.ttl_s
        cost += rest
    return {"cost": cost, "no_cache_cost": base, "hits": hits, "writes": writes,
            "savings": 1 - cost / base if base else 0.0}


def break_even_reads(write_mult: float, read_mult: float) -> int:
    if read_mult >= 1:
        raise ValueError("reads must be cheaper than uncached input")
    return max(0, math.ceil((write_mult - 1) / (1 - read_mult) - 1e-12))

Try it first.

TimeO(n log n) for the sort, O(1) per request

SpaceO(distinct prefixes)

tests.py · 10 passing

tests.pypython19 lines

P = Pricing(input_per_mtok=3.0, output_per_mtok=15.0)
close = lambda x, y: abs(x - y) < 1e-12
one = simulate([Request(0, "sys", 10_000, 0, 0)], P)
assert close(one["cost"], 0.0375) and close(one["no_cache_cost"], 0.03) and one["savings"] < 0
two = simulate([Request(0, "sys", 10_000, 0, 0), Request(100, "sys", 10_000, 0, 0)], P)
assert close(two["cost"], 0.0405) and close(two["no_cache_cost"], 0.06) and two["hits"] == 1
kept_warm = simulate([Request(t, "sys", 10_000, 0, 0) for t in (0, 200, 400, 650)], P)
assert kept_warm["hits"] == 3 and kept_warm["writes"] == 1
cold = simulate([Request(t, "sys", 10_000, 0, 0) for t in (0, 301, 602)], P)
assert cold["hits"] == 0 and cold["writes"] == 3
assert simulate([Request(0, "x", 10_000, 0, 0), Request(1, "y", 10_000, 0, 0)], P)["hits"] == 0
small = simulate([Request(0, "s", 500, 100, 50), Request(1, "s", 500, 100, 50)], P)
assert close(small["cost"], small["no_cache_cost"]) and small["writes"] == 0
mixed = simulate([Request(0, "sys", 2000, 300, 200), Request(10, "sys", 2000, 500, 100)], P)
expected = 2000 * 3e-6 * 1.25 + 300 * 3e-6 + 200 * 15e-6 + 2000 * 3e-6 * 0.1 + 500 * 3e-6 + 100 * 15e-6
assert close(mixed["cost"], expected)
assert break_even_reads(1.25, 0.1) == 1
assert break_even_reads(2.0, 0.1) == 2
assert break_even_reads(1.0, 0.1) == 0

04Staff-level follow-up

Real caches match the longest cached prefix at block boundaries, so model several breakpoints (tools, system prompt, documents, conversation history) and order prompt sections from most stable to most volatile. Read the provider's usage fields (cache read and write token counts) and alert when the hit rate drops after a deploy; that is how you catch a prompt change that put a timestamp at the top. Price and TTL differ by provider and model, so keep them in config, never in code.