How would you attribute LLM and GPU costs to teams on a shared platform?
- 5 min — spend types and who pays today
- 10 min — units: API tokens, weighted pool tokens, GPU-hours
- 10 min — tagging enforcement and call-graph attribution
- 10 min — incentives, idle capacity, stability
- 5 min — rollout: showback → chargeback, ownership
Pick the unit per spend type. API spend: tokens × price per model, straight from gateway logs. Shared inference pools: allocate the pool's cost by weighted tokens (uncached input × w_in + cached input × w_cache + output × w_out), since GPU time per output token is several times that of an input token. Dedicated GPUs: by allocation per Kubernetes namespace (OpenCost). Training: GPU-hours per job. Idle headroom: either spread pro-rata or shown as an explicit platform line.
None of it works without mandatory metadata — team, app, feature, environment — enforced at the gateway by rejecting untagged requests. See cost per query.
$4M/month: $2.6M self-hosted inference pool, $0.9M API, $0.5M training. The gateway emits per-request records (team, app, model, uncached/cached/output tokens, latency) to the warehouse; a daily job computes cost per team and feature. Weights: output = 5× input, cached input = 0.1×. Dashboards show $ per team, per feature and per 1k requests; budgets alert at 80%.
- Incentive-compatible: teams should save money by doing the right thing — so cached tokens and preemptible batch are charged less.
- Explainable: a team lead can reproduce their bill from raw logs and a published rate card.
- Stable: fix rates monthly so one team's usage doesn't swing another's unit price; the platform absorbs variance.
- Failure modes: shared service accounts that dump everything on 'platform' (keep unattributed spend <3%); agent call graphs spanning teams (propagate an originating-feature header); charging idle-capacity batch at full price, which discourages using idle GPUs.
- Org scope: FinOps and the platform team own the model; finance agrees chargeback; run showback for a quarter first. Build on gateway logs + OpenCost; buy FinOps tooling for the cloud bill.
Assumptions: shared pool fully loaded at $2.6M/month (GPUs, network, power; illustrative); 40 teams; monthly pool traffic 400B input tokens (60% cached) and 60B output; weights input 1, cached 0.1, output 5.
- Weighted units: 160B × 1 + 240B × 0.1 + 60B × 5 = 160 + 24 + 300 = 484B units.
- Rate: $2.6M / 484B ≈ $5.37 per M units → $5.37/M input, $0.54/M cached, $26.9/M output — compare against API list prices to decide what to self-host.
- Example team: 20B input (50% cached) + 2B output → 10 + 1 + 10 = 21B units ≈ $113k/month.
- Idle: at 65% average utilisation, ~35% of every team's bill is shared headroom; publish that split rather than burying it.
- Tooling: ~1B request records/month × ~300 B ≈ 300 GB/month in the warehouse.
What the answer sounds like at each level
Tags requests by team and multiplies tokens by API prices.
Builds a weighted-token rate card for shared pools, handles caching and idle capacity, and enforces tags at the gateway.
Designs incentive-compatible, explainable, stable pricing, sequences showback before chargeback, and aligns finance, platform and product leaders.
Follow-ups the interviewer will push
1A team disputes its bill.
Show the raw request logs and the published rate card. If the model is wrong — e.g. long contexts use more KV-time than tokens suggest — fix the rate card for everyone next month, never ad hoc.
2One user action triggers 40 calls across three teams' services.
Propagate a trace id and an originating-feature header through the call graph; attribute to the originating feature and show internal services as a breakdown.
3Should long contexts cost more per token?
Yes: add a KV-time surcharge above 32k tokens, since a 128k request holds ~30× the memory of a 4k one for its whole lifetime.
Red flags interviewers listen for
- Allocates shared-pool cost by request count.
- No enforcement of tags, so 'unattributed' is a large bucket.
- Hides idle capacity in rates without telling anyone.
- Jumps straight to chargeback with no showback period.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Cost model | Weighted tokens for pools with caching discount; GPU-hours for training; explicit idle line. |
| Data | Gateway-enforced tags; originating-feature propagation. |
| Incentives | Rates reward caching, smaller models and idle-capacity use. |
| Change management | Showback first, published rate card, dispute process. |
Getting 40 teams to accept a GPU chargeback
Illustrative exampleOur shared inference platform cost about $3.5M a month, all on the platform budget, and spend had grown 2.4× in two quarters. Finance wanted per-team accounting, but 31% of requests carried no team tag.
As the staff engineer on the six-person platform team, I owned delivering chargeback within a quarter without breaking any team.
I traced the untagged 31% to 14 shared service accounts myself and worked with each owner, then made tags mandatory at the gateway in two steps: four weeks of warnings, then rejection. I wrote and published the rate card — input, cached and output weights with the formula. The head of the support-copilot org pushed back hard: she argued chargeback would penalise the team carrying the most customer value and that the weights were arbitrary. Instead of debating, I ran showback for a full quarter, gave her team the raw logs to verify their bill, and when their data showed caching was under-credited, I lowered the cached-token weight from 0.25 to 0.1 for everyone.
Across 40 teams, untagged traffic fell from 31% to 1.8%. Over the next two quarters total spend stayed flat while traffic grew 40% — unit cost down ~29% — driven by teams adopting prompt caching (cached share 22% → 58%) and moving three workloads to smaller models. I would publish headroom as a separate line from day one: embedding it in the rate caused a 15% rate jump in a low-traffic month and a week of escalations.
Sources
- OpenCost documentation — OpenCost (CNCF), 2025docs
- Prompt caching — Anthropic, 2025docs