Trade-off framing
Choosing one option knowing what it costs you on another axis. Senior answers name the axes, the deciding constraint, and when they'd switch.
A trade-off is a choice where improving one thing makes another worse: faster but more expensive, more accurate but slower, simpler but less flexible. In system design interviews and real engineering, there's rarely a 'best' option, only the best one for given constraints. Good engineers say what they're giving up, why it's acceptable here, and what would change their mind.
Choosing between a sports car and a van: neither is better until you say whether you're moving house or racing.
Answering 'it depends' and stopping. Interviewers want the dependency named and resolved: 'it depends on QPS; below 20k/day hosted wins, above it self-host.'
A useful trade-off statement has four parts: the options, the axes (latency, cost, quality, complexity, reversibility, team skill), the deciding constraint with a number ('p95 must be under 1.5 s and we have 600 ms of budget for retrieval'), and the switch condition ('if traffic exceeds 200 QPS, self-hosting becomes cheaper'). Classify the decision's reversibility: Amazon's one-way vs two-way door framing; two-way doors (prompt changes, model choice behind a gateway) deserve fast decisions, one-way doors (data model, tenancy tier, vendor lock-in on embeddings) deserve written analysis. Use evidence, not adjectives: an Oracle experiment or a quick cost/latency table beats 'X is more scalable'. Record significant decisions as Architecture Decision Records (Nygard, 2011) so the reasoning survives. Error budgets from SRE make one common trade-off explicit: reliability vs velocity, measured against an SLO & SLA.
For any significant choice: a short options × axes table with numbers, the binding constraint, the switch condition, and a one-way/two-way door label; ADR for one-way doors; two-way doors decided by the owning engineer within days.
Choosing a reranker for a legal-contracts search: Option A, a hosted rerank API: +7 points NDCG@10, +180 ms p95, $1.00 per 1k searches (illustrative). Option B, self-hosted bge-reranker-v2-m3 on one L4: +6 points, +140 ms for 50 candidates, ~$600/month fixed. At 15k searches/day, A costs ~$450/month and B $600, so A wins today; the switch point is ~20k searches/day, or immediately if a customer forbids sending contract text to a third party. It's a two-way door (behind an interface), so ship A and revisit at 15k/day growth.
I'd frame every design choice as options × axes, pick using the binding constraint with a number, say what I'm giving up, and name the condition that would flip the decision. I'd also say whether it's a one-way or two-way door, because that sets how much analysis it deserves. Follow-up: 'what if you're wrong?' For a two-way door, the cost is a migration I've already made cheap; for a one-way door, I'd have prototyped first.
| Hosted rerank | Self-host bge-v2-m3 | |
|---|---|---|
| NDCG@10 gain | +7 | +6 |
| p95 added | +180 ms | +140 ms |
| Cost at 15k/day | ~$450/mo | ~$600/mo fixed |
| Data leaves VPC | yes | no |
| Switch when | >20k/day or residency | — |
Numbers worth knowing
- Analysis for a two-way door
- hours to dayscheap to reverse; bias to action
- Analysis for a one-way door
- a written doc + prototypedata model, tenancy tier, core vendor
- Axes worth naming
- 3–5more than that and you haven't found the binding constraint
How to decide
| When | Do this |
|---|---|
| Decision is reversible behind an interface | Pick the simplest option that meets the constraint; revisit at a stated trigger. |
| Decision is hard to reverse | Write an ADR, prototype, and get review from the teams affected. |
| Two options are within noise on your eval | Choose on cost and operational simplicity. |
| A constraint is legal or contractual | It's binding; it overrides cost and quality. |
Cost maths
Break-even = fixed cost ÷ (per-unit hosted cost − per-unit self-hosted variable cost). $600/month ÷ ($1.00/1k − ~$0/1k) = 600k searches/month ≈ 20k/day. Below that, hosted is cheaper; above it, self-hosting wins on cost, before counting the engineer time to operate it.
Operating it
- A metric crosses the stated switch condition
- A new constraint appears (residency, latency SLO tightened, cost target)
- Repeated re-litigation of the same decision: write the ADR
- Metrics named in switch conditions (traffic, cost, latency): review monthly against thresholds
- ADRs with stale assumptions: review quarterly
- Decision reversals: track count and cost to calibrate how much analysis decisions get
When it goes wrong
A team spent four months building a self-hosted embedding and rerank stack to save money; total spend went up.
Quarterly cost review: GPU nodes at 12% utilisation, plus 1.5 engineers on maintenance.
The trade-off was argued on per-token price alone at a projected 10× traffic that never arrived; no switch condition was written down.
Returned to hosted APIs behind the same interface, with an ADR stating the traffic threshold for revisiting self-hosting.
Where it shows up
Sources
- Documenting Architecture Decisions — Michael Nygard, 2011blog
- Site Reliability Engineering, ch. 3: Embracing Risk — Google, 2016book
- An Elegant Puzzle: Systems of Engineering Management — Will Larson, 2019book
Check yourself
Three questions. Every option is explained once you answer.