STAR story bank

Our financial-analytics assistant (≈1.1M questions/month) had been moved wholesale to a reasoning model after two wrong answers reached an executive review. Wrong-answer reports fell, but blended cost went from $0.011 to $0.052 per question and p95 latency rose from 4 s to 31 s against a 10 s SLO.

I was the tech lead on a team of 4 engineers and 1 analyst. My job was to keep the quality gain and get back under the SLO and a $0.02 cost ceiling. The head of finance analytics opposed any rollback: "one more wrong number in a board pack costs more than the whole model bill."

I sampled 2,000 questions from one week and had the analyst label difficulty; 78% were single-table lookups. The team built a 450-item golden set (300 easy, 150 hard) with verified SQL results. I designed the routing experiment and wrote the router; two engineers built the eval harness and a fallback path; one handled gateway budgets. On hard items the reasoning model scored 91% vs 74% for the fast model; on easy items both scored 97%. We shipped an intent classifier routing only hard questions to the reasoning tier, with an 8k thinking cap and a 20 s timeout falling back to the fast model plus a "verify" banner. To win over the finance lead I showed her the 300 easy items side by side — identical answers — and gave her a weekly misroute report.

Hard-item accuracy stayed at 90% (n=150), easy at 97% (n=300); blended cost fell to $0.017 per question and p95 to 8.5 s. The router misrouted 3.2% of hard questions to the fast tier in the first month; we cut it to 1.4% by adding the 60 worst misroutes as training examples. What I'd do differently: build the misroute metric before launch — for two weeks we only learned about misroutes from user complaints.

Our research-agent feature (beta, 120 tenants, ~38k tasks/week) had been priced from a demo averaging 4 calls per task at $0.09. Two weeks into beta, spend was running at $0.41 per task — 4.6× forecast — and the GA price was already announced.

I owned the feature's economics as staff engineer on a 6-person team (4 engineers, 1 PM, 1 data scientist). I had to produce an accurate model and bring mean cost under $0.15 without dropping task success below 85%. Our PM argued for simply raising the price; sales opposed that because 30 tenants were mid-negotiation.

I exported 9,400 task traces from the gateway and plotted calls and tokens per task: p50 5 calls, p95 22, and the top 6% of tasks consumed 41% of tokens because re-sent context grew every step. I wrote the per-task cost model and the budget design; the data scientist built the 300-task golden set; two engineers implemented stop conditions (12-step cap, 150k-token budget) and tool-result summarisation; one moved the stable 5k-token system prompt to prompt caching and routed extraction steps to a smaller model. I persuaded the PM by showing the price rise needed to cover p95 tenants would be 3×, which sales confirmed would lose the 30 deals.

On the golden set (n=300) task success went from 87% to 86%; in production mean cost fell to $0.13 and p95 to $0.48 per task over the next 4 weeks (n≈150k tasks). Per-tenant budgets with 80% alerts went into the launch checklist, and the forecast template now models a step distribution. What I'd do differently: instrument per-step cost before beta — we lost two weeks reconstructing it from logs.

A product team asked my platform group to stand up a vector pipeline for their onboarding assistant. They had budget approved and a vendor shortlist.

I was the reviewing engineer on the design. My job was to sign off on infra, but the design didn't justify itself.

I counted the corpus: 340 pages, changing a few times a quarter. I built a two-day spike that stuffed the whole thing into context with prompt caching and ran both approaches against 120 questions the support team had collected. Flat context scored higher on faithfulness and cost less per query at their projected volume. I wrote a one-page memo with both numbers and a threshold — "revisit this when the corpus passes roughly 1,500 pages or we need per-document ACLs."

They shipped six weeks earlier with no vector infrastructure to operate. The threshold memo got reused by two other teams. About a year later one of them did cross the line and came back with the memo in hand, which made that review trivial.

Our customer-facing docs assistant started quoting a deprecated API parameter that we had removed from the documentation months earlier. Support escalated it as a retrieval bug.

I owned the retrieval layer, so it landed on me, but I didn't think it was retrieval.

Before touching the index I pulled 40 failing traces and checked whether the correct chunk was in context. It was, in 37 of 40 — the model was overriding it with pretrained knowledge of our older public API. I ran a small A/B: one arm added an explicit precedence instruction and a refusal clause, the other reordered context so the authoritative chunk sat last. Both helped; together they took the contradiction rate from 9% to under 1% on our eval set. I also added a groundedness check that flags answers containing entities absent from the context, which gave us a standing metric instead of waiting for support tickets.

Fixed in under a week with no index changes. The broader win was process: we split our eval dashboard into retrieval metrics and generation metrics, and three later "retrieval bugs" turned out to be prompt issues too.

Our enterprise document assistant had a stubborn quality floor. Recall looked acceptable in aggregate, but a specific set of customers reported the assistant "didn't know" entire documents they had uploaded.

I picked it up as a quality investigation after two sprints of retrieval tuning had produced nothing.

Instead of looking at retrieval, I audited the ingestion output: I sampled 500 ingested documents and compared extracted character count against file size, flagging outliers. About 12% of PDFs produced almost no text — they were scans, and our extractor returned empty strings without erroring. A further set were multi-column reports being read straight across, producing word salad. I added an OCR fallback triggered by a text-density check, switched to a layout-aware parser for multi-column detection, and — the part that mattered most — added a per-document ingestion quality score with an alert, so a bad parse became a visible failure instead of a silent one.

Recall on the affected customer set went from roughly 40% to the high 80s. The durable change was cultural: ingestion got its own dashboard and its own SLO, and "is it in the index correctly?" became the first question in every quality investigation rather than the last.

Our embedding provider announced deprecation of the model backing a 40-million-chunk index serving several internal products, with a hard cutoff about four months out.

I led the migration across three consuming teams. The constraint was no quality regression and no read downtime, on a corpus that changed continuously.

I started with the eval, not the infra: we froze a 400-query golden set with labelled relevance and benchmarked four candidate models, including a smaller one at half the dimensions. The smaller model scored within noise of the largest on our domain, which cut index memory roughly 45%. For the cutover I built a parallel index and a dual-write path so live edits landed in both, backfilled over nine days with a rate-limited worker pool, then ran a week of shadow reads diffing top-10 overlap and reranker scores. We ramped behind a per-tenant flag. The step I had to argue for was re-calibrating the abstention thresholds on the new space — the old values would have silently doubled our refusal rate.

Cut over with no downtime and a small quality improvement on the golden set, at lower ongoing cost. I wrote the dual-write and shadow-compare pieces as a reusable migration module; the next model swap, a year later, took one engineer about two weeks instead of a quarter.

We added a cross-encoder reranker to our search assistant. Answer quality improved clearly on the eval set, but p95 latency went from about 1.4s to 2.1s and we breached the SLA the product team had committed to.

Keep the quality gain, get back under budget. Rolling back was the easy option and I didn't want to take it.

I profiled the stage and found two things: we were reranking 200 candidates when a sweep showed quality plateaued around 60, and reranking ran strictly after retrieval completed on both indexes even though the dense results arrived much earlier. I cut the candidate pool to 64, started reranking dense results while the lexical leg was still running and merged late arrivals, and added a semantic cache on query-document pairs that hit on roughly 30% of traffic from repeated support questions. I also distilled a smaller reranker and kept the large one behind a flag for a quality-sensitive tenant.

p95 came down to about 1.5s, keeping almost all of the nDCG improvement — we lost under a point. More usefully, the candidate-count sweep became a standing dashboard, so the next team adding a reranker started from a tuned number instead of a default.

Support flagged that our assistant kept missing answers that sat plainly in our FAQ pages, while finding long-form guides reliably. Individually each miss looked like a one-off ranking issue.

I owned ranking quality that quarter and wanted the class-level cause rather than a per-document patch.

I grouped a quarter's worth of low-rank complaints by source document type, and the pattern was immediate — FAQ-format content was over-represented by a wide margin. The cause was our chunker splitting on headings, which separated each question from its answer: the question-chunk matched user queries but contained no answer, and the answer-chunk contained the answer but matched nothing. I changed the chunker to treat a Q&A pair as an atomic unit, backfilled that content type, and added a chunk-level sanity check that flags chunks ending in a question mark with no following body.

Recall on FAQ-sourced questions roughly doubled and the complaint category disappeared from the support queue. The broader change was that we started bucketing quality complaints by document type as standard triage, which surfaced two more structural issues over the following months.

We were launching a semantic cache to cut LLM spend on our multi-tenant assistant. The design keyed cached answers on the normalised query embedding, and it was already in staging with good hit rates.

I was reviewing it as the engineer responsible for the retrieval platform. The cost savings were real and the team was close to shipping.

The cache key had no tenant component, which meant two customers asking the same question would share an answer built from one customer's documents. I wrote a failing test that demonstrated the leak concretely rather than describing it in review comments — that changed the conversation in about ten minutes. We added tenant to the key, which cut the hit rate roughly in half, so I proposed recovering it by caching the expensive reranking stage per tenant and caching only non-grounded general answers globally. Then I went looking for the same class of bug elsewhere and found two: trace logs capturing chunk text without tenant redaction, and an internal eval tool running with a service account that could read every namespace. I added cross-tenant adversarial tests to the shared CI template so every service inherits them.

We shipped the cache with about 70% of the projected savings and no isolation gap. The adversarial test template has since caught two similar mistakes in other teams' services before they reached staging, which is a better outcome than the original catch.

On a Monday our assistant's answer quality dropped noticeably — vague answers, more abstentions — with no deploy in the preceding four days. I was on call.

Restore quality, find the cause, and close the detection gap that let it run over a weekend unnoticed.

I started from traces rather than hypotheses. Comparing Friday and Monday traces for the same queries, retrieval scores were visibly lower across the board, which pointed at the index rather than the prompt or the model. The ingestion log showed a bulk re-import from one source system over the weekend: they had changed their export to include navigation chrome and footers on every page, so thousands of chunks were now mostly boilerplate, and that boilerplate was crowding candidate pools. I rolled the index back to the Friday snapshot to mitigate within the hour, then added a boilerplate-stripping step and a per-source chunk-quality check — mean content-to-template ratio — that fails ingestion on a sharp drop. For detection, I added a daily automated eval run against the golden set with an alert on regression, which is what should have caught it on Saturday.

Quality was back within about 90 minutes and the root cause was fixed the same day. The daily eval alert has since caught three regressions before any customer noticed, including one from an upstream model change we hadn't been told about. The postmortem also changed how we treat data changes — we started gating bulk ingests the same way we gate code deploys.

Our internal developer-docs assistant generated weekly changelog digests for 40 engineering teams. It had shipped as an agent with 8 tools. Across 1,200 runs over six weeks, steps per run ranged from 4 to 25, cost per digest varied 6× (p50 $0.18, p95 $1.10), and 9.5% of runs ended on the 25-step cap after looping on the same search tool.

I was the tech lead on a team of four. The goal: make digests predictable without losing quality, and decide whether agency was earning its keep. The original author, a senior engineer, disagreed strongly — he argued the agent “handled weird weeks” and a chain would be brittle.

I pulled 300 traces and did the error analysis myself: 91% of successful runs followed the same four steps in the same order; nearly all variation was failure. Rather than argue, I asked the dissenting engineer to pick the 30 “weird weeks” he worried about, and we put them in the golden set alongside 120 normal weeks. Two engineers rebuilt the core as a chain — fetch merged PRs, fetch release notes, draft, evaluator-optimizer check against a 5-point rubric with at most 2 revisions. The dissenter built the escape hatch: a bounded agent (8-step cap) for weeks a classifier flagged as anomalous. I wrote the comparison harness and owned the go/no-go.

On the 150-task golden set (n=150, 3 trials each), rubric pass rate went from 82% to 85%, cost per digest p95 fell from $1.10 to $0.21, and step-cap exits went from 9.5% to 0. The anomaly branch triggered on 7% of weeks. The team adopted the rule that new features start as workflows. What I'd do differently: I spent two weeks arguing from intuition before running the error analysis; I'd run it on day one, and I'd have let the dissenter own the golden-set selection from the start — his 30 hard cases were the most valuable part of the eval.

We built an agent to resolve account-access tickets — about 31% of our support volume, ~1,100 tickets/day, median human resolution time 4 h 10 min. Offline it passed 92% of a 180-task golden set (pass^3 84%). The head of support wanted it on for all access tickets within two weeks to hit a quarterly backlog target.

I was tech lead for a team of five (three engineers, one data scientist, one support-ops analyst). My job: launch without a single wrongly unlocked account and produce evidence for how much autonomy each action class should get. The head of support was the dissenting voice; the blast radius of a mistake was account takeover for real customers.

I proposed a 2-week shadow and committed to a date for the first canary so the timeline stayed concrete. Engineers built the shadow diff; the data scientist owned the comparison; I reviewed every disagreement category myself. The shadow ran on 2,300 tickets: 91% agreement with human decisions, but 14 proposals (0.6%) unlocked accounts after only weak identity signals (email match without device or recent-login evidence) — a class absent from the golden set. I took those 14 cases to the head of support; seeing real accounts that would have been unlocked changed the conversation in one meeting. We tightened the verification tool, added 40 tasks for that class, then ran a 5% canary with approval on every action and automated abort thresholds.

Canary: 1,140 actions, 0 wrong actions (95% upper bound ≈ 0.26%), 6% of proposals edited by approvers. We auto-approved unlocks with strong signals; MFA resets stayed gated. Six weeks later 38% of access tickets resolved automatically with median time 6 minutes, and the backlog target was met three weeks after the original date. The shadow-diff tooling became the platform default for four later agent launches. What I'd do differently: bring the fraud team in before shadow — they recognised the weak-signal pattern immediately and could have saved a week.

I was tech lead on a 7-engineer team running an internal developer-docs assistant with ~4,000 weekly users. We shipped prompt changes about three times a week after spot-checking a handful of outputs. In one quarter, 9 of 23 prompt changes caused a regression that users reported, and nobody could say whether quality was trending up or down.

My goal was to make prompt changes measurable without slowing the team. I owned the design and the first version of the harness; two engineers owned labelling and CI wiring with me.

I pulled 140 bug reports from the previous two months and turned them, with one other engineer, into 210 cases with 7 binary rubric criteria. For 4 criteria an LLM judge reached κ ≥ 0.7 against our labels on 150 items, so we automated those; the other 3 became deterministic checks in code. I wired an 80-item smoke set into CI as an eval gate (under 6 minutes) and ran the full suite nightly with per-slice confidence intervals posted to the team channel. Our PM disagreed: she expected the gate to halve shipping velocity. I proposed a two-week trial with an override label and agreed to roll it back if median PR cycle time rose by more than half a day; we tracked it publicly.

Median PR cycle time went from 1.6 to 1.9 days, inside the agreed bound, so the gate stayed. User-reported regressions dropped from 9 of 23 changes to 2 of 26 the following quarter, and the override was used 3 times, each with a written reason. What I'd do differently: I left the judge prompt outside version control for the first month; an untracked edit shifted baselines by 3 points and we spent a week chasing a regression that didn't exist.

Twelve features across six teams ran on one hosted model snapshot when the provider announced its retirement 90 days out. Prompts had been tuned to that model for a year, and there was no shared eval: each team checked quality by hand.

I led the platform side as the gateway tech lead, with three platform engineers; the twelve feature owners owned their prompts and sign-off. My goal was every feature migrated before day 75, with no user-visible quality drop and a reusable process.

I used gateway logs rather than our prompt registry to inventory call sites and found 31, four of which were in no registry (two cron jobs, two internal tools). I set up a shared harness and asked each team for a golden set of 50–80 items from logged traffic, labelled with a common rubric — 740 items in total. Offline, 8 features were within 1 point; 4 regressed: two from JSON format drift, one from answers 18% longer, and one — insurance-claim notes — where refusals rose from 0.4% to 2.1%. The finance-reporting team lead pushed back: he wanted to skip this model and wait for the next release. I showed him the retirement date was contractual, offered to keep the old model as fallback on his feature until the last week, and paired one of my engineers with his team on the prompt fix, which won him over.

All 12 features migrated by day 61, with zero user-facing incidents attributed to the migration and cost per query down 22% on the new model. When the next upgrade came, the same 740-item suite let us qualify the candidate in 9 days instead of 2 months. What I'd do differently: I let two teams skip shadow traffic to save time, and one had a downstream regex parser break at the 5% canary for 40 minutes before rollback — shadowing would have caught it for free.

Our serving platform ran ~25 internal applications on ~300 H100s. Autoscaling keyed on GPU utilisation kept the fleet pinned at maximum replicas around the clock, yet we still breached the TTFT p95 SLO about 9 times a month at peak.

I was tech lead of the four-person serving team and owned fixing both the cost and the SLO breaches without new hardware.

I pulled 30 days of engine metrics and showed utilisation sat at 97–100% at both 50 and 400 QPS, while the waiting-queue gauge rose ~90 s before the TTFT breach in 11 of 12 incidents. I proposed KEDA on waiting-per-replica plus KV usage, with a warm node pool. The SRE lead objected that custom metrics put Prometheus in the scaling path. We agreed a fail-static rule — if the metric is missing for 2 minutes the scaler holds replicas and pages — plus an HA Prometheus pair. I wrote the scaler config and a load-test harness; two engineers built the warm pool with weights on local NVMe. We rolled out to 2 apps, then 10, then all 25 over five weeks.

Over the next 60 days SLO breaches fell from ~9 to 1 per month, average GPUs in use dropped from ~300 to ~210, and the freed capacity ran batch evals (~65k GPU-hours/month). Cold start went from 6.5 min to 80 s. In hindsight I would have split long-context traffic out first: three long-context tenants triggered needless KV-driven scale-outs in week one.

Our shared inference platform cost about $3.5M a month, all on the platform budget, and spend had grown 2.4× in two quarters. Finance wanted per-team accounting, but 31% of requests carried no team tag.

As the staff engineer on the six-person platform team, I owned delivering chargeback within a quarter without breaking any team.

I traced the untagged 31% to 14 shared service accounts myself and worked with each owner, then made tags mandatory at the gateway in two steps: four weeks of warnings, then rejection. I wrote and published the rate card — input, cached and output weights with the formula. The head of the support-copilot org pushed back hard: she argued chargeback would penalise the team carrying the most customer value and that the weights were arbitrary. Instead of debating, I ran showback for a full quarter, gave her team the raw logs to verify their bill, and when their data showed caching was under-credited, I lowered the cached-token weight from 0.25 to 0.1 for everyone.

Across 40 teams, untagged traffic fell from 31% to 1.8%. Over the next two quarters total spend stayed flat while traffic grew 40% — unit cost down ~29% — driven by teams adopting prompt caching (cached share 22% → 58%) and moving three workloads to smaller models. I would publish headroom as a separate line from day one: embedding it in the rate caused a 15% rate jump in a low-traffic month and a week of escalations.

Our VP of Product wanted to launch AI clause extraction for contract review to all 1,200 customer accounts on a fixed date tied to a sales event. On our 500-contract golden set, extraction was 91% correct overall but only 74% on indemnity and limitation-of-liability clauses — the two that matter most to lawyers.

I was the tech lead for a team of six. My job was to give an honest launch recommendation and, if I disagreed, a plan the VP could accept.

I didn't argue in the launch meeting; I asked for 30 minutes the next day and brought a one-page breakdown by clause type with ten real failure examples. The VP's objection was that 91% was better than the competitor's demo. I agreed it was, and proposed launching on the date to 10% of accounts with low-confidence indemnity extractions flagged for human review, ramping to 100% when that slice reached 88%. I wrote the rollout plan; the team built the confidence flag in eight days and a per-clause dashboard. I also committed publicly that if we missed the ramp criteria I'd present the reasons myself.

We launched on the date to 120 accounts. Indemnity accuracy reached 89% after three weeks of error analysis on flagged cases, and we went to 100% in week five. In that window we caught 31 extractions that would have been wrong in front of customers. What I'd do differently: I raised the concern four weeks before launch; raising it at spec time would have made it a plan input, not a late objection.

Five product teams were calling three model providers directly with their own keys. Finance couldn't attribute ~$180k/month of LLM spend, and a key leaked in a client bundle once. I was on the platform team; none of the five reported to me.

Move all five to a shared gateway with per-team budgets, rate limits and fallback within one quarter, without a mandate.

I started with the data each team lacked: I pulled 30 days of provider invoices and showed each team its own cost per feature, which none of them had. I wrote a two-page RFC and asked each team's lead to comment before it was final. The loudest dissenter was the search team's lead, who worried about added latency on a p95-sensitive path. Instead of arguing, I benchmarked the gateway on their traffic replay: +6 ms p50, +11 ms p95, and prompt-cache routing that cut their p95 by 140 ms. I built the client library and migration PRs myself for the first two teams; two engineers on my team of four did the rest.

Four of five teams migrated in 10 weeks; 96% of LLM traffic went through the gateway, spend became attributable per feature, and fallback absorbed a 47-minute provider outage with no user-facing errors. The fifth team migrated the next quarter. I'd do one thing differently: I under-invested in the on-call story for the gateway, and the first incident page went to a team that didn't know it was now on the critical path.

I had pitched and led a multi-agent assistant for insurance claims adjusters: a planner, a policy-lookup agent, and a drafting agent. After four months with a team of five, it resolved 62% of cases on our 400-case eval set at ~$1.10 per case and 40 s median latency.

We'd agreed up front that it had to beat the existing rules-plus-template flow by 10 points on resolution or halve adjuster handling time. I had to decide whether to keep going.

A junior engineer on the team built a single-call prompt chain baseline over a weekend; I insisted we evaluate it on the same 400 cases rather than dismiss it. It scored 59% at $0.08 per case and 6 s. The claims operations director — our sponsor — pushed to keep the agent because it demoed better. I showed him the side-by-side on 20 real cases with adjusters timing both; the 3-point gap didn't change handling time. I recommended stopping, wrote the decision doc, and moved the policy-lookup tool into the simpler chain.

We shipped the chain in three weeks; it cut median handling time from 14 to 9 minutes across ~80 adjusters. Two engineers moved to a fraud-signals project that shipped that quarter. My regret: the kill criteria were in a slide, not in the eval harness, so it took a weekend experiment to force the comparison we should have run in month one.

I shipped a prompt revision to our internal-wiki assistant used by ~9,000 employees. It improved answer fluency on our eval set but, as we found later, removed the instruction ordering that kept the model citing sources. Uncited answers went from 3% to 41% of responses.

I was the author and the on-call engineer that week. I needed to mitigate, find the cause, and close the gap that let it through.

A support lead flagged it 26 hours after deploy from user complaints. I rolled back via the prompt flag within 15 minutes of the report, confirmed citation rate recovered on live traffic, then diffed outputs on 200 sampled queries. Our eval measured answer quality but had no citation-presence check — that was the real gap, and I had written that eval. I added a citation-rate metric to the eval gate (fail below 95%), a live dashboard with an alert at 10% uncited over an hour, and made prompt changes go through a 5% canary. I wrote the postmortem and named my missing check as the contributing cause.

Re-shipped the improved prompt with a fixed ordering two days later: fluency gains kept, uncited rate 2.5%. In the following six months the citation gate blocked two other prompt changes from other teams. My regret: 26 hours of detection time. I'd have caught it in one hour if the canary had existed, and I'd argued against canaries for prompts months earlier as too heavy.

A mid-level engineer, eight months on the team, was primary on-call for the first time when our clinical-guidelines search started returning empty results for ~18% of queries at 2 p.m. on a weekday. About 600 clinicians used it daily.

I was secondary on-call and her mentor. I had to keep the incident short without taking it over.

I joined the call and said: 'You're incident lead; I'm here as a second pair of eyes; if we pass 30 minutes I'll suggest we roll back.' I asked questions rather than giving answers — 'what changed in the last 24 hours?', 'what does the empty slice have in common?'. She found the empty queries all carried a new department filter whose metadata field had been renamed by an ingestion change. She proposed reverting the ingestion config; I confirmed it was safe and she executed it. Recovery took 34 minutes. Afterwards I paired with her on the postmortem, and she presented it.

Empty-result rate went back to 0.4%. She added a filter-schema contract test that blocked a similar rename three months later. Within a year she was running our on-call onboarding, and the team's median time-to-mitigate on her rotations fell from 52 to 31 minutes. My regret: I let it run four minutes past the 30-minute line I'd set, because she was close — I should have held the line I'd stated.

Leadership gave me and three engineers a charter of 'use AI to transform customer support' for a developer-tools company handling ~22,000 tickets a month. No metric, no budget line, and three directors each with a different idea: a chatbot, auto-triage, and an agent that fixes customer configs.

Turn it into a plan with a measurable goal within six weeks, and get the three directors to agree.

I spent two weeks reading 500 tickets with two support leads and tagging them. 38% were answerable from existing docs, 21% needed account data, and only 4% needed a config change. I proposed one metric — tickets resolved without a human, with a satisfaction floor — and a sequence: docs-grounded answers first, account-aware triage second, config agent deferred. The director who wanted the config agent objected strongly. I showed that the 4% slice was where a wrong action could break production for a customer, and proposed a read-only diagnostic for it instead. I wrote the charter; the directors signed it.

Phase one resolved 24% of tickets without a human at CSAT 4.3/5 against a 4.2 baseline within one quarter, measured on ~66,000 tickets. Triage cut first-response time by 40%. My regret: I let the charter omit an explicit cost-per-ticket target, and we spent a month later justifying model spend that a target would have pre-approved.

We had six weeks to ship semantic search for a retailer's catalog of ~2.4M products before a seasonal peak. The full design had hybrid retrieval plus a cross-encoder reranker; the reranker alone needed GPU capacity we wouldn't have for nine weeks.

As tech lead for a team of four, I had to decide what to ship on the date.

I measured both on 1,500 labelled queries: hybrid alone reached nDCG@10 0.61 versus 0.68 with the reranker, and both beat the old keyword search at 0.49. I proposed shipping hybrid-only on the date, and wrote the trade-off down: 7 points of relevance given up, repaid by week 11, owner me. The merchandising lead dissented — she worried about poor results on high-margin categories. I added a curated boost list for her top 200 queries, which she maintained, and we agreed to roll back any category whose conversion fell below the old search during the first two weeks.

Search shipped on time; conversion from search rose 9% over the peak period versus the prior year. No category tripped the rollback. The reranker shipped in week 10, lifting nDCG to 0.69. My regret: the boost list outlived its purpose by four months because I didn't set an expiry, and it began overriding better reranked results.

Our finance director asked for an autonomous agent to process supplier invoices end to end — read the PDF, match it to a purchase order, and approve payment — after seeing a vendor demo. Volume was ~7,000 invoices a month and a mismatch cost real money.

I led the AI engineering group of seven. I had to decide what to build and, if not an agent, persuade the director.

I mapped 300 historical invoices with an AP clerk: 88% followed one of four fixed paths (exact PO match, partial delivery, price variance under 2%, missing PO). That's a workflow, not an open-ended task. I built a two-day prototype: an extraction step with structured output, deterministic matching rules, and an LLM only for line-item reconciliation, with every payment above $5,000 routed to human approval. On the 300 invoices it was correct on 94% versus 81% for an agent prototype my colleague built in parallel, at a tenth of the cost. The director's concern was that it wouldn't handle the odd 12%; I routed those to a clerk queue with the extraction pre-filled.

Shipped in five weeks. Clerks' handling time fell from 11 to 4 minutes per invoice on average across ~7,000 a month, with zero erroneous payments in the first quarter. We wrote an agent-vs-workflow checklist that the group now uses for every request. My regret: I framed my first reply as 'agents are risky', which put the director on the defensive; leading with the prototype comparison would have saved a week.

Four teams shipping RAG features over financial filings each reported 'faithfulness' with different definitions — two with LLM judges, one with string overlap, one with weekly manual spot checks. Leadership saw four green dashboards while customers reported unsupported numbers in answers.

As the staff engineer for applied AI, I needed one standard the teams would actually use, with no authority over three of them.

I ran a calibration exercise: 250 answers sampled from all four products, labelled by two analysts for claim-level support. Inter-annotator agreement was κ 0.81. Against those labels, the four team metrics ranged from κ 0.32 to 0.71. I shared the results with each lead privately first. The team with the overlap metric objected that judges were too expensive at their volume. I proposed tiers: a claim-level judge on a 2% sample online plus a full run on a 400-item golden set per release, which cost them ~$90 a week. I wrote the standard with the four leads as co-authors and built the harness integration; my team of three maintained it.

All four teams adopted it within two months; the judge reached κ 0.76 against analyst labels. The standard surfaced a numeric-hallucination bug in one product that had been reporting 97% faithful — real rate 88% — fixed within a sprint. My regret: I didn't include the analysts in the standard's governance, and when their labelling guidelines changed six months later, the calibration set drifted without anyone noticing.

I was convinced we needed to fine-tune an open-weights model for our legal-research assistant to get jurisdiction-specific answers right. I'd argued for it in planning; a senior engineer on another team argued that better retrieval over the statute corpus would get us further for less.

I was leading the effort with two engineers and had a quarter of GPU budget allocated on my recommendation.

Before spending the budget, I set up a fair comparison on 600 questions across six jurisdictions, labelled by two contract attorneys: a LoRA fine-tune, the base model with improved hybrid retrieval and a reranker, and both together. I pre-registered the decision rule: pick the cheapest option within 2 points of the best. Retrieval alone scored 81%, fine-tune alone 72%, both 82%. Retrieval won by the rule. I presented the result myself, credited the other engineer by name in the meeting, and returned 70% of the GPU budget.

We shipped the retrieval improvements in five weeks; jurisdiction-correct answers went from 64% to 80% in production sampling (n = 1,000). The returned GPU budget funded a red-teaming effort that found two prompt-injection paths before launch. My regret: I'd defended my position for a month in planning before testing it — a two-week experiment at the start would have saved both teams friction.