Design a customer-support copilot that resolves what it can and hands off to a human agent cleanly when it can't.
- 5 min — scope: channels, intents, volume, regulated data
- 10 min — bot: retrieval, tools, auth, server-side limits
- 10 min — handoff triggers and packet contract; agent assist
- 10 min — metrics, liability, failure modes
- 5 min — estimate, rollout and ownership
Two surfaces on one platform: a customer-facing bot that resolves with RAG over help content plus tools (order lookup, claim status, refunds within server-enforced limits), and agent assist that drafts replies and summaries for human agents.
Handoff is a first-class state transition with a contract — intent, collected fields, actions already taken, sentiment, summary, transcript — so the customer never repeats themselves. Triggers: explicit request, low confidence or abstention, policy (refund above a limit, legal threat, vulnerability signals), two failed loops, negative sentiment.
Insurance claims: the customer asks for claim status → the bot authenticates by OTP, calls the claims API and answers. The customer disputes a denial → policy routes disputes to humans → the bot writes a handoff packet into the helpdesk queue (Zendesk or Salesforce Service Cloud) with priority, tells the customer the real wait time, and keeps the conversation open. The human sees the summary and an AI-drafted reply with citations, edits and sends; the edit is logged as a training signal.
- The bot's words bind the company. In Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024) the airline was held liable for its chatbot's wrong bereavement-fare answer. Policy answers must be grounded or refused, and tool limits are enforced server-side, never by prompt.
- Containment is not success. Measure resolution (no recontact within 7 days), CSAT, handoff rate, re-escalation after handoff, and human handle time with assist.
- Failure modes: deflection loops that trap customers (always offer a human after two failed turns); off-hours handoff with no one there (async callback); context lost in transfer.
- Org scope: support operations owns policies and queue routing; CX engineering owns helpdesk integration; the AI platform owns models and evals. Buy the helpdesk, build the policy and tool layer. Roll out agent assist first — low risk and it produces labelled data — then the bot on 10% of the top five intents.
Assumptions: 2M conversations/month; 8 bot turns each; per turn ~6k input (4k cached system prompt, tools and policy; 2k new history and passages) and 300 output; illustrative $3/M input, $15/M output, $0.30/M cache reads; 40% handed off; $5 blended cost per human contact (illustrative).
- Turns: 16M/month → ~6/s average, ~25/s peak; at ~5 s per turn ≈ 125 concurrent calls.
- Model $: uncached 32B × $3 = $96k + cached 64B × $0.30 = $19k + output 4.8B × $15 = $72k ≈ $187k/month ≈ $0.09 per conversation.
- Human side: 800k handoffs × $5 = $4M/month. If assist cuts handle time 20%, that is ~$800k/month — over 4× the bot's entire model bill. Lead the business case with assist.
- Containment value: 1.2M bot-resolved × $5 ≈ $6M/month gross, minus recontacts — measure the 7-day recontact rate to keep this honest.
What the answer sounds like at each level
A RAG chatbot over the help centre with a 'talk to a human' button.
Tools with server-side limits, explicit handoff triggers and a structured handoff packet; measures resolution, not deflection.
Frames liability and policy grounding, leads the business case with agent assist, defines rollout by intent and ownership across support ops, CX engineering and AI platform.
Follow-ups the interviewer will push
1An outage sends 10× traffic, all asking 'is it down?'.
Detect the incident intent, answer from the status page through a semantic cache keyed by intent, and protect the human queue; scale bot capacity rather than humans.
2A bank becomes a tenant.
Step-up authentication per tool, PII redaction in logs, no training on transcripts, in-region processing, every action audited, and complaint detection that routes regulated complaints to the right queue.
3The model provider is down.
Fail to the human queue with an apology and an honest wait time; agent assist degrades to search-only. Fallback to a second provider only for flows already evaluated on it.
Red flags interviewers listen for
- Refund or policy limits enforced only in the prompt.
- Optimises containment rate with no recontact measure.
- Handoff drops context and the customer repeats everything.
- No path to a human after repeated failure.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Handoff | Explicit triggers and a structured packet; customer never repeats themselves. |
| Safety and liability | Grounded policy answers, server-side limits, cites the 2024 tribunal case or equivalent. |
| Metrics | Resolution with recontact window, re-escalation, handle time, CSAT. |
| Business case | Shows agent assist savings rival or exceed bot model spend. |
Sources
- Building effective agents — Anthropic, 2024blog
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper