Intent classification & routing
Labelling each incoming query by what kind of request it is, so it can be routed to the right pipeline: RAG, SQL, tool call, chit-chat or refusal.
Not every question should go through the same steps. 'What's our parental leave policy?' needs document search; 'how many tickets did we close last week?' needs a database query; 'thanks!' needs no search at all. Intent classification decides which kind of request a message is, so the system can route it to the right path — and decline ones it shouldn't answer.
A hospital triage nurse who decides whether you need the pharmacy, a GP, or the emergency room before anyone treats you.
Optimising overall accuracy. 95% accuracy can hide that every 'how many/how much' question goes to RAG and gets a hallucinated number. Weight the confusion matrix by the cost of each mistake.
A router sits in front of retrieval with 4–12 intents (e.g. doc_qa, analytics → Text-to-SQL, action → tool use, compare → Query decomposition, smalltalk, out_of_scope → Abstention & relevance thresholds). Options by cost: rules/regex (< 1 ms, brittle); embedding + nearest-centroid or logistic regression on 200–500 labelled examples per intent (2–10 ms, 90–95% accuracy on clear intents); fine-tuned small encoder like DeBERTa-v3 (5–15 ms on CPU); or an LLM with a JSON-schema output (150–500 ms, best on long-tail phrasing, easiest to change). Adaptive-RAG (Jeong et al. 2024) routes by question complexity (no retrieval / single-step / multi-step) and cut cost while keeping accuracy. Always include an unknown class with a confidence threshold: low-confidence queries take the safest general path (hybrid RAG) rather than a guess. Routing errors are asymmetric — sending an analytics question to RAG gives a plausible-sounding wrong number, which is worse than a slow correct path.
Embedding + logistic-regression classifier (200–500 examples/intent) with a confidence threshold of 0.7; LLM (small model, JSON schema) for the low-confidence tail; unknown falls back to hybrid RAG; all routing decisions logged with confidence.
Internal analytics + wiki assistant, 40k queries/day. Embedding classifier (bge-small, logistic regression, 2,400 labelled examples) at 4 ms: 93% accuracy overall, but 78% on analytics vs doc_qa confusions like 'what was churn last quarter?' (a metric) vs 'how is churn defined?' (a doc). Adding an LLM fallback only when classifier confidence < 0.7 (14% of traffic, +300 ms) raised analytics accuracy to 94%.
Recommend a cascade: "Cheap classifier first, LLM only for the low-confidence tail, an explicit unknown class that falls back to the safest path." Then measure the confusion matrix, not just accuracy — the costly errors are specific pairs. Expect: how do you add a new intent? Label 200+ examples, retrain, shadow it before routing traffic.
| → doc_qa | → analytics | → action | → out_of_scope | |
|---|---|---|---|---|
| doc_qa | 0.95 | 0.03 | 0.01 | 0.01 |
| analytics | 0.18 | 0.78 | 0.02 | 0.02 |
| action | 0.04 | 0.01 | 0.93 | 0.02 |
| out_of_scope | 0.12 | 0.01 | 0.01 | 0.86 |
Numbers worth knowing
- Embedding classifier latency
- 2–10 mssmall encoder on CPU + linear head
- LLM router latency
- 150–500 mssmall hosted model, ~400-token prompt, JSON output
- Labelled data needed
- 200–500 examples per intentfor ≥ 90% accuracy on distinct intents
How to decide
| When | Do this |
|---|---|
| ≤ 6 well-separated intents and labelled examples | Embedding classifier; no LLM on the hot path. |
| Long-tail phrasing or intents change weekly | LLM router with schema output, cached by normalised query. |
| Classifier confidence < threshold | Route to the safest general path, never to an action/tool path. |
Cost maths
Cascade cost = all × cheap + tail × LLM. 1M queries/month: embedding classifier ≈ fixed CPU (~$50/month); LLM on the 15% tail at ~500 tokens × $0.25/M + 30 tokens × $1.25/M ≈ $0.00016 → ~$24/month. LLM-for-everything ≈ $160/month and +300 ms on every query.
Operating it
- Low-confidence share > 25% — label more data or merge overlapping intents
- A new product surface ships — add an intent and shadow it for a week
- Specific confusion pair > 10% — add contrastive examples for that pair
- Per-pair confusion rate on a weekly labelled sample — alert if analytics→doc_qa > 5%
- Low-confidence share — alert above 20%
- Intent distribution drift vs last 4 weeks — alert on ±30% for any intent
When it goes wrong
Finance users received confident but wrong revenue figures from the assistant.
Spot-check by an analyst; tracing showed 'what did we bill Acme-segment customers in Q2' was routed to document RAG, which summarised a stale board deck.
Classifier trained mostly on 'how many/what is the total' phrasings for analytics; 'what did we bill' fell to doc_qa at 0.62 confidence, above a too-low threshold of 0.5.
Raised threshold to 0.7 with LLM fallback, added 300 billing/revenue phrasings, and made numeric answers from doc_qa carry a 'from document dated X' caveat.
Related terms
Where it shows up
Sources
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity — Jeong et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.