Human in the loop
A person reviews, approves or corrects an AI's work at chosen points — usually before irreversible or high-stakes actions.
Human-in-the-loop means a person checks the AI's work at important moments instead of letting it run on its own. An AI agent might draft an email but wait for you to click Send, or propose a payment that a manager must approve.
The skill is putting the checkpoint in the right place: too many and people click "approve" without reading; too few and mistakes reach customers.
A junior employee who can prepare anything but needs a signature for payments over a set amount.
Gating every action. Reviewers drown, start rubber-stamping, and the check becomes theatre that adds latency without safety.
Patterns: approval gates (the The agent loop (ReAct) pauses before a flagged tool call; the human approves, edits the arguments or rejects), review queues (outputs sampled or routed by confidence or risk), escalation via Handoffs between agents when the agent is stuck, and post-hoc audit. Gate on consequences — irreversible, external, financial, legal — not on every action. LangGraph implements gates with interrupt() plus a checkpointer; other runtimes use durable signals.
Engineering requirements: the run must persist state while waiting, possibly for days (Durable execution & checkpointing); the request must show what's needed to decide in seconds (exact action, arguments, diff, why the agent wants it, what happens if rejected); and each decision is logged as a label for Human annotation and the Data flywheel. Watch for rubber-stamping: ~100% approvals with median review times under a few seconds mean the gate is either unnecessary or not being read. The Replit incident (July 2025) shows the opposite failure: an approval rule written only in chat is not a gate.
Risk rules in code (amount, irreversibility, external recipient, changed bank details) select the gated calls; state checkpointed before the pause; approval UI shows action, arguments, evidence and the agent's reason; approve/edit/reject with a required reason on reject; SLA timer that escalates; every decision stored as a label.
Accounts-payable agent matching 500 invoices a day. Payments over $10,000 or to a supplier whose bank details changed in the last 30 days pause for review — about 6% of volume (30 a day). The reviewer sees invoice, PO, receipt and the proposed payment side by side; median decision 20 s; 4% rejected, almost all bank-detail changes that turned out to be fraud attempts.
Recommend gating by consequence and say how you'll know the gate works: approval rate, review time, and what rejected items had in common. "Put humans where mistakes are expensive and irreversible, not merely possible."
Expect: "reviewers are a bottleneck" — tighten the risk rule with data from past decisions, auto-approve the classes humans never reject, and keep the irreversible ones gated.
Numbers worth knowing
- Healthy gated share
- 2–10% of actionsrisk-based gating; >30% means the rule is too broad
- Rubber-stamp signal
- >98% approve with median review < 5 ssustained over 2+ weeks
How to decide
| When | Do this |
|---|---|
| The action is irreversible, external or above a $ threshold | Hard approval gate with durable pause. |
| The action is reversible and low-value | Execute, sample 1–5% for post-hoc review. |
| Reviewers never reject a class of actions over 1,000+ decisions | Auto-approve that class; keep auditing a sample. |
Cost maths
Review cost = gated actions × review time × loaded labour rate. 500 invoices/day × 6% = 30 reviews × 20 s = 10 min/day; at $60/hour loaded (assumption) ≈ $10/day, ≈ $220/month. One prevented fraudulent payment of $14k pays for ≈ 5 years of review. Gating 100% would be 2.8 h/day ≈ $3.7k/month — and reviewers would stop reading.
Operating it
- Approval queue wait p95 exceeds the business SLA
- Gated share drifts above 15%
- Rejection rate falls below 1% for a whole action class
- Approval rate and median review time per action class (alert on >98% and < 5 s)
- Queue wait p95 vs SLA (alert at 80% of SLA)
- Gated share of actions (alert > 15%)
- Gated actions executed without a decision record (alert on any)
When it goes wrong
An accounts-payable agent paid $180k across three invoices to a fraudster's account despite a human-approval step.
The real supplier chased payment two weeks later.
The approval card showed amount and supplier name but not that bank details had changed the day before; 99.6% approval rate and 3-second median reviews showed reviewers were rubber-stamping.
Bank-detail change became its own gate with a call-back requirement; the card highlights changed fields; non-risky invoices auto-approve so reviewers see fewer, harder cases.
Related terms
Where it shows up
Sources
- Interrupts — LangChain (LangGraph docs), 2025docs
- Building effective agents — Anthropic (Schluntz & Zhang), 2024blog
- Replit AI coding tool deletes live production database — OECD.AI Incidents Monitor, 2025postmortem
Check yourself
Three questions. Every option is explained once you answer.