How do you get reliable structured output from an LLM?
- 2 min — three levels of strength and how constrained decoding works
- 2 min — validation layer with business rules
- 2 min — metrics and failure routing
- 1 min — schema design rules
Three strengths. Prompting for JSON: parses 95–99% of the time on strong models; fails on extra prose, truncation or a stray comma. Provider structured outputs (e.g. OpenAI strict: true JSON schema, tool-use input schemas): decoding constrained to the schema, so it parses unless truncated by max_tokens. Constrained decoding in self-hosted servers (vLLM / SGLang with XGrammar, Outlines or llguidance): at each step, tokens that would violate the grammar are masked out — syntactic validity is guaranteed. Valid shape is not correct content: values can still be wrong.
An invoice-extraction service for a European AP team needs vendor, invoice_date, currency (ISO 4217), total, vat_amount and line items. Schema-constrained output makes parse failures ~0. Then business validation: line items sum to total within 0.01, VAT rate is one of the country's valid rates, date not in the future. On 2,000 real invoices: 0% parse failures, 3.8% validation failures — those route to human review instead of the ledger.
- Keep schemas flat and small; describe every field; avoid 200-value enums (route to a lookup instead). Deep nesting measurably hurts accuracy.
- Hard constraints can degrade reasoning: put a free-text
reasoningfield first in the schema so the model thinks before committing values. - Track three separate rates: parse failure, validation failure, field-level accuracy — each has a different fix.
- Tool calling is the same mechanism: the tool schema is a structured-output contract.
- Org scope: a shared schema/validator library (Pydantic or Zod) owned by the platform team, reused across extraction products; build the validators, buy the constrained decoding.
- Where it breaks: treating schema validity as success. A perfectly shaped JSON with a hallucinated total is the dangerous failure because nothing downstream flags it.
What the answer sounds like at each level
Uses JSON mode or a schema and parses the result with retries.
Knows constrained decoding masks invalid tokens, separates parse from validation failures, and adds business-rule checks.
Standardises schemas and validators across teams, tracks three failure rates, and designs schemas that preserve reasoning quality.
Follow-ups the interviewer will push
1Output is valid JSON but truncated. Why and what do you do?
Hit max_tokens mid-object; constrained decoding cannot fix length. Size max_tokens from p99 output length plus margin, detect stop_reason = max_tokens, and retry with a higher cap or paginate line items.
2Now 10× volume. Is retry-on-failure still OK?
At 3.8% validation failure, retries add ~4% cost — fine. But retries rarely fix semantic errors; route those to review and track the review queue SLA instead.
Red flags interviewers listen for
- Parses with regex and hopes.
- Equates 'valid JSON' with 'correct data'.
- Puts the reasoning field after the answer fields.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Explains token masking against a grammar and its limits (truncation, semantics). |
| Validation | Domain-specific checks after parsing, with a human route. |
| Metrics | Separate parse, validation and field accuracy rates. |
Sources
- Efficient Guided Generation for Large Language Models — Willard & Louf, 2023paper