Structured output / decoding
Constraining generation so output always matches a schema or grammar (e.g. valid JSON) by masking tokens that would break it.
Normally an LLM can write anything, which is a problem when your code expects exact JSON with specific fields. Structured decoding forces the model to only produce output that fits a format you define, so the result always parses.
APIs call this 'structured outputs' or 'JSON schema mode', and tool calls rely on the same idea.
A form with fixed boxes and drop-downs instead of a blank page: the writer can only fill in valid values.
Assuming schema-valid output is correct output. The format is guaranteed; the content is not.
At each decode step a grammar engine, compiled from a JSON Schema, regex or context-free grammar, computes which vocabulary tokens are valid given the output so far; all other Logits & probabilities are set to −∞ before sampling. Outlines (Willard & Louf, 2023) framed this as a finite-state machine over the tokenizer vocabulary; XGrammar (2024) precomputes most of the mask for pushdown automata so per-token overhead is negligible, and is a backend in vLLM and SGLang. Hosted equivalents: OpenAI structured outputs with strict: true, and JSON-schema tool definitions on the major providers.
It guarantees syntax, not semantics: an enum field will hold a valid value, possibly the wrong one. Tight formats can cost reasoning accuracy (Tam et al., 2024, 'Let Me Speak Freely?'), so let the model reason in free text or a leading reasoning field, then constrain the final object. Complex schemas (deep recursion, large anyOf) compile slowly on first use, so cache compiled grammars. Always validate business rules after parsing; this underpins tool calling, Text-to-SQL, extraction and LLM-as-judge verdicts.
Hosted: strict structured outputs or JSON-schema tools. Self-hosted: vLLM structured outputs with the XGrammar backend. Schemas kept flat with enums for closed sets; a leading reasoning string field when the task needs thought; Pydantic or Zod validation of business rules after parse; grammars compiled once and cached per schema version.
Clinical intake extraction with schema { triage: enum[urgent, routine, self_care], allergies: string[], current_meds: [{ name, dose_mg: number, frequency }] }. On 10,000 notes, free-form JSON prompting produced 1.8% unparseable outputs; strict decoding produced 0. A business-rule check still caught 2.8% semantic errors, mostly doses in micrograms written into dose_mg, routed to a nurse for review.
Commit: use the provider's strict schema mode or a grammar backend for anything code parses, and still validate. Separate 'parses' from 'correct' explicitly, and say where reasoning happens relative to the constrained block. The follow-up is 'the schema changes next quarter, what breaks?': version the schema with the taxonomy it encodes and monitor the class distribution, because enums silently force new categories into 'other'.
| Step | Output so far | Allowed next | Chosen |
|---|---|---|---|
| 1 | (empty) | { | { |
| 2 | { | "triage" | "triage" |
| 3 | {"triage": | "urgent" | "routine" | "self_care" | "routine" |
| 4 | …"routine", | "allergies" | "allergies" |
| 5 | …"allergies": | [ | [ |
| 6 | …[ | string or ] | "penicillin" |
Numbers worth knowing
- Parse failures, free-form JSON
- 1–3%prompt-only JSON on 10k-item extraction runs (illustrative range)
- Parse failures, constrained
- 0%strict schema decoding, excluding truncation at max_tokens
- Per-token mask overhead
- negligible vs decode stepXGrammar-style precompiled masks; first compile of a complex schema can take 100 ms–seconds
How to decide
| When | Do this |
|---|---|
| Code parses the output | Use strict schema decoding; never rely on 'please return JSON'. |
| The task needs multi-step reasoning | Reason in free text or a leading field, then constrain the final object. |
| A category set changes over time | Version the enum with its source taxonomy and alert on 'other' share. |
| Schema uses features the backend can't compile | Flatten it; split into two calls rather than drop constraints. |
Cost maths
Cost of not constraining = failure_rate × (retry cost + latency tail). At 2% parse failures and a $0.01 call, retries add ~2% spend, but each retry doubles that request's latency, which lands in the p99. The bigger cost is semantic review: 2.8% flagged × 10,000 notes/day × $0.50 per human review = $140/day, which is the number to drive down.
Operating it
- Business-rule validation failures above 2%
- 'other' or fallback enum share up 2× from baseline
- Schema compile time appearing in p99 latency
- Enum class distribution vs 30-day baseline: alert on 2× shift in any class
- Business-rule validation failure rate: alert above 2%
stop_reason = max_tokenson structured routes: alert above 0.1% (truncated objects)
When it goes wrong
Cyber-incident claims at an insurer were routed to the general property queue for five weeks.
Monthly review found the 'other' share of the claim-type enum had risen from 4% to 19%.
A new 'cyber' product launched, but the extraction schema's enum was not updated; constrained decoding forced every cyber claim into 'other', which defaulted to property.
Enum generated from the product taxonomy service at build time, schema version logged per call, and an alert on 'other' share above 2× baseline.
Where it shows up
Sources
- Efficient Guided Generation for Large Language Models — Willard & Louf, 2023paper
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models — Dong et al., 2024paper
- Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models — Tam et al., 2024paper
- Structured model outputs — OpenAI, 2026docs
Check yourself
Three questions. Every option is explained once you answer.