Structured Outputs: Schema-Valid JSON & Strict Tool Use from LLMs

Q: Response format or strict tool use — which do I use?

If the model's answer is the data you want (extract these fields, classify this text, generate this object), use response format (output_config.format). If the model is deciding to call a function and you need its arguments to be valid, use strict tool use (strict: true). When an agent both acts and reports, use both.

Q: Does a structured output mean the answer is correct?

No. It guarantees the output matches your schema's shape and types — not that the values are true. A schema-valid extraction can still pull the wrong number; a valid classification can still be mislabeled. Always add value-level validation and, for quality, evaluate the outputs.

Q: What JSON Schema features are supported?

Core types, enum, const, anyOf/allOf, $ref/$defs, additionalProperties: false, and string formats (date-time, email, uri, uuid, etc.) are widely supported. Numeric ranges, string-length limits, complex array constraints, and recursive schemas are often not enforced by constrained decoding — SDKs may strip and validate them client-side. Check your provider's docs and don't rely on unsupported constraints for hard guarantees.

Q: Why did I get truncated or invalid JSON?

Almost always max_tokens — the model hit the output cap mid-object (stop_reason: "max_tokens"). Raise max_tokens, or stream and check the stop reason. A refusal stop reason can also break conformance. Check stop_reason before parsing.

Structured Outputs

Structured outputs constrain an LLM's response to match a schema you define — guaranteeing valid, parseable data instead of prose you have to scrape. When your application needs the model to return a category, extract fields into a database, or produce arguments for a function, structured outputs are how you make the response programmatically reliable rather than "usually JSON, until it isn't."

Before structured outputs, developers coaxed JSON out of models with prompt tricks ("respond only with JSON") and prayed — then wrote defensive parsers for the times the model added a "Here's your JSON:" preamble, a trailing comment, or a hallucinated field. Constrained decoding replaced hope with a guarantee: the output will conform to the schema.

TL;DR

Structured outputs make the model return data that provably matches a JSON schema — no scraping, no fragile parsing.
Two mechanisms: response format (output_config.format) constrains the whole reply to a schema; strict tool use (strict: true) guarantees valid tool arguments.
Use response format when the model's answer is data; use strict tool use when the model is calling a function.
Design tight schemas: additionalProperties: false, explicit required, enum for fixed choices. The schema is your validation contract.
Structured outputs guarantee shape, not truth — a schema-valid answer can still be wrong. Validate values, not just structure.
Watch the edge cases: max_tokens truncation produces incomplete JSON, and a refusal can break schema conformance.

Quick Example

Extract structured data from unstructured text with a response-format schema — the model's reply is guaranteed to be JSON matching it:

The first text block is guaranteed valid JSON conforming to the schema — no preamble, no stray fields.

Core Concepts

How Constrained Decoding Works

At each generation step a model produces a probability distribution over next tokens. Constrained decoding masks that distribution to only tokens that keep the output valid against the schema — so the model literally cannot emit an unclosed brace, a missing required field, or a value outside an enum. The guarantee is structural and enforced at the decoding layer, not coaxed via the prompt.

Two Mechanisms, Two Jobs

They can be combined: an agent can be forced to call search_flights with valid arguments (strict tool use) and return a final summary in a fixed shape (response format).

Strict Tool Use

Setting strict: true on a tool definition guarantees the tool_use.input the model produces validates against the tool's input_schema exactly. The schema needs additionalProperties: false and a required list. This removes an entire class of bugs — no more defensive coercion of a passengers field that arrived as the string "two".

Designing Good Schemas

The schema is simultaneously your prompt, your validator, and your API contract. Design it deliberately:

Lock the shape. Always set additionalProperties: false so the model can't invent fields, and list every field you require in required.
Constrain values with enum and const. For categorical fields, an enum both guides the model and rejects invalid values structurally. "status": {"enum": ["open", "closed", "pending"]} beats a free-text status.
Use string formats. date-time, date, email, uri, uuid, and others are supported and push the model toward well-formed values.
Describe fields. A description on each property is prompt engineering — it tells the model what to put there. {"type": "string", "description": "ISO 3166 country code, e.g. US"}.
Know the limits. Numeric ranges (minimum/maximum), string lengths (minLength/maxLength), and recursive schemas are typically not enforced by constrained decoding — the SDK may strip them and validate client-side, or they're simply ignored. Don't rely on them for hard guarantees; validate those in code.
Keep it flat where you can. Deeply nested schemas are harder for the model and slower to compile. Flatten when the data model allows.

💡 Tip: A new schema incurs a one-time compilation cost on its first request; subsequent requests with the same schema are cached. Keep schemas stable to benefit.

Best Practices

Prefer SDK Parse Helpers

SDKs offer helpers (Pydantic in Python via messages.parse(), Zod in TypeScript) that generate the schema from a typed model and parse the response into a typed object — you get schema generation and validation in one step, with editor autocomplete on the result. Use them over hand-writing JSON schemas.

Validate Values, Not Just Shape

Structured outputs guarantee the response matches the schema. They do not guarantee it's correct. An extraction can return a schema-valid but wrong email; a classification can return a valid-but-mistaken label. Add semantic validation (does the email exist in your CRM? is the amount within range?) on top of structural validation.

Give the Model Room

Structured output still costs tokens. If max_tokens is too low, the model can hit the cap mid-object and return truncated, unparseable JSON with stop_reason: "max_tokens". Budget generously for large or nested objects.

Handle Refusals

If the model refuses a request for safety reasons (stop_reason: "refusal"), the output may not conform to the schema — the refusal takes precedence over the format constraint. Check stop_reason before parsing.

Reserve Schemas for When You Need Them

Constraining output slightly limits the model's expressiveness and adds compile overhead. For open-ended generation (writing, reasoning, explanation) let the model produce prose. Use structured outputs specifically where a machine consumes the result.

Common Mistakes

Parsing Without Checking the Stop Reason

Leaving the Schema Open

Prompt-Coaxing Instead of Constraining

Asking "respond only with JSON" in the prompt and hoping is the old, fragile way. If a machine consumes the output, use a schema constraint — the guarantee is structural, not a polite request the model might ignore.

FAQ

Response format or strict tool use — which do I use?

If the model's answer is the data you want (extract these fields, classify this text, generate this object), use response format (output_config.format). If the model is deciding to call a function and you need its arguments to be valid, use strict tool use (strict: true). When an agent both acts and reports, use both.

Does a structured output mean the answer is correct?

No. It guarantees the output matches your schema's shape and types — not that the values are true. A schema-valid extraction can still pull the wrong number; a valid classification can still be mislabeled. Always add value-level validation and, for quality, evaluate the outputs.

What JSON Schema features are supported?

Core types, enum, const, anyOf/allOf, $ref/$defs, additionalProperties: false, and string formats (date-time, email, uri, uuid, etc.) are widely supported. Numeric ranges, string-length limits, complex array constraints, and recursive schemas are often not enforced by constrained decoding — SDKs may strip and validate them client-side. Check your provider's docs and don't rely on unsupported constraints for hard guarantees.

Why did I get truncated or invalid JSON?

Almost always max_tokens — the model hit the output cap mid-object (stop_reason: "max_tokens"). Raise max_tokens, or stream and check the stop reason. A refusal stop reason can also break conformance. Check stop_reason before parsing.

Can I use structured outputs with streaming and tools?

Yes — structured outputs work with streaming, tool use, and extended thinking. They're incompatible with a few features (notably citations on some providers). Consult your provider's compatibility notes.