Designing Output Schemas That Models Actually Follow

TL;DR: Output schema prompting is the design work underneath any structured-output feature: field names the model can't misread, nesting only where fields genuinely belong together, enums where the answer space is closed, and an explicit rule for when a field is allowed to come back empty. Get those four wrong and no amount of API-level enforcement fixes the data you get back.

What does "designing a schema" actually mean, separate from enforcing it?

Two different things get called "JSON prompting," and conflating them is why so much advice about it doesn't help. One is the enforcement layer: a parameter like response_format or output_config.format that forces a model's output to validate against a schema through constrained decoding. Our structured outputs guide covers that layer: what OpenAI, Anthropic, and Google each guarantee, and where they diverge on refusals and size limits.

This post is the other half: the schema itself. Enforcement guarantees shape, meaning the right keys and the right types, nothing extra. It says nothing about whether priority: "high" is the correct priority, whether a nested object should have been flat, or whether the model just picked the closest wrong enum value because none of your options actually fit the input. A perfectly enforced schema built with vague field names and no escape hatches still produces confidently wrong data. It's just wrong in a shape your code can parse without complaining. If you haven't already, our guide to JSON prompts covers the base technique of asking for structured output at all, and our guide to choosing an output format covers whether JSON is even the right shape for the job before you get this far; this page assumes you're past both and dealing with a schema that validates but still comes back wrong.

Most of what follows applies whether or not you're using a vendor's enforcement mode at all. A model asked to fill a badly designed schema in a plain chat window fails for the same reasons a strict: true request does. Enforcement just makes the failure easier to see, because it removes "the output wasn't even valid JSON" as an excuse and leaves you looking straight at a wrong value sitting in a perfectly formed field.

Why does a model fill some fields correctly and others reliably wrong?

The field name and its description are most of the signal a model has for what belongs in a slot. A field literally named status or data gives it almost nothing to work with beyond context, and under ambiguity a model doesn't fail randomly: it fails consistently, in whatever direction the surrounding text nudges it. That consistency is actually useful, because it means the fix is nearly always in the name and description, not in some deeper model limitation you can't touch.

OpenAI's own structured-outputs guide states this as the first piece of design advice it gives, not a footnote: "To maximize the quality of model generations, we recommend the following", and the first two items on that list are "Name keys clearly and intuitively" and "Create clear titles and descriptions for important keys in your structure" (developers.openai.com/api/docs/guides/structured-outputs, fetched Sept 5, 2026). That's the whole trick, stated plainly by the vendor whose own enforcement mode you're probably using: a field named ticket_priority with a description spelling out the exact allowed meanings needs almost no inference from the model. A field named priority with no description needs the model to guess what scale you mean, and it will guess a different scale on a different run.

The same principle applies to naming collisions across nested levels. Two fields both called id, one for a customer and one for an order, force the model to track which id means what purely from position in the tree. Prefix them (customer_id, order_id) even when the nesting technically disambiguates them for your parser. It doesn't disambiguate them for the model doing the filling, and the model is the part of this system you can't unit-test in advance.

Should a schema be flat, or should related fields nest?

Nest only when fields genuinely belong to a sub-object the model needs to reason about as one unit: an address, a line item that can repeat, a pair of coordinates. Flatten everything else, even when a nested shape looks tidier on paper or matches your database's table structure more closely.

The reason is debuggability, not aesthetics. When a flat field comes back wrong, you know exactly which one, and you can fix its name or description in isolation without touching anything else. When a nested object comes back wrong, you often can't tell whether the model skipped the whole object, filled it with plausible-looking defaults because it didn't understand the sub-schema, or got one field right and two wrong: the failure is smeared across everything inside that node, and your validator usually just reports that the object failed rather than pointing at the one field that actually broke. A schema three levels deep multiplies that ambiguity at every level, which is part of why the size and nesting ceilings on enforcement APIs exist in the first place. Design-wise, treat depth as a cost you pay only where the grouping earns it, never as a default you reach for because it looks more "correct" on the page.

When should a field be an enum instead of free text?

Constrain a field to an enum the moment the set of valid answers is closed and you can actually enumerate it: status codes, categories, severity tiers, yes/no/unknown. An enum kills synonym drift, where the same real-world answer comes back as "urgent" in one run, "high priority" in the next, and "critical" in a third, all meaning the same thing to a human and three different things to whatever code reads that field downstream. Free text stays right for content that's genuinely open: a person's name, a summary, a quoted excerpt. Forcing an open-ended field into an enum just relocates the ambiguity into a list that's never quite complete.

That incompleteness is the trap worth planning for before it bites you. A schema-constrained model doesn't have the option to say "none of these fit" unless you gave it one. OpenAI's own docs are direct about the failure mode: "The model will always try to adhere to the provided schema, which can result in hallucinations if the input is completely unrelated to the schema" (developers.openai.com/api/docs/guides/structured-outputs, fetched Sept 5, 2026).

How do you handle a field that's genuinely optional?

Treat "optional" as something you design for, not something the schema format gives you for free. Several enforcement modes don't support true field omission at all. Under OpenAI's Structured Outputs, every property has to appear in required, full stop, so there's no native concept of "skip this key entirely." Anthropic's mode is more permissive here; the structured outputs guide linked above has the exact per-vendor mechanics.

The portable pattern, regardless of which API you're on, is a nullable type plus an explicit rule: define the field as accepting either its real type or null, and spell out in the prompt exactly when it should be null versus filled. "Leave discount_code null if the message doesn't mention one" is unambiguous, in a way that leaving the decision to pure inference isn't. Hoping the model figures out on its own when a field doesn't apply is a much less reliable ask than having it fill an always-present field correctly, because now it's making a meta-decision about the schema itself instead of just extracting a value it can see sitting in the input.

A worked before-and-after

Take a support-ticket extractor. Here's a schema built the way most first drafts look: short field names, one deep nesting level, no descriptions, no escape hatch.

{
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "info": {
      "type": "object",
      "properties": {
        "type": { "type": "string" },
        "level": { "type": "string" }
      }
    },
    "notes": { "type": "string" }
  },
  "required": ["id", "info", "notes"]
}

Every field name here needs outside context to mean anything, info.type and info.level are free text standing in for what should be closed sets, and there's no field for "this ticket doesn't clearly fit any category." Redesigned with the principles above:

{
  "type": "object",
  "properties": {
    "ticket_id": { "type": "string", "description": "The ticket or case number as written in the source text." },
    "category": {
      "type": ["string", "null"],
      "enum": ["billing", "technical", "account_access", "other", null],
      "description": "Closest matching category. Use null only if the ticket text gives no basis to classify it."
    },
    "priority": {
      "type": "string",
      "enum": ["low", "medium", "high", "urgent"],
      "description": "Stated or clearly implied urgency. Default to 'medium' if unstated."
    },
    "summary": { "type": "string", "description": "One-sentence plain-language summary of the customer's issue." }
  },
  "required": ["ticket_id", "category", "priority", "summary"]
}

Nothing here is nested that doesn't need to be, every enum has a documented default or an escape value, and a human reading the schema alone, with no accompanying prompt, can tell what each field is for. That last property matters more than it looks: if a field's job isn't obvious from its name and description, it isn't obvious to the model either.

The four design choices behind that rewrite generalize past this one example:

How do you fix a schema that keeps coming back wrong?

Run it as a loop, not a one-shot rewrite. Generate against the schema, then validate the values, not just the shape. A JSON Schema validator confirms priority is a string from your enum; it says nothing about whether "high" was actually correct for that particular ticket. That second check has to come from a human reviewing a sample, or from an eval you build against known-correct answers. OpenAI's own guidance points at exactly this gap: "Structured Outputs can still contain mistakes. If you see mistakes, try adjusting your instructions, providing examples in the system instructions, or splitting tasks into simpler subtasks" (same source, fetched Sept 5, 2026). Schema enforcement was never claimed to fix correctness, only shape.

When you find a recurring wrong value, don't respond by re-explaining the field's rule more thoroughly. Feed back the specific failure: the exact wrong value the model produced, the input that caused it, and what the right value should have been. A concrete corrected example moves a model further per token than a longer abstract instruction does, and it's diagnostic in a way a vague "try to be more accurate" note isn't. If the same category of error keeps showing up against different concrete examples, that's a sign the schema itself needs a field renamed or an enum value added, not that the prompt needs to be sterner.

Run the fix against a representative batch of real inputs before calling it done, not one lucky retry. A schema that now handles the single ticket you were staring at can still fail on the next twenty, especially when the fix was shaped narrowly around that one failure. Keep a running set of inputs that have broken the schema before, and re-check the whole set every time you change a field name, an enum, or a description. A schema that stops regressing on old failures while it's being extended for new ones is the real sign the design has converged, not just that today's test happened to pass.

One more habit worth adopting even outside the retry loop: generate your JSON Schema from the same type definitions your code already validates against, a Pydantic model, a Zod schema, rather than hand-writing the schema separately. OpenAI's docs flag this directly: "To prevent your JSON Schema and corresponding types in your programming language from diverging, we strongly recommend using the native Pydantic/zod sdk support" (fetched Sept 5, 2026). A schema and a validator that started life as two separate hand-typed documents drift apart quietly, and the failure you end up debugging is sometimes just that mismatch, not the model at all.