Streaming Responses (And When Not To)

TL;DR: A streaming LLM API sends the model's output as a sequence of small events instead of one blocking response, which lowers the time to the first visible token but not the total generation time. It complicates schema validation, retries, and error handling, and a reasoning model can stream silently for a long time before returning nothing at all. Turn it off for structured extraction, moderation-sensitive output, and unattended background jobs.

What does a streaming LLM API actually change?

It changes when you see output, not how much work the model does. Without streaming, the server buffers the entire generation and returns it in one HTTP response once the model is done. With streaming, the server opens the connection and starts sending typed events as soon as the first tokens exist, and the client assembles them as they arrive.

That distinction matters because the two numbers people conflate, time to first token and total generation time, move independently. Streaming can cut the first number dramatically, since a user watching a chat window sees text appear almost immediately instead of staring at a spinner. It does close to nothing for the second number. The model still has to generate the same number of tokens at the same underlying speed; framing each chunk as its own event adds a small amount of overhead rather than removing any.

OpenAI states the trade-off directly in its own streaming guide: "By default, when you make a request to the OpenAI API, we generate the model’s entire output before sending it back in a single HTTP response. When generating long outputs, waiting for a response can take time. Streaming responses lets you start printing or processing the beginning of the model’s output while it continues generating the full response" (developers.openai.com/api/docs/guides/streaming-responses, accessed September 3, 2026). Read carefully, that's a claim about perceived latency, not about the model finishing faster.

What does the raw stream actually look like on the wire?

Underneath every vendor's SDK, streaming responses are server-sent events, or SSE: a normal HTTP response that the server keeps open, sending discrete data: lines instead of closing the connection after one payload. OpenAI's Chat Completions reference describes its own stream parameter this way: "Whether to stream back partial progress." Tokens are sent as data-only server-sent events as they become available, and, in the spec's exact wording, "with the stream terminated by a data: [DONE] message" (raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml, master branch, accessed September 3, 2026).

Anthropic's Messages API frames the same idea with named events rather than a single generic delta. Its streaming reference shows the shape directly:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_1nZ...", "type": "message", "role": "assistant", "content": [], "model": "claude-opus-5", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}}

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}}

event: message_stop
data: {"type": "message_stop"}

(platform.claude.com/docs/en/build-with-claude/streaming, accessed September 3, 2026.) Every SDK on the market wraps a version of this loop. If you ever have to debug streaming without the SDK, this is the actual shape you are parsing.

What event types do OpenAI and Anthropic actually emit?

The two vendors solve the same problem with different granularities, and the difference matters if you are handling errors or tool calls mid-stream rather than just printing text.

OpenAI's own guide is explicit that the event set is intentional, not incidental: "The Responses API uses semantic events for streaming. Each event is typed with a predefined schema, so you can listen for events you care about" (developers.openai.com/api/docs/guides/streaming-responses, accessed September 3, 2026). If you are building against either API directly, read the event list before you write the parsing loop. Guessing the shape from a blog post, including this one, is how a client silently drops the first response.created event and starts its UI a beat late.

When does streaming genuinely earn its complexity?

Streaming is worth the extra client-side work in exactly one shape of product: a human is watching, in real time, and the answer is long enough that the wait would otherwise feel dead. A chat interface is the obvious case. So is a live code-completion panel, or anything narrating a multi-step agent process where a user benefits from seeing progress rather than a spinner.

It is close to worthless anywhere the output is consumed by a program rather than a person. A server action that calls a model, gets a JSON object back, and writes it to a database gains nothing from streaming, because nothing downstream reads a partial value. The added surface area, an open connection, a parsing loop, a state machine for partial content, buys latency that no one is there to perceive.

Why do streaming and strict structured output fight each other?

Because a syntactically incomplete JSON document is not valid JSON, and a schema check has nothing to run against until the closing brace arrives. You can absolutely stream the bytes of a structured response: OpenAI's structured-outputs guide describes its SDK stream helpers this way: "You can use streaming to process model responses or function call arguments as they are being generated, and parse them as structured data." The same guide adds, "That way, you don’t have to wait for the entire response to complete before handling it" (developers.openai.com/api/docs/guides/structured-outputs, accessed September 3, 2026). What you cannot do is validate the object against its json_schema before it is complete, because the schema check is a well-formedness question and a half-written array is not well-formed by definition.

In practice this means the SDK gives you two different things that look similar: a best-effort parsed snapshot, useful for a progress UI, and a final, schema-checked completion, which only exists once the stream ends. OpenAI's guidance leans into that split directly: "We recommend relying on the SDKs to handle streaming with Structured Outputs" (same page, accessed September 3, 2026), which is a polite way of saying that hand-rolled incremental JSON parsing against a partial document is exactly the kind of code you do not want to own.

There is a second, separate reason to avoid streaming a structured response: moderation. OpenAI states this directly, in the context of production apps that need to check content before showing it to a user: "streaming the model’s output in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate." On the same page, moderation scores themselves "arrive after the full generated output is available. They aren’t included with partial output deltas" (developers.openai.com/api/docs/guides/streaming-responses, accessed September 3, 2026). If your product has to check output before a user sees it, streaming does not remove that requirement; it just makes it awkward to enforce mid-stream.

What happens when a reasoning model streams?

Something genuinely different from a non-reasoning model, and it is the sharpest argument in this whole post for turning streaming off. On both major vendors, the tokens a model spends reasoning before it writes a visible answer count against the same output ceiling as the visible text.

OpenAI is explicit: you can "limit the total number of tokens the model generates, including reasoning tokens, visible output tokens, and non-visible formatting tokens" with max_output_tokens. Its guide continues: "If the generated tokens reach the context window limit or the max_output_tokens value you’ve set, you’ll receive a response with a status of incomplete and incomplete_details with reason set to max_output_tokens. This might occur before any visible output tokens are produced, meaning you could incur costs for input and reasoning tokens without receiving a visible response" (developers.openai.com/api/docs/guides/reasoning, accessed September 3, 2026).

Anthropic documents the same shape of constraint for extended thinking: "Thinking tokens count toward the max_tokens limit for the turn, so the budget must leave room for the final response" (platform.claude.com/docs/en/build-with-claude/extended-thinking, accessed September 3, 2026).

Streaming does not fix this. It just changes the failure from a slow single response into a stream that goes quiet and then closes, which is arguably worse, because a quiet stream looks a lot like a stream that is about to produce something.

Why is retrying a stream harder than retrying a single response?

Because a single response either arrives or it doesn't, and a naive retry just repeats the whole call. A stream can fail partway through, after already sending half a coherent-looking answer, with no standard mechanism across vendors for resuming from where it stopped. Your options are to discard everything received so far and retry from scratch, wasting the tokens already streamed past, or build your own logic for detecting a partial answer and asking the model to continue, an app-specific state machine a single blocking call never required.

This gets worse once retries for rate limits or transient errors enter the picture. A background job that calls an LLM and retries on failure is simple to write correctly: call, check status, retry the whole call if it failed. The same job built on a stream must decide, on every reconnect, whether it is starting fresh or resuming, and whether tokens already billed for the failed attempt should be assumed lost.

Can a truncated stream look like a finished answer?

Yes, and this is the failure mode that is easiest to ship without noticing. A single blocking response either comes back complete or the request fails outright, which is a clear signal either way. A stream that is cut off, by a hit max_output_tokens, a dropped connection, or a server-side timeout, can still have delivered several well-formed sentences before it stopped. Read on a screen, an answer that stops mid-list or mid-paragraph often still reads like a complete thought, especially if the model happened to end on a period.

The fix is mechanical, not visual: check the terminal status the API actually sends, rather than trusting that the text looks done. OpenAI's Responses object carries an explicit status field that can read incomplete with an incomplete_details.reason, and Anthropic's final message_delta event carries a stop_reason you can inspect the same way. A client that only watches for the last text chunk and never checks the terminal event is trusting layout, not data.

Is the stop parameter supported on every reasoning model?

No, and this is a genuinely useful example of a fact that lives in one place and nowhere else. OpenAI's OpenAPI specification defines the stop parameter's constraints under a schema called StopConfiguration, and its description reads, verbatim: "Not supported with latest reasoning models o3 and o4-mini. Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence" (raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml, master branch, accessed September 3, 2026).

That sentence does not appear anywhere in OpenAI's prose reasoning guide. A search of that guide's own rendered text for the word "stop" returns zero results. So a developer reading only the human-facing documentation for reasoning models has no way to discover the gap; the constraint exists exclusively in the machine-readable spec that tools like SDK generators consume.

Streaming vs batch: which should you use?

A short decision list, in order of what to check first:

1. Is a person watching the response arrive, in real time? If yes, and the answer is more than a sentence or two, stream it.
2. Does anything downstream need a complete, schema-checked object before it can act? If yes, treat streaming as optional plumbing at most: the parsing and validation still happen once, at the end.
3. Is the model a reasoning model? If yes, budget max_output_tokens generously and check the terminal status field explicitly. Streaming will not warn you about an incomplete result any more clearly than a single response would.
4. Does the output need to clear a moderation or safety check before a user sees it? If yes, do not stream it to the user directly; generate, check, then release.
5. Is this a background job with no UI at all? If yes, use a single blocking call. There is no perceived-latency benefit for streaming to buy, and you remove an entire class of partial-failure bugs for free.

None of this argues against streaming in general. A chat product without it feels broken today, and for good reason: the perceived-latency win is real. The argument is narrower: stream when a human benefits from watching tokens arrive, and reach for a single call, structured output, and an explicit status check everywhere else. If you're writing the prompt behind the structured side of that split, see our guide to JSON prompts and the free JSON prompt generator. For the sampling controls shaping what comes out of either path, top-p vs top-k and RAG vs fine-tuning vs prompting cover the decisions next to this one, and few-shot vs zero-shot prompting covers the input side of the same request.