Skeleton-of-Thought Prompting (Outline, Then Fill)

TL;DR: Skeleton of thought prompting means asking a model for a short outline first, then expanding each outline point, ideally in parallel rather than one after another. The 2023 research behind it (Ning et al.) measured real speed-ups above 2x on most tested models, but the gain depends on actually parallelizing the expansion step, and the paper is explicit that the technique fails on questions needing step-by-step reasoning.

What Is Skeleton-of-Thought Prompting?

Skeleton-of-thought prompting splits answering a question into two separate steps: first get a short skeleton, a numbered list of the points the answer should cover, then expand each point into full content before assembling the final response.

It comes from a specific paper: Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation (Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang), first posted to arXiv in July 2023 and accepted at ICLR 2024. The method the paper proposes guides the model to generate a skeleton of the answer first, then conducts parallel API calls or batched decoding to complete the contents of each skeleton point at the same time, rather than one after another.

That last clause is the part worth reading twice: the technique isn't really about the outline. It's about doing the expansion step in parallel instead of writing the whole answer token by token, start to finish, in one continuous pass. The outline is just what makes parallel expansion possible in the first place, since each point becomes an independent unit of work once you know what it is.

How Does the Two-Stage Process Actually Work?

The paper breaks it into exactly two stages, and the mechanics matter for understanding where the speed actually comes from.

Stage 1: Skeleton. The model receives a prompt asking it to output only a short outline, typically 3 to 10 points, each just a few words, not full sentences. This is a normal, single, sequential generation, no different from any other prompt.

Stage 2: Point-expanding. Each point from the skeleton becomes its own separate request, along the lines of expand only point 3, briefly. Crucially, the paper runs these requests in parallel, either as concurrent calls to an API-based model or as a single batched-decoding pass on a self-hosted model. The individual expansions are then concatenated in order to produce the final answer.

Stage 1 (skeleton):
Give me a short outline, 3-6 numbered points, for
answering: [your question]. Points only, no explanation yet.

Stage 2 (point-expanding — fire these concurrently, not
one after another, to get the actual speed-up):
Point 1 of this outline: [point 1 text]. Expand only this
point, in 1-2 sentences, for the question: [your question].

Point 2 of this outline: [point 2 text]. Expand only this
point, in 1-2 sentences, for the question: [your question].

[...repeat per point, run concurrently...]

How Much Faster Is It, Really?

Enough to matter for anything serving real users. Testing across 12 models released in 2023, the paper states plainly: "We test SoT on 12 recently released LLMs." Not only does SoT provide considerable speed-ups, but it can also improve answer quality in several question categories, and the paper reports a greater-than-2x speed-up, up to 2.39x, on 8 of the 12 models tested.

The paper's own worked example, measured directly rather than estimated: answering the same question took 22 seconds normally with Claude, dropping to 12 seconds with skeleton-of-thought, a 1.83x speed-up. On Vicuna-33B running locally on an NVIDIA A100 GPU, the same question took 43 seconds normally and 16 seconds with the technique, a 2.69x speed-up.

The mechanism behind the open-source-model numbers is straightforward: normal decoding generates one token at a time, sequentially, no matter how much spare GPU capacity is sitting idle. Batching several point-expansion requests together lets the GPU do that same decoding work for multiple points at once, which is closer to how the hardware wants to be used in the first place.

Can You Get the Speed-Up Just by Asking for an Outline in ChatGPT?

Not the latency win specifically, and this is the part worth being careful about. If you ask for an outline first in a normal chat window, then ask the model to expand point 1, then point 2, then point 3, each in the same conversation, you have not parallelized anything. You've just added extra round trips to a process that was already sequential, which can easily make the whole interaction slower, not faster.

The actual speed-up in the research requires firing the point-expansion requests at the same time, either as concurrent calls to an API you're building against, or as a single batched pass through a model you're running yourself. That's an engineering pattern, not a chat-window habit. If you're building a pipeline and can issue several API calls concurrently, skeleton-of-thought is a genuine, measured latency technique. If you're a single person typing into ChatGPT or Claude's web interface, the parallel-expansion machinery isn't available to you, so the honest benefit is different: a forced outline step still tends to produce a more organized, better-structured answer, which is a real but separate benefit from the one the paper's headline numbers measure.

Does Skeleton-of-Thought Work for Every Kind of Question?

No, and the paper's own limitations section is direct about where it breaks: "The current SoT is suitable for questions that require a long answer whose structure can be planned ahead, while not suitable for questions that require step-by-step reasoning or only need a short answer." The paper calls this out as the technique's single biggest weakness, stating plainly that "the biggest limitation is that SoT is not suitable for questions that require step-by-step reasoning".

That failure mode makes structural sense once you see how the technique works: point-expansion happens independently and in parallel, with each point unable to see what the others produced. A math problem, a logic puzzle, or any task where step 3 depends on the actual result of step 2 cannot be split this way without breaking the dependency the reasoning relies on. This is the opposite failure mode from chain-of-thought prompting, which exists specifically to make each reasoning step build on the last one; skeleton-of-thought works precisely when the points don't need to depend on each other. If your task genuinely needs branching, backtracking reasoning rather than either a strict chain or independent parallel points, that's closer to what tree-of-thought prompting is built for.

Does Splitting the Answer Hurt Coherence?

Sometimes, and the paper measured this rather than assuming it away. Their own answer-quality investigation reports: "As for coherence and immersion, SoT is not worse than the normal generation around 60% of the time." Read the other side of that number: on roughly 40% of the answers they evaluated for those two dimensions, the skeleton-based answer actually read worse than a normally generated one. Splitting an answer into independently written pieces and concatenating them is not free; it can leave visible seams, especially on questions where flow between points matters more than complete coverage.

That real quality cost is exactly why the paper doesn't recommend applying skeleton-of-thought blindly to every question, which is what the router variant below exists to fix.

What Is SoT-R, and Why Does It Exist?

SoT-R (Skeleton-of-Thought with Router) is the paper's answer to the two problems above: the technique fails outright on reasoning questions, and it measurably hurts coherence on some others. Rather than applying skeleton-of-thought everywhere, SoT-R adds a router step that decides, per question, whether to use it at all.

The paper tests two router designs: prompting a strong model (GPT-4) to judge whether a given question's ideal answer is "in a list of independent points" and a small, purpose-trained classifier (a 120-million-parameter RoBERTa model) fine-tuned to make the same call more cheaply. Either way, the router's job is the same: send reasoning questions and short-answer questions to normal generation, and send genuinely structurable questions, the ones where parallel points don't need to depend on each other, through the skeleton-and-expand path.

Does This Still Apply to 2026 Models?

Worth flagging plainly: the paper's numbers come from 2023-era models, tested via the specific mechanics of parallel API calls and batched local decoding available at the time, on models like GPT-4, Claude, ChatGPT, and several 7B-33B LLaMA-based open-source models. Two things about 2026 are different enough to matter.

First, several providers now offer their own server-side techniques (speculative decoding, more aggressive batching) that already attack the same slow-sequential-decoding problem the paper targets, from the provider's side rather than the prompt's side. How much additional headroom skeleton-of-thought still buys on top of infrastructure a provider already runs internally isn't something this paper measured, since that infrastructure didn't exist in its current form in 2023.

Second, the paper's core limitation, that step-by-step reasoning tasks can't be parallelized this way, is a structural fact about dependent reasoning steps, not a fact about any particular model generation. It's reasonable to expect that limitation to hold on any future model for the same reason it held in 2023: you cannot generate step 3 of a dependent chain before you know the result of step 2, regardless of how capable the model doing the generating is. The speed-up numbers are a 2023 snapshot; the shape of where the technique does and doesn't apply is closer to a permanent property of the approach.

When Should You Actually Reach for This?

Given the trade-offs above, skeleton-of-thought earns its place in a narrower set of situations than a general promise to make answers faster suggests on its own.

• You're building a pipeline, not typing into a chat window. The speed-up requires firing concurrent requests, so this belongs in code you control, not a manual habit for everyday prompting.
• The output is naturally list-shaped. Product descriptions, FAQ answers, comparison summaries, and explanations with independent facets are exactly the "long answer whose structure can be planned ahead" case the paper describes. A narrative or a proof is not. Generating test cases for several independent functions in one pass is a reasonable fit for the same reason: each function's tests don't depend on another function's tests.
• Latency, specifically, is the metric you're optimizing. If you're already comfortable with normal generation time and mainly care about token cost, this technique doesn't help; it can even cost more tokens for the same content, since the paper is explicit that parallel calls trade latency for request volume, not the other way around.
• You can tolerate, or actively check for, some coherence loss. Given the paper's own 60%-of-the-time figure, run the router variant or your own quality spot-check rather than assuming every output reads as smoothly as a normally generated one.

If none of those apply, plain generation, or plain chain-of-thought for anything with dependent steps, is the simpler and safer default.

The Bottom Line

Skeleton-of-thought prompting is a real, measured technique for a real problem: sequential decoding is slow, and the paper's own numbers, an 1.83x speed-up on Claude and 2.69x on Vicuna-33B in their worked example, up to 2.39x across 12 tested models, back that up directly. But the win is specifically an engineering one: it requires actually running the point-expansion step in parallel, not just asking for an outline in a normal chat conversation. And it has a hard boundary the paper names outright — it doesn't work for step-by-step reasoning, and it costs real coherence on some fraction of the questions it's applied to, which is exactly why the paper's own recommended version adds a router rather than applying the technique universally.

By Nafiul Hasan — Founder of Prompt Architects, building prompt tooling used across ChatGPT, Claude, and Gemini, writing from hands-on testing of production prompts.