Analogical Prompting (Let the Model Generate Its Own Examples)

TL;DR: Analogical prompting asks the model to generate its own relevant worked examples before answering your actual question, instead of you writing few-shot examples yourself. It is a real, published ICLR 2024 method, not a folk technique, and its own benchmark tables show a small, consistent accuracy gain over zero-shot and few-shot chain-of-thought.

What Is Analogical Prompting?

Analogical prompting is a technique that has a model recall and solve a handful of problems similar to the one you actually asked, before it produces its final answer. The model generates its own few-shot examples on the fly, tailored to your specific question, rather than you writing them out or retrieving them from a fixed library ahead of time.

That distinction matters more than it sounds. Ordinary few-shot prompting is static: you pick two or three example question-answer pairs once, and every user gets the same ones regardless of what they actually asked. Analogical prompting flips that. The model looks at your problem first, invents its own related problems, works through those, and only then answers yours, using its own solved examples as nearby scaffolding. It is built on top of chain-of-thought reasoning, not a replacement for it, and it is worth reading the full chain-of-thought explainer if that foundation is new to you.

Where Does Analogical Prompting Come From?

The technique is not folk wisdom; it is the subject of a specific paper: "Large Language Models as Analogical Reasoners", by Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou, published at ICLR 2024 (arXiv:2310.01714, first submitted October 3, 2023). The paper's own abstract describes the motivation directly: chain-of-thought prompting "demonstrates impressive performance across reasoning tasks, but typically needs labeled exemplars of the reasoning process." Analogical prompting is introduced to remove that requirement by having the model "self-generate relevant exemplars or knowledge in the context, before proceeding to solve the given problem."

Mechanically, the paper describes augmenting a target problem with an instruction block. In its own words, given a target problem, the prompt adds:

# Problem: [your problem]
# Relevant problems:
Recall three relevant and distinct problems. For each problem,
describe it and explain the solution.
# Solve the initial problem:

The model then generates several worked examples in response to the "# Relevant problems:" instruction, before moving on to solve your actual problem using its own generated context. The paper reports results across three base models — GPT-3.5-turbo, text-davinci-003, and PaLM 2 — on two math benchmarks, GSM8K and MATH:

The gain over 5-shot chain-of-thought is modest on GSM8K with GPT-3.5-turbo (76.7% to 77.8%) and larger on the older text-davinci-003 model (54.0% to 61.0%). The paper also tested code generation on Codeforces and additional reasoning tasks in BIG-Bench, but this project verified only the math benchmark tables above against the paper's own text, so treat the code-generation and BIG-Bench claims as directionally consistent rather than independently confirmed here.

Why Self-Generated Examples Beat a Fixed Few-Shot Set

The paper's own case for the method is adaptability, not just convenience. As its abstract puts it, the approach "can also tailor the generated exemplars and knowledge to each problem, offering adaptability." A fixed set of few-shot examples written in advance has to be general enough to cover whatever question shows up, which means it is rarely a close match to any specific one. A self-generated set is produced after the model has already seen your actual problem, so the examples it invents can be shaped around the same structure, the same kind of numbers, the same category of reasoning step your problem needs.

This also sidesteps a real practical cost: building and maintaining a curated few-shot library, or a retrieval system that picks the closest match from one, is work. Analogical prompting needs neither. The trade-off, covered below, is that the model is generating the examples itself, from the same knowledge that will answer your question, so it is not adding outside information the model did not already have.

The paper also checked whether the examples need to be generated in one continuous pass or whether sampling them separately and then re-feeding them to the model helps. Its own conclusion: an "alternative approach" of independently generating exemplars and re-prompting with all of them "does work," but the single-pass version, generate the examples and the answer in one response, "achieves comparable performance" while being simpler to run. That is a useful thing to know before over-engineering an implementation: the plain version, one prompt, one response, is what the paper itself validates as sufficient.

Does the Diversity Instruction Actually Matter?

Yes, and the paper measured it directly rather than asserting it. It explains why the instruction is not decorative: "This step is crucial as some LLMs have a tendency to repetitively generate identical problems, which can be misleading when solving the target problem." Left unprompted, a model asked for "relevant examples" can generate three versions of essentially the same problem, which gives it no more useful scaffolding than one example would.

The paper's own ablation table quantifies the effect on GPT-3.5-turbo:

The fix in the prompt is the same phrase the paper actually uses. Its own example instruction reads "generate problems that are distinct from each other", added right alongside the "# Relevant problems:" instruction above. That single line is responsible for most of the gap between a mediocre implementation of this technique and the one the paper's headline numbers describe.

Does Analogical Prompting Stack With Self-Consistency?

It does, per a separate ablation in the same paper. Combining analogical prompting with self-consistency — sampling multiple independent answers and taking the most common one — pushed the GPT-3.5-turbo numbers from 77.8% to 85.3% on GSM8K and from 37.3% to 46.0% on MATH. The paper's own caption for that result states plainly: "Self-consistency can be used to further boost the performance of our analogical prompting approach."

That is a genuinely large jump, but it comes at a real cost: self-consistency means running the whole generate-examples-then-answer sequence multiple times per question and comparing the results, which multiplies both latency and token spend on top of what analogical prompting already adds. It is worth knowing the ceiling exists, and worth being honest that reaching it is not free.

A Worked Example, Step by Step

To make the mechanics concrete, here is a hypothetical walkthrough, invented for this post and not run against a live model, so treat it as an illustration of the shape of the technique rather than a real transcript. Say your actual question is: "A store marks up a wholesale item by 40% and then applies a 15% discount at checkout. What's the net percentage change from wholesale to sale price?"

Under analogical prompting, before touching that question, the model is asked to recall three distinct, relevant problems. A plausible self-generated set: (1) a problem applying a single markup with no discount, (2) a problem applying two sequential discounts with no markup, and (3) a problem combining a markup and a discount in the opposite order (discount first, then markup). The model works each one to completion, which forces it to practice the specific move your real question needs — treating percentage changes as multipliers rather than as numbers you can just add or subtract — three times before it ever writes an answer to your question.

That repetition is the entire mechanism. It is not retrieving outside knowledge; it is rehearsing the applicable move using the model's own existing knowledge, in a form close enough to your problem that the rehearsal actually transfers.

Where Analogical Prompting Falls Short

Three limits are worth naming plainly. First, cost: every self-generated example is additional output the model has to produce before it reaches your actual answer, which means more tokens and more latency than a zero-shot prompt, for a gain the paper's own tables put in the low single digits of accuracy on most of its GSM8K comparisons. Second, it adds no new knowledge. The model is drawing on the same training data to invent its examples that it would use to answer your question directly, so it cannot rehearse its way to a fact it never learned. Third, the technique was validated on math word problems, code generation, and BIG-Bench reasoning tasks specifically — open-ended, subjective, or fact-lookup tasks are a different shape of problem, and this project did not verify results for those categories from the paper.

How Is This Different From Step-Back and Self-Ask Prompting?

Analogical prompting sits alongside two other named techniques that add a structured step before the real answer, and it is worth being precise about which is which rather than treating them as interchangeable. Step-back prompting has the model derive a more abstract, higher-level question or principle from your specific one, answer that first, then use it to answer your actual question — it moves up a level of abstraction. Self-ask prompting has the model explicitly decompose your question into follow-up sub-questions, answer each one in turn, and combine them — it moves sideways into your question's parts.

Analogical prompting does neither of those. It moves outward, into neighboring problems, generating separate-but-similar cases and solving those first, rather than abstracting upward or decomposing the one you actually asked. All three are legitimate, separately published approaches to the same underlying goal — giving the model useful context to reason with before it commits to a final answer — and nothing stops you from combining them where they don't conflict.

A Copy-Paste Template You Can Use Today

Here is a template built on the same shape the paper describes, with the diversity instruction included from the start rather than added as an afterthought:

I have the following problem:

[your problem or question]

Before solving it, recall three relevant problems that are
distinct from each other and from the problem above. For each
one:
- Describe the problem
- Work through the full solution

Then, using what those three examples demonstrate, solve my
original problem above, showing your reasoning.

Swap "problems" for whatever unit fits your task, cases, scenarios, examples, and it generalizes past math. The instruction to make the recalled items distinct from each other is not optional decoration; per the paper's own ablation above, it is most of the measured gain.

Analogical prompting is a genuine, published technique with real numbers behind it, not a step change, and worth knowing precisely rather than as a vague "let it think first" habit. If persona and framing choices interest you as a related lever on what a model is willing to assert before it even reaches a reasoning step, does emotional prompting work covers a different, more contested corner of the same broader question: what you put in the prompt before the actual task changes what comes out of it.