Few-Shot Prompting for Classification and Tagging

TL;DR: Few shot classification needs more than good examples. Once you're sorting real items into real categories, you also need label definitions precise enough to resolve boundary cases, a real answer for items that don't fit any label, and a way to check the prompt isn't quietly biased toward one answer before you run it on more items than you can check by hand.

What Does Few-Shot Classification Actually Require, Beyond Examples?

If you haven't already decided whether examples are the right call for your task at all, start with few-shot vs zero-shot prompting. If you know you're using examples and want to know how many, how many examples should a few-shot prompt have covers that directly, and how to choose good few-shot examples covers which ones to pick. This post assumes both of those are settled and covers what's specific to classification and tagging as a task: the label design, the edge cases, and the checks that a generic few-shot guide doesn't get into.

Classification and tagging share one property most few-shot tasks don't: a fixed, checkable answer space. That's what makes them tractable to automate at volume, and it's also exactly where the failure modes below live. A support-ticket router, a content tagger, or a lead-scoring prompt isn't graded on whether the output reads well. It's graded on whether the label is right, applied the same way twice, and defensible on the item that doesn't cleanly fit anywhere.

How Do You Write Label Definitions the Model Can Actually Apply?

A label name alone under-specifies the task. billing, bug, and feature-request look self-explanatory to you, because you already know where the boundaries sit. The model only knows what your examples and any definition you write actually show it.

Compare two versions of the same category:

Weak:
billing — payment or subscription issues

Better:
billing — the customer is disputing a charge, asking about
a refund, or reporting a payment method problem. Does NOT
include: feature requests about billing UI, or bugs that
happen to occur during checkout (those are "bug").

The second version does something the first doesn't: it names the adjacent categories your label gets confused with and says explicitly which side of the line each ambiguous case falls on. That's the actual content of a label definition, not a longer synonym for the label's name. Do this for every label where you can predict the confusion, not just the one or two you happen to think of first, and you'll cut down on the ambiguous middle far more than adding another example of the easy, unambiguous case ever does.

Is Multi-Label Tagging the Same Problem as Single-Label Classification?

No, and conflating them is one of the more common design mistakes in a tagging prompt. Single-label classification assumes every item belongs to exactly one bucket. Multi-label tagging assumes an item can carry zero, one, or several tags simultaneously, which is a genuinely different shape of task, not a harder version of the same one.

The failure mode is worth sitting with, because it's easy to misdiagnose. If you force a genuinely multi-label task (a blog post that's both AI and marketing) into a single-label prompt, the model has to arbitrarily commit to one tag when both are correct. That output pattern looks like the model being inconsistent from run to run. It's actually the prompt asking a question that doesn't have a single right answer, and no amount of better examples fixes a task-shape mismatch.

If your real content genuinely needs multiple tags, say so directly in the instruction and show it in your examples:

Assign every tag below that applies to this article. If none
apply, reply with an empty list. Do not force a single choice.

Tags: [AI, marketing, engineering, product, opinion]

Article: "How we rebuilt our onboarding flow using GPT-4
for personalized copy, and what it did to activation."
Tags: [AI, marketing, product]

What Should Happen When an Item Doesn't Fit Any Label?

Give it somewhere to go. Every real label set eventually meets an item that's ambiguous, malformed, or genuinely outside the taxonomy, and a prompt with no fallback forces the model to pick the closest wrong label rather than admit the mismatch.

Without an explicit other or unclear option, a model under pressure to output a valid label from your set tends to default toward whichever label appeared most often in your examples, since that's the strongest signal it has to go on. That quietly inflates your most common category with items that don't actually belong there, and the inflation is invisible unless you're specifically checking the boundary cases, because every individual output still looks like a plausible answer.

Add the fallback explicitly, and treat it as a real category with its own place in your examples, not an afterthought:

Categories: billing, bug, feature-request, other

Use "other" only when the ticket genuinely doesn't fit the
first three categories, or when you can't tell from the text
given. Do not use "other" for tickets that are simply hard
to read — reason through those and pick the best fit.

That last line matters. Without it, other becomes a dumping ground for anything the model finds slightly difficult, which defeats the purpose of having a taxonomy at all.

What Does All of This Look Like Put Together?

Here's a single support-ticket tagging prompt that applies the label-definition, multi-label, and fallback rules above at once, rather than as separate abstract pieces:

Assign each ticket exactly one category from the list below.
Use "other" only if it genuinely fits none of the first four —
do not use it just because the ticket is oddly worded.

billing — disputing a charge, asking about a refund, or
  reporting a payment method problem. NOT feature requests
  about billing UI, and NOT bugs that happen during checkout
  (those are "bug").
bug — something in the product is broken or behaves
  unexpectedly. NOT a request for something that doesn't
  exist yet (that's "feature-request").
feature-request — asking for something the product doesn't
  currently do.
account — login, password, or account-access issues only.
other — genuinely doesn't fit above, or too unclear to
  confidently place.

Ticket: "I was charged twice for my subscription this month
and I can't find a way to get a refund from the settings page."
Category: billing

Ticket: "Export button does nothing when I click it, and
the page doesn't show an error either."
Category: bug

Ticket: "Would love a dark mode option for the dashboard."
Category: feature-request

Ticket: "[your real ticket text]"
Category:

Notice what each part of this prompt is doing a distinct job: the label definitions resolve the billing/bug boundary that would otherwise be genuinely ambiguous on a checkout-related bug report, the other category has an explicit instruction against overusing it, and each worked example was chosen to sit near a real boundary rather than in the easy middle of its category. None of those three choices come from picking better examples in isolation. They come from designing the taxonomy the examples are demonstrating, before you write a single example.

How Do You Check Whether Your Classification Prompt Is Actually Consistent?

Run it on an input that should not favor any label at all, and see what comes back. This is a real, tested technique, not an intuition: Calibrate Before Use: Improving Few-Shot Performance of Language Models (Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh, ICML 2021) found that GPT-3's few-shot classification prompts carry hidden bias baked in by the examples themselves, independent of what you actually ask it to classify. Their diagnostic: feed the prompt a content-free test input, like a blank field or the literal text N/A, and see what it predicts. On one sentiment task, they report that "the model’s biases cause it to score this input as 61.8% Positive" — a meaningless input, scored as if it meant something, purely because of skew in the examples and their order.

That's a genuinely useful, cheap check you can run on any classification prompt before trusting it: swap your real input for a blank, a nonsense string, or literally the text N/A, and see whether the model still confidently lands on a specific label. If it does, your examples (their order, their label balance, or their wording) are pulling the model toward that answer regardless of the actual input, and you should rebalance or reorder them before running the prompt on real data.

A second, complementary check is simpler and doesn't require a nonsense input at all: run the exact same real item through your prompt several times and see if the label holds. This is the same idea behind self-consistency prompting applied to a classification task instead of a math problem — if the same input gets different labels across runs, that instability is real signal, not noise, and it usually means either your label definitions still have a gap or two of your labels genuinely overlap for that input.

When Does This Need to Become a Production Pipeline Instead of a Prompt?

Once you're tagging more items than you can spot-check by eye, or a wrong label costs something real, treat the prompt as a small system rather than a one-off. A few things earn their place at that point that don't matter for a prompt you run occasionally by hand:

• A confidence or ambiguity signal. Ask the model to flag when it's uncertain, not just to always commit to one answer. A tagging prompt that can flag itself for review is more useful in production than one that always sounds confident.
• A sampled human review loop. Pull a small, random percentage of outputs for a person to check periodically, rather than assuming the prompt that passed your fifty test cases keeps passing at ten thousand.
• A re-test trigger. Whenever you add, rename, or merge a label, re-run your test set. A label taxonomy change invalidates the boundary language your examples were teaching, even if none of the example text itself changed.
• Version the prompt, not just the label list. If the taxonomy and the prompt drift out of sync because only one of them got updated, every classification made in that gap is silently wrong in a way nothing will flag on its own.

None of this is exotic infrastructure. It's the difference between a prompt that worked in testing and a system you can actually trust unattended.

What Happens When Your Label Taxonomy Changes?

It will, and most classification prompts aren't built with that in mind. A category set that made sense with fifty tickets a week often stops making sense at five thousand, once new categories genuinely need to exist and old ones need to split or merge.

Two things go wrong if you don't plan for this. First, adding a new category without touching your examples leaves the model with no demonstration of what belongs there, so it keeps routing those items into whichever old category is the closest fit, even after the new label technically exists in your instruction. A label the model has never seen an example of is, in practice, a label it will rarely choose, no matter how clearly you define it in prose. Second, splitting one category into two (say, bug becoming bug-critical and bug-minor) silently invalidates every past classification made under the old single category, and nothing about the system tells you that happened unless you're specifically tracking taxonomy versions against classification history.

The practical fix is to treat the label list as a versioned artifact, not a static constant baked into the prompt. When you change it, budget time to add at least one worked example for anything genuinely new, re-run your test set (including the content-free check above) before trusting the updated prompt, and decide explicitly whether historical items classified under the old taxonomy need to be re-run or just left as-is with a note about which taxonomy version produced them. Skipping that decision doesn't avoid it — it just means old and new data get silently compared as if they meant the same thing, when they don't.

The Bottom Line

Good examples are necessary for few-shot classification, but they're not sufficient once real categories, real edge cases, and real volume show up. The layer that actually holds a tagging system together is the one most few-shot guides skip: label definitions precise enough to resolve the boundary, an honest fallback for the item that fits nowhere, and a habit of checking the prompt against a meaningless input before trusting it against a real one.

Build the taxonomy and the fallback before you write a single example, run the content-free check before you run the prompt on real data, and treat any drift between your label list and your prompt as a bug, not a detail. That's the part of few-shot classification that doesn't show up in a guide about picking good examples, because it isn't about the examples at all.

By Nafiul Hasan — Founder of Prompt Architects, building prompt tooling used across ChatGPT, Claude, and Gemini, writing from hands-on testing of production prompts.