Prompting for ML Experiments

TL;DR: An ML experiment prompt that starts with "write me a training script" has no way to tell a real result from a plausible one. Start with a falsifiable hypothesis, fix and log every random seed you can name, and never ask a model to summarize results it wasn't actually given — that's the fastest route to a fabricated metric in a report someone will act on.

"Train a model on this dataset and see if it works better" is not an experiment. It's a request with no wrong answer, which means whatever comes back, code that runs and a number that looks reasonable, can't be checked against anything. The prompt did the damage before the model wrote a line, by never stating what "better" meant or what would count as evidence against the idea.

Why Should an ML Experiment Prompt Start with a Hypothesis, Not a Task?

Because a hypothesis is falsifiable and a task isn't. "Train a model" succeeds by definition the moment code runs without erroring. "Adding character n-grams to the feature set improves F1 on the minority class by more than 2 points over the current baseline" can fail, cleanly, and that failure is informative. The second framing also forces you to name the baseline you're comparing against, which a bare task request routinely skips.

A useful ML experiment prompt names four things before it asks for any code:

1. The hypothesis, stated so it could turn out false.
2. The baseline it's being compared against, specifically enough to rerun.
3. The metric that would confirm or reject it, and the size of change that would count as meaningful rather than noise.
4. What's held constant — data split, seed, hyperparameters not under test — so the comparison isn't confounded by something that changed incidentally.

Hypothesis: adding TF-IDF character n-grams (3-5) to the existing
word-level feature set improves macro F1 on the minority class ("churn")
by at least 2 points over the current baseline model.

Baseline: logistic regression, word-level TF-IDF only, as in
train_baseline.py, same train/val split (seed=42).

Held constant: same train/val split, same seed, same regularization
strength (C=1.0), same class weighting.

Write the training script for the variant model only. Log the seed,
the exact feature configuration, and macro F1 on both classes to a
results.json file. Do not print or estimate a result — the script
should only report numbers it actually computed after running.

Reproducibility: What "Set a Random Seed" Actually Needs to Cover

A single random_state=42 is rarely the only source of run-to-run variation in an ML pipeline. A prompt that asks for "reproducible results" without naming which seeds it means tends to fix one and leave the rest to chance.

import random
import numpy as np
import torch

SEED = 42

def set_all_seeds(seed: int = SEED) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)

set_all_seeds()

# scikit-learn estimators and train_test_split take their own random_state:
# LogisticRegression(random_state=SEED)
# train_test_split(X, y, test_size=0.2, random_state=SEED)

A prompt that says "make this reproducible" gets more consistent results if it lists the specific calls that take a seed argument in your stack, rather than trusting the model to remember every one of them for your particular combination of libraries.

Prompting for an Ablation Instead of a Single Experiment

An ablation asks a narrower, more useful question than "does this help": given an effect you've already observed, which specific component is responsible? A vague "try some variations and see what helps" prompt tends to produce inconsistent baselines across variants, which makes the comparison meaningless before you even get to results.

State the ablation explicitly:

Ask for one training script parameterized by a --variant flag rather than four separate scripts, so the only thing that changes between runs is the thing you're actually testing. A prompt that lets the model write four independent scripts risks small, unintentional differences (a slightly different preprocessing step, a different default) that confound the comparison you were trying to isolate in the first place.

Designing the Experiment Before You Write Any Training Code

The most useful place to bring a model in is earlier than most people do: before any code exists, while the question is still being shaped. Describe the question, the data you actually have, and the real constraint you're working under, and ask for the smallest experiment that would genuinely answer it, before asking for a training script.

Question: does adding the customer's support-ticket history improve
churn prediction, or is it redundant with the usage-based features
we already have?

Data available: 18 months of usage logs, 6 months of support-ticket
text (shorter history, only 40% of customers have any tickets).

Constraint: one afternoon of compute, no GPU cluster access this week.

Propose the smallest experiment that would give a real answer to the
question above, given the constraint. Flag any reason the ticket data's
shorter history or partial coverage could bias the result before I
design anything around it.

This kind of prompt catches a design flaw (in this case, the ticket data's uneven coverage) before it becomes a training run you have to redo, and it does so more cheaply than catching the same flaw after the fact in a results writeup. Treat "design this experiment" and "write this training script" as two separate prompts with two separate failure modes: the first can produce a flawed plan that runs without error, and the second can produce broken code that never gets far enough to show you the plan was flawed in the first place.

Is the Difference Real, or Is It Noise?

A model comparing two numbers will, if asked plainly "did this help," tend to answer yes whenever the new number is higher, without checking whether the gap is larger than the run-to-run variation you'd see from the seed alone. That's a real risk specific to ML experiments: unlike a deterministic unit test, a single training run's metric is a sample, not a fact, and a 0.3-point F1 improvement from one run of each variant is often indistinguishable from noise.

Ask for the check explicitly rather than trusting a single-run comparison:

I ran the baseline and the variant 5 times each with different seeds
(42, 43, 44, 45, 46). Here are the macro F1 scores for each:

Baseline: [paste 5 numbers]
Variant: [paste 5 numbers]

Report the mean and standard deviation for each. State whether the
difference between the two means is larger than roughly one standard
deviation of the noisier group. Do not describe the variant as
"clearly better" if it isn't.

This doesn't require a formal statistical test to be useful. Even a rough "is the gap bigger than the spread" check, applied honestly, catches the most common version of this mistake: reporting a single lucky run as a confirmed improvement, then building on a conclusion that a second run wouldn't reproduce.

Prompting for Data Leakage Checks Before You Trust a Result

A model asked to write feature-engineering code has no independent way to know which columns in your dataset were computed using information that wouldn't have been available at prediction time. Leakage is one of the most common reasons an experiment reports a great result that never survives contact with production, and it's exactly the kind of thing worth naming explicitly rather than assuming the model will catch it unprompted.

"Review this feature engineering code for data leakage: does any feature use information from after the prediction point, or information computed across the full dataset (including validation/test rows) rather than fit only on training data?" is a specific, checkable question. A generic "review my code" prompt is far less likely to surface it, because leakage doesn't look like a bug in the code itself, it looks like a feature column with a slightly-too-good name.

Turning Real Results Into an Honest Writeup

This is the step where fabrication risk is highest, because it's the one place a plausible-sounding sentence is easy to generate and hard to spot. The fix is mechanical: give the model the actual output, and forbid it from adding anything that isn't in that output.

Here is the actual content of results.json from the experiment above:
[paste the real file contents]

Summarize these results in three sentences for a team update. Only
state numbers that appear in this file. Do not estimate, round in a
way that changes the conclusion, or add a comparison to a number that
isn't present here. If a metric I'd expect (e.g. precision on the
minority class) is missing, say so instead of estimating it.

The instruction to name a missing metric rather than fill the gap is the important line. A model asked to write three confident sentences about an experiment will fill a hole with something plausible unless it's told explicitly that an admitted gap beats an invented number.

How This Connects to Verification and Self-Critique Prompting

The discipline here (state what would falsify the claim, then check the actual output against it) is the same one behind verification prompting, applied to a specific artifact: a results file instead of a general claim. If you want the model to generate its own test cases for a hypothesis before you run anything, analogical prompting is the relevant technique for pressure-testing a hypothesis's edge cases before committing compute to it. And when an experiment's outcome is ambiguous, self-ask prompting is a useful structure for working through which follow-up experiment would actually resolve the ambiguity, rather than jumping straight to another training run.

Prompting for the Negative Result You'd Rather Not Report

An experiment that disproves the hypothesis is still a real result, and it's the one a prompt is most likely to soften without being asked to. Given ambiguous or unfavorable numbers, a model summarizing them for a team update will often reach for hedged, optimistic language ("shows promise," "warrants further investigation") that reads better than it informs. If the hypothesis wasn't supported, say so as plainly as the confirmed case: "state clearly whether this result supports or contradicts the stated hypothesis, even if the answer is that it doesn't help" is worth adding to any results-summary prompt, because the alternative is a writeup that's technically accurate in every individual sentence while leaving the actual conclusion unclear.

A Short Checklist Before You Trust an Experiment's Conclusion

1. Was the hypothesis stated before the code was written, in a form that could have failed?
2. Are all the seeds that affect this run (data split, model init, library-level randomness) fixed and logged?
3. Did every variant in a comparison share the same baseline and held-constant settings?
4. Is every number in the writeup traceable to an actual logged result, not a plausible-sounding estimate?
5. If a metric is missing, does the writeup say so, instead of quietly working around the gap?