llm experimentation

The Complete Guide to LLM Experimentation: Compare Prompts, Models, and Agents

Jun 14, 2026·12 min read

Kritin Vongthongsri
Co-founder @ Confident AI. LLM Evals & Safety Wizard. Previously ML + CS @ Princeton researching self-driving cars.

Every AI team eventually ends up in the same debate: is this new prompt actually better, or do we just like the 3 examples we happened to look at?

This is what LLM experimentation is for. It gives you a controlled way to compare two or more versions of your AI app against the same dataset, using the same metrics, before you push changes to production.

No vibes, no cherry-picked examples, and no gut feel. Instead, you get a repeatable process for deciding which prompt, model, tool set, or agent configuration should actually ship.

In this guide, I'll walk through the full LLM experimentation workflow: what LLM experimentation means, how it differs from evaluation, how to choose what to optimize, how to curate datasets and metrics, how to interpret results, how to extend experiments into production, and how Confident AI supports the loop end-to-end.

TL;DR

LLM evaluation answers "how good is this version?" LLM experimentation answers "which version is better?"
The only fair comparison uses the same dataset and evaluation metrics. If prompt A and prompt B see different inputs, you are not running an experiment. You are collecting anecdotes.
Change one variable at a time. Compare prompt vs prompt, model vs model, tool set vs tool set, or retrieval config vs retrieval config. If everything changes at once, you will not know what worked.
Use the 2 + 3 rule for metrics. Start with 2 general-purpose metrics and 3 custom metrics that encode what good means for your product.
Look beyond LLM-as-a-judge scores. Latency, cost, tool correctness, retries, token usage, and escalation rate can matter just as much as answer quality.
Experiments do not end after the report. Winners should be versioned, A/B tested when appropriate, monitored in production, and used to improve future datasets.

What Is LLM Experimentation?

LLM experimentation is the process of comparing multiple versions of an LLM app under controlled conditions. Each version runs against the same evaluation dataset and gets scored by the same metric collection so you can see which version performs better.

The basic experimentation process looks like this:

Pick the parameter you want to optimize: prompt, model, tool set, retrieval config (top-k, chunking, reranker), agent architecture (planner, memory, reflection loop), etc.
Choose a fixed dataset that represents the use case.
Prepare a collection of metrics that evaluate what you care about.
Run every variant against the same dataset and metric collection.
Compare aggregate scores, inspect failures, and decide what to ship or investigate next.

Choosing a winner is also not always straightforward. If version A performs better on 50% of test cases and version B wins on the other 50%, you need to inspect the trade-offs before deciding what to ship.

LLM Experimentation vs LLM Evaluation

An eval tells you whether one version passed your quality bar.
An experiment tells you which version performed best against one or more alternatives.

That distinction matters because evals are easy to mistake for experiments. If you run prompt A today, prompt B tomorrow, compare the pass rates, and call it a decision, the comparison is noisy unless both versions ran on the same dataset, with the same metrics, under the same conditions.

LLM Experiments vs A/B Tests

An LLM experiment is usually done offline in development or staging. You can compare many prompt, model, tool, retrieval, or agent variants against the same dataset and evaluation metrics.

An A/B test compares two versions, usually A vs B. In production, that usually means splitting live traffic across two LLM app versions and measuring product outcomes.

In practice, offline experiments and A/B tests in production are usually used together to continuously improve an LLM app or agent.

Running an LLM Experiment

Running an LLM experiment should be easy once you have the right setup. A good setup is controlled in the right ways: clear baseline, clear variants, fixed dataset, and fixed metrics.

The minimum workflow for an LLM app or agent looks like this:

Choose what to optimize.
Choose a baseline app or agent version.
Create one or more app or agent variants.
Run every version on the same evaluation dataset.
Score every version with the same metric collection.

This lets you answer the practical question: if we swapped production to this version, what would likely get better and what would likely get worse?

Choosing What to Optimize

Most LLM apps have a lot of knobs. Prompts, models, tools, retrieval, temperature, memory, response format, and planner instructions can all change behavior.

Parameter	When to Optimize	What You Compare
Prompt	You want better wording, instructions, examples, formatting, or refusal behavior	Two different task prompts, such as a concise instruction prompt vs a few-shot prompt
System prompt	You want to change the app's role, policy, tone, or global constraints	Two different system prompts, such as a strict policy-first prompt vs a more conversational prompt
Model	You want to trade off quality, cost, latency, or reliability	Different models, such as GPT-4o vs Claude vs Gemini
Tool set	You want to know whether an agent can complete tasks more reliably with different tools	Different tool configurations, such as the current tools vs a version with a refund lookup tool added
Retrieval config	You want better grounding or context selection in a RAG workflow	Different retrieval settings, such as top-k 3 vs top-k 8, or no reranker vs reranker
Temperature	You want to test creativity, determinism, or consistency	Different decoding settings, such as temperature 0 vs 0.3 vs 0.7
Memory strategy	You want a multi-turn agent to remember useful context without drifting	Different memory approaches, such as no memory vs short-term memory vs summarized memory
Agent planning	You want to reduce loops, wrong tool calls, or unnecessary reasoning	Different planning instructions, such as a direct tool-use planner vs a step-by-step planner

Curating Your Evaluation Dataset

Your dataset is the test bed your variants have to run across. If it is too easy, every version wins. If it is random, the result is noisy. If it is not representative, you optimize for a fake product.

A good experiment dataset usually includes a few general categories, plus use-case-specific cases for the kind of LLM app you are testing:

Common successful workflows
Known failure cases from production, logs, support tickets, or manual review.
Ambiguous inputs where the app should ask for clarification.
Policy-sensitive cases if the app handles security, compliance, safety, escalation, or other strict rules.
RAG cases if the app uses retrieval and needs expected context or grounding requirements.
Agentic tasks if the app uses tools, multi-step sequencing, or final-answer quality depends on intermediate actions.

Choosing Your Metrics

Your metrics are the scoreboard. If the scoreboard is wrong, the experiment is wrong.

For most LLM experiments, start with the 2 + 3 rule: use 2 out-of-the-box metrics for common failure modes and 3 custom metrics for the use-case-specific behavior that makes your app good or bad.

Not every metric needs to apply to every test case.

Signal	Why It Matters
Latency	A higher-quality answer can still be unusable if it is too slow
Cost	A variant can win on quality but be too expensive to ship broadly
Tool retries	The final answer may look correct while the trace shows instability
Escalation rate	A prompt that sounds better may still fail to resolve the task
Abstention or refusal rate	The variant may become too cautious or not cautious enough

Comparing Prompts Side by Side

That is why you usually should not send every prompt idea straight into a full experiment. Before scoring 100+ test cases, run each candidate prompt on one representative test case and inspect the answer it generates.

Interpreting Experiment Results

A good experiment report does more than tell you "B scored higher." It shows whether B is better in the ways that matter.

When you read the report, look for:

Clear winners: one variant wins across the metrics and test cases that matter most.
Close calls: the difference is small enough that you should collect more test cases before shipping.
Trade-offs: one prompt improves relevance but hurts concision, tone, faithfulness, cost, or latency.
Failure clusters: the average looks fine, but one slice fails.
Outliers: a narrow but important failure could be worse than a small aggregate improvement.

Running Experiments in Production

Offline experiments should catch obvious regressions. Production experiments tell you whether the strongest candidate still works under real traffic, real retrieval data, and real business constraints.

The relationship should look like this:

Use offline experiments to eliminate weak variants.
Ship the strongest candidate behind versioning.
Run a canary, shadow deployment, or A/B test if the change is meaningful.
Monitor online evals and product metrics.
Feed production failures back into the next offline experiment.

A/B Testing Prompts and Models

Use offline experiments to eliminate bad candidates, then use shadow traffic, canaries, or production A/B tests to confirm the strongest version under real traffic.

Improving Your Experiments

The best LLM experiments get more useful over time. Each production run should teach you what to add to the evaluation dataset, what to measure next, and which metrics need sharper criteria.

Expanding Your Evaluation Dataset for Experiments

Your evaluation dataset should get better every time production teaches you something.

Expanding Your Metric Suite

Production should also teach you when your metrics are incomplete or misaligned.

Common LLM Experimentation Pitfalls

Most LLM experimentation failures come from bad experimental design:

Changing too many variables at once.
Using different datasets per variant.
Optimizing for one metric only.
Trusting averages too much.
Skipping the smoke test.
Ignoring close calls.
Forgetting cost and latency.
Not versioning the experiment.

Why Confident AI Is the Best Platform to Run LLM Experiments

A useful LLM experimentation platform is not just a playground. Confident AI lets teams compare various aspects of their AI apps against the same dataset and metric collection.

Conclusion

LLM experimentation is not complicated, but it does require discipline. Compare variants against the same dataset. Score them with the same metrics. Change one variable at a time. Read both the aggregate scores and the individual failures. Then choose the version that wins on the quality bar you actually care about.