free · no signup · 1 run a day
A/B Prompt Tester
Paste two versions of a prompt, run both against the same model, and see which one actually wins — with a judged score per criterion, not a hunch.
1 free run a day · no signup · prompts are not stored
How it works
01
Paste two versions of your prompt. Optionally add one test input both will run against.
02
Both prompts run on the same model, same settings. An LLM judge scores each output on 3 criteria.
03
See the winner, the score gap, and the one-line reason. Iterate and re-run.
Why A/B test prompts at all
Most prompt changes are judged by eyeballing one output — which is how regressions ship. A vaguer variant can produce a nicer-looking answer on one lucky run and fall apart on the next ten inputs. Running both variants under identical conditions and scoring the outputs against fixed criteria is the smallest unit of honest prompt testing: it forces the difference to show up in the result, not in your impression of it.
Frequently asked questions
What is A/B testing for prompts?
A/B prompt testing runs two variants of a prompt against the same model and the same input, then compares the outputs on defined criteria. It replaces "this output feels better" with a measured comparison: which variant followed its instructions, produced usable output, and kept the result constrained. It is the same discipline as A/B testing landing pages, applied to prompt engineering.
Is this A/B prompt tester really free? Do I need an API key?
Yes. You get one A/B run per day, no signup, no API key, no card. The prompts run on our infrastructure and are not stored. A free account unlocks the full prompt evaluator (0-100 structural score, 3 evaluations a month); the Basic plan adds the Playground, where you run A/B tests against up to 20 inputs with custom criteria using your own API key.
How does the judge decide the winner?
An LLM judge receives both prompts and both outputs, and scores each side 0-100 on three criteria: instruction adherence (did the output follow the prompt's own rules), output quality (correct, complete, usable), and precision (no filler, no drift, no guessed format). The winner is the prompt that produced the more reliable result — not the prettier prose.
One run is a small sample. How do I test properly?
Correct — a single run tells you about one input, and LLMs are non-deterministic. Treat this tool as the fast first pass: it catches structural differences (missing format spec, vague constraints) immediately. For a statistically meaningful comparison, run both variants against a batch of representative inputs with defined pass criteria — that is what the Playground A/B on the Basic plan does, with up to 20 inputs per session, running on your own API key.
See how batch A/B testing works →Which models does it work with?
The free tester runs both prompts on the same fast production model, so the comparison is apples to apples. Prompt-vs-prompt differences (structure, specificity, constraints) transfer across models: a variant that wins on one model because it specifies format and scope usually wins on GPT-4o, Claude, and Gemini for the same reason.
Do you store my prompts?
No. The tester returns the outputs and the verdict and keeps no prompt content tied to you. A cached copy of the anonymous result may be kept so identical re-runs are instant.
Want the winner scored structurally? Run it through the free AI prompt checker. Setting up a real testing routine? Read how to A/B test prompts properly. Watching token cost? Try the free token counter.