What is the difference between pass@k and pass^k?

Both describe repeated tries at the same task. Pass@k is the chance of at least one success in k tries, so it rises as k grows. Pass^k is the chance that all k tries succeed, so it falls. Anthropic says pass^k suits customer-facing agents, where users expect reliable behavior every time.

How many test cases do I need to start?

Fewer than most teams think. Anthropic's January 2026 post says 20 to 50 simple tasks drawn from real failures is a great start, because early changes to an agent usually have a large, visible effect. More mature agents may need larger and harder sets to detect smaller changes. Start small and keep adding.

Can a benchmark score tell me if an agent is good for my business?

Not by itself. A benchmark such as SWE-bench or tau-bench measures one fixed set of tasks under one setup. The papers' authors warn that passing a benchmark's checks may not be enough: the SWE-bench paper says execution-based testing alone cannot guarantee reliable performance. Build a small eval from your own tasks as well.

How Do You Test Whether an AI Agent Is Any Good?

Q: What is an eval for an AI agent?

An eval is a test for an AI system. In Anthropic's definition, you give the AI an input, then apply grading logic to its output to measure success. For an agent, the grading often checks the final state of the environment, such as whether a record changed correctly, and not only the words the agent used.

Imagine the Lagos fabric shop from the first post has an agent that answers “where is my order?” questions. One Tuesday the developer rewrites its instructions to make it friendlier. She tries three chats, all look good, and she ships it. Two weeks later the owner finds that the agent has started promising delivery dates the courier never gave. Nobody can say when it began. Three chats could not have caught it. What would have?

(This is an illustrative scene, not a real business.)

Why is trying it a few times not enough?

Because an agent can behave differently each time, and a change that fixes one thing can break another. Anthropic’s engineering team, in a post of 9 January 2026, says teams can get “surprisingly far” with manual testing and intuition when they start. But once an agent is in production, the team can end up “flying blind”: users say it feels worse after changes, and there is no way to check except to guess. In Anthropic’s words, without tests, “debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed.”

Agents make this harder. As the third post showed, they take many steps, and one early mistake can build on the next. The usual fix is an eval.

What is an eval for an AI agent?

An eval, short for evaluation, is a test for an AI system. Anthropic defines it as a test where you “give an AI an input, then apply grading logic to its output to measure success”. Anthropic’s post defines the other words you will meet, and we use them throughout:

A task is “a single test with defined inputs and success criteria”.
A trial is each attempt at a task. Because outputs vary between runs, you run several trials of each task.
A grader is “logic that scores some aspect of the agent’s performance”. A task can have several.
A transcript is the complete record of a trial, including outputs, tool calls, reasoning and intermediate results.
The outcome is “the final state in the environment at the end of the trial”.
An evaluation harness is the software that runs the evals from start to finish: it gives the agent its tools, runs the tasks, records every step, grades the results and adds them up. This is our paraphrase of Anthropic’s definition.

In our reading, for the small agent in the eighth post, the harness would be a second script that feeds it test questions and scores what comes back.

Should you grade what the agent said or what it did?

Mostly the result it produced or changed. Anthropic gives a clear example: a flight-booking agent might say “Your flight has been booked” at the end of the transcript, but the outcome is whether a reservation actually exists in the database. The words can be wrong while sounding right.

Anthropic also warns against grading the exact steps. It found that checking for a fixed sequence of tool calls was “too rigid”, because agents “regularly find valid approaches that eval designers didn’t anticipate”. Its advice: “it’s often better to grade what the agent produced, not the path it took.”

The two benchmarks from the third post show outcome grading at work. A benchmark, as defined there, is a fixed set of test tasks with a scoring rule, so different systems can be compared on the same work.

tau-bench (customer service). A language model plays the customer, and the agent uses tools on a database. The paper says it compares “the database state at the end of a conversation with the annotated goal state”, meaning the correct end state its authors wrote down for each task. The authors also note the limit: their pass signal, a reward of 1, “might be a necessary but not sufficient condition for a successful episode”, for example, “the agent might issue the return without explicit user confirmation, which violates the policy.”
SWE-bench (coding). The agent’s fix is applied to the code, and the project’s tests are run. If the patch applies and all the tests pass, the issue counts as resolved. Anthropic describes the rule as fixing the failing tests without breaking existing ones. The authors add that relying only on execution-based testing “is insufficient to guarantee reliable performance of model generations”, because generated code can be less comprehensive, efficient or readable than human-written code.

In our reading, the lesson is not to copy these benchmarks. It is that even well-built end-state checks leave gaps, so an eval needs more than one kind of check.

Who does the grading?

Anthropic describes three kinds of grader, and says agent evals typically combine them.

Code-based graders are ordinary programs: string matches, tests, checks of the final state. Anthropic lists them as fast, cheap, objective and reproducible, but “brittle to valid variations”.
Model-based graders use a language model to score the work, often against a rubric, which in our plain words is a written list of what good looks like. This is also called an LLM-as-judge. They are flexible and can handle open-ended answers, but are “non-deterministic”, cost more than code and need calibrating against human graders.
Human graders are experts or reviewers. Anthropic calls them the gold standard but expensive and slow, and they are used to calibrate model-based graders.

Anthropic recommends “deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously”.

Graders can be wrong too. Anthropic reports that Claude Opus 4.5 first scored 42% on a benchmark called CORE-Bench, until a researcher found a grader that marked “96.12” wrong when it expected “96.124991…”, among other problems. After fixing bugs and loosening the setup around the model, the score was 95%. A low score may mean a broken test.

How many times should you run each task?

More than once, because the same agent can pass a task on one run and fail it on the next. Two measures help, both defined in Anthropic’s post.

pass@k is the chance of at least one success in k attempts. It rises as k grows.
pass^k is the chance that all k attempts succeed. It falls as k grows. The tau-bench paper introduced it, and the third post covered it.

Anthropic’s own example: “If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%.” Which measure to use depends on the job. Anthropic suggests pass@k when one success is enough, and pass^k “for agents where consistency is essential”.

Illustrative example. Imagine the shop’s order agent answers a test question correctly on 3 of 4 runs, a 75% success rate. Run three times, it gets all three right only about 42% of the time. By our own arithmetic, assuming independent runs, it gets at least one right about 98% of the time. In our reading, a customer should not be asked to bet on the 98%. These numbers are made up.

How do you build your first eval set?

Start small and start from real failures. Anthropic says teams delay evals because they think they need hundreds of tasks, but “20-50 simple tasks drawn from real failures is a great start”. Its advice, in our words:

Use what you already check by hand. Turn the manual checks you do before each release, and bug reports or support complaints, into test cases.
Write unambiguous tasks. A good task is “one where two domain experts would independently reach the same pass/fail verdict”. Make a reference solution for each, a known working output, to prove it can be solved.
Balance the set. Test cases where the behavior should happen and where it should not. If you only test whether the agent searches when it should, you may build one that searches for almost everything.
Isolate each trial. Each should be “isolated” by starting from a clean environment.
Read the transcripts. When a task fails, the transcript shows whether the agent erred or your grader rejected a good answer. Anthropic adds that, with frontier models, a 0% pass rate across many trials is “most often a signal of a broken task, not an incapable agent”.

Anthropic separates capability evals, which ask what the agent can do well and start with a low pass rate, from regression evals, which ask whether it still handles what it used to and should pass nearly everything. Tasks can graduate from one to the other.

Illustrative example. Imagine the shop’s first set has six tasks, each from a real complaint. In one, an order is held at the depot because of an incomplete address. A code-based grader checks that no message was sent to the customer without the owner’s approval, and another confirms that the reply gives no delivery date, because the courier gave none. A model-based grader with a short rubric checks the tone. Another task is a plain status question where the agent should send nothing, which balances the set. Each task runs three times. Tested this way, the Tuesday rewrite’s invented dates would have failed a check at once. All of this is made up.

What else tells you whether an agent is any good?

Evals are one layer. Anthropic lists the others: production monitoring, A/B testing (comparing two versions on real users), user feedback, manual transcript review, and systematic human studies. It lists “false confidence” as a risk of automated evals that do not match real usage, and says production monitoring “catches issues that synthetic evals miss”. It adds: “no single evaluation layer catches every issue”.

Anthropic also says it does not “take eval scores at face value until someone digs into the details of the eval and reads some transcripts”. The third post made the same point about published scores. A team of agents, as in the ninth post, needs the same tests.

For the business side, our guide to measuring AI ROI covers how to judge whether an AI project pays off.

What to remember

An eval is a set of tasks with graders. Start with 20 to 50 tasks from real failures, and run each more than once.
Grade the outcome, such as the final state of a database, more than the wording or the exact path. Use code checks where you can, model judges where you must, and humans to calibrate.
A single pass can mislead. Pass^k asks whether the agent succeeds every time, and no eval replaces watching a live agent.

Next in the series: how to keep an AI agent safe, including prompt injection, permissions and human approval.