What Can AI Agents Do Today, and Where Do They Still Fail?

Imagine it is Monday morning again at the Lagos fabric shop from the first post, and order 1042 has still not arrived. Before handing the problem to an AI agent, the owner reads two headlines. One says agents can now fix real software bugs. The other says they fail many customer-service tasks. Both can be true, and this post shows why.

(This is an illustrative scene, not a real business.)

What can AI agents do well today?

AI agents do best on tasks where the steps are within reach, the tools are well defined, and the result can be checked. If you want a refresher on what an agent is, see What Is an AI Agent, and How Is It Different from a Chatbot?. The loop it follows is covered in How Does an AI Agent Actually Work, Step by Step?.

Anthropic’s December 2024 post “Building effective agents” names two promising areas: customer support and coding. It says agents add the most value for tasks that “require both conversation and action, have clear success criteria, enable feedback loops, and integrate meaningful human oversight”. On coding, it says “Code solutions are verifiable through automated tests”. Anthropic has since added a note to that page saying “Much of the tooling landscape described in this post has changed since December 2024.” We therefore use only its examples of where agents help, not its advice on tools.

Vendors describe what their coding agents are built to do. As one example, Anthropic’s documentation for its coding tool Claude Code says, as of October 2026, that it “reads your codebase, edits files, runs commands, and integrates with your development tools”. That is a product description, not independent proof of how well it performs. For that we need tests.

How do we measure what agents can do?

We use benchmarks. A benchmark is a fixed set of test tasks with a scoring rule, so different systems can be compared on the same work. A leaderboard is a public table of the scores that teams have submitted. This post uses two benchmarks: SWE-bench, which tests coding, and tau-bench (pronounced “tau bench”), which tests customer service.

How well do coding agents do on SWE-bench as of October 2026?

On the SWE-bench leaderboard, as of 12 October 2026, the best agents resolve a large share of the test problems. SWE-bench was introduced in a 2023 research paper. It contains 2,294 problems taken from real GitHub issues, which are written bug reports or feature requests, across 12 popular Python projects. The system is given the issue and the code, and must produce a fix.

A problem counts as resolved only if the fix passes the tests. Each problem has at least one test that failed before the fix and must pass after it. The paper says a median of 51 other tests also run, to check that existing features still work.

The paper’s best result was Claude 2 resolving “a mere 1.96%” of the problems. The official leaderboard at swebench.com now tells a different story. As of 12 October 2026:

• The “Full” board, the same 2,294 problems, has a top entry of 52.62%, dated 19 December 2025.
• The “Verified” board is a 500-problem subset that the site describes as human-filtered. Its default “Bash Only” view runs every model in the same simple agent setup, and the top entry there is 76.8%, dated 17 February 2026.
• Across all submitted agents on Verified, the top score is 79.2%, dated December 2025.

The newest result anywhere on the page is from February 2026. The scores are those the leaderboard reports, and we have not re-run any of them. The SWE-bench team also marks which entries it ran or checked itself, and the 52.62%, 76.8% and 79.2% entries are not marked.

In our reading, do not subtract 1.96 from 76.8 and call it progress. The 2023 setup gave a model some retrieved files and asked for a fix in one go, with no agent loop. And Verified is a different, smaller set of problems. What the numbers do show is a large move on a test where passing tests is the measure. Anthropic’s post adds a caution: “human review remains crucial for ensuring solutions align with broader system requirements.”

Where do AI agents still fail?

In our reading, they are weakest on long tasks with rules, and above all on consistency. The tau-bench paper (June 2024) tests an agent in two settings, retail and airline customer service. A language model plays the customer, and the agent has tools for the company’s database and a written policy, the company’s rules. The score compares the database at the end of the conversation with the correct end state.

The paper reports that “even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)”. Function calling is the way a model asks for a tool to be run, which the next posts explain. In the paper’s tables the agent managed about 61% in retail and about 35% in airline on a single try.

The consistency number comes from the paper’s metric. Pass^k is “the chance that all k i.i.d. task trials are successful, averaged across tasks”, where “i.i.d.” means repeated independent tries. Pass^1 asks whether one try works. Pass^8 asks whether eight tries on the same task all work.

Newer results on tau-bench’s own leaderboard show the same shape. As of 12 October 2026, the top entry on its banking board, a text benchmark where agents retrieve information from a knowledge base, scores 55.2% on pass^1 and 35.1% on pass^4, with four tries per task. An entry from March 2026 (Alibaba Cloud’s Qwen3.5-397B-A17B) scores 84.4% on retail pass^1 and 59.7% on pass^4. The top banking entry is Qwen 3.8 Max, from Qwen. These are not like for like with the 2024 paper, as the last section explains, but the gap between “works once” and “works every time” stayed wide.

What do those failures look like in practice?

In a sample of 115 runs of its best retail agent, the paper found 36 failures that were the agent’s own fault, and sorted them into three kinds. The paper gives percentages; the counts below are our conversion of its percentages.

• Wrong input or information (20 of the 36). The agent picks the right tool but fills in an input wrongly, or gives the customer wrong or missing information. An example of the first is choosing the wrong product option when exchanging a lamp.
• Wrong decision (9 of the 36). The agent breaks a rule. In one task the policy said the exchange tool could be used only once, but the agent exchanged one item first, so the second never got exchanged.
• Half-finished request (7 of the 36). The user asked for several things and the agent did only some. In Task 42 the user wanted all orders checked for wrong addresses, and only one was fixed.

Illustrative example. Imagine the shop’s agent on that Monday morning. Order 1042 shipped last week and is held at the depot because the delivery address is incomplete. The tool names are made up. Each failure kind could look like this:

• Wrong input: it calls lookup_order(1024), a typo away from the right order, and reports on a stranger’s parcel.
• Wrong decision: the shop’s rule says the owner approves refunds, yet the agent offers one to the customer.
• Half-finished request: the customer says two parcels have address problems, and the agent fixes one.

In our reading, none of these is exotic. Each is a small slip, and a slip once in a while is what makes pass^k low.

Why do benchmark numbers need care?

A score is only meaningful next to the test that produced it. Keep four points in mind:

• Different tests share a name. SWE-bench Full and SWE-bench Verified are different problem sets.
• Tests get revised. In February 2026 the tau-bench team wrote that it had “audited and fixed 50+ tasks across the airline and retail domains”, correcting incorrect expected actions, ambiguous instructions, and impossible constraints. Scores from before and after are not like for like, and neither are the 2024 paper’s numbers and the 2026 leaderboard’s.
• The setup matters. Who plays the customer, how the agent is prompted, and how many tries are allowed all change the result.
• A benchmark is not your business. Your data, rules, and customers are different.

If you are weighing whether an agent is worth trying, our guide on how to measure AI ROI covers how to judge the return on an AI project.

What should you expect from an AI agent today?

In our reading, expect a capable assistant that needs checking. Choose tasks where the result can be verified, such as tests passing or a record updated correctly, and keep a person in charge of anything risky.

OpenAI’s practical guide says “Actions that are sensitive, irreversible, or have high stakes should trigger human oversight until confidence in the agent’s reliability grows”, and gives “canceling user orders, authorizing large refunds, or making payments” as examples. For coding, Anthropic’s Claude Code security documentation says, as of October 2026, “You’re responsible for reviewing proposed code and commands for safety before approval.”

What to remember

• Agents do well when the task is bounded and the result can be checked, and coding with tests is the clearest case.
• They are far less consistent on long, rule-heavy tasks. Pass^k measures that gap, and it is wide.
• Read benchmark numbers with their date, test, and setup, and keep a person approving risky actions.

Next in the series: how to decide whether you need an AI agent at all, or whether a simple automation is enough.