As AI agents move from research demos to production deployments, one question has become impossible to ignore: how do you actually know if an agent is good? Perplexity scores and MMLU leaderboard numbers tell you very little about whether a model can navigate a real website, resolve a GitHub issue, or reliably handle a customer service workflow across hundreds of interactions. The field has responded with a wave of agentic benchmarks — but not all of them are equally meaningful.
One important caveat before diving in: agent benchmark scores are highly scaffold-dependent. The model, prompt design, tool access, retry budget, execution environment, and evaluator version can all materially change reported scores. No number should be read in isolation, context about how it was produced matters as much as the number itself.
With that in mind, here are seven benchmarks that have emerged as genuine signals of agentic capability, explaining what each one tests, why it matters, and where notable results currently stand.
1. SWE-bench Verified
🔗 Leaderboard & details: swebench.com
What it tests: Real-world software engineering. SWE-bench evaluates LLMs and AI agents on their ability to resolve real-world software engineering issues, drawing from 2,294 problems sourced from GitHub issues across 12 popular Python repositories. The agent must produce a working patch — not a description of a fix, but actual code that passes unit tests. The Verified subset is a human-validated collection of 500 high-quality samples developed in collaboration with OpenAI and professional software engineers, and is the version most commonly cited in frontier model evaluations today.
Why it matters: The benchmark’s trajectory makes it one of the most reliable long-run progress trackers in the field. When it launched in 2023, Claude 2 could resolve only 1.96% of issues. In vendor-reported late-2025 and early-2026 results, top frontier models crossed the 80% range on SWE-bench Verified — though exact scores vary meaningfully by scaffold, effort setting, tool setup, and evaluator protocol, and should not be compared directly across vendors without accounting for those differences. A consistent pattern has emerged: closed-source models tend to outperform open-source ones, and performance is heavily shaped by the agent harness as much as the underlying model.
One caveat worth flagging: high SWE-bench scores do not guarantee a general-purpose agent. They indicate strength in software repair tasks specifically — not universal autonomy — which is precisely why it must be used alongside the other benchmarks in this list.
2. GAIA
🔗 Leaderboard & details: huggingface.co/spaces/gaia-benchmark/leaderboard
What it tests: General-purpose assistant capabilities that require multi-step reasoning, web browsing, tool use, and basic multimodal understanding. GAIA tasks are deceptively simple in phrasing but require a chain of non-trivial operations to complete correctly — the kind of compound task a real assistant would face in the wild.
Why it matters: GAIA is widely referenced in agent evaluation research and maintains an active Hugging Face leaderboard where teams across the community submit results. Its design resists shortcut-taking: an agent cannot guess its way through. It has become one of the standard suites for exposing tool-use brittleness and reproducibility gaps in real agent evaluations — surfacing failure modes that narrower benchmarks miss entirely. For teams evaluating general-purpose assistants rather than task-specific agents, GAIA remains one of the most honest signal generators available.
3. WebArena
🔗 Leaderboard & details: webarena.dev
What it tests: Autonomous web navigation in realistic, functional environments. WebArena creates websites across four domains — e-commerce, social forums, collaborative software development, and content management — with real functionality and data that mirrors their real-world equivalents. Agents must interpret high-level natural language commands and execute them entirely through a live browser interface. The benchmark consists of 812 long-horizon tasks, and the original paper’s best GPT-4-based agent achieved only 14.41% end-to-end task success, against a human baseline of 78.24%.
Why it matters: Progress on WebArena has been substantial. By early 2025, specialized systems were reporting single-agent task completion rates above 60% — IBM’s CUGA system reached 61.7% on the full benchmark (February 2025), and OpenAI’s Computer-Using Agent achieved 58.1% in its January 2025 technical report. These gains reflect a broader pattern in stronger web agents: explicit planning, specialized action execution, memory or state tracking, reflection, and task-specific training or evaluation loops. The remaining gap to human performance — 78.24% per the original paper — reflects harder unsolved problems like deep visual understanding and common-sense reasoning. WebArena is one of the most widely used benchmarks for testing true web autonomy, not scripted automation.
4. τ-bench (Tau-bench)
🔗 Leaderboard & code: github.com/sierra-research/tau-bench
What it tests: Tool-agent-user interaction under real-world policy constraints. τ-bench emulates dynamic, multi-turn conversations between a simulated user and a language agent equipped with domain-specific API tools and policy guidelines. The benchmark covers two domains — τ-retail and τ-airline — and simultaneously evaluates three things: whether the agent can gather required information from a user across multiple exchanges, whether it correctly follows domain-specific policy rules (e.g., rejecting non-refundable ticket changes), and whether it behaves consistently at scale via the pass^k reliability metric.
Why it matters: τ-bench exposes a reliability crisis that most one-shot benchmarks are completely blind to. Even state-of-the-art function calling agents like GPT-4o succeed on fewer than 50% of tasks, and their consistency is far worse — pass^8 falls below 25% in the retail domain. That means an agent that can handle a task in one trial cannot reliably handle the same task eight times in a row. For any real deployment handling millions of interactions, that inconsistency is disqualifying. By combining reasoning, tool-use, policy adherence, and repeatability into a single evaluation framework, τ-bench fills a gap that outcome-only benchmarks leave wide open.
5. ARC-AGI-2
🔗 Leaderboard & competition: arcprize.org/leaderboard
What it tests: Fluid intelligence — the ability to generalize to genuinely novel visual reasoning puzzles that resist memorization or pattern-matching from training data. Each task presents the agent with a small number of input-output grid examples and asks it to infer the underlying abstract rule, then apply it to a new input. Created by François Chollet, the benchmark is the centerpiece of the ARC Prize competition.
Why it matters: Context is essential here. ARC-AGI-1 has been effectively saturated: by 2025, frontier models reached 90%+ through brute-force engineering and benchmark-specific training. ARC-AGI-2, released in March 2025, is the current and substantially harder version designed to close those loopholes. The ARC Prize 2025 Kaggle competition attracted 1,455 teams, with the top competition score reaching 24% using NVIDIA’s NVARC system — a specialized synthetic data generation and test-time training approach on a 4B parameter model. Among commercial frontier models, the score landscape has evolved quickly: GPT-5.2 reached 52.9%, Claude Opus 4.6 reached 68.8%, and Gemini 3.1 Pro achieved a verified score of 77.1% following its February 2026 release — more than double the performance of its predecessor Gemini 3 Pro (31.1%). These results show rapid progress on ARC-AGI-2, but human comparison should be interpreted carefully: the ARC Prize 2025 technical report states that ARC-AGI-2 tasks were validated as solvable by independent non-expert human testers, rather than presenting a single fixed “human baseline” percentage.
The benchmark’s hardest moment came with ARC-AGI-3, launched in March 2026 with an interactive video game format requiring agents to explore novel environments, infer goals, and plan action sequences without explicit instructions. The ARC-AGI-3 technical report states directly: humans can solve 100% of the environments, while frontier AI systems as of March 2026 score below 1%. That result is not a flaw in the benchmark — it is the point. Four major AI labs — Anthropic, Google DeepMind, OpenAI, and xAI — have established ARC-AGI as a standard benchmark on their public model cards, making it the field’s clearest North Star for tracking genuine generalization progress.
6. OSWorld
🔗 Leaderboard & code: os-world.github.io
What it tests: Cross-application computer use on real operating systems. OSWorld provides 369 computer tasks spanning real web and desktop applications, OS file I/O, and cross-app workflows across Ubuntu, Windows, and macOS. Agents must interact through actual GUI interfaces using raw keyboard and mouse control — not through clean APIs or text-only channels. Each task includes a custom execution-based evaluation script for reliable, reproducible scoring.
Why it matters: Most agentic benchmarks operate in text-only or API-only environments. OSWorld tests whether a model can actually operate a computer, making it uniquely relevant for computer-use agents being deployed in enterprise and productivity workflows. At the time of its original publication at NeurIPS 2024, humans could accomplish over 72.36% of tasks, while the best model achieved only 12.24% — a stark and revealing gap. The benchmark has since been upgraded to OSWorld-Verified, which addresses over 300 reported issues and improves evaluation reliability through enhanced infrastructure, fixed web environment changes, and improved task quality. The multimodal demands — combining visual grounding, operational knowledge, and multi-step planning across real operating systems — make OSWorld significantly harder than code-only evaluations.
7. AgentBench
🔗 Code & details: github.com/THUDM/AgentBench
What it tests: Breadth. AgentBench evaluates LLMs as agents across eight distinct environments: OS interaction, database querying, knowledge graph navigation, digital card games, lateral-thinking puzzles, household task planning, web shopping, and web browsing. Rather than going deep on one task domain, it assesses how well a model generalizes across fundamentally different agentic settings within a single evaluation framework.
Why it matters: A model that scores impressively on SWE-bench may completely collapse in a database query environment or a web navigation task. AgentBench is best used to compare agent architectures and identify where capability transfer breaks down — not to predict production performance directly. That cross-domain diagnostic view is valuable signal especially when selecting a base model for a multi-purpose agent system or when diagnosing which environment types expose a specific model’s weaknesses. No other benchmark in this list offers this kind of breadth-first diagnostic view in a single run.
Conclusion
No single benchmark tells the full story. SWE-bench Verified measures software engineering competence with real GitHub issues; GAIA tests compound tool-use and multi-step reasoning across domains; WebArena evaluates true web autonomy with 812 long-horizon tasks; τ-bench surfaces the reliability crisis that one-shot benchmarks miss entirely; ARC-AGI-2 probes genuine generalization and fluid intelligence — with ARC-AGI-3 showing the frontier hasn’t come close to solving it; OSWorld evaluates full-stack computer control across real operating systems; and AgentBench diagnoses breadth across eight fundamentally different environments. Used together, and interpreted with awareness of scaffold dependencies, these seven provide the most honest picture currently available of where an agent actually stands.
As agentic systems move deeper into production, the teams that understand these distinctions — and evaluate against all of them — will build more reliably, and report capabilities more honestly.
Key Takeaways:
SWE-bench Verified tracks the most dramatic progress curve in AI: from 1.96% (Claude 2, 2023) to above 80% in vendor-reported late-2025/early-2026 results — but scores are not directly comparable across vendors due to scaffold, tool, and evaluator differences
τ-bench reveals a reliability crisis most benchmarks ignore: even top models score below 50% success and fall under pass^8 of 25% on the same retail tasks
ARC-AGI-1 is saturated at 90%+; ARC-AGI-2 is the current test, with Gemini 3.1 Pro leading at 77.1% (verified, Feb 2026); ARC-AGI-3 launched March 2026 and all frontier systems score below 1%
WebArena has seen major progress — from 14.41% baseline to 61.7% (IBM CUGA) by early 2025 — driven by modular Planner-Executor-Memory architectures, not a single model breakthrough
OSWorld is the most rigorous test of real computer use: 369 cross-app tasks with a 60-point gap between human and AI performance at launch
GAIA is widely referenced in agent evaluation research and maintains an active community leaderboard on Hugging Face
Agent benchmark scores are highly scaffold-dependent — model, tool access, retry budget, and evaluator version all materially affect reported numbers
Be the first to comment