AI Still Can’t Beat the On-Call Engineer: Here’s Why

AI companies keep pitching autonomous site reliability engineer agents—AI that investigates production incidents in place of humans. Datadog ran the actual benchmark on real outages, and the best AI models can’t yet beat the engineers they’re supposed to replace.

The benchmark is ARFBench (Anomaly Reasoning Framework Benchmark), a joint project from Datadog and Carnegie Mellon. Built from 63 real production incidents, extracted from engineers’ own Slack threads during live emergencies—750 multiple-choice questions covering 142 monitoring metrics and 5.38 million data points, every question verified by hand. No synthetic data. No textbook scenarios.

“Trillions of dollars are lost each year due to system outages,” the researchers write. The benchmark tests whether AI can actually help change that.

“Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice,” the paper reads.

Questions come in three tiers. Tier I: Does an anomaly exist in this chart? Tier II: When did it start, how severe is it, what type?

The Tier III—the hardest—requires cross-metric reasoning: Is this chart causing the problem in that other chart? That’s where AI falls apart. GPT-5 scores just 47.5% F1 on Tier III questions, a metric that penalizes models for gaming answers by picking the most common class.

“Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice,” the researchers write.

How every model stacked up

GPT-5 led all existing models at 62.7% accuracy—on a test where random guessing gets 24.5%. Gemini 3 Pro scored 58.1%. Claude Opus 4.6: 54.8%. Claude Sonnet 4.5: 47.2%.

Domain experts scored 72.7% accuracy. Non-domain experts—time series researchers at Datadog without extensive observability experience—still hit 69.7%.

No AI model beat either human baseline.

The model that actually topped the full leaderboard was Datadog’s own hybrid: Toto—their internal time series forecasting model—combined with Qwen3-VL 32B. Toto-1.0-QA-Experimental scored 63.9% accuracy, edging past GPT-5 while using a fraction of its parameters. On anomaly identification specifically, it outperformed every other model by at least 8.8 percentage points in F1.

A purpose-built domain model, trained on observability data, outperforming a frontier general-purpose system at this specific task is the expected outcome. That’s the point.

The most valuable finding isn’t which model scored highest.

“We observe substantially different error profiles between leading models and human experts, suggesting that their strengths are complementary,” the researchers write. Models hallucinate, miss metadata, and lose domain context. Humans misread precise timestamps and occasionally fail on complex instructions. The mistakes barely overlap.

Model a theoretical “Model-Expert Oracle”—a perfect judge that always picks the right answer between the AI and the human—and you get 87.2% accuracy and 82.8% F1. Way above either alone.

That’s not a product. It’s a documented target—built from real emergencies, not curated datasets—that quantifies exactly how much better human-AI collaboration could perform. The leaderboard is live on Hugging Face. GPT-5 sits at 62.7%. The ceiling is 87.2%.