2% of production LLM calls failed in March 2026, according to Datadog's own telemetry from thousands of companies running AI in production, and nearly a third of those failures, close to 8.4 million of them in a single month, were rate limit errors rather than anything the model got wrong. That distinction matters more than the headline failure rate. A team that reads "AI is unreliable" and responds by testing a different model, rewriting prompts, or waiting for the next frontier release is fixing a problem the data says usually isn't the one they have. The more common outage is a capacity problem: an API budget sized for a pilot, now taking production traffic.
What Datadog actually measured
This isn't a survey. Datadog's State of AI Engineering 2026 report, first published April 21, 2026 and updated with monthly breakdowns since, is built from anonymized request-level telemetry captured by its LLM Observability product across thousands of customers running LLM calls and agentic workloads in production. That's a different kind of evidence than the self-reported adoption surveys most "state of AI" reports run on: it's what actually happened on the wire, span by span, not what an engineer remembered or estimated afterward when a survey asked.
The report's February 2026 window found 5% of all LLM call spans returned an error, and 60% of those errors traced to exceeded rate limits. By March 2026, the overall error rate had fallen to 2%, but rate limiting still accounted for close to a third of it, nearly 8.4 million individual rate-limit errors across the dataset that month. The failure rate improved. The reason for failure didn't move.
The failure rate is dropping. The cause behind it isn't
Halving an error rate sounds like a story about better models or better prompting. Read against Datadog's own numbers, it reads more like a story about growing request volume against fixed quota: the same companies are asking their AI providers for more, more often, and running into the same capacity ceiling at a larger absolute scale even as the percentage improves. Token usage per request more than doubled for the median customer year over year, and quadrupled for the 90th-percentile power users, per the same report. A quota sized for last year's traffic doesn't survive that kind of growth, and a rate-limit error looks identical to a hallucination in a support ticket that just says "the AI feature broke." It doesn't look identical in an incident review, where a 429 status code and a retry-after header are a completely different fix than a prompt rewrite. Teams that route every AI complaint straight to a prompt engineer are solving the wrong ticket a meaningful share of the time.
Why more than 70% of teams now run three or more models
The same report found OpenAI's share of LLM traffic at 63% in its dataset, down from 75% a year earlier, while Google's Gemini and Anthropic's Claude each gained 20 and 23 percentage points of adoption over the same window. More than 70% of organizations now use three or more models in production, and the share running six or more nearly doubled year over year. The obvious read is cost and capability shopping. The read that matters for reliability is different: a second and third provider is also a failover path. A single-provider setup has no answer to that provider's rate limit except to wait or fail the request. A multi-provider setup can shift a burst of traffic sideways the moment one provider's quota gets tight, turning what would be a user-facing error into an invisible reroute.
The architecture pattern that turns a rate limit into an outage
Most teams aren't built to do that shifting, because most AI features aren't built with anywhere to shift to. Datadog's telemetry found 59% of agentic application requests made exactly one service call end to end, and only 18% made three or more. A single-call agent has one shot: it calls a provider, and either that call succeeds or the request fails outright, with no intermediate step where a retry against a different model or provider could absorb the failure. A multi-step architecture, by contrast, has natural seams where a fallback provider, a cached response, or a degraded but real answer can sit. The same report found only 28% of LLM call spans showed any cached-read input tokens, meaning most calls reprocess a full prompt every time, and that system prompts alone made up 69% of all input tokens in the traces Datadog analyzed. Caching a system prompt that doesn't change between calls is close to free capacity: fewer tokens spent per request against the same rate limit, which is the same ceiling causing most of the 8.4 million failures in the first place.
A worked example
Picture a 30-person Series A company that shipped an AI copilot feature behind its core product this year, tested it for weeks against a few hundred internal users, and watched it work close to flawlessly. Launch day brings a spike to several thousand concurrent users, and the error rate climbs within the first hour. The on-call engineer's first instinct is to check whether the model regressed or whether a recent prompt change introduced a bug, because that's the failure mode everyone on the team has a mental model for. Nothing there. What actually happened: token volume per request had grown through weeks of feature additions nobody re-benchmarked against the account's quota, the app called a single provider with no fallback, and launch traffic pushed it past a per-minute cap the team had never had reason to test. The fix wasn't a smarter model. It was a rate-limit alert wired in before launch, a second provider configured as a fallback, and prompt caching turned on for the system prompt that hadn't changed in months.
What this changes about how you plan a launch
Treat AI request reliability as its own line item, separate from whether the model's output is any good, and budget for it before a launch rather than during the incident that follows one. That means load-testing against actual provider rate limits at the traffic a launch is expected to bring, not the traffic a pilot had; wiring an alert on rate-limit errors specifically, so they don't get lumped into a generic "AI errors" dashboard where they're indistinguishable from an output-quality problem; and treating a second model provider as failover infrastructure, not just a cost comparison exercise to run once a quarter. None of that is where the fix belongs if the actual complaint is about output quality, wrong answers, hallucinated citations, a tone that's off. That's an evaluation problem, and building the eval harness that catches it is a different project with a different owner. Confusing the two is exactly how a rate-limit outage turns into a monthslong argument about which model to switch to.
If your team is past the pilot and staring down a launch that needs its AI feature to hold up at real traffic, our Silicon Valley team can scope a production-readiness review against your actual provider setup, retry logic and quota headroom, distinct from the evaluation work that follows once the infrastructure holds. Get in touch and we'll tell you honestly which of the two you need first.
Sources
- Datadog: State of AI Engineering 2026 (published April 21, 2026, with monthly data through March 2026)
- Datadog: AI Is Hitting Operational Limits as Companies Rush to Scale (press release, April 21, 2026)