That gap points to a weakness in how organizations evaluate databases. Most evaluations measure speed, capacity, uptime and scalability under favorable conditions. In production, traffic comes in bursts, data volumes grow and nodes fail. The real test is whether the database holds a consistent service level when conditions stop cooperating.
That property is predictability. A predictable database delivers the same response time at the 99.9th percentile (p99.9) under three times its expected load, while a node is down and after two years of data growth.
Performance is nonlinear, which is why benchmarks mislead.
Queueing behavior is nonlinear. At lower utilization, added load may have a modest effect on latency. As utilization rises, the curve steepens. Requests arrive faster than they drain; queues build and response times that were single-digit milliseconds become hundreds. The system still works, but it’s no longer fast enough to be useful.
That’s why benchmarks running at comfortable utilization can be misleading. They measure the flat part of the curve. The harder test comes when demand spikes, such as when concert tickets go on sale, the news breaks or market activity floods the same small set of keys.
Capacity added during a surge takes time to provision, join and rebalance, and rebalancing consumes resources. It’s better to ask whether the system can absorb a sudden multiple of normal demand at the promised service level and whether adding hardware raises sustained throughput without degrading response time.
The slowest request defines the experience.
Modern applications rarely make one database call. A fraud check reads customer history, device fingerprints and transaction context before approving a payment, while a recommendation engine may fetch dozens of records before a page renders. The customer experiences the whole interaction, which changes the arithmetic.
Suppose one call in 100 is slow. An interaction that depends on 100 calls has about a 63% chance of encountering at least one slow call. In other words, roughly two out of three user interactions can be affected even when 99% of individual calls are fast. Jeff Dean and Luiz Barroso described this effect in “The Tail at Scale“ in 2013.
The math gets more revealing as the tail improves. If one call in 1,000 is slow, the chance of a 100-call interaction encountering one drops to about 10%. At one in 10,000, it falls to about 1%.
That’s why averages—and even p99—can hide the experience users actually have. For applications with significant fan-out, leaders should ask for p99.9 and p99.99 performance: how the slowest one in 1,000 or 10,000 requests behaves under realistic load. Those tail latencies can determine whether a transaction finishes in time or a customer ends up waiting.
AI raises the cost of variability.
AI workloads are less forecastable by design. Traditional applications follow predetermined execution paths. Agentic systems decide at runtime how much context to retrieve, which tools to invoke and how many steps to take. One request may resolve in a single hop while the next runs 40 sequential retrievals.
That variability makes capacity planning difficult. In a sequential agent loop, tail latency can become end-to-end latency because there’s limited parallelism to absorb a slow step. A slow lookup at step 31 can still produce a poor experience, regardless of the quality of the model’s answer.
Scope compounds the challenge. A data layer selected for a chatbot pilot may be asked six months later to support retrieval, personalization, real-time decisioning and an agent fleet at a significantly higher workload. Decisions made for a small workload can carry over to a much larger one.
Unpredictability has a bill, and someone pays it.
When a database’s behavior is inconsistent, engineering teams often compensate with caches, read replicas and headroom. Each adds cost and operational complexity.
A cache sitting in front of an unpredictable database is a bet on the hit rate. A cold start, a surge on keys that weren’t pre-warmed or an expired hot key can push more traffic to the database. Teams then add cache-warming, tiering and invalidation logic, increasing complexity.
Each added layer slows delivery. A feature change touches several systems, and senior engineers spend more time tuning, diagnosing and explaining tail latency. Alongside the cloud bill is the question of where your best people spend their time. More predictable database behavior can reduce that operational burden, making capacity planning more straightforward and freeing engineering teams to spend more time on the product.
Predictability also comes with trade-offs. Bounding the tail may mean carrying unused headroom, giving up some peak throughput or accepting architectural constraints in exchange for more consistent behavior. Unpredictable performance can create extra costs from excess infrastructure, operational complexity and engineering time.
Here’s what leaders should ask.
Executives don’t need to design the benchmark. They do need to make sure their teams are testing the right things:
• What is our p99.9 at three times the expected load, with production-scale data?
• What happens to that number when a node fails mid-test?
• If we add 50% more nodes, do we get 50% more sustained throughput, and what does the tail do while rebalancing?
• How long does it take for the system to fully recover and rebalance after a failure?
• Are we measuring at the client or the server?
Run these tests before a live surge. Make sure someone is watching customer-experienced latency alongside database health, because the two diverge exactly when it counts.
Make it part of the decision.
Database decisions shape customer experience, infrastructure spend and engineering productivity for years. A strong evaluation should establish how the system behaves as the application grows, traffic gets lumpy and components fail.
When the data layer holds steady through growth, change and routine failures, the business can plan with confidence and the engineering organization can get back to building.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
