Latency Percentiles (p50/p95/p99)

Latency percentiles — p50, p95, p99 — describe the response time that a given percentage of requests fall at or under: p50 is the median request, p95 means 95 percent of requests were at least that fast, and they matter because an average response time hides exactly the slow outliers that make a product feel broken to real users.

A mean latency can look perfectly healthy while a meaningful slice of actual requests sit through waits several times longer than typical. That's not a flaw in the math — it's what averaging does by design, and it's precisely why percentiles, not means, are the standard way latency gets measured, reported, and put into service-level agreements.

How it works

Percentiles are computed on real, sorted samples, not modeled from an assumed distribution shape. Sort every measured latency from fastest to slowest, then read off the value at the desired rank — for the 95th percentile of 20 samples, that's the 19th-fastest value (the nearest-rank method). This matters because latency distributions are never symmetric: they have a long tail of unusually slow requests caused by cold starts, garbage collection pauses, retries, or network hiccups, and that tail is exactly what a percentile exposes and a mean smooths over.

Why it matters

An LLM product can have a fine-looking mean latency and a terrible p99 at the same time, and users remember the terrible experiences, not the typical ones. That's why serious systems set alerting thresholds and SLAs on percentiles rather than averages — a mean can stay flat while p99 quietly triples, and nobody would notice from the average alone until enough users complained.

A worked example

Twenty real request latencies, mostly clustered around 118–131ms, include three slow outliers at 600ms, 650ms, and 900ms. The mean across all twenty is 213.2ms — worse than every single one of the seventeen normal requests, dragged upward by the three outliers, and a poor description of what a typical request actually felt like. The percentiles tell a clearer story: p50 is 125ms (the honest typical case), p95 is 650ms, and p99 is 900ms — the numbers that describe what the worst-treated slice of users actually experienced, which the mean alone never surfaced.

How Miatz teaches it

Learners compute p50/p95/p99 by hand from a raw list of sample latencies before ever using a monitoring dashboard that does it for them — sorting the list, applying the nearest-rank formula, and comparing their answer to the mean of the same data, so the gap between the two is something they've derived rather than been told. The systems track then has them instrument a real API call and watch their own hand-computed percentiles show up in production tooling, closing the loop between the arithmetic and the dashboard.