I Was Measuring LLM Latency Wrong

For a long time, I recorded one latency number for every LLM request: latency_ms = response_finished_at - request_started_at It looked useful. I could calculate an average, add a p95 chart, and see whether a model was getting slower. But whenever a user complained that an AI feature felt slow, that number rarely told me what to fix. Sometimes the request waited in a queue before reaching the provider. Sometimes the first token was slow, even though the rest of the response streamed quickly.…