LLM Inference Latency: Why Your 7B Model Gets 15 tok/s on a T4 but 3,500 tok/s on an H100

`NVIDIA's spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster. So a 7-billion-parameter LLM should be 15× faster on an H100, right? It's 150× faster. The T4 struggles at ~15 tok/s . The H100 cruises at ~2,200 tok/s — and with continuous batching, north of 3,500 . The 15× TFLOPS gap doesn't explain the 150× throughput gap. The missing variable is the one thing NVIDIA's marketing pages bury on line three of the…