Why memory bandwidth matters more than TFLOPS for LLM inference

Your GPU's TFLOPS rating does not decide how fast it generates tokens. Memory bandwidth does. Here's the spec that actually separates A100-era inference from H200-era inference. Key takeaways HBM3e reaches up to 9.6 Gb/s per pin versus HBM2e's 3.2 Gb/s, tripling per-pin signalling speed with the same 1,024-bit interface width. Total GPU memory bandwidth jumps from 2.0 TB/s on the A100 (HBM2e) to 4.8 TB/s on the H200 (HBM3e), a 140% increase, and 8.0 TB/s on the B200. LLM token generation…