fp16 doesn't just lose precision, it loses range — gradients underflow to zero, and loss scaling borrows exponent headroom

Modern GPUs run 16-bit matmuls on tensor cores roughly twice as fast as fp32, and a 16-bit tensor is half the bytes. So training in fp16 / bf16 nearly halves activation memory and doubles throughput — for free, it seems. Except a 16-bit float doesn't only lose precision. fp16 loses range , and that's the failure that makes naive 16-bit training NaN within a few steps. I built a live tool that lays out the formats bit-by-bit and rescues underflowing gradients with a slider; here's the whole…