Gradient checkpointing: keep N activations, recompute the rest — memory falls from O(N) to O( N) for one extra forward pass

1Key Takeaways

• Train a deep network and the thing that blows up your VRAM usually isn't the weights — it's the activations .
• Backprop needs every layer's forward output to compute that layer's gradient, so the naive training step stores all of them, and activation memory grows as O(N) with depth (and with batch size, and with sequence length).
• Gradient checkpointing trades a little compute for a lot of that memory: keep only a handful of activations and recompute the rest on demand during the backward pass.….