One Gradient Spike, Six Batches to NaN: Debugging a Deterministic Neural-Network Training Failure

The first visible NaN appeared in the final linear layer at batch 46. By then, the model was already effectively destroyed. Six batches earlier, every tensor was still finite. The loss was finite. The gradients were finite. The momentum buffers were finite. Nothing had crossed the clean boundary between a valid floating-point value and NaN. But the training state was already in a runaway regime. This is how an aggregate-analysis failure led me backward—from a corrupted checkpoint, to one…