When the arithmetic becomes the dynamics
Preview note.
A training algorithm is implemented in finite precision. Small numerical effects can change the dynamics we observe, so the behavior of a real computation can differ from its idealized equations.
This question motivates our work on Slingshot loss spikes. We study how finite-precision errors can disrupt gradient balance and drive rapid parameter growth during long-term training.