Publications

2026

1. NeurIPS

   

   Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes

   Hanqing Liu, Jianjun Cao, Yuanze Li, and Zijian Zhou

   2026

   Accepted to NeurIPS 2026

   Abs arXiv Bib PDF

   Finite-precision errors can break the zero-sum constraint of cross-entropy gradients. The resulting drift creates a feedback loop between classifier and feature means, termed Numerical Feature Inflation. This offers a numerical explanation for rapid norm growth and Slingshot loss spikes, while also predicting that parameter growth can occur without visible spikes.

   @misc{liu2026grokking,
     title = {Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes},
     author = {Liu, Hanqing and Cao, Jianjun and Li, Yuanze and Zhou, Zijian},
     year = {2026},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
     url = {https://arxiv.org/abs/2605.06152},
     note = {Accepted to NeurIPS 2026},
   }