A look at Aurora optimizer's hyperparameters — learning rate, weight decay, and pp_beta — across 14M / 50M / 100M / 300M decoder-only transformers, looking for better loss.

This is a first of many (hopefully) series of writings on a wide variety of topics. I will try to release as consistently, and with the upmost highest quality, although I can't promise the latter.