Attention in the y = abx Toy Model
Learning dynamics of a scalar linear attention layer on in-context regression
Badges come from GitHub once the author claims this feed.
Learning dynamics of a scalar linear attention layer on in-context regression
Training transformer baselines at 30M, 60M, and 125M to measure how much random seed alone moves final validation loss.