Watch it learn
During training we kept copies of the model at different ages. The same openings, with the same random draw, completed by each of them.
Loss curve
The loss measures how surprised the model is by the real next token (cross-entropy, in nats). It starts near ln(2,048) ≈ 7.6: the model spreads its bet evenly over every token.
Temperature 0.8, top-k 40, fixed seed: only the weights change from one age to the next.
What the curve shows
The first few hundred steps do most of the work: the model learns which tokens are common, then which ones follow each other. After that, every gain costs far more compute.
A loss of is a “perplexity” of : on average, the model hesitates as if picking among that many equally likely tokens. Much better than 2,048, and very far from a human reader.
Training details are on the How it works page.