Research: The improvement in pre-training efficiency from 2019 to 2025 mainly comes from data

Dwarkesh Patel and Jerry Han published research on September 8, 2026, breaking down the contributions of data and model improvements in pre-training progress from 2019 to 2025. The study was conducted on a smaller scale, focusing on pre-training, corresponding to the publicly released model recipes and publicly available data corpora for each year, and was trained at a computational scale of up to 1e19 FLOPs.

The results show that under the 1e19 FLOPs computational budget, the computational efficiency gain from data improvements is 12.0 times, while model improvements yield 3.7 times, with the data-side gain being approximately 3.24 times that of the model side. The gains from data and model improvements are largely independent and interact very little, with the additive effect of both explaining 88% of the variance in OLMES scores under a linear model.

On the model side, the evolution from GPT-2 to OLMo-2 encompasses optimizers, positional encoding, normalization, activation functions, and initialization, among others. On the data side, the corpus evolved from approximately 9 billion tokens of OpenWebText in 2019 to larger and more finely filtered corpora like UltraFineWeb by 2025.