The Foundation Era · Research · 2017
Transformer
By replacing recurrence entirely with self-attention, the Transformer became the universal engine of modern AI.
Before the Transformer, sequence modeling was dominated by recurrent neural networks (RNNs) and their variants, including LSTMs and GRUs. These architectures processed tokens one at a time in sequence, making parallelization during training difficult and causing gradients to vanish or explode over long dependencies. By 2017, despite years of engineering refinement, RNNs remained fundamentally bottlenecked by their sequential nature, limiting both training speed and the ability to capture relationships between distant tokens.
The Transformer did not merely improve on existing methods — it replaced the dominant computational paradigm for sequence modeling wholesale. Its scalability turned out to be its most consequential property: as researchers scaled Transformers with more data and more parameters, performance improved in ways that were not anticipated at the time of publication. This scaling behavior, later formalized in papers on neural scaling laws, underpins every large language model deployed today, from GPT-4 to Claude to Gemini.