The Scale Era · Research · 2020
GPT-3
At 175 billion parameters, GPT-3 demonstrated that scaling a language model could unlock emergent capabilities no one had explicitly trained for.
When OpenAI released the GPT-3 paper in May 2020, it arrived in a field already transformed by its predecessors. GPT-2 (2019) had shown that large autoregressive language models could produce surprisingly coherent text, but it remained clearly machine-generated to careful readers. GPT-3 changed the subjective experience of interacting with a language model. Trained on roughly 45 terabytes of filtered text from the internet, books, and Wikipedia, it dwarfed every public model that came before it. Its 175 billion parameters were more than 100 times those of GPT-2.
GPT-3 established the empirical case for the scaling hypothesis — the idea that increasing model size and training data, without architectural reinvention, could yield qualitative jumps in capability. This reoriented research priorities across the entire field. Labs that had focused on clever architectural improvements began devoting resources to simply training larger models. The paper's few-shot learning results suggested that a sufficiently large model could perform new tasks from natural language instructions alone, foreshadowing the instruction-tuning and prompt-engineering paradigms that followed.