GPT-2

GPT-2 was developed by OpenAI and introduced in February 2019 as a large-scale unsupervised language model trained to predict the next word in a sequence. It was trained on a dataset called WebText, comprising roughly 40 gigabytes of text scraped from outbound links on Reddit that had received at least three upvotes — approximately 8 million documents. The model demonstrated a striking ability to generate coherent, extended passages of text from a short prompt, a capability that surprised even its creators.

The full GPT-2 model contained 1.5 billion parameters, making it one of the largest language models publicly discussed at the time. It used the transformer decoder architecture introduced by Vaswani et al. in 2017, stacking 48 layers with 1,600-dimensional embeddings and a context window of 1,024 tokens. OpenAI released the model in stages — first a 117 million parameter version in February, then progressively larger versions — withholding the full 1.5B model until November 2019, citing concerns about potential misuse for generating disinformation at scale.

The staged release was itself a landmark event in AI policy discourse. OpenAI published a blog post explaining their reasoning, framing GPT-2 as a model whose text-generation quality was high enough to warrant caution before broad public access. This decision was controversial: many researchers argued the caution was overstated or served primarily as publicity, while others praised it as a responsible precedent. By the time the full model was released in November 2019, independent researchers and other organizations had already replicated comparable capabilities, and no significant misuse tied directly to the staged release was documented.

GPT-2 established that scaling transformer language models on large, diverse internet text could yield qualitatively impressive generative capabilities without task-specific training. This finding directly motivated the subsequent scaling efforts that produced GPT-3 and the broader class of large language models that now underpin products used by hundreds of millions of people. The model demonstrated zero-shot and few-shot task performance on reading comprehension, summarization, and translation benchmarks — not by design, but as emergent properties of scale.

Beyond technical impact, GPT-2 introduced a new dimension to AI development: the public deliberation over whether and how to release powerful models. The staged release sparked ongoing debates about responsible disclosure, dual-use risk, and the obligations of AI laboratories — conversations that have only intensified with subsequent systems. GPT-2 was, in this sense, the first high-profile case study in what the field now calls 'responsible scaling,' establishing a template for the difficult tradeoffs that define modern AI governance.