BERT

Before BERT, the dominant paradigm for natural language processing relied on unidirectional language models — systems that read text either left-to-right or right-to-left, but never both at once during pretraining. Word embeddings like Word2Vec and GloVe captured static word meanings, while models like ELMo introduced context-sensitive representations by concatenating outputs from two separately trained unidirectional LSTMs. These approaches left a fundamental gap: no single representation could attend to both left and right context simultaneously when building an understanding of a word or sentence.

BERT, which stands for Bidirectional Encoder Representations from Transformers, was introduced by researchers at Google AI Language in October 2018. It addressed the bidirectionality problem by adapting the Transformer encoder architecture — originally described in the 2017 'Attention Is All You Need' paper — and pretraining it on two novel self-supervised objectives. The first, Masked Language Modeling (MLM), randomly masks 15% of input tokens and trains the model to predict them using surrounding context from both directions. The second, Next Sentence Prediction (NSP), trains the model to determine whether two sentences are consecutive in the original text, equipping it with inter-sentence reasoning ability.

BERT was pretrained on the BooksCorpus (800 million words) and English Wikipedia (2.5 billion words) using significant computational resources, then fine-tuned on specific downstream tasks with minimal architectural modification. Upon release, BERT-Large achieved state-of-the-art results on 11 NLP benchmarks simultaneously, including the Stanford Question Answering Dataset (SQuAD 1.1 and 2.0) and the General Language Understanding Evaluation (GLUE) benchmark. Its release triggered an immediate and sweeping transformation of how the NLP research community approached transfer learning, making large-scale pretraining followed by task-specific fine-tuning the new standard methodology.

BERT established pretraining-then-fine-tuning as the dominant paradigm for NLP in a way that no prior model had achieved at scale. It demonstrated conclusively that a single, general-purpose language representation — trained on raw, unlabeled text — could be adapted to diverse tasks ranging from question answering and sentiment analysis to named entity recognition and textual inference, often surpassing systems that had been carefully engineered for those specific tasks over years. This shifted the field's center of gravity away from task-specific architectures and toward foundation models built on massive unsupervised pretraining.

The downstream consequences of BERT were enormous and rapid. Google confirmed in 2019 that BERT had been integrated into Google Search, affecting nearly every English-language query processed. It spawned a generation of successor models — RoBERTa, ALBERT, DistilBERT, XLNet, and many others — each refining aspects of BERT's design, and it catalyzed the broader research program that eventually led to GPT-3 and the large language model era. BERT's open release of pretrained weights on GitHub democratized access to high-quality language representations, enabling researchers and engineers worldwide to build on its foundation without requiring Google-scale compute.