Llama 3

When Meta released Llama 3 in April 2024, it represented the culmination of a deliberate strategy to build open-weights large language models capable of matching proprietary systems. The Llama lineage had begun in February 2023 with a model designed primarily for researchers, but each successive generation expanded both capability and ambition. By the time Llama 3 arrived, Meta's AI Research team had grown substantially, and the project reflected industrial-scale investment: new tokenizers, larger and cleaner pretraining datasets, and instruction-tuning pipelines refined over two prior model generations.

Llama 3 was released in two initial sizes — 8 billion and 70 billion parameters — with a 405-billion-parameter model following in July 2024. All variants used a new tokenizer with a 128,000-token vocabulary, an improvement over Llama 2's 32,000-token vocabulary, enabling more efficient encoding of code and multilingual text. The models were pretrained on over 15 trillion tokens of data, roughly seven times the data used for Llama 2, sourced from a filtered and deduplicated web corpus. Grouped-query attention, first introduced at scale in Llama 2's larger variants, was applied across all Llama 3 sizes to improve inference efficiency.

The instruction-tuned variants — branded Meta Llama 3 Instruct — were fine-tuned using supervised fine-tuning, rejection sampling, proximal policy optimization, and direct preference optimization. On standard benchmarks including MMLU, HumanEval, and MT-Bench, the 70B instruct model matched or surpassed GPT-3.5 and approached GPT-4-class performance in several evaluations. The 405B model, released under a custom community license permitting broad commercial use, became one of the most capable openly available models ever released and set a new reference point for what the open ecosystem could achieve.

Llama 3 demonstrated conclusively that the performance gap between open-weights and closed proprietary models was not a fundamental technical barrier but a function of resources and iteration. Before Llama 3, the conventional wisdom held that frontier capability required the secrecy and scale of labs like OpenAI or Anthropic. The 405B model's competitive performance on reasoning, coding, and instruction-following tasks forced a reassessment of that assumption, and catalyzed downstream development across the open-source ecosystem as fine-tuners, tool builders, and researchers gained access to a genuinely competitive base.

The broader consequence was a structural shift in how organizations approached AI deployment. Enterprises and governments that had been locked into API-dependent relationships with closed providers now had a credible open alternative they could run on their own infrastructure, audit, and modify. Meta's decision to release under a permissive license — rather than the more restrictive terms sometimes used for large models — meant that Llama 3 became the foundation for thousands of derivative projects, specialized fine-tunes, and embedded applications within months of release, reshaping the competitive dynamics of the entire industry.