The Llama 3 Herd of Models
Today's paper introduces the Llama 3 Herd of Models, a new set of foundation models for language developed by Meta AI. These models support multilinguality, coding, reasoning, and tool usage, with the largest model boasting 405B parameters and a context window of up to 128K tokens. The paper presents an extensive evaluation of Llama 3, comparing its performance to leading language models like GPT-4.
Overview
The Llama 3 models are developed using a two-stage approach: pre-training and post-training. In the pre-training stage, the models are trained on a massive scale using next-token prediction. This stage involves processing about 15T multilingual tokens, which is significantly more than the 1.8T tokens used for Llama 2.
The post-training stage involves fine-tuning the models to follow instructions, align with human preferences, and improve specific capabilities such as coding and reasoning. This stage focuses on enhancing the models' ability to understand and execute complex tasks. There are several rounds of alignment, each of which involving supervised finetuning (SFT) on instruction tuning data and Direct Preference Optimization.
The development process emphasizes three key factors: data, scale, and managing complexity. The team improved both the quantity and quality of data used for training, implementing more careful pre-processing and curation pipelines. They also scaled up the model size significantly, with the largest model having 405B parameters and being trained using 3.8 × 10^25 FLOPs, which is almost 50 times more than the largest version of Llama 2.
Additionally, the model’s capabilities can be extended by incorporating image, video, and speech functionalities through a compositional approach. This method involves three key stages:
1. Multi-Modal Encoder Pre-Training - Separate encoders for images and speech are trained. The image encoder learns from large datasets of image-text pairs, understanding the relationship between visual content and its description in natural language. The speech encoder employs a self-supervised approach, masking parts of speech inputs and reconstructing them using discrete-token representation, thereby learning the structure of speech signals.
2. Vision Adapter Training - An adapter integrates the pre-trained image encoder with the language model via cross-attention layers, aligning image representations with language representations. This adapter is further trained on text-image pairs, updating only the image encoder parameters while keeping the language model parameters unchanged. Additionally, a video adapter is trained on paired video-text data to aggregate information across frames.
3. Speech Adapter Training - A speech adapter converts speech encodings into token representations compatible with the language model. The adapter and encoder parameters are jointly updated in a supervised fine-tuning stage, enhancing speech understanding. A text-to-speech system is also integrated without altering the language model parameters.
By using these approaches, the resulting models are capable of recognizing image and video content and supporting speech interaction. However, these models with additional capabilities are still under development and not yet released.
Results
The paper reports that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a wide range of tasks. The models show strong performance in multilinguality, coding, reasoning, and tool usage.
The experiments with multi-modal integration (image, video, and speech) also demonstrate competitive performance with state-of-the-art models on related recognition tasks.
Importantly, the paper announces the public release of Llama 3, including pre-trained and post-trained versions of the 405B parameter language model. They also release Llama Guard 3, a model designed for input and output safety.
Conclusion
The Llama 3 Herd of Models represents a significant advancement in the field of large language models. By focusing on data quality and quantity, scaling up model size, and managing complexity, the team at Meta AI has developed a set of models that rival leading competitors in performance across various language-related tasks. The public release of these models and the ongoing work on multi-modal integration suggest a promising future for AI research and applications.
For more information please consult the full paper.
Congrats to the authors for their work!
Llama Team. "The Llama 3 Herd of Models." AI @ Meta, 23 July 2024, https://llama.meta.com/.