Large Language Model (LLM) Talk

Context engineering is the system-level discipline of architecting the dynamic information environment for AI models. Unlike prompt engineering, which focuses on phrasing specific instructions, context engineering programmatically assembles the model's "working memory" using retrieved data, tool outputs, and conversation history. It employs strategies like selection, compression, and ordering to manage token limits and prevent "context rot." By orchestrating how information is filtered and presented at runtime, context engineering ensures LLMs remain grounded and reliable for complex, long-horizon tasks, effectively serving as the operating system for agentic AI.

Manus AI is a general-purpose autonomous agent designed to function as a digital worker rather than a passive chatbot. Developed by Monica and acquired by Meta, it utilizes a Planner-Executor architecture to orchestrate foundation models like Claude and Qwen within cloud-based sandboxes. Manus excels at complex, asynchronous tasks—including app deployment, massive parallel research, and data analysis—by autonomously planning workflows and executing actions via a virtual file system and browser. Its unique Context Engineering and multi-agent approach enable it to manage long-horizon tasks efficiently without constant human oversight.

Kimi K2, developed by Moonshot AI, is an open agentic intelligence model built on a Mixture-of-Experts (MoE) architecture. It features 1 trillion total parameters, with 32 billion active during inference. Trained on 15.5 trillion tokens using the stable MuonClip optimizer, Kimi K2 is optimized for advanced reasoning, coding, and tool use. It offers strong performance and significantly lower pricing than many competitors, making cutting-edge AI accessible and fostering innovation.

Mixture-of-Recursions (MoR) is a unified framework built on a Recursive Transformer architecture, designed to enhance the efficiency of large language models. It achieves this by combining three core paradigms: parameter sharing (reusing shared layers across recursion steps), adaptive computation (dynamically assigning different processing depths to individual tokens via lightweight routers), and efficient Key-Value (KV) caching (selectively storing or sharing KV pairs). This integrated approach enables MoR to deliver large-model quality with significantly reduced computational and memory overhead, improving efficiency for both training and inference.

MeanFlow models introduce the concept of average velocity to fundamentally reformulate one-step generative modeling. Unlike Flow Matching, which focuses on instantaneous velocity, MeanFlow directly models the displacement over a time interval. This approach allows for highly efficient one-step or few-step generation using a single network evaluation. MeanFlow is built on a principled mathematical identity between average and instantaneous velocities, guiding network training without requiring pre-training, distillation, or curriculum learning. It achieves state-of-the-art performance for one-step generation, significantly narrowing the gap with multi-step models.

Mamba is a novel deep learning architecture that achieves linear scaling in computation and memory with sequence length, addressing Transformers' quadratic limitations. Its selective State Space Model (SSM) layer dynamically adapts to input context, allowing it to "forget" or "remember" information. Optimizations include a hardware-aware parallel algorithm for its recurrent "selective scan", employing kernel fusion for efficient GPU memory usage and recomputation to reduce memory footprint during training. This results in significantly faster inference (up to 5x throughput) and superior long-context handling.

LLM alignment is the process of steering Large Language Models to operate in a manner consistent with intended human goals, preferences, and ethical principles. Its primary objective is to make LLMs helpful, honest, and harmless, ensuring their outputs align with specific values and are advantageous to users. This critical process prevents unintended or harmful outputs, mitigates issues like specification gaming and reward hacking, addresses biases and falsehoods, and manages the complexity of these powerful AI systems. Alignment is vital to transform unpredictable models into reliable, trustworthy, and beneficial tools, especially as AI capabilities advance.

The "Why We Think" from Lilian Weng, examines improving language models by allocating more computation at test time, drawing an analogy to human "slow thinking" or System 2. By treating computation as a resource, the aim is to design systems that can utilize this test-time effort effectively for better performance. Key approaches involve generating intermediate steps like Chain-of-Thought, employing decoding methods such as parallel sampling and sequential revision, using reinforcement learning to enhance reasoning, enabling external tool use, and implementing adaptive computation time. This allows models to spend more resources on analysis, similar to human deliberation, to achieve improved results.

Deep Research is an autonomous research agent built into ChatGPT. It performs multi-step online research over several minutes, behaving like a human researcher by searching, reading, analyzing, and synthesizing information from multiple sources. It produces detailed, cited reports. Unlike standard ChatGPT's single-step responses, Deep Research uses an agent architecture orchestrating specialized reasoning models (like o3-mini) and generalist models (like GPT-4).

vLLM is a high-throughput serving system for large language models. It addresses inefficient KV cache memory management in existing systems caused by fragmentation and lack of sharing, which limits batch size. vLLM uses PagedAttention, inspired by OS paging, to manage KV cache in non-contiguous blocks. This minimizes memory waste and enables flexible sharing, allowing vLLM to batch significantly more requests. As a result, vLLM achieves 2-4x higher throughput compared to state-of-the-art systems like FasterTransformer and Orca.

Qwen3 models introduce both Mixture-of-Experts (MoE) and dense architectures. They utilize hybrid thinking modes, allowing users to balance response speed and reasoning depth for tasks, controllable via parameters or tags. Developed through a multi-stage post-training pipeline, Qwen3 is trained on a significantly expanded dataset of approximately 36 trillion tokens across 119 languages. This enhances its multilingual support for global applications. The models also feature improved agentic capabilities, notably excelling in tool calling, which increases their utility for complex, interactive tasks.

RAGEN is a modular system for training and evaluating LLM agents using multi-turn reinforcement learning. Built on the StarPO framework, it implements the full training loop including rollout generation, reward assignment, and trajectory optimization. RAGEN serves as research infrastructure to analyze LLM agent training dynamics, focusing on challenges like stability, generalization, and the emergence of reasoning in interactive environments.

DeepSeek-Prover-V2 is an open-source large language model designed for formal theorem proving in Lean 4. Its training relies heavily on synthetic data, generated by using DeepSeek-V3 to decompose problems into subgoals, which are then recursively solved by a smaller 7B prover model. The model uses a two-stage training process, including supervised fine-tuning and reinforcement learning (GRPO), to bridge informal reasoning with formal proofs. It achieves state-of-the-art performance, particularly with its high-precision Chain-of-Thought mode.

The DeepSeek-Prover project aims to advance large language model capabilities in formal theorem proving by addressing the scarcity of training data. It uses autoformalization to convert informal high school and undergraduate math competition problems into formal statements, generating a large dataset of 8 million synthetic proofs. Quality filtering and formal verification with Lean 4 ensure data reliability. An iterative process enhances the model, leading to state-of-the-art performance on miniF2F and FIMO benchmarks, outperforming models like GPT-4.

The Model Context Protocol (MCP), introduced by Anthropic in November 2024, is an open protocol standardizing how applications provide context to LLMs. Acting like a "USB-C port for AI applications," it provides a standardized way to connect AI models to different data sources and tools. MCP employs a client-server architecture to overcome the complex "MxN integration problem" by establishing a common interface, reducing complexity to M+N. This allows for more robust and scalable AI applications by eliminating the need for custom connectors and fostering a unified ecosystem for LLM integration.

LLM post-training is crucial for refining the reasoning abilities developed during pretraining. It employs fine-tuning on specific reasoning tasks, reinforcement learning to reward logical steps and coherent thought processes, and test-time scaling to enhance reasoning during inference. Techniques like Chain-of-Thought (CoT) and Tree-of-Thoughts (ToT) prompting, along with methods like Monte Carlo Tree Search (MCTS), allow LLMs to explore and refine reasoning paths. These post-training strategies aim to bridge the gap between statistical pattern learning and human-like logical inference, leading to improved performance on complex reasoning tasks.

Agent AI refers to interactive systems that perceive visual, language, and environmental data to produce meaningful embodied actions in physical and virtual worlds. It aims to create sophisticated and context-aware AI, potentially paving the way for AGI by leveraging generative AI and cross-reality training. Agent AI systems often use large foundation models (LLMs and VLMs) for enhanced perception, reasoning, and task planning. Continuous learning is crucial for these agents to adapt to dynamic environments, refine their behavior through interaction and feedback, and achieve self-improvement.

FlashAttention-3 accelerates attention on NVIDIA Hopper GPUs through three key innovations. It achieves producer-consumer asynchrony by dividing warps into producer (data loading with TMA) and consumer (computation with asynchronous Tensor Cores) roles, overlapping these critical phases. Second, it hides softmax latency by interleaving softmax operations with asynchronous GEMMs using techniques like pingpong scheduling and intra-warpgroup pipelining. Lastly, FlashAttention-3 leverages hardware-accelerated low-precision FP8 GEMM, employing block quantization and incoherent processing to enhance throughput while mitigating accuracy loss. This summary is based on the provided sources.

FlashAttention-2 builds upon FlashAttention to achieve faster attention computation with better GPU resource utilization. It enhances parallelism by also parallelizing along the sequence length dimension, optimizing work partitioning between thread blocks and warps to reduce shared memory access. A key improvement is the reduction of non-matmul FLOPs, which are less efficient on modern GPUs optimized for matrix multiplication. These enhancements lead to significant speedups compared to FlashAttention and standard attention, reaching higher throughput and better model FLOPs utilization in end-to-end training for Transformers.

FlashAttention is an IO-aware attention mechanism designed to be fast and memory-efficient, especially for long sequences. Its core innovation is tiling, where input sequences are divided into blocks processed within the fast on-chip SRAM, significantly reducing reads and writes to the slower HBM. This contrasts with standard attention, which materializes the entire attention matrix in HBM. By minimizing HBM access and recomputing the attention matrix in the backward pass, FlashAttention achieves faster Transformer training and a linear memory footprint, outperforming many approximate attention methods that overlook memory access costs.

PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that balances simplicity, stability, sample efficiency, general applicability, and strong performance. PPO replaced TRPO (Trust Region Policy Optimization) as the default algorithm at OpenAI due to its simpler implementation and greater computational efficiency, while maintaining comparable performance. PPO approximates TRPO by clipping the policy gradient and using first-order optimization, avoiding the computationally intensive Hessian matrix and strict KL divergence constraints of TRPO. The clipping mechanism in PPO constrains policy updates, prevents excessively large changes, and promotes stability during training. Its surrogate objectives and clip function enable the reuse of training data, making PPO sample efficient, especially for complex tasks.

Andrej Karpathy's tech talk (youtube), provides a comprehensive yet accessible overview of Large Language Models (LLMs) like ChatGPT. The talk details the process of building an LLM, including pre-training, data processing, and neural network training.Key stages include downloading and filtering internet text, tokenizing the text, and training neural networks to model token relationships. The discussion covers the distinction between base models and assistants, highlighting fine-tuning to create conversational AIs. It also addresses challenges like hallucinations and mitigation strategies, such as knowledge-based refusal and tool use. The talk further explores reinforcement learning and the emergence of "thinking" in models.

Andrej Karpathy's talk, "Intro to Large Language Models," demystifies LLMs by portraying them as systems with two key components:a parameters file (the weights of the neural network) anda run file (the code that runs the network). The creation of these files starts with a computationally intensive training process, where a large amount of internet text is compressed into the model's parameters. The scaling laws show that LLM performance depends on the number of parameters and the amount of training data.Karpathy reviews how LLMs are evolving to incorporate external tools and multiple modalities. He presents his view of LLMs as the kernel process of an emerging operating system and also discusses the security challenges of LLMs, including jailbreak attacks, prompt injection attacks, and data poisoning.

DeepSeek-V2 is a Mixture-of-Experts (MoE) language model that balances strong performance with economical training and efficient inference. It uses a total of 236B parameters, with 21B activated for each token, and supports a context length of 128K tokens. Key architectural innovations includeMulti-Head Latent Attention (MLA), which compresses the KV cache for faster inference, andDeepSeekMoE, which enables economical training through sparse computation. Compared to DeepSeek 67B, DeepSeek-V2 saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts maximum generation throughput by 5.76 times. It is pre-trained on 8.1T tokens of high-quality data and further aligned through Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

Matrix calculus is essential for understanding and implementing deep learning. It provides the mathematical tools to optimize neural networks using gradient descent. The Jacobian matrix, a key concept, organizes partial derivatives of vector-valued functions. The vector chain rule simplifies derivative calculations in nested functions, common in neural networks. Automatic differentiation, used in modern libraries, relies on these principles. Grasping matrix calculus allows for a deeper understanding of model training and the implementation of custom neural networks.

'S1' refers to simple test-time scaling, an efficient approach to enhance language model reasoning with minimal resources. It involves training a model on a small, carefully curated dataset like s1K and using budget forcing to control test-time compute. Budget forcing enforces maximum or minimum thinking tokens by appending delimiters or the word "Wait". The s1-32B model, developed using this method, outperforms other models on competition math questions. The approach combines a curated dataset with a straightforward test-time technique, leading to strong reasoning performance and effective test-time scaling.

Reinforcement Learning from Human Feedback (RLHF) incorporates human preferences into AI systems, addressing problems where specifying a clear reward function is difficult. The basic pipeline involves training a language model, collecting human preference data to train a reward model, and optimizing the language model with an RL optimizer using the reward model. Techniques like KL divergence are used for regularization to prevent over-optimization. RLHF is a subset of preference fine-tuning techniques. It has become a crucial technique in post-training to align language models with human values and elicit desirable behaviors.

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that enhances mathematical reasoning in large language models (LLMs). It is like training students in a study group, where they learn by comparing answers without a tutor. GRPO eliminates the need for a critic model, unlike Proximal Policy Optimization (PPO), making it more resource efficient. It calculates advantages based on relative rewards within the group and directly adds KL divergence to the loss function. GRPO uses both outcome and process supervision, and can be applied iteratively, further enhancing performance. This approach is effective at improving LLMs' math skills with reduced training resources.

Model/Knowledge distillation is a technique to transfer knowledge from a cumbersome model, like a large neural network or an ensemble of models, to a smaller, more efficient model. The smaller model is trained using "soft targets," which are the class probabilities produced by the larger model, rather than the usual "hard targets" of correct class labels. These soft targets contain more information, including how the cumbersome model generalizes and the similarity structure of the data. A temperature parameter is used to soften the probability distributions, making the information more accessible for the smaller model to learn. This process improves the smaller model's generalization ability and efficiency. Distillation allows the smaller model to achieve performance comparable to the larger model with less computation.

Qwen2.5 is a series of large language models (LLMs) with significant improvements over previous models, focusing on efficiency, performance, and long sequence handling. Key architectural advancements include Grouped Query Attention (GQA) for better memory management, Mixture-of-Experts (MoE) for enhanced capacity, and Rotary Positional Embeddings (RoPE) for effective long-sequence modeling. Qwen2.5 uses two-phase pre-training and progressive context length expansion to enhance long-context capabilities, along with techniques like YARN, Dual Chunk Attention (DCA), and sparse attention. It also features an expanded tokenizer and uses SwiGLU activation, QKV bias and RMSNorm for stable training.

The Qwen2 series of large language models introduces several key enhancements over its predecessors. It employs Grouped Query Attention (GQA) and Dual Chunk Attention (DCA) for improved efficiency and long-context handling, using YARN to rescale attention weights. The models utilize fine-grained Mixture-of-Experts (MoE) and have a reduced KV size. Pre-training data was significantly increased to 7 trillion tokens with more code, math and multilingual content, and post-training involves supervised fine-tuning (SFT) and direct preference optimization (DPO). These changes allow for enhanced performance, especially in coding, mathematics, and multilingual tasks, and better performance in long-context scenarios.

Qwen-1, also known as QWEN, is a series of large language models that includes base pretrained models, chat models, and specialized models for coding and math. These models are trained on a massive dataset of 3 trillion tokens using byte pair encoding for tokenization, and they feature a modified Transformer architecture with untied embeddings and rotary positional embeddings. The chat models (QWEN-CHAT) are aligned to human preferences using Supervised Finetuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). QWEN models have strong performance, outperforming many open-source models, but they generally lag behind models like GPT-4.

OpenAI's o1 is a generative pre-trained transformer (GPT) model, designed for enhanced reasoning, especially in science and math. It uses a 'chain of thought' approach, spending more time "thinking" before answering, making it better at complex tasks. While not a successor to GPT-4o, o1 excels in scientific and mathematical benchmarks, and is trained with a new optimization algorithm. Different versions like o1-preview and o1-mini are available. Limitations include high computational cost, occasional "fake alignment," and a hidden reasoning process, and potential replication of training data.

GPT-4o is a multilingual, multimodal model that can process and generate text, images, and audio and represents a significant advancement over previous models like GPT-4 and GPT-3.5. GPT-4o is faster and more cost-effective, has improved performance in multiple areas, and natively supports voice-to-voice. GPT-4o's knowledge is limited to what was available up to October 2023. It has a context length of 128k tokens. The cost of training GPT-4 was more than $100 million, and it has 1 trillion parameters.

Kimi k1.5 is a multimodal LLM trained with reinforcement learning (RL). Key aspects include: long context scaling to 128k, improving performance with increased context length; improved policy optimization using a variant of online mirror descent; and a simplistic framework that enables planning and reflection without complex methods. It uses a reference policy in its off-policy RL approach, and long2short methods such as model merging and DPO to transfer knowledge from long-CoT to short-CoT models, achieving state-of-the-art reasoning performance. The model is jointly trained on text and vision data.

DeepSeek-R1 is a language model focused on enhanced reasoning, employing reinforcement learning (RL) and building upon the DeepSeek-V3-Base model. It uses Group Relative Policy Optimization (GRPO) to reduce computational costs by eliminating the need for a separate critic model, which is commonly used in other algorithms such as PPO. The model uses a multi-stage training pipeline including an initial fine-tuning with cold-start data, followed by reasoning-oriented RL, and supervised fine-tuning (SFT) using rejection sampling, and a final RL stage. A rule-based reward system avoids reward hacking. DeepSeek-R1 also employs a language consistency reward during RL to address language mixing. The model's reasoning capabilities are then distilled into smaller models. DeepSeek-R1 achieves performance comparable to, and sometimes surpassing, OpenAI's o1 series on various reasoning, math, and coding tasks.

Claude 3 is a family of large multimodal AI models developed by Anthropic, with a focus on safety, interpretability, and user alignment. The models, which include Opus, Sonnet, and Haiku, excel in reasoning, math, coding, and multilingual understanding. They are designed to be helpful, honest, and harmless assistants and can process text, audio, and visual inputs. Claude 3 models use Constitutional AI principles, aiming for more ethical and reliable responses. They have improved abilities in long context comprehension, and have shown strong performance in various tests, often outperforming previous Claude models and sometimes matching or exceeding GPT models in some benchmarks.

GPT-4, or Generative Pre-trained Transformer 4, is a large multimodal language model created by OpenAI, and the fourth in the GPT series. It is a significant advancement over previous models such as GPT-3, with improvements in model size, performance, contextual understanding, and safety. GPT-4 uses a Transformer architecture, a deep learning model that has revolutionized natural language processing. It can process both text and images, and it has a larger context window than GPT-3, enabling it to handle longer documents and more complex tasks. GPT-4 was trained using a combination of publicly available data and licensed third-party data, and then fine-tuned using reinforcement learning and human feedback. It also has increased reasoning and generalization abilities, making it more reliable for advanced and specialized applications.

Training large language models (LLMs) is challenging due to the large amount of GPU memory and long training times required. Several parallelism paradigms enable model training across multiple GPUs, and various model architecture and memory-saving designs make it possible to train very large neural networks. The optimal model size and number of training tokens should be scaled equally, with a doubling of model size requiring a doubling of training tokens. Current large language models are significantly under-trained. Techniques such as data parallelism, model parallelism, pipeline parallelism, and tensor parallelism can be used to distribute the training workload. Other strategies include CPU offloading, activation recomputation, mixed-precision training, and compression to save memory.

MiniMax-01 is a series of large language and vision-language models that use lightning attention and a mixture of experts (MoE) to achieve long context processing. The models, MiniMax-Text-01 and MiniMax-VL-01, match the performance of top-tier models, like GPT-4o and Claude-3.5-Sonnet, while offering 20-32 times longer context windows, reaching up to 4 million tokens during inference. The models use a hybrid architecture, with linear and softmax attention mechanisms, and are trained on large datasets of text, code, and image-caption pairs. They also use a multi-stage training process with supervised fine-tuning and reinforcement learning to optimize their capabilities in long-context and real-world scenarios.

DeepSeek-V3 is a large Mixture-of-Experts (MoE) language model, trained ~10x less cost, with 671 billion total parameters, of which 37 billion are activated for each token. It uses Multi-head Latent Attention (MLA) and DeepSeekMoE architectures. A key feature of DeepSeek-V3 is its auxiliary-loss-free load balancing strategy and multi-token prediction training objective. The model was pre-trained on 14.8 trillion tokens and underwent supervised fine-tuning and reinforcement learning. It has demonstrated strong performance on various benchmarks, achieving results comparable to leading closed-source models while maintaining economical training costs.

The Tree of Thoughts (ToT) framework enhances problem-solving in large language models (LLMs) by using a structured, hierarchical approach to explore multiple solutions. ToT breaks down problems into smaller steps called "thoughts", generated via sampling or proposing. These "thoughts" are evaluated using value or voting strategies, and search algorithms like breadth-first or depth-first search navigate the solution space. This allows LLMs to backtrack and consider alternative paths, improving performance in complex decision-making tasks.

Large language models (LLMs) demonstrate some reasoning abilities, though it's debated whether they truly reason or rely on information retrieval. Prompt engineering enhances reasoning, employing techniques like Chain-of-Thought (CoT), which involves intermediate reasoning steps. Multi-stage prompts, problem decomposition, and external tools are also used. Multi-agent discussions may not surpass a well-prompted single LLM. Research explores knowledge graphs and symbolic solvers to improve LLM reasoning, and methods to make LLMs more robust against irrelevant context. The field continues to investigate techniques to improve reasoning in LLMs.

LangChain is an open-source framework that simplifies the development of applications using large language models (LLMs). It offers tools and abstractions to enhance the customization, accuracy, and relevancy of LLM-generated information. LangChain allows developers to connect LLMs to external data sources, and create applications like chatbots, question-answering systems, and virtual agents. Key components include model interfaces, prompt templates, chains, agents, retrieval modules, and memory. LangChain enables the creation of complex, context-aware applications by combining different components.

LlamaIndex is an open-source framework for building LLM applications by connecting custom data to LLMs. It excels in Retrieval-Augmented Generation (RAG), data storage, and retrieval. It works by ingesting data from various sources, indexing it (often into vector embeddings), and querying it with a language model. LlamaIndex has tools to evaluate the quality of retrieval and responses. It supports AI agents for automated tasks. The framework facilitates the creation of custom knowledge bases for querying with LLMs.

Chain of Thought (CoT) is a prompting technique that enhances the reasoning capabilities of large language models (LLMs) by encouraging them to articulate their reasoning process step by step. Instead of providing a direct answer, the model breaks down complex problems into smaller, more manageable parts, simulating human-like thought processes. This method is particularly beneficial for tasks requiring complex reasoning, such as math problems, logical puzzles, and multi-step decision-making. CoT can be implemented through prompting, where the model is guided to "think step by step," or it can be an automatic internal process in some models. CoT improves accuracy and transparency by providing a view into the model's decision-making.

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by connecting them to external knowledge sources. It works by retrieving relevant documents based on a user's query, using an embedding model to convert both into numerical vectors, then using a vector database to find matching content. The retrieved data is then passed to the LLM for response generation. This process improves accuracy and reduces "hallucinations" by grounding the LLM in factual, up-to-date information. RAG also increases user trust by providing source attribution, so users can verify the information.

Fine-tuning is a machine learning technique that adapts a pre-trained model to a specific task or domain. Instead of training a model from scratch, fine-tuning uses a pre-trained model as a starting point and further trains it on a smaller, task-specific dataset. This process can improve the model's performance on specialized tasks, reduce computational costs, and broaden its applicability across various fields. The goal of fine-tuning can be knowledge injection or alignment, or both. Fine-tuning is often used in natural language processing. There are many ways to approach fine-tuning, including supervised fine-tuning, few-shot learning, transfer learning, and domain-specific fine-tuning ...

Scaling laws describe how language model performance improves with increased model size, training data, and compute. These improvements often follow a power-law, with predictable gains as resources scale up. There are diminishing returns with increased scale. Optimal training involves a balance of model size, data, and compute, and may require training large models on less data, stopping before convergence. To prevent overfitting, the dataset size should increase sublinearly with model size. Scaling laws are relatively independent of model architecture. Current large models are often undertrained, suggesting a need for more balanced resource allocation.

LLaMA-3 is a series of foundation language models that support multilinguality, coding, reasoning, and tool usage. The models come in different sizes, with the largest having 405B parameters and a 128K token context window. The development of Llama 3 focused on optimizing data, scale, and managing complexity, using a combination of web data, code, and mathematical text, with specific pipelines for each. The models underwent pre-training, supervised finetuning, and direct preference optimization to enhance their performance and safety. Llama 3 models have demonstrated strong performance in various benchmarks and aim to balance helpfulness with harmlessness.