Llama 4: Meta launches open-source model with massive memory

Imagine being able to analyze Proust’s entire “À la recherche du temps perdu” in one go, or have an AI examine your entire application’s source code… What seemed impossible just a few months ago is now becoming reality with Meta Llama 4, a spectacular breakthrough in the world of artificial intelligence.

On April 5, 2025, Meta unveiled its new Llama 4 family of models, marking a decisive turning point in the world of open source large language models (LLMs).

Listen to the AI podcast :

This announcement radically transforms the possibilities of generative AI with native multimodal functionality and monumental context windows of up to 10 million tokens – the equivalent of around 7500 pages of text!

“The new Llama 4 architecture completely redefines the boundaries of what generative AI can do, combining unprecedented efficiency with advanced multimodal capabilities.”

Introduction to the Llama 4 family

Meta presents a range of three distinct models, each designed to meet specific needs while sharing a revolutionary common architecture based on the “mixture of experts” (MoE) principle.

Llama 4 Scout: The little giant in a titanic context

Llama 4 Scout, although described as a “compact” model, shatters records with its 17 billion active parameters spread across 16 experts (totaling 109 billion parameters).

To give you an idea, it’s as if every question you ask it activates a team of specialized experts rather than a single giant brain!

His most impressive feature? A context window of 10 million tokens, an absolute record that far exceeds Gemini’s 2 million tokens.

To put this in perspective, it’s like going from a memory capable of holding a novel to one capable of encompassing an entire library!

The icing on the cake: it can run on a single NVIDIA H100 GPU (with Int4 quantization), making it much more affordable than its competitors.

Llama 4 Maverick : The super-powered all-rounder

Llama 4 Maverick steps up a gear with its 17 billion active parameters spread across 128 experts, for a total of 400 billion parameters.

This multimodal model outperforms rivals such as GPT-4o and Gemini 2.0 Flash in code, reasoning and image comprehension benchmarks.

“Llama 4 Maverick offers an exceptional performance/cost ratio: at just 19 to 49 cents per million tokens, it’s almost 10 times cheaper than GPT-4o, which costs $4.38.”

This economic feat could well democratize access to advanced AI for many companies and developers.

Llama 4 Behemoth: The Teaching Colossus

The real behemoth of the family is Llama 4 Behemoth with its 288 billion active parameters spread across 16 experts, reaching nearly 2 trillion (2000 billion) parameters in total.

Still in training and not available to the public, Meta claims it already outperforms GPT-4.5, Claude Sonnet 3.7 and Gemini 2.0 Pro on several scientific and mathematical benchmarks.

This model takes on the role of teacher to pass on its knowledge to the smaller models (Scout and Maverick) via a process called “distillation” – imagine a professor emeritus training the next generation of teachers!

Technical innovations that are changing the game

The Mixture of Experts (MoE) architecture: AI that optimizes its brain

The major innovation of Llama 4 lies in its MoE architecture, a first for the Llama family. Unlike traditional models where all parameters are activated for each token (like using your whole brain to think about the color of an apple), MoE models activate only a fraction of the parameters per token processed.

To simplify, imagine a gigantic brain which, instead of using all its resources for each task, activates only the most relevant “experts”.

This enables remarkable energy efficiency while maintaining, or even improving, the quality of results.

Llamama 4’s multimodality explained

The multimodality is a fundamental feature of all Llama 4 models, enabling them to simultaneously understand and analyze text and images like never before.

What is native multimodality?

In contrast to previous approaches that treated different modalities (text on one side, images on the other) separately before merging them, Llama 4 uses a revolutionary “early fusion” technique that integrates textual and visual tokens directly into the model architecture.

How does this early merging work?

1. Improved visual encoder: Meta has perfected Llama 4’s visual encoder, based on MetaCLIP but trained specifically to better adapt to LLM
2. Simultaneous processing: The model can process up to 48 images during the pre-training phase, with positive results tested down to 8 images in the post-training phase
3. Unified contextual understanding: This deep integration enables models to understand and reason about complex multimodal content, such as video sequences or sets of linked images, while maintaining contextual understanding of the associated text

Practical applications

This native multimodality transforms the way AI can interact with visual content:

• Temporal analysis of activities: Understanding the evolution of a scene through several images
• Image rounding: Ability to align user prompts with visual concepts and anchor responses to specific image regions
• Understanding complex scenes: Describe in detail what’s going on in an image and answer specific questions about its content

These capabilities pave the way for novel applications such as the creation of virtual assistants capable of “seeing” and interacting with the visual world in a more natural and intuitive way.

iRoPE technology: The secret of titanic contexts

To achieve the record context capacity of 10 million tokens, Meta has developed a new architecture christened iRoPE.

This technology combines interlaced attention layers without positional embeddings and inference attention temperature adjustment.

In needle-in-a-haystack tests, where the model has to retrieve precise information from an ocean of data, Llama 4 Scout shows near-perfect results even with 10 million context tokens.

It’s as if you could instantly find a specific phrase in an entire library!

Massive, multilingual training

The pre-training of Llama 4 models required major innovations to handle the massive scale of the data. Meta trained these models on more than 30 trillion tokens, more than double the amount used for Llama 3.

On the linguistic front, Llama 4 is fluent in 200 languages, including more than 100 with over a billion tokens each. This massive approach to multilingualism aims to make models truly versatile on a global scale.

“Llama 4’s training on 200 different languages represents a decisive step towards truly global AIs, capable of communicating with users from all walks of life.”

Game-changing real-world applications

Massive document processing and code analysis

With its context window of 10 million tokens, Llama 4 Scout enables:

• Analyze simultaneously hundreds of documents (contracts, reports, knowledge bases)
• To explore entire codebases to detect bugs or generate documentation
• To synthesize massive corpora of scientific literature

For a developer, it’s like having a colleague who has read and memorized all your company’s code and could help you understand any part of the system!

Multimodal analysis and visual comprehension

Llamama 4’s native multimodality transforms image analysis:

• Understanding relationships between multiple images
• Time analysis of activities (as in a video)
• “Grounding” of images – ability to precisely answer questions about specific areas of an image

For a doctor, this could mean an AI capable of analyzing and comparing series of medical images while taking into account the patient’s textual history.

The challenges that remain

Despite its spectacular advances, Llama 4 is not without its challenges:

Licenses not so “open”

Although described as “open source”, Llama 4 imposes certain restrictions, particularly for companies with more than 700 million users.

These constraints maintain a certain amount of Meta control over large-scale use, which differs from truly open licenses like MIT or Apache.

Substantial hardware requirements

Even the “smallest” version (Scout) cannot be run on standard consumer GPUs due to its size, limiting its adoption by individual developers or small businesses.

“Hyper-quantization techniques (up to 1.58 bit) are under discussion and could potentially make these models accessible on more modest hardware.”

Persistent but improving bias

Meta acknowledges that its models have historically exhibited biases, particularly on political and social topics.

With Llama 4, the company claims to have made significant progress, reducing unequal refusals to less than 1% on a range of controversial thematic issues.

Competitive comparison

Llama 4 Behemoth positions itself directly against private frontier models such as GPT-4.5 from OpenAI and Claude Sonnet 3.7 from Anthropic, even outperforming them on several scientific benchmarks.

Llama 4 Maverick, meanwhile, compares favorably with Gemini 2.0 Flash on a wide range of benchmarks, including image comprehension and reasoning.

The experimental version ranks second on LMArena, just behind Gemini 2.5 Pro.

Finally, Llama 4 Scout clearly dominates Mistral 3.1 (24B) and Gemma 3 (27B) on virtually all benchmarks, despite a similar number of active parameters.

Towards a more open and intelligent future

Meta reaffirms its commitment to openness as a driver of innovation, by making Llama 4 Scout and Llama 4 Maverick available for download.

The company has also implemented several ethical safeguards, including:

• Llama Guard: A security model capable of detecting whether inputs or outputs violate established policies
• Prompt Guard: A classification model trained to detect malicious prompts
• CyberSecEval: Assessments that help understand and reduce cybersecurity risks

A turning point in the history of AI

The arrival of Llama 4 marks a new era in which openness, efficiency and multimodality become as important as raw performance.

This shift is redefining the balance of power in the AI industry and accelerating the widespread adoption of these transformative technologies.

“As these models spread across diverse applications and industries, they promise to fundamentally transform how we interact with digital information, further blurring the boundaries between text, image and deep contextual understanding.”

As Meta continues to innovate with projects like “Llama 4 Reasoning” on the horizon, one thing is certain: we’re entering an exciting new phase of artificial intelligence, where the possibilities seem truly limitless.

And how do you imagine using this unprecedented power of processing and understanding in your projects?