First Principles Series · Part 2

A fair comparison of World Models and JEPA (since I couldn't find one)

The math, the costs, the production case. The article I would have wanted to read a month ago.

If you follow the AI debate, you have heard about World Models and JEPA. They are two of the most discussed architectures in robotics and autonomous systems. The discussion is often confused, though: the terms are used interchangeably, the differences are not made clear, and the technical details that would let you understand what these systems actually do are almost always missing.

This article is an attempt to clear that up. We will start from the problem both architectures try to solve, then see how they solve it in fundamentally different ways, with concrete numerical examples. After a public production case (BADAS 2.0) that grounds the discussion, we will spend a long section on the representation collapse problem, which is the actual technical heart of JEPA and which is usually dismissed in one line. We will close with the part that is most often left out of similar articles: the hardware, VRAM, and cost numbers you need to actually train these models, and what to do with all of this on Monday morning.

The problem: predicting what comes next

Think about how your brain works when you cross a street. You do not just react to what you see right now. You imagine where the car will be in two seconds. You predict what happens if you speed up. You simulate alternative scenarios in your head. This capacity for mental simulation is what World Models and JEPA try to give to machines.

Why is it important? Because a system that can only react to present stimuli is structurally limited. A robot that grasps an object needs to predict where its hand will be 100 milliseconds from now. A self-driving car needs to anticipate the behaviour of other vehicles. An AI agent that plans needs to reason about the consequences of its actions before acting.

The technical problem is: how do you teach a machine to predict the future when you do not have "ground-truth future" labels? The answer both architectures give is the same. Train by self-supervision. Show the model a sequence, hide part of it, ask the model to guess the hidden part. If it succeeds, it has learned something about the structure of the world.

The two architectures diverge on what to hide and what to ask as the prediction. World Models hide future frames and ask the model to reconstruct them pixel by pixel. JEPA hides patches of an image or video and asks the model to predict only the latent representation of those patches. The difference looks subtle and is actually deep.

World Models: simulating the world internally

The concept of World Model was formalised by Ha and Schmidhuber in 2018 in a paper of the same name [1]. The idea is to build an internal simulator of the world that lets an agent "imagine" what happens if it takes a given action, without having to actually take it.

The fundamental equation

A World Model is, in its simplest form, a function that takes the current state of the world and an action, and produces the next state:

The observation x_t is too large to manipulate directly. A 256×256 RGB image has 196,608 numbers. A 1-second video at 30 fps has nearly 6 million. World Models therefore work in a compressed latent space — the state s_t — produced by an encoder, with three distinct components.

The three components

Encoder e_φ. Compresses the high-dimensional observation into a compact latent representation. Typically a CNN followed by an MLP for images, a transformer for video. A 256×256 RGB image (196,608 numbers) becomes a vector of 256 or 512 numbers. This compression forces the model to discard noise.

Dynamics Model f_θ. The core of the World Model. It takes the latent state s_t and the action a_t and predicts s_t+1. Typically an RNN (GRU, LSTM) or a transformer. Dreamer uses a variant called RSSM (Recurrent State-Space Model) that is worth looking at in detail.

Decoder d_ψ. Reconstructs pixels from the predicted latent state. Its main role is during training: the reconstruction loss (pixel-wise MSE) is the signal that forces encoder and dynamics to preserve visual information.

RSSM: the Dreamer trick

The dynamics model in Dreamer does not produce a single monolithic state. It decomposes the state s_t into a deterministic part h_t (a GRU hidden state that accumulates history) and a stochastic part z_t (a sample from a categorical distribution). The motivation is that the world is intrinsically uncertain, and modelling that uncertainty explicitly improves long-horizon prediction.

Notation reminder. In Dreamer's RSSM we are now operating inside the latent state: s_t = (h_t, z_t). The symbol z_t below refers specifically to the stochastic categorical component, following the DreamerV3 paper. It is one of the two components of the state, not a re-labelling of the encoder output we used earlier in the simple WM diagram.

The DreamerV3 [4] total loss is a weighted sum of three terms: reconstruction loss (pixel MSE), KL divergence between prior and posterior (forces the prior to be predictive), and reward loss (a separate MLP predicts the reward from s_t). This decomposition follows directly from the variational lower bound (ELBO) for sequential models. DreamerV3 adds KL balancing: the prior is allowed to move faster towards the posterior than the posterior towards the prior, with separate stop-gradients on each side. It is a small detail in the loss code and one of the reasons DreamerV3 trains stably across very different environments without per-task hyperparameter tuning.

Numerical example

The "dream to learn" loop

Once the world model is trained, Dreamer uses it to generate imagined trajectories. The policy is not trained on real data (slow, expensive) but on the world model's dreams. A real robot typically interacts with the environment at tens of steps per second; the world model can produce on the order of thousands of imagined transitions per second on a single GPU, depending on architecture. This is the source of Dreamer's data efficiency.

The pixel reconstruction problem

World Models have a structural limitation: the decoder. To train the model you need a loss signal, and that signal is pixel-wise MSE between the predicted frame and the real frame. The gradient flowing back through encoder and dynamics is computed with respect to visual fidelity, not with respect to representation utility.

The practical consequence is that the encoder spends capacity preserving information the decoder can then reconstruct. Reflections, shadows, textures, background details: all of these enter the loss with the same weight as the things that actually matter for decisions.

This observation is what led Yann LeCun to propose JEPA. If the problem is that the decoder forces reconstruction of useless pixels, the solution is to remove the decoder.

JEPA: predicting without reconstructing

In 2022 LeCun published "A Path Towards Autonomous Machine Intelligence" [6], a 60-page position paper criticising the generative approach and proposing an alternative: JEPA, the Joint-Embedding Predictive Architecture.

The idea: instead of predicting future pixels (or masked patches), predict the latent representation of what is hidden. The loss is no longer pixel-wise MSE. It is a distance in embedding space.

The architecture

JEPA uses three components.

Context encoder e_φ. Receives the context (visible patches of the image, or past frames of a video) and produces its representation z_ctx.

Target encoder e_φ'. Receives the target (masked patches, or future frames) and produces z_tgt, which acts as "ground truth" in latent space. Critical detail: the parameters φ' are not updated by the gradient. They are updated as an exponential moving average (EMA) of φ.

Predictor p_ψ. Receives z_ctx and tries to predict ẑ_tgt. Typically much lighter than the encoders (6 transformer layers against the 24 of the encoder for ViT-L).

The formal loss, expanded

Let us write it out properly. Let x be an image, M(x) the context (visible patches), N(x) the target (masked patches). Let φ be the online encoder parameters, φ' the target encoder parameters, ψ the predictor parameters.

Numerical example: I-JEPA on a 224×224 image

BADAS 2.0: JEPA in production

Before we go deeper into the collapse problem, it is worth checking that the JEPA approach actually works at production scale. The most concrete and well-documented public case is BADAS 2.0 by Nexar [17] [18], a collision anticipation system that predicts road incidents before they happen. The Nexar site publishes full benchmarks and direct comparisons with a generative competitor, which is rare and lets us check JEPA's thesis on a safety-critical task.

BADAS 2.0 uses V-JEPA 2 as its backbone: a ViT-L Vision Transformer with 300M parameters, fine-tuned end-to-end on roughly 200,000 labeled videos (~2M windowed clips) collected from a 350,000-dashcam Nexar fleet. It is used by commercial fleets, insurance companies, and AV development programs.

The controlled comparison with Cosmos-Reason2

The most interesting experiment Nexar has published is a controlled comparison. They took NVIDIA Cosmos-Reason2-2B (a 2B-parameter vision-language model post-trained from Qwen3-VL-2B-Instruct, designed for embodied reasoning) and fine-tuned it on the same training data as BADAS 2.0. Same training set, same protocol, same task.

The most telling metric is early warning recall: 91.3% for BADAS against 48.3% for Cosmos. BADAS detects danger ahead of time nine times out of ten. Cosmos does so less than half the time. In the context of collision anticipation, "late" equals "wrong".

One clarification, because Nexar's own published table compresses it. The 99.4% AP and 91.3% recall above belong to the main BADAS 2.0 model, the 300M V-JEPA 2 ViT-L. BADAS also ships distilled variants, and the smallest of them, Flash Lite at 22M parameters, scores a slightly lower 98.4% AP. So the headline accuracy and the headline tiny model are technically two different models in the same family. Both comparisons against Cosmos hold, they are just not the same row.

That 22M variant is the one that matters most. A 22M-parameter model beating a 2B-parameter model on the same task, with 91× fewer parameters, is where the architectural difference stops being a technical detail and becomes a fundamental efficiency gap.

Why JEPA wins here

Two reasons, both worth separating. The first is the prior each model carries into fine-tuning. V-JEPA 2 was pre-trained on more than 1M hours of internet video [9], internalising at scale the regularities of physical motion — objects falling, pedestrians changing direction, vehicles braking. Cosmos-Reason2 was post-trained from a generalist vision-language model with no comparable exposure to long-form video physics. Same fine-tuning data on top, very different priors underneath.

The second reason is structural — what each architecture is even computing. To predict accidents you do not need to reconstruct what the scene will look like in the next frame. You need to capture whether that car is braking, whether that pedestrian is committing to crossing. V-JEPA 2 optimises directly for those semantic invariances, in latent space. Cosmos-Reason2, even after fine-tuning, is still organised around generating tokens of free-form reasoning, with the latency, verbosity, and overconfidence problems that follow.

A detail worth noting on generalisation: BADAS 2.0 also works on scenarios that have nothing to do with road driving — drones in flight, cleaning robots, forklifts. The model learned general physics during V-JEPA 2 pre-training on millions of hours of video, not just road rules. Nexar's own framing is "Beyond the Road": any machine with a camera that moves through the physical world can use BADAS, not because it was trained on every environment, but because it learned how the physical world works. Yann LeCun, who sits on the Nexar board, comments: "Models don't emerge from abstractions alone. They come from sustained exposure to reality."

That JEPA wins this comparison at all rests on one technical condition: during pre-training, the system did not collapse into a useless trivial solution. This is the topic of the next section, and it is the part of the story most articles skip in a sentence. It is worth slowing down on.

The collapse problem, in detail

This is the section worth reading in full. Representation collapse is the central problem of all non-contrastive self-supervised methods, and understanding it means understanding why JEPA architectures look the way they do and could not have been designed differently.

The problem, formulated

Consider the JEPA loss with both encoders sharing parameters (no EMA for a moment): L(φ, ψ) = ||p_ψ(e_φ(M(x))) − e_φ(N(x))||². What is the trivial minimum?

A neural network minimising this loss without constraints will happily converge to the constant solution. All representations collapse onto the same point, the loss goes to zero, and the model has learned nothing. This is complete collapse.

There is also a more subtle phenomenon, dimensional collapse, characterised by Jing, Vincent, LeCun and Tian in 2022 [13]. Here the representations do not collapse to a single point, but they live in a subspace whose dimension is much smaller than the embedding space. Empirically you detect it by computing the covariance matrix of the representations across a batch and counting how many eigenvalues are significantly above zero. If the encoder outputs vectors in ℝ¹⁰²⁴ but only 30 eigenvalues are non-trivial, there is dimensional collapse.

Why collapse is a property of the gradient flow

Tian, Chen and Ganguli in 2021 [12] gave a theoretical analysis of the problem for the linear case. Their key result: if both encoders are identical and updated with the same gradient, gradient flow inevitably leads to collapse. The steepest descent direction always points towards the constant solution.

To avoid this, you need asymmetries. The three families of solutions modern architectures use are:

JEPA belongs to the asymmetric architecture family and inherits directly from BYOL (Grill et al., DeepMind 2020 [10]), the work that first showed a non-contrastive method can be trained without collapsing. Let us look at why in detail.

The three asymmetries of BYOL/JEPA

Asymmetry 1 — Stop-gradient. The target encoder receives zero gradient. It cannot chase the online encoder; it can only be chased. Without this asymmetry both encoders would converge to the same trivial minimum. Chen and He 2021 [11] showed empirically that removing the stop-gradient in SimSiam causes immediate collapse to the trivial solution. The loss drops to zero and the per-channel standard deviation of the embeddings goes to zero with it.

Asymmetry 2 — EMA. The target encoder parameters are updated as a moving average of the online parameters: φ' ← τφ' + (1−τ)φ. With τ = 0.996, the target moves at 0.4% of the online encoder's rate per step. This means the target encoder is a "slow" version of the online encoder and provides a stable training signal. SimSiam [11] showed that EMA is not strictly required (just stop-gradient + predictor can suffice in some regimes), but every JEPA-family work has retained it because empirically it stabilises training significantly.

Asymmetry 3 — Predictor. The predictor p_ψ transforms z_ctx before comparison with z_tgt. This transformation is learned and updated by the gradient. Tian, Chen, Ganguli [12] showed analytically — in the linear gradient-flow regime — that removing the predictor causes the dynamics to collapse onto the trivial solution. Empirically the same holds far outside that regime: every published JEPA-style training that removes the predictor collapses. The predictor cannot be the identity function; it has to learn to map online-encoder representations to a target that lags behind it.

Why V-JEPA fights a worse collapse

In video the problem is more severe. Two consecutive frames at 30 fps differ by milliseconds and are visually almost identical. If the model learns to "copy" the previous frame's representation, it scores a low loss without learning anything meaningful. The V-JEPA paper does not name this failure mode; it is descriptive shorthand to call it temporal shortcut or temporal collapse. The mitigations are explicit in the paper even if the name is not.

V-JEPA [8] fights this with three additional mechanisms compared to I-JEPA:

1. Aggressive masking: up to 90% of the video is masked (against 75% in I-JEPA). This forces the model to use long-range information.
2. Spatio-temporal block masking: masks are contiguous blocks extending across several consecutive frames. You cannot guess a masked pixel from its neighbours if those neighbours are also masked.
3. Multi-block targets: the predictor has to predict several independent target blocks from the same context. This increases signal diversity and reduces the risk of learning shortcuts specific to one target.

Despite these mitigations, V-JEPA still shows partial dimensional collapse: the effective rank of the representation covariance matrix is significantly lower than the embedding dimension. The problem is not closed, it is managed at an acceptable level.

VICReg: an explicit alternative

Worth mentioning VICReg (Bardes, Ponce, LeCun, ICLR 2022 [14]) as a conceptually different alternative. Instead of preventing collapse with asymmetry, VICReg prevents it with explicit regularisation. The loss has three terms:

VICReg is cleaner mathematically (no architectural tricks, just explicit loss terms) but requires careful tuning of the coefficients λ, μ, ν. JEPA won in popularity because it trains more easily, even if the mechanism is less transparent. The choice between these families is still an open research question.

LLMs, Diffusion, and JEPA: the three families

The AI generative debate often frames the world as LLMs vs diffusion as if those were the only options. When the task is predicting the future, there are three fundamentally different approaches, and understanding their differences clarifies where JEPA sits.

Autoregressive (LLMs)

LLMs work by autoregressive prediction. Given x_1:t, they model p(x_t+1 | x_1:t) as a categorical distribution over a discrete vocabulary. They generate one token at a time, sampling each from this distribution.

The approach can be applied to images and video by tokenising the input (VQ-VAE, VQGAN, or more recent discrete tokenisers). Models like iGPT, Parti, and NVIDIA Cosmos [16] use this paradigm. Their strength is composability with the transformer techniques matured for language: scaling laws, instruction tuning, RLHF.

Diffusion

Diffusion models (Stable Diffusion, DALL-E 3, Sora) implement an inverse stochastic process. They start from pure noise x_T ~ N(0, I) and "denoise" it iteratively across T steps (typically 20–50 in modern models, up to 1000 in original DDPMs). The model is trained to predict the noise added at each step.

The structural difference from LLMs: diffusion models operate non-autoregressively. All pixels are refined simultaneously at each step. This enables exceptional visual quality but makes the model non-interactive in the classical sense: you cannot hand it an action and ask what happens.

JEPA (non-generative)

JEPA does not generate anything. It produces no pixels, no discrete tokens. It produces only latent representations. For this reason it is not directly comparable to LLMs and diffusion as a "generative model"; it is a separate category, that of discriminative self-supervised models.

The philosophical difference is deep. LLMs and diffusion try to reconstruct the exact output (the next token, the noise-free image). To do that they must model the full distribution p(x) or p(x | y), including all irrelevant details. JEPA only tries to capture the structure of the input, discarding details by construction.

For applications that care about what will happen rather than what it will look like, JEPA is theoretically more suitable. This is the exact intuition that led Nexar to build BADAS 2.0 on V-JEPA 2 instead of a generative model.

Which architecture to use, when

The structural differences summarised:

Use a World Model when: you need an internal simulator to plan actions; you want to "dream" trajectories for reinforcement learning; you need visualisable predictions for debug or for explainability to non-technical users.

Use JEPA when: you want to pretrain an efficient encoder for downstream use; you care about classification, detection, or anomaly detection more than generation; your task is one where visual detail is noise (collision anticipation, anomaly detection); you have limited compute and abundant unlabeled data.

Combine them when: you want both. A JEPA-pretrained encoder can sit inside a World Model, replacing the standard VAE. This is exactly what recent work explores — a pure perception module (JEPA) + a dynamics module with action conditioning (World Model). V-JEPA 2-AC [9], the action-conditioned variant Meta released in June 2025, is a concrete step in this direction.

Hardware, VRAM, costs: the real numbers

The architectural discussion stays abstract until you confront the practical problem: what does it actually take to train one of these models? How much VRAM, on what hardware, for how long, with how much data? Answers change radically depending on whether you are doing pre-training from scratch, fine-tuning, or inference only. Worth doing the arithmetic.

The memory formula, from first principles

For a model with N parameters trained with Adam in mixed precision (BF16 activations + FP32 master weights, the modern standard), GPU memory breaks down as follows:

Without FlashAttention, an extra term proportional to L · b · a · s² appears (a = attention heads), which dominates for long sequences. Flash-Attention is essentially mandatory for video-scale sequence lengths today. Two more techniques can reduce memory by an order of magnitude. Gradient checkpointing reduces activations to O(√L) at the cost of ~30% extra time recomputing. FSDP / ZeRO-3 shards parameters, gradients, and optimizer state across N GPUs, reducing per-GPU memory by ~N at the cost of significant inter-GPU traffic.

Worked example: fine-tuning V-JEPA 2 ViT-L (300M params)

Same calculation for Cosmos-Reason2-2B

Hardware: when to use what

GPU choice is not just "bigger is better". The relevant axes are VRAM (capacity), bandwidth (for data-intensive training), and numerical format support (FP8 changes the picture on H100). Cloud prices below reflect mid-2026 market rates on specialised GPU providers (Lambda, RunPod, Jarvislabs, Spheron, Thunder Compute); hyperscalers (AWS, GCP, Azure) typically charge 2–6× more. Spot pricing can be 40–60% cheaper than on-demand. Prices drift weekly.

The A100 → H100 step usually pays for itself for pure training: H100 costs roughly 1.5–2× more per hour but trains roughly 3× faster in BF16 and 6× faster in FP8 (relative to A100 BF16). Payback is fast on large jobs. For inference, A100 and L40S remain competitive.

Three concrete scenarios, with costs

Data efficiency: JEPA vs World Model

Data efficiency is a different metric from compute efficiency, and the architectures separate interestingly here.

I-JEPA [7] reaches competitive ImageNet linear probing with a ViT-H/14 trained for 300 epochs in roughly 1,200 GPU-hours. MAE (Masked Autoencoder, generative reconstruction) needs about 10× more compute for similar linear probing on the same architecture. The I-JEPA paper reports being "over 10× more efficient than a ViT-H/14 pretrained with MAE". On paper: I-JEPA is significantly more compute- and sample-efficient because there is no decoder to train and the loss is in latent space.

V-JEPA 2 was pre-trained on more than 1M hours of internet video on the order of tens of millions of clips. Fine-tuning on the BADAS task requires 200k labelled videos. For comparison, Sora (diffusion video) is estimated to have been trained on tens of millions of hours of video with resources estimated at hundreds of millions of dollars.

DreamerV3 has a particular data-efficiency profile: it shines with little data because most of the learning happens in the world model's dreams. On standard pixel-based control benchmarks like the DMC Suite (DeepMind Control), it reaches strong performance within 1M environment steps; model-free baselines like PPO typically need an order of magnitude more interactions to reach comparable scores. On Atari with a 100k-step budget, DreamerV3 is among the most sample-efficient agents reported. The world model acts as a data amplifier.

What this means: for builders, investors, decision-makers

Different readers leave this article with different obligations. The technical bottom line is the same; the practical move is not.

If you build

For perceptual tasks that have to run safely and at the edge — collision anticipation, anomaly detection, visual quality control, surgical assistance, drone perception — JEPA is now the default starting point. Take a public V-JEPA 2 checkpoint, fine-tune on your domain data, distil to a small variant for deployment. Resist the temptation to use a 2B+ vision-language model as the perception layer just because it is the easiest API call: the BADAS-vs-Cosmos comparison shows the structural penalty. Reserve World Models for cases where you genuinely need to roll out actions in imagination — robotics control, RL with expensive simulators, agentic policies that have to plan multi-step interventions. For everything in between, the hybrid V-JEPA 2-AC and similar action-conditioned JEPAs are the architecture to watch through 2026.

If you invest

The compute moat for foundation-model-quality perception is concentrating in two places: pre-training on >1M hours of video (Meta, NVIDIA, a handful of others) and high-quality labelled fine-tuning data with privileged access (Nexar's 350k dashcam fleet, Tesla's vehicle fleet, certain medical imaging consortia, the autonomous logistics players). The middle layer — the people who fine-tune and deploy on existing checkpoints — is where margins are healthier and where most defensible companies will be built. Be careful of pitches that promise to "train a foundation model from scratch" outside the very specific niches where that economic argument holds.

If you decide budgets

Three numbers to internalise from this article. Pre-training a serious JEPA from scratch: about $86k of compute, plus team and data costs that dwarf that. Fine-tuning an existing checkpoint to a real task: about $2.3k of compute, days of work. Running an RL world model for robotics on a single GPU: $150–700. The implication is that the right question for most organisations is not "should we train a foundation model?" but "do we have the labelled data and the engineering bandwidth to fine-tune an existing one well, and the inference budget to deploy it at our scale?" The bottleneck is rarely compute and almost always data and integration.

Conclusion

World Models and JEPA are two different answers to the same problem: how to teach machines to predict the future by self-supervision. World Models do it by generating pixels, constrained by a decoder and a reconstruction loss. JEPA does it by staying in latent space, managing the representation collapse problem with carefully calibrated architectural asymmetry.

Pixel reconstruction is more intuitive and produces interpretable output, but it wastes representational capacity on irrelevant details. Latent prediction is more efficient and captures semantic structure, but its output is not directly interpretable and it requires the full anti-collapse arsenal we discussed.

Neither is "better" in absolute terms. They are different tools for different problems. The choice depends on what you need: a simulator to plan with (World Model), an encoder to perceive with (JEPA), or both combined in the hybrid architectures emerging right now. V-JEPA 2-AC and similar action-conditioned variants are the first concrete steps in this direction.

What unites both is the underlying intuition that has driven the field for years: the ability to predict the future is central to intelligence. A system that can only react to the present is structurally limited. The next generation of AI, which will operate in the physical world and have to reason about the consequences of its own actions, will need to imagine. The architectures we are building today are the first serious attempts to give it that ability.

Luigi Simeone is a Technology Executive, Independent Researcher, and Fractional CTO based in Turin, Italy. He holds a PhD in Signal Processing & Nonlinear Dynamics from the University of Southampton and is currently looking for the right problem to solve — preferably one that doesn't fit in a single box.