ora-0069

4.8.2 Beyond tokens: JEPA and predictive grounding

A second objection strengthens the first. Perhaps token-predictive Transformers are only an early implementation, while joint-embedding predictive architectures already address the deeper complaint that learned composition is not grounded in a physical world. A JEPA predicts a target representation from a context representation rather than reconstructing every target pixel or token. I-JEPA applies this principle to spatial blocks of an image [ Assran et al. , 2023 ] ; V-JEPA 2 combines large-scale video prediction with an action-conditioned latent predictor that can be used for model-predictive robot control [ Assran et al. , 2025 ] . This program is explicitly motivated as a route from self-supervised perception to world models, planning, and autonomous intelligence [ LeCun , 2022 ] .

The basic architecture has a direct categorical reading. Let \(\mathcal W\) be a candidate world category, let \(O:\mathcal W\to \mathcal X\) be a sensory observation functor, and let \(E:\mathcal X\to \mathcal Z\) be a learned encoder. The predictor operates in the latent category \(\mathcal Z\), comparing a predicted target \(\widehat z_T\) with the target-encoder value \(z_T\).

Figure 4.1 A JEPA as a latent predictive diagram. The learned comparison can discard sensory detail, but the training loss alone does not determine whether the retained quotient supports the queries, interventions, and invariances of the discovered world.

Prediction in \(\mathcal Z\) is a powerful form of grounding only relative to what the encoder preserves. Suppressing unpredictable texture may expose object motion and physical regularity; suppressing a distinction needed for a later counterfactual or social query destroys relevant structure. The choice of latent representation is therefore a choice of observational quotient, not a neutral window onto the world.

Definition 4.7 Query-sufficient grounded representation

For a declared query family \(\mathcal Q\), the latent representation \(E\) is \(\mathcal Q\)-sufficient when every query \(q:\mathcal W\to \mathcal Y\) in \(\mathcal Q\) factors, up to a declared equivalence, as

\[ q\simeq \bar q\circ E\circ O \]

for some latent query \(\bar q:\mathcal Z\to \mathcal Y\). It is interventionally coherent when each admissible world action \(a:w\to w'\) has a latent predictor \(P_a\) for which the observation, encoding, and prediction square commutes up to the declared comparison.

This definition gives the JEPA proposal a demanding but fair ORACLE test. A low representation-prediction loss is evidence that some target information is predictable from context. It is not by itself evidence that all \(\mathcal Q\)-relevant distinctions factor through the representation, that different viewpoints induce coherent maps, or that action-conditioned rollouts identify the correct intervention. Conversely, if these factorization and coherence conditions do hold, JEPA may be an especially natural computational substrate for an ORACLE learner.

The six Spelke systems sharpen the comparison. Object continuity and contact, viewpoint geometry, and persistent shape should become tests of latent invariance and action coherence. Approximate number requires preservation of magnitude structure rather than merely decodability by a probe. Goal-directed agency requires counterfactual sensitivity to actions. Social and linguistic structure require communicative compositions that video prediction alone need not identify. Thus “grounded” is not a single property: it is indexed by the core fragment, observer, action class, and query family.

JEPA also reveals why the Transformer-versus-world-model opposition is too coarse. Contemporary JEPAs use Transformer encoders and predictors, while multimodal systems can align latent video models with language models. The scientific comparison concerns objectives, presentations, and preserved structure—not brand names for network components.