5.8.3 COMBO: composition under decentralized observation

5.8.3 COMBO: composition under decentralized observation

COMBO moves the same empirical program into embodied multi-agent planning [ Zhang et al. , 2025 ] . Its agents receive partial egocentric RGBD observations, yet must reason about the consequences of joint action in a shared world. A generative reconstruction module first estimates an overall world state. The framework then assigns three roles to vision–language models: an Action Proposer generates candidate actions, an Intent Tracker predicts what the other agents may do, and an Outcome Evaluator scores imagined states. A tree search combines these modules with a video world model to plan over longer cooperative horizons.

The compositional world model exploits the factorization of a joint action \(a=(a_1,\ldots ,a_n)\). If \(X\) is a future video and \(x_0\) the estimated current state, its idealized product takes the form

The corresponding diffusion scores are composed at inference time. This lets one learned dynamics model simulate joint actions for varying team sizes. The reported evaluation uses three embodied cooperation benchmarks with two to four agents, including cooking and visually specified puzzle tasks in ThreeDWorld.

COMBO is especially revealing through the Witsenhausen lens developed later in this book. Each agent has a local information field, the available actions depend on that information, and another agent’s intention is latent rather than directly observed. The estimated global image is therefore not simply “the world.” It is a comparison object assembled from partial views, temporal memory, and generative completion. Likewise, the Intent Tracker’s output is a defeasible belief about another decision mechanism, not an observation of that mechanism.

An ORACLE formulation would keep these warrants typed. Egocentric observation categories would map into a candidate shared-world category; overlap maps would test whether local reconstructions agree; agent-indexed action objects would combine into a joint-action object; and rollout would ask for a coherent extension of those local data through time. When a predicted joint outcome fails, repair could then be localized to perception, cross-view gluing, intent inference, action composition, dynamics, or evaluation. COMBO currently implements these functions as a powerful planning architecture, but it does not require the comparison maps to satisfy a categorical universal property or certify which previously established conclusions survive a repair.

Doctrinal reading.

COMBO fixes a decentralized observation doctrine, a factorization of joint actions, a generative comparison object for the shared state, and role-specific interfaces for proposing, tracking, and evaluating. Tree search solves the planning problem inside that package. A persistent inconsistency among local views may instead require revising the shared-state construction, the intent ontology, or the joint-action factorization itself.

Thus COMBO adds an important dimension to the distinction above. Compositional world modeling is not only about recombining objects or skills; in a team it must also reconcile multiple information histories and models of other agents. That is exactly where UOCL, Witsenhausen information structures, and the book’s later theory of coherent repair can provide a foundation beyond architectural modularity.

The same analysis applies to multimodal foundation models. Cross-modal alignment supplies comparison maps between local presentation categories; grounding requires those maps to commute with action and observation in the world, not merely to place paired samples near one another in a latent metric.