For the past two or three years, multimodal has been at once the least controversial and the least visible direction in AI. Large language models can look at pictures, read tables, and process video, and their capabilities keep pushing outward. The industry drew the natural conclusion: the language model is already the backbone, so multimodal is just a matter of wiring images, audio, and video into it so that it understands the world more completely. Plenty of people go further and assume the next generation of foundation models can do without the multimodal path altogether.
That conclusion is not wrong. But as someone with formal training in computer vision, and who deals with AI companies of every kind in my investment work, I have always had the nagging sense that it is only half right. Much of the progress in this last round of multimodal did not come from models genuinely getting better at “seeing” or “understanding.” It came from models getting better at “talking.” Images were plugged in, video was plugged in, but the part that actually organizes information, calls on knowledge, and produces the answer is still mainly the language model. So what we get is a system that keeps getting better at polished descriptions of visual content, without necessarily having done solid modeling at the visual level, and without understanding the world the way humans and animals do. On the surface the industry is doing multimodal. Underneath, it may still be running on an extension of language-centrism.
Yesterday I watched Shengbang Tong, a PhD student of Yann LeCun and Saining Xie at NYU, defend his thesis, Searching for Multimodal Intelligence, and it hit home. Put it alongside Vincent Sitzmann’s essay The flavor of the bitter lesson for computer vision, Saining Xie’s public talk on visual representation, and Zhang Xiaojun’s two long interviews with Xiangyu Zhang and Saining Xie on multimodal, and one conclusion keeps getting sharper: what the next stage of multimodal has to fill in is not the interface but the foundation. Not a few more kinds of input wired into the language model, but a rebuilt way for machines to represent the world. We were too quick to demote vision to a preprocessing module for the language model. What truly matters next may be precisely the opposite: lifting vision back up to the position of underlying infrastructure.
I. What a Multimodal Foundation Model Actually Is, and How to Train One
The most valuable thing about Shengbang Tong’s line of work is that he never settled for “vision isn’t good enough yet.” He broke the problem into specifics. His early work on MMVP was essentially proving one thing: the trouble with many of today’s multimodal models is not that they can’t answer, but that they don’t see clearly. They can of course identify an “elephant,” a “car,” or a “person” in an image, and they can write a paragraph of explanation that reads beautifully. But on more basic visual patterns they err constantly: orientation, count, local state, relative position, viewpoint change, occlusion, text inside the image. The nastiest part is that these are usually not silent errors. They are errors that arrive with a fluent explanation. The model does not lack language ability; on the contrary, its language ability is too strong — strong enough to mask the shortfall in visual grounding.
This is why Tong keeps stressing that language-supervised encoders like CLIP are strong, but much of the time they are strong in semantics, not necessarily in vision. A few years ago that would have sounded counterintuitive, because the previous generation of multimodal systems took CLIP as the default answer for the visual front end. But look at the finer-grained visual evaluations and the problem is plainly there: the model knows roughly what something is, but cannot necessarily tell what is actually happening. Once that premise holds, the whole industry’s understanding of multimodal has to take a step back. If vision itself is far from solved, then the belief that a stronger LLM naturally brings stronger multimodal needs re-evaluating in its own right.
What deserves even more attention is that Tong’s later work on language-free visual representation didn’t just swing the answer to “then go back to pure visual self-supervision.” What he really did was something more important: control the variables as tightly as possible and compare the real differences between visual representation routes. The received wisdom in the industry has been that the language-supervised route suits multimodal and the self-supervised route suits pure vision tasks. But that claim has always mixed in too many other factors — different data volumes, different distributions, different downstream interfaces — so the comparisons never fully spoke to the methods themselves. The real value of this body of work is that it asked the question again under a fairer setup, which is why its conclusions cut deeper.
The conclusion is not a blunt one. On one hand, language-supervised encoders remain strong on many general multimodal tasks, especially OCR, charts, document understanding, and knowledge-heavy question answering; that advantage has not gone away. On the other hand, language-free visual representation has not hit the end of its road, as some people assumed. It still has real headroom on vision-centric tasks, and as models and data keep scaling, nobody has touched the ceiling yet. More importantly, this work surfaces a bigger question: what really sets the ceiling of multimodal may not be supervision type but data composition. Put bluntly: what matters most is not whether there is language supervision, but what exactly you let the model look at.
The implication for the industry is bigger than the methodological debate. When people talked about data moats in the past, they usually meant who has more data. Going forward, what matters more may be who better understands the data recipe. Which image-text data helps grounding; which text-dense images significantly lift OCR and chart understanding; which video helps the model form priors about world dynamics; which action data belongs in alignment rather than in the main body of pretraining — all of these questions are becoming critical. So what will truly be scarce is not necessarily data scale, but the systematic capability built around data composition: knowing how to mix it, knowing where it works, knowing how to measure it, knowing at which stage failures occur. That capability is worth far more than the sentence “we have massive multimodal data,” because it sits closer to the lever that actually steers how models evolve.
Take one more step and the question shifts from “how should visual representation be done” to “why have understanding and generation still not been genuinely unified.” For the past two years, one of the industry’s favorite directions has been unifying understanding and generation, as if putting image understanding and image generation into the same model would by itself move multimodal into its next stage. Xiangyu Zhang’s judgment on this in his interview was exactly right: the reason it has never truly worked is not just the sheer engineering effort. It is that underneath there is no shared representation space strong enough. Language’s control over vision is not fine-grained, image-text alignment is not solid, and the result is an understanding module and a generation module placed side by side on the surface while in practice behaving like a hybrid bolted together from two systems. Understanding can hardly feed generation, and generation can hardly improve understanding. That is why the “unified” model has never made the qualitative leap.
Follow that question down and the parts of Tong’s thesis on MetaMorph and RAE become especially critical, because what he asks is not “how do we make one model both answer questions and draw pictures.” He asks something more fundamental: if understanding and generation face the same world, why can’t they share the same deeper visual representation? Once that question is on the table, “unification” stops being a stacking of features at the product level and becomes a genuine systems problem. If understanding and generation are ever truly unified, it will not be by bolting two modules together, but by bringing them back to the same latent interface, the same internal representation. At that point generation is no longer merely an output capability; it becomes a reasoning interface inside the model. The model can compare candidates in representation space, filter paths, evaluate consistency, even do more effective test-time scaling. This direction is still early, but it addresses a structural problem, which makes it far more worth watching than yet another “large model that can also draw.”
I have always believed that unifying generation and understanding is the prerequisite for multimodal reasoning. In LLMs that is exactly how it happened: because generation and understanding could happen in one space, the model gradually acquired reasoning ability, and RL could then unlock more powerful intelligence. Go one step further: a world model should, by its very nature, unify generation and understanding. Only then can it do long-horizon, reliable next-state prediction.
II. World Models and 3D Representation
One more step brings us to the bigger question: will world models become the real battleground of multimodal? I lean more and more toward yes, and this is no longer the personal preference of a few researchers; it is a shared direction that different routes are gradually converging on. LeCun has talked about world models for years. Sitzmann’s line has moved from neural scene representation toward video planning and latent action. Saining Xie has kept reminding the industry that visual representation and visual search remain the core bottleneck, not a supporting role that gets swallowed automatically once the language model is strong enough. What gives the second half of Tong’s thesis its weight is that it has already begun testing this systematically: if you do unified multimodal pretraining from scratch, putting text, images, video, and actions into one sequence framework, does world-modeling ability grow out on its own?
At this point the importance of video changes completely. People have always treated video as “more complex images,” but going forward it is more likely a kind of world signal. Video naturally carries state change, temporal dependency, patterns of motion, causal cues, and the outcomes of actions; it is closer to how the real world actually evolves than any static image can be. This is why more and more top teams have started treating video as the key pretraining resource for the next stage of foundation models, rather than one more add-on modality.
More importantly, this judgment directly rewrites the research logic of robotics and embodied AI. The industry’s default assumption has been that action data is the scarcest and therefore the most valuable. That is not wrong, but it is only half right. The fuller statement is this: action data matters, of course, but it is not necessarily suited to being the main body of pretraining; it is more likely suited to the alignment layer, the policy layer, and the calibration layer. The bigger base underneath may come from video world models, unified representation, and long-horizon prediction. If that holds, then the strongest robotics teams of the future will not necessarily be the ones with the most teleoperation data, but more likely the ones that are first to wire up the whole chain end to end: video pretraining, state modeling, action alignment, policy extraction. Recent work from NVIDIA and Generalist, along with Chinese startups such as Dexmal and Wujie (无界动力), already counts as promising attempts down this path.
Sitzmann’s essay contains another judgment that is easy to misread but actually very important: where 3D sits. Many readers come away with an oversimplified conclusion — 3D no longer matters. I think that is reading it backwards. What is no longer important may not be 3D itself, but the old paradigm that treats explicit 3D as the bottleneck every system must pass through. Sitzmann himself was one of the key drivers of the neural-scene-representation line of work — from DeepVoxels to SIREN to much of what followed, he has gone deeper than most into the question of how to let a model represent the world in a more continuous, more geometric, more 3D way. Precisely because of that, when he later began to question the central status of explicit 3D in embodied intelligence, the signal deserves special attention. This is not someone who never did 3D heckling from outside. It is someone who pushed that road to the frontier, now re-judging it on a longer time scale: spatial intelligence of course remains critical, but the way machines acquire it does not have to follow the route of human-predefined intermediate representations all the way through.
So what will truly exit the stage is not spatial intelligence, but the old consensus that “explicit 3D must occupy the very center of the whole system.” It will of course still matter, especially in high-reliability settings such as robotics, industry, autonomous driving, and manufacturing, but it is more likely to retreat gradually into the interface layer, the constraint layer, and the verification layer — no longer the trunk line every capability must pass through. This does not contradict Professor Fei-Fei Li’s emphasis on spatial intelligence; one could even say the two are talking about the same thing. Spatial ability is extremely important; the only question is whether machines will acquire it along the intermediate-task path humans are familiar with.
If you truly put the threads from Tong, Sitzmann, Xie, and Zhang together, I think you arrive at a very clear judgment for the industry: over the next few years, what will truly be re-valued in multimodal is not “who shipped another demo that looks more human,” but underlying visual representation, data composition capability, video world models, and multimodal architecture. The competition of the future will not be about who plugged in more modalities, but about who is closer to genuinely building an internal representation system that is more faithful to the world. This is why I increasingly feel that the next hard fight in multimodal is not in the chat box but in the foundation underneath. Whoever has more solid visual grounding, a more unified representation interface, stronger video and action pretraining, and a more sensible capacity allocation mechanism has the better chance of truly defining the next generation of foundation models.
In one sentence: this last round of multimodal was essentially the language model expanding outward; the next round will be the model genuinely re-learning the world. Language matters, of course, but language is not the foundation, and it is not everything. Nor is vision an add-on. It is becoming the foundation again, just as it has been across hundreds of thousands of years of biological evolution — and once that happens, a Cambrian era of intelligence may arrive sooner than anyone expects.
References
- Shengbang Tong, PhD defense materials, Searching for Multimodal Intelligence
- Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- Tong et al., Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Fan et al., Scaling Language-Free Visual Representation Learning
- Tong et al., MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
- Tong et al., Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
- Tong et al., Beyond Language Modeling: An Exploration of Multimodal Pretraining
- Vincent Sitzmann, The flavor of the bitter lesson for computer vision
- Related work by Vincent Sitzmann: DeepVoxels, SIREN, Scene Representation Networks, True Self-Supervised Novel View Synthesis is Transferable, Large Video Planner Enables Generalizable Robot Control
- Saining Xie, talk: The multimodal future: Why visual representation still matters
- Xie et al., V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
- Zhang Xiaojun’s full interview with Xiangyu Zhang
- DreamLLM: Synergistic Multimodal Comprehension and Creation
- Rich Sutton, The Bitter Lesson
- Yann LeCun, A Path Towards Autonomous Machine Intelligence
- Fei-Fei Li, From Words to Worlds: Spatial Intelligence is AI’s Next Frontier