Biological organisms perceive, learn from, and act within their environment through multiple sensory modalities such as vision and hearing, seamlessly integrating complementary signals to form a richer understanding of their surroundings. Building neural networks with similar capabilities has long been a goal in machine learning, yet has proven difficult due to challenges in multitask learning, requirements for modality-specific architectural choices, and a lack of large-scale aligned multimodal datasets.
In this thesis we develop a framework for training neural networks that accept arbitrary combinations of modalities as input and produce arbitrary modalities as output. A key insight enabling unified and stable training across diverse types of modalities is the use of tokenization. By encoding each modality (such as images, geometry, semantics, text, structured data, etc.) into sequences of discrete tokens, we establish a common format that enables joint training with a single loss function. At the core of our framework is a multimodal masked modeling objective (MultiMAE, 4M), where a network learns to predict random subsets of tokens across all modalities from other random subsets. With this formulation, networks learn to fuse information across modalities and to model their distributions, resulting in capable any-to-any multimodal generative models. Due to the lack of large-scale and diverse aligned multimodal datasets, we resort to pseudo labeling on web-scale image-caption datasets, and generate high-quality aligned data using off-the-shelf specialist networks.
We demonstrate that this framework scales to tens of diverse modalities and tasks and to models with billions of parameters (4M-21). The resulting any-to-any multimodal models express a broad set of capabilities: out of the box, they are able to solve a range of classical vision tasks, perform any-conditional generation, and transfer well to novel downstream tasks. Because inputs and outputs share the same structure, generated modalities can be fed back as inputs, enabling self-consistent multimodal ``chain-of-thought'' generation.
Finally, we question the standard practice of representing modalities as fixed-size token sequences, and investigate ways to adapt the representations to a sample's inherent complexity. We introduce a flexible tokenizer (FlexTok) that encodes images into variable-length 1D token sequences, where early tokens capture high-level semantics and later tokens add details. This formulation enables autoregressive image generation to describe images using a coarse-to-fine "visual vocabulary'", paving the way for more sample-efficient multimodal training.
