NVIDIA open-sources PixelUMM, an encoder-free model that reads and writes raw pixels to understand and generate images and video

Why it matters

【Key points】

NVIDIA researchers and the University of Waterloo open-sourced PixelUMM, one model that both understands and generates images and video. The paper went up on Sept 29, followed by Apache 2.0 code on GitHub and weights on Hugging Face. It drops two standard parts of today's multimodal models: no VAE latent space and no separate vision encoder (ViT). It reads and writes raw pixels directly.

【Verifiable】

Per the paper and model card, PixelUMM builds on Qwen3-8B as a Mixture-of-Transformers: understanding and generation have separate experts but share one self-attention over text, clean pixels and noisy pixels. Images become 16x16 patches and videos 16x16x4-frame tubelets, each fed in through a single linear projection; text is autoregressive and visuals use pixel-space flow matching. The card lists about 15.2B parameters. The project page reports MMMU 41.67 (BAGEL 55.30, Qwen3-VL-Instruct 8B 69.60), AI2D 80.12 (BAGEL 89.20) and DocVQA 90.42 (Qwen2.5-VL 7B 94.90). The team notes the linear pixel head leaves faint grid-aligned steps in smooth areas, worse at high guidance, and the released checkpoint uses that head. Weights are noncommercial, research and evaluation only.

【Our read】

PixelUMM shows a single Transformer working directly on pixels can handle image and video understanding and generation in one architecture, cutting two components that are usually trained and aligned separately. The cost is still visible: on a similar 8B-class backbone, reasoning-heavy understanding trails encoder-based models by a wide margin, and generation needs heavier output heads to remove artifacts. It is a proof of direction, not a replacement, and noncommercial weights limit near-term use. Watch whether the gap closes at larger scale; if it does, next-generation multimodal models may drop the VAE and vision encoder too.