ReadPaper Blog
Scalable Visual Pretraining for Language Intelligence
The paper argues that foundation models lose important scientific knowledge when visually rich documents are reduced to plain text before pretraining. It introduces Visual Pretraining, an unsupervised autoregressive approach that trains on rendered document pages as visual tokens and predicts next visual latents in a frozen visual-feature space. Across matched corpora, multiple model backbones, and scientific reasoning benchmarks, the study finds that preserving document visuals improves language reasoning, scaling efficiency, and cross-modal alignment.
Source: Scalable Visual Pretraining for Language Intelligence
The Hidden Knowledge in Pages
Scalable Visual Pretraining for Language Intelligence asks whether language intelligence can be learned more effectively from scientific documents in their native visual form rather than from extracted text alone. The paper starts from the observation that papers, textbooks, and technical reports encode knowledge not only in words but also in figures, tables, typeset equations, and page-level layout. Its central claim is that converting these sources into linear token sequences is a lossy projection of a richer representational structure. The authors connect this motivation to cognitive science evidence about diagrammatic and spatial reasoning, as well as to the Platonic Representation Hypothesis, which suggests that different modalities can converge toward shared abstractions. The implication is that visual documents are not merely containers for text, but direct sources of trainable structure for foundation models.
What Gets Lost
The paper identifies a destructive bottleneck in prevailing pretraining pipelines for scientific corpora. At the data level, visually rich documents are commonly parsed from HTML, LaTeX, or PDFs into plain text, sometimes using neural document-parsing systems such as MinerU2.5 before the language model ever sees the material. This conversion can discard geometric constraints, symbolic topology, table structure, formula layout, and correspondences between captions, diagrams, and surrounding prose. At the training level, existing multimodal models may use images as conditioning context for text-token prediction, but the paper argues that this still leaves visual content outside the core predictive objective. By contrasting these regimes with direct learning from rendered pages, the study isolates document representation as a key variable in scientific reasoning performance.
Visual Pretraining
The proposed method, Visual Pretraining, trains foundation models directly on raw document pages without text extraction or image-text pairing supervision. In the VP pathway, pages from the same scientific-PDF corpus are rendered as images, filtered into foreground visual patches, and represented as visual tokens. The model is trained autoregressively with next visual latent prediction in a frozen visual-feature space, so visual structure becomes part of the predictive process rather than an auxiliary input. This design is compared against Text Pretraining, where the same underlying documents are converted into parsed text and used for text-token prediction. Because the corpus source is matched across VP and TP, the experiments are designed to test whether retaining visual form provides learning signal beyond the textual content alone.
The Scoreboard
The empirical results show that Visual Pretraining improves text-only scientific reasoning under matched document sources and identical supervised fine-tuning. The paper evaluates across Qwen 3.5, Qwen 3, Llama 3.2 Vision, and Llama 3.1, covering both native multimodal and language-only backbones. On benchmarks including MMLU-Pro, GPQA Diamond, AIME 2025, and HLE, VP consistently outperforms the matched TP baseline, with reported gains including up to 3.22 points on GPQA Diamond and up to 2.1 points on MMLU-Pro. The study reports that VP reaches these benefits while using about 20B visual tokens compared with roughly 80B text tokens for TP, corresponding to only 25% of the token budget. The authors interpret the stronger gains on GPQA and MMLU-Pro than on HLE as evidence that VP particularly helps where scientific knowledge is embedded in visual and structural document cues.
The Takeaway
The broader conclusion is that visual documents can serve as scalable pretraining data for language intelligence, not only as inputs for downstream multimodal tasks. The paper reports that VP gains increase with retained visual-token budget, reaching normalized improvements over TP of 1.27x on MMLU-Pro, 2.02x on GPQA, and 2.88x on AIME-25 relative to the starting checkpoint. It also links better visual-feature cosine similarity during next-token visual-feature prediction to larger downstream multimodal gains, suggesting that improved visual-space modeling supports transfer. The visual-structure-density analysis further strengthens the mechanism: VP performs similarly to TP on text-dominant pages but gains more on examples rich in figures, equations, tables, and layout. These findings support the paper’s claim that preserving the native visual form of scientific documents can improve reasoning efficiency and cross-modal alignment without relying on explicit image-text pairing supervision.
