ReadPaper Blog
From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models
This paper argues that dense prediction with text-to-image backbones should produce task-native pixel fields directly, rather than treating depth, normals, masks, mattes, saliency, or pose as image-like targets to be generated. Its method, ReChannel, keeps the pretrained DiT input pathway, adapts the frozen model with task LoRA, and reads out each spatial token through a small linear head, yielding strong accuracy and faster inference than target-side latent decoding.
Source: From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models
The Mission
The paper starts from the observation that many dense vision problems share the same input constraint: a single RGB image must be converted into pixel-aligned fields such as depth, surface normals, alpha mattes, referring segmentation masks, saliency maps, or pose heatmaps. Wang, Lin, Li, Jiang, and Li argue that these outputs differ fundamentally from RGB synthesis because they are not new visual content to be rendered, but task-native quantities defined on the same image plane. This distinction motivates the paper’s central question: whether a large text-to-image model should be used as a generator of annotation-like images or as a pretrained organizer of spatial evidence. The authors propose that a pretrained DiT already arranges the RGB input into a patch-aligned token lattice, making dense prediction a field readout problem. The implication is that generative pretraining can provide semantic, structural, and geometric priors without forcing dense targets through an RGB-generation output interface.
The Detour
The paper identifies a specific inefficiency in prior generative and editing approaches to dense prediction: annotations are often encoded into the latent space of an RGB-trained VAE and decoded back into pixels as if they were images. This target-side VAE path is natural for RGB synthesis, where the model must reconstruct color, texture, illumination, and fine appearance, but it is indirect for outputs evaluated as pixel-space quantities. The authors frame this as an inherited interface problem rather than a limitation of generative pretraining itself. Dense prediction supervision and metrics operate on task-native fields, so forcing depth, normals, masks, mattes, or heatmaps through an RGB reconstruction module can add unnecessary representation and decoding constraints. The paper’s diagnostic using participation ratio further supports this view by noting that task-adapted token fields occupy compact subspaces compared with the higher-dimensional RGB input field.
ReChannel
ReChannel implements the paper’s readout formulation by preserving the pretrained text-to-image model’s RGB input route while removing the target-side decoder entirely. The RGB image is encoded with the model’s VAE encoder so that the DiT receives inputs in the distribution and coordinate system it was pretrained on. The DiT backbone is then run deterministically in zero-noise mode, with its original weights frozen and task-specific LoRA parameters providing lightweight adaptation. For each task, the adapted spatial token field is mapped by a shared token-local linear projection into a corresponding p x p x K_t pixel-space patch. Because this head has no spatial mixing and only about 33K parameters, the method makes the DiT token lattice responsible for spatial organization while the readout merely reinterprets token channels as task-native quantities.
The Fight Test
The empirical case for ReChannel is built on FLUX-Klein across six dense prediction tasks and more than a dozen benchmarks. The paper reports new state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation, while remaining competitive on surface normals, saliency, and pose. These tasks span continuous geometry, high-frequency opacity estimation, language-conditioned masks, and multi-channel heatmaps, which strengthens the claim that the same readout interface is not narrowly specialized. Controlled comparisons indicate that performance comes from the output interface rather than from a larger prediction head: a 13x larger head does not improve results, and training from scratch collapses. In a matched 4B setting, ReChannel is also reported as more accurate and 2.48x faster than an edit-plus-latent-decode counterpart.
The Verdict
The paper’s broader implication is that text-to-image pretraining is valuable for dense perception because it learns rich image-formation priors, but dense prediction should not inherit the full machinery of RGB target generation. ReChannel separates the useful pretrained RGB prior from the unnecessary target-side rendering path, replacing generation with direct field readout on the DiT’s patch-indexed spatial lattice. This reframing aligns the model interface with the way dense tasks are supervised and evaluated: as pixel-space fields rather than decoded images. The method also suggests a minimalist design principle for future dense predictors using generative backbones: keep the input compatibility that the pretrained model expects, adapt the token field lightly, and read out task-native outputs directly. The reported accuracy and speed gains support the paper’s conclusion that dense perception can benefit from generative scale without adopting a generative output interface.
