ReadPaper Blog
Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
The paper proposes SpectraReward, a training-free reward function for text-to-image reinforcement learning that uses a pretrained multimodal large language model to measure how recoverable the original prompt is from a generated image. Instead of collecting human preference labels, fine-tuning a reward model, asking for scalar judgments, or decomposing prompts into verification questions, SpectraReward computes the image-conditioned prompt log-likelihood in a single teacher-forced forward pass. The result matters because the method provides an efficient, off-the-shelf reward signal that improves text-to-image generation across multiple models, algorithms, and out-of-distribution benchmarks.
Source: (none provided)
Read It Back
The paper addresses a central bottleneck in reinforcement learning for text-to-image generation: the reward model often determines how far a generator can be improved, but strong reward models are expensive to build and hard to calibrate. Existing approaches based on human preference annotations require costly data collection and reward-model training, while zero-shot MLLM judges can be noisy when asked to emit scalar quality scores. SpectraReward reframes the problem by asking whether the generated image contains enough information for a pretrained MLLM to assign high likelihood to the original prompt. This turns image-text alignment knowledge already learned during MLLM pretraining into a reward signal without additional supervision. The paper’s core claim is that prompt recoverability from an image is a practical proxy for compositional fidelity and instruction following in text-to-image reinforcement learning.
The Semantic Spectrum
SpectraReward conditions a frozen pretrained MLLM on a generated image and then evaluates the original text prompt with teacher forcing. For each prompt token, the model produces an image-conditioned log-likelihood, and the resulting token-level profile is named the semantic spectrum. The scalar reward used for reinforcement learning is the average image-conditioned prompt-token log-likelihood, so the reward is continuous, deterministic, and obtained from one forward pass. This design avoids direct judge calibration problems because the MLLM is not asked to invent a score, and it avoids VQA-style decomposition because the prompt is evaluated in its original form. The paper positions this likelihood-based signal as a dense measure of how well specific textual requirements are supported by the generated image.
Self-Reward in One Brain
The paper further introduces Self-SpectraReward for unified multimodal models, where the same architecture contains both an understanding branch and a generation branch. In this setting, the policy’s own understanding branch supplies the image-conditioned prompt likelihood used to train its generation branch. This creates a closed-loop self-improving framework that does not depend on an external reward MLLM or external knowledge. The authors argue that this internal reward source has a structural advantage because it shares tokenizer, vision encoder, and pretraining distribution with the generator being optimized. The hypothesis is that reward-policy alignment can make the reward signal more useful than a larger but distributionally mismatched external evaluator.
What the Experiments Say
The experimental study evaluates SpectraReward broadly rather than as a single-model demonstration. The paper reports reinforcement-learning experiments over two generator backbones, including SD3.5-M and BAGEL, three RL algorithms, and nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters. Generalization is tested on out-of-distribution text-to-image benchmarks including TIIF-Bench, GenEval, GenEval2, WISE, and DPGBench. Both SpectraReward and Self-SpectraReward consistently improve downstream generation performance relative to the corresponding baselines. The reported results include gains for BAGEL of +10.0 on TIIF-Bench and +4.3 or +5.5 on GenEval, and SpectraReward is reported to outperform AlphaGRPO by 6.3 on TIIF-Bench and 2.1 on GenEval.
Punchline: Bigger Isn’t Always Better
A key finding of the paper is that reward-model scale does not monotonically translate into better image-generation reinforcement learning. Among external MLLM backbones, the paper reports that larger models are not always stronger reward sources, with Qwen3-VL-30B-A3B identified as the best external MLLM in the described comparison rather than the largest 235B-scale option. Self-SpectraReward with BAGEL’s own understanding branch matches or surpasses much larger external reward models in several comparisons. The paper reports Self-SpectraReward improvements over the best external SpectraReward backbone of +1.2 on GenEval, +2.1 on GenEval2, and +2 on WISE. The implication is that effective reward design for text-to-image RL depends not only on evaluator capacity, but also on how well the reward model’s perception and distribution align with the generator being trained.
