Search and recommendation systems increasingly need to return a set of results, not one best match. A query like ‘camping gear’ should return a tent, a sleeping bag, a stove, and a headlamp. It should not return 10 near-identical tents.
Google Research has introduced Retrieve-for-Train (R4T) for this problem. The framework uses reinforcement learning (RL) once, offline, to learn good query fan-out. It then distills that behavior into a small diffusion model that produces all retrieval directions in 1 pass.
Why Standard Fan-Out Falls Short
Query fan-out splits one broad prompt into several sub-queries. The research team identifies 2 problems when a generic LLM does this at inference time.
The first is paraphrastic collapse. For ‘Bohemian festival style,’ zero-shot Qwen3-4B wrote ‘bohemian festival fashion’ and ‘festival bohemian clothes.’ These near-synonyms retrieve a homogeneous slate.
The second is latency. Autoregressive generation plus repeated retrieval calls is slow. Best-of-N sampling improves quality but multiplies inference cost.
The 3-Step R4T Pipeline
Fan-out LM training: A fan-out language model (FOLM) generates k sub-queries. A frozen dense retriever executes them. A set-level reward scores the whole retrieved set, not each item alone.
Supervision synthesis: The trained FOLM samples 128 fan-outs per query at temperature 0.9. These become (query, target set) training pairs with no human labels. For open-ended tasks, targets are retrieved-content embeddings. For compositional tasks, targets are sub-query embeddings.
Diffusive retriever training: A 53.9M-parameter diffusion transformer learns to map a query embedding to a full set of target embeddings. It uses a variance-exploding formulation within the EDM framework. At inference, it generates all embeddings in a single non-autoregressive pass. Nearest-neighbor search then maps each embedding to database items.
Reward Design and Reward Hacking
For open-ended abstract retrieval (OAR), the reward combines 3 weighted terms:
Groundedness (λg = 0.6): penalizes distance between each sub-query embedding and its nearest database item.
Diversity (λd = 0.2): the Vendi Score over representative retrieved items, such as the top-1 item per sub-query.
Alignment (λa = 0.2): mean cosine similarity between each sub-query and the original query.
For weakly supervised compositional retrieval (WSCR), the reward is the fraction of reference-set items the fan-out retrieves.
The ablation explains why all 3 OAR terms are important. With groundedness alone, Gemma3-4B converged to strings like ‘line ending line ending line ending.’ Adding alignment made collapse even faster, as the policy repeated paraphrases of the query. Adding diversity closed both shortcuts.
Training uses GRPO with soft PPO regularization, which adds forward and reverse KL penalties. Key settings include group size 8, learning rate 1×10⁻⁷, and global batch size 512.
Interactive Explainer