ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling
A single-step, transformer-free generative model trained with IMLE that reaches an FID of 2.56 on ImageNet 256 without iterative denoising.
Abstract
Generative models have undergone many generations of evolution, from VAEs/GANs to diffusion/flow matching. Along the way, the underlying techniques have become more complicated and various beliefs about what drives strong empirical performance have taken hold. Due to the success of diffusion models and flow matching, one of the more common beliefs is the importance of transforming the noise distribution to the data distribution gradually through many small transformations.
We ask whether this is truly necessary, and take a minimalist approach to designing a competitive generative model. We start with the bare-bones essentials, namely just a training objective and a model. We purposefully make both simple. For the training objective, we choose Implicit Maximum Likelihood Estimation (IMLE), and eschew more complicated alternatives such as variational inference, adversarial training and numerical integration. For the model, we eschew transformers and instead choose a moderately sized convolutional network. Then we judiciously added elements that are truly essential, which surprisingly do not include iterative denoising. The result is a single-step parameter-efficient generative model that produces high quality samples at fast speed: it achieves an FID of 2.56 on ImageNet 256 and simultaneously attains good precision and recall.
Setting the Stage
Why Modern Models Take Many Steps
Diffusion and flow matching models, both members of a broader family known as stochastic interpolant models, currently define the state of the art in image generation. They succeeded earlier approaches such as GANs, which are unstable to train and prone to mode collapse, and VAEs, which train stably but tend to produce blurry, low-fidelity samples.
Their advantage is commonly attributed to how they decompose generation. Instead of mapping the prior distribution to the data distribution in one large transformation, they break the mapping into a sequence of smaller, simpler transformations, each carried out by a separate step during sampling. This decomposition is widely believed to be the key to their sample quality, and it is also why generating a single sample takes many steps instead of one.
During training, each prior sample is randomly coupled with a data sample. Many different coupling may intersect at a given point, the network can only learn to predict their average there. Thus during inference, a single step regresses to the data mean rather than a real sample. Multi-step iterative sampling is required to yield an accurate sample.
Generating in a Single Step with IMLE
Implicit Maximum Likelihood Estimation takes a direct route: a neural generator maps samples from the prior to data samples in a single pass. In IMLE training, the coupling between prior and data is learnt rather than randomly assigned. Hence during inference, a single step suffices.
Training such a generator directly is generally difficult because the likelihood of the data under it is intractable. IMLE provides a workaround, implicitly maximizing that likelihood without requiring it to be known explicitly. IMLE optimizes a likelihood-based objective akin to regression, without the variational bounds and encoders of VAEs, the adversarial training of GANs, or the iterative numerical integration used by stochastic interpolant models.
The algorithm decomposes into two phases. In the matching phase, latent codes are sampled from the prior and each real image is matched to its nearest generated sample, establishing a coupling between the prior and the data. In the optimization phase, the parameters are updated to bring each generated sample closer to its matched real image. The two phases repeat until convergence.
Can Minimalist Be Competitive?
This paper asks a simple question: is all of the machinery that modern generative models have accumulated, from elaborate architectures to sophisticated sampling schemes and intricate training objectives, truly necessary? Rather than add to it, we take a minimalist approach: we start from the simplest generative model we can, IMLE, and add only the components that prove essential. In line with the same philosophy, we use a ConvNeXt-based architecture rather than the transformer-based ones that now dominate. We identify one improvement to each of IMLE’s two phases.
Improving IMLE
Our two contributions improve the optimization and matching phases of IMLE respectively.
Supervising Every Stage
A Different View of Diffusion/Flow Matching Models
Stochastic interpolant methods are commonly understood to perform iterative denoising. At training time, a denoising network is trained to convert a noisy image into a clean one. At test time, that network is applied repeatedly to convert pure noise into a generated sample. This iterative application is their key distinguishing feature, and it is what they are commonly believed to owe their success to. It also creates a mismatch between training and test: the network is applied once per sample during training, but iteratively during generation.
We consider an alternative view of this mismatch. Rather than viewing the model as one instance of the denoising network, we unroll the iterations of the sampler and view the model as their composition, \(h_\theta = g^1_\theta \circ \cdots \circ g^T_\theta\), where \(g^t_\theta\) is the update applied at iteration \(t\). Under this view the model is applied just once at test time, and it is training that deviates from the norm, evaluating only part of the model at a time. Each update is now an intermediate layer of \(h_\theta\). We term the conventional view the training-centric view, and this one the testing-centric view.
Read this way, the training objective reveals something specific. These methods supervise the denoising network’s prediction at every time step, with equal weight applied to all steps, so every intermediate layer of \(h_\theta\) receives a direct training signal. We call this per-stage supervision, and we hypothesize that the sample quality of many-step models stems from it rather than from iterative sampling itself.
Bringing It to IMLE
The unrolled view applies naturally to IMLE. Its generator can also be seen as a prior-to-data network composed of stages, only executed in a single forward pass rather than through iterative sampling. We build it as a stack of upsampling blocks, \(f_\theta = f_{\theta^{L}} \circ \cdots \circ f_{\theta^{1}}\), in which each stage operates at a progressively higher resolution. This recasts IMLE in a form similar to the diffusion construction, with each stage now analogous to \(g^t_\theta\), only indexed by resolution rather than noise level.
The architecture of existing IMLE methods is already compositional, but the training is not. The loss is computed only at the final output, which directly supervises only the last stage, and earlier stages receive a training signal only through gradients that flow back from that loss. We address this by supervising every stage directly, at the resolution at which it operates. Each stage is given an output head that produces an image at its own resolution, and its target is the real image downsampled to match.
Making Matching Robust
Randomness in sampling causes outliers
For the IMLE algorithm, the model distribution changes during training, so the matching between model samples and ground truth data points needs to be recomputed. Due to randomness in the sampling, there may be no samples drawn from a mode of the model distribution, and as a result, a ground truth data point that can otherwise be explained by that mode would not be well matched. Note that this is different from mode collapse: the poor matching results from randomness in the sampling, not a deficiency in the model distribution.
A Robust Loss
The standard squared $l_2$ loss lets these mismatches dominate the mean loss. In essence this is a problem of outliers: a small number of large-residual mismatched pairs contribute disproportionately to the mean loss compared to a much larger pool of well-matched pairs.
The classical remedy in robust statistics is to replace the squared $l_2$ loss with a function that grows sub-quadratically in the residual, so that a mismatch has less influence and the well-matched pairs dominate the update. Empirically, we find that Geman-McClure loss works best in our case.
Putting It All Together
We call the resulting method ROMS-IMLE, short for Robust Multi-Stage Implicit Maximum Likelihood Estimation. We ablate our contribution on Oxford Flowers dataset (1000 images $256 \times 256$).
| Configuration | FID ↓ | Precision ↑ | Recall ↑ |
|---|---|---|---|
| Vanilla IMLE | 62.61 ± 2.28 | 0.45 ± 0.03 | 0.46 ± 0.01 |
| IMLE + Multi-Stage | 31.24 ± 1.08 | 0.79 ± 0.01 | 0.72 ± 0.02 |
| IMLE + Multi-Stage + Geman-McClure (ours) | 13.90 ± 0.08 | 0.93 ± 0.00 | 0.73 ± 0.02 |
Results
Our approach works well in both Pixel and Latent space.
Takeaway
A single-step generative model can retain the sample quality and diversity commonly exhibited by stochastic interpolant models, without decomposing generation into gradual transformations.
BibTeX
@article{vashist2026romsimle,
title={ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling},
author={Vashist, Chirag and Li, Ke},
journal={arXiv preprint arXiv:2607.19332},
year={2026}
}