ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling

Which parts of modern generative modelling actually matter? Starting from IMLE and adding back only the essentials: we achieve FID 2.56 on ImageNet 256 in a single step.

Abstract

Generative models have undergone many generations of evolution, from VAEs/GANs to diffusion/flow matching. Along the way, the underlying techniques have become more complicated and various beliefs about what drives strong empirical performance have taken hold. Due to the success of diffusion models and flow matching, one of the more common beliefs is the importance of transforming the noise distribution to the data distribution gradually through many small transformations.

We ask whether this is truly necessary, and take a minimalist approach to designing a competitive generative model. We start with the bare-bones essentials, namely just a training objective and a model. We purposefully make both simple. For the training objective, we choose Implicit Maximum Likelihood Estimation (IMLE), and eschew more complicated alternatives such as variational inference, adversarial training and numerical integration. For the model, we eschew transformers and instead choose a moderately sized convolutional network. Then we judiciously added elements that are truly essential, which surprisingly do not include iterative denoising. The result is a single-step parameter-efficient generative model that produces high quality samples at fast speed: it achieves an FID of 2.56 on ImageNet 256 and simultaneously attains good precision and recall.

Background

Why Modern Models Take Many Steps

Diffusion and flow matching models, both members of a broader family known as stochastic interpolant models, currently define the state of the art in image generation. They succeeded earlier approaches such as GANs, which are unstable to train and prone to mode collapse, and VAEs, which train stably but tend to produce blurry, low-fidelity samples.

Their advantage is commonly attributed to how they decompose generation. Instead of mapping the prior distribution to the data distribution in one large transformation, they break the mapping into a sequence of smaller, simpler transformations, each carried out by a separate step during sampling. This decomposition is widely believed to be the key to their sample quality, and it is also why generating a single sample takes many steps instead of one.

During training, each prior sample is randomly coupled with a data sample. Many different couplings may intersect at a given point, so the network can only learn to predict their average. Thus during inference, a single step regresses to the data mean rather than a real sample. Multi-step iterative sampling is required to yield an accurate sample.

Random Coupling in Diffusion/Flow matching models: During training, each prior sample is randomly coupled with a data sample. Thus during inference, multi-step iterative sampling is required to yield an accurate sample.

Generating in a Single Step with IMLE

Implicit Maximum Likelihood Estimation takes a direct route: a neural generator maps samples from the prior to data samples in a single pass. In IMLE training, the coupling between prior and data is learnt rather than randomly assigned. Hence during inference, a single step suffices.

Learnt coupling in IMLE: The coupling is not randomly assigned but learnt through training. This means the model can directly produce accurate samples in one single step.

Training such a generator directly is generally difficult because the likelihood of the data under it is intractable. IMLE provides a workaround, implicitly maximizing that likelihood without requiring it to be known explicitly. IMLE optimizes a likelihood-based objective akin to regression, without the variational bounds and encoders of VAEs, the adversarial training of GANs, or the iterative numerical integration used by stochastic interpolant models.

The algorithm decomposes into two phases. In the matching phase, latent codes are sampled from the prior and each real image is matched to its nearest generated sample, establishing a coupling between the prior and the data. In the optimization phase, the parameters are updated to bring each generated sample closer to its matched real image. The two phases repeat until convergence.

Overview of IMLE algorithm: First, each real data point is coupled with its nearest generated sample. Next, the model's parameters are updated to reduce the distance within each coupled pair.

Can Minimalism Be Competitive?

This paper asks a simple question: is all of the machinery that modern generative models have accumulated, from elaborate architectures to sophisticated sampling schemes and intricate training objectives, truly necessary? Rather than add to it, we take a minimalist approach: we start from the simplest generative model we can, IMLE, and add only the components that prove essential. In line with the same philosophy, we use a ConvNeXt-based architecture rather than the transformer-based ones that now dominate. We identify one improvement to each of IMLE’s two phases.

Improvements to IMLE

Our two contributions improve the optimization and matching phases of IMLE respectively.

Supervising Every Stage

A Different View of Diffusion/Flow Matching Models

Stochastic interpolant methods are commonly understood to perform iterative denoising. At training time, a denoising network is trained to convert a noisy image into a clean one. At test time, that network is applied repeatedly to convert pure noise into a generated sample. This iterative application is their key distinguishing feature, and it is what they are commonly believed to owe their success to. It also creates a mismatch between training and test: the network is applied once per sample during training, but iteratively during generation.

We consider an alternative view of this mismatch. Rather than viewing the model as one instance of the denoising network, we unroll the iterations of the sampler and view the model as their composition, \(h_\theta = g^1_\theta \circ \cdots \circ g^T_\theta\), where \(g^t_\theta\) is the update applied at iteration \(t\). Under this view the model is applied just once at test time, and it is training that deviates from the norm, evaluating only part of the model at a time. Each update is now an intermediate layer of \(h_\theta\). We term the conventional view the training-centric view, and this one the testing-centric view.

Read this way, the training objective reveals something specific. These methods supervise the denoising network’s prediction at every time step, with equal weight applied to all steps, so every intermediate layer of \(h_\theta\) receives a direct training signal. We call this per-stage supervision, and we hypothesize that the sample quality of many-step models stems from it rather than from iterative sampling itself.

Testing-centric view: Unrolled, the sampler's steps form one deep compositional network, and the training objective supervises every layer of it directly. We hypothesize that this per-stage supervision, and not iterative sampling, is what drives sample quality.

Adapting It for IMLE

The unrolled view applies naturally to IMLE. Its generator can also be seen as a prior-to-data network composed of stages, only executed in a single forward pass rather than through iterative sampling. We build it as a stack of upsampling blocks, \(f_\theta = f_{\theta^{L}} \circ \cdots \circ f_{\theta^{1}}\), in which each stage operates at a progressively higher resolution. This recasts IMLE in a form similar to the diffusion construction, with each stage now analogous to \(g^t_\theta\), only indexed by resolution rather than noise level.

The architecture of existing IMLE methods is already compositional, but the training is not. The loss is computed only at the final output, which directly supervises only the last stage, and earlier stages receive a training signal only through gradients that flow back from that loss. We address this by supervising every stage directly, at the resolution at which it operates. Each stage is given an output head that produces an image at its own resolution, and its target is the real image downsampled to match.

Optimization phase improvement: Vanilla IMLE computes its loss only at the final output, so earlier stages are trained only through gradients flowing back from it. We give every stage an output head and supervise it against the real image downsampled to its own resolution.

Making Matching Robust

Outliers from Randomness in Sampling

For the IMLE algorithm, the model distribution changes during training, so the matching between model samples and ground truth data points needs to be recomputed. Due to randomness in the sampling, there may be no samples drawn from a mode of the model distribution, and as a result, a ground truth data point that can otherwise be explained by that mode would not be well matched. Note that this is different from mode collapse: the poor matching results from randomness in the sampling, not a deficiency in the model distribution.

Randomness in Sampling: Matching is recomputed every round from a fresh set of samples, so an unlucky draw can leave a real data point with no sample near it. This is a sampling artefact rather than mode collapse.

A Robust Loss

The standard squared $l_2$ loss lets these mismatches dominate the mean loss. In essence this is a problem of outliers: a small number of large-residual mismatched pairs contribute disproportionately to the mean loss compared to a much larger pool of well-matched pairs.

The classical remedy in robust statistics is to replace the squared $l_2$ loss with a function that grows sub-quadratically in the residual, so that a mismatch has less influence and the well-matched pairs dominate the update. Empirically, we find that Geman-McClure loss works best in our case.

Matching phase improvement: Replacing squared $l_2$ loss with a robust loss like Geman-McClure loss makes the matching phase robust to outliers caused by randomness in sampling.

Putting It All Together

We call the resulting method ROMS-IMLE, short for Robust Multi-Stage Implicit Maximum Likelihood Estimation. We ablate our contributions on Oxford Flowers dataset (1000 images $256 \times 256$).

Configuration	FID ↓	Precision ↑	Recall ↑
Vanilla IMLE	62.61 ± 2.28	0.45 ± 0.03	0.46 ± 0.01
IMLE + Multi-Stage	31.24 ± 1.08	0.79 ± 0.01	0.72 ± 0.02
IMLE + Multi-Stage + Geman-McClure (ours)	13.90 ± 0.08	0.93 ± 0.00	0.73 ± 0.02

Ablation on Oxford Flowers: We report the mean and standard deviation over three seeds, all evaluated at the same training iteration.

Vanilla IMLE: FID 62.61

IMLE + Multi-Stage: FID 31.24

IMLE + Multi-Stage + Geman-McClure: FID 13.90

Results

Our approach works well in both pixel and latent space.

ImageNet 256 [Latent]: FID 2.56, 310M params, 1 NFE

CelebA-HQ [Pixel]: FID 6.70, 138M params, 1 NFE

CIFAR-10 [Pixel]: FID 3.93, 1 NFE

Interpolating in prior space: Each tile in the 7 × 4 grid is a separate interpolation. Moving linearly between two prior samples produces plausible images that vary smoothly, suggesting the generator has learned a coherent mapping from the prior.

BibTeX

@article{vashist2026romsimle,
  title={ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling},
  author={Vashist, Chirag and Li, Ke},
  journal={arXiv preprint arXiv:2607.19332},
  year={2026}
}