Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers

Diffusion models have quietly become the most interesting corner of open-source AI over the past two years, with releases like FLUX.1-dev and Wan 2.1 pushing text-to-image and text-to-video into serious territory. The problem was never inference — Diffusers handled that beautifully. The problem was training these things at scale without babysitting checkpoint formats or rewriting your pipeline every time a new architecture showed up.

NVIDIA's NeMo Automodel, now integrated with Hugging Face's Diffusers library, is aimed squarely at that gap. Point it at any Diffusers model ID on the Hub — FLUX.1-dev, Wan 2.2, HunyuanVideo 1.5, Qwen-Image — and it trains using the actual Diffusers model classes and pipelines. No separate training format, no conversion step before or after. A fine-tuned checkpoint drops right back into a DiffusionPipeline for inference or straight up to the Hub for sharing, and it keeps working with quantization, LoRA adapters, and custom samplers downstream.

The real trick is that parallelism becomes a config choice instead of a code rewrite. FSDP2, tensor parallel, context parallel, pipeline parallel — you declare it in a YAML file rather than hand-coding sharding logic for a 12-billion-parameter FLUX model or a 13-billion-parameter HunyuanVideo. NVIDIA walked through a concrete example: fine-tuning FLUX.1-dev on the 78-image Rider-Waite tarot dataset across 8 H100 GPUs, using pre-cached VAE latents and a trigger token ('trtcrd') to steer style. After 200 steps, prompts using the trigger picked up the tarot card's cream-red-black palette and ink-line aesthetic, while identical prompts without the trigger stayed photographic — proof the adaptation is localized rather than a wholesale rewrite of the base model.

The benchmark numbers, all on a single node of 8× H100 80GB cards, give a sense of what this costs in practice. Full fine-tuning of FLUX.1-dev at 512×512 runs about 0.9 seconds per step and 4.4 images/s per GPU, chewing through nearly 64 GiB of peak memory. Swap in LoRA rank 64 and memory climbs slightly but throughput jumps to 6.7 images/s per GPU, since you're updating a fraction of the parameters. Video is heavier, unsurprisingly — full fine-tuning of Wan 2.1's 14B variant takes 3.8 seconds per step and produces roughly a quarter of a clip per second per GPU, even with activation checkpointing turned on.

What matters longer-term isn't this one tarot demo, it's the promise that new Diffusers models can plug into this training stack with minimal extra code — a data handler and a model adapter, according to NVIDIA, rather than a bespoke training script. NeMo Automodel currently only supports flow-matching objectives, so it's not universal, but for the flow-matching models that dominate the current open-source video and image scene, that's a meaningful chunk of coverage. A fully typed Python API is also on the way, for teams who'd rather compose recipes in code than maintain YAML files.