nanoVLM: The simplest repository to train your VLM in pure PyTorch

Hugging Face just open-sourced nanoVLM, a minimalist toolkit for training vision-language models that fits comfortably on a free-tier Colab notebook. The project takes its name and spirit from Andrej Karpathy's nanoGPT, applying the same philosophy — strip a complex model down to its readable essentials — to the messier world of multimodal AI.

The architecture is a straightforward mashup of two things people already know well. The vision side borrows Google's SigLIP encoder, while the language side follows the Llama 3 design, in this case using HuggingFaceTB's SmolLM2-135M. Bridging the two is a Modality Projection module, which runs image embeddings through a pixel-shuffle operation and a linear layer before concatenating them with text embeddings and handing everything off to the language decoder. Pixel shuffle matters more than it sounds: it cuts down the number of image tokens, which keeps training fast on transformer decoders that get expensive fast as input length grows.

Getting started is almost aggressively simple: clone the repo, run python train.py, done. Under the hood, the training script handles dataset loading through Hugging Face's load_dataset API, splits data into train and validation sets, and initializes the model either from scratch or from a checkpoint via VisionLanguageModel.from_pretrained. One nice touch — because the Modality Projection layer starts untrained while the vision and language backbones are already pretrained, the optimizer uses two separate learning rates, a higher one for the projector and a gentler one for the backbones, so the new component can catch up without wrecking what the backbones already know.

Hugging Face put its own build through the paces too, training a nanoVLM checkpoint on roughly 1.7 million samples from the Cauldron dataset in about six hours on a single H100 GPU. It's not meant to challenge state-of-the-art VLMs, and the team is upfront about that. The point is transparency — a model small and simple enough that you can actually trace how image and text inputs turn into generated text, down to the tokenizer, the image processor, and the generate() call.

Inference is just as stripped-down: run generate.py with an image and a prompt, or poke at the trained model through a hosted Hugging Face Space if you'd rather skip local setup entirely.