nanoVLM: The simplest repository to train your VLM in pure PyTorch

Hugging Face Blog · 1 year ago

nanoVLM is a lightweight PyTorch toolkit for training Vision Language Models that can process images and text to generate text outputs. The entire codebase runs on free-tier Google Colab notebooks and uses a vision transformer backbone (Google's SigLIP) aligned with a Llama 3-based language model through a modality projection module. Users can train a custom VLM by running two commands—cloning the repository and executing train.py—with a reference model trained on 1.7 million samples in approximately 6 hours on a single H100 GPU.