A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi

arXiv cs.AI · 6 hours ago

Researchers deployed MiniCPM-V-4.6, a multimodal AI model with vision and language capabilities, entirely on a 2011 NVIDIA Tesla C2075 GPU with 6GB memory by optimizing CUDA kernels and quantization strategies. The all-GPU implementation achieved 114 tokens/second at 2k context length and 1.7 seconds end-to-end latency for image question-answering, with critical optimizations including 8-bit weight dequantization, chunked delta-rule recurrent layers (2.8x faster), and per-head GEMM calls that reduced quadratic attention scaling by 17x. The work demonstrates that careful kernel-level optimization and stage-validated porting enable modern multimodal models to run on decade-old hardware within memory constraints.