Framework & Infrastructure

32 summarised stories about Framework & Infrastructure, each linking back to the original source. Browse all topics →

Thursday, 16 January 2025

Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference

Hugging Face Blog 1 year ago

Text Generation Inference now supports multiple inference backends including TensorRT-LLM and vLLM through a unified interface, allowing users to switch between different execution engines without changing their deployment code. The vLLM backend integration is scheduled for Q1 2025, with additional backends for llama.cpp, AWS Neuron, and Google TPU also in development. Users can now select the optimal backend for their specific hardware and performance requirements rather than being locked into a single inference solution.

Efficiency & Optimization Framework & Infrastructure Hardware Hugging Face

Timm ❤️ Transformers: Use any timm model with transformers

Hugging Face Blog 1 year ago

The Hugging Face transformers library now supports integration with timm (PyTorch Image Models), allowing users to leverage timm's 200K+ daily-downloaded computer vision models within the transformers ecosystem. An 8-bit quantized timm ViT model reduced from 346.27 MB to 88.20 MB (74.53% reduction) while maintaining nearly identical inference accuracy. Users can now fine-tune timm models with the Trainer API, apply LoRA for parameter-efficient training, use pipeline APIs for inference, and round-trip fine-tuned models back to timm's native format.

Computer Vision Efficiency & Optimization Framework & Infrastructure Hugging Face