In-House LLM Serving at Netflix
TLDR Dev
Netflix built an in-house system to serve large language models using vLLM and NVIDIA Triton, integrating it into their existing JVM-based serving infrastructure rather than using external APIs. The platform supports both gRPC and OpenAI-compatible HTTP endpoints, with deployment strategies including red-black and versioned rollouts to handle model updates without dropping requests. The system implements constrained decoding via vLLM's logits processor interface to generate compliant outputs by construction, though this required optimization across vLLM versions to handle batching efficiently at scale.
Why it matters
Netflix has developed an in-house LLM serving platform that integrates model deployment and inference within its existing production environment, using a unified serving system supported by vLLM and Triton with features like A/B testing and efficient model routing.
