Deploying 🤗 ViT on Kubernetes with TF Serving

Hugging Face is back with part two of its Vision Transformer serving series, and this time the toy example grows up. The first post showed how to get a ViT model running locally with TensorFlow Serving, handling image preprocessing and gRPC requests on a single machine. This post picks up where that left off and walks through what it actually takes to put that same model in front of real users, at scale, using Docker and Kubernetes.

The workflow is refreshingly unglamorous: package the model into TensorFlow Serving's SavedModel format, drop it into a directory structure that looks like models/hf-vit/1, then build a custom Docker image on top of the official tensorflow/serving base image using docker cp and docker commit. Nothing exotic there. The image gets pushed to Google Container Registry, and from there it's off to Google Kubernetes Engine, though the author is careful to note that Minikube or EKS would work just as well since none of the underlying Kubernetes concepts change.

What's more interesting is the reasoning behind choosing this path over something like SageMaker or Vertex AI, which promise ML deployment out of the box. Hugging Face's answer is basically: this stuff is battle-tested, widely adopted, and gives you finer control without forcing you to abstract away decisions you might actually care about, like tensorflow_inter_op_parallelism and tensorflow_intra_op_parallelism settings that let you tune how the model uses CPU threads on your specific hardware.

The actual Kubernetes setup leans on three YAML manifests. deployment.yaml pins the Docker image, opens ports 8500 and 8501 for gRPC and REST, and sets CPU resource requests at 800m (80% of a core) per container. service.yaml exposes those ports externally through a LoadBalancer. And hpa.yaml configures the Horizontal Pod Autoscaler to keep between one and three replicas running, scaling up whenever average CPU utilization crosses 80%. Applying all three with kubectl, or bundling them with Kustomize once things get unwieldy, is what finally turns a local demo into something that can survive a traffic spike.

It's a fairly standard MLOps recipe, but the value here is in the specifics: the exact GKE provisioning commands, the machine types, the parallelism flags, the autoscaling math. That's the kind of detail that turns a blog post into something you actually copy into a terminal.