Running Privacy-Preserving Inferences on Hugging Face Endpoints

Zama, the open source cryptography outfit behind Concrete ML, just made fully homomorphic encryption a lot less painful to touch. For 18 months the company has been building tooling that lets scikit-learn, PyTorch, TensorFlow and ONNX models run inference directly on encrypted data, without decrypting it, without a private key anywhere near the server. That's the whole promise of FHE: math on ciphertext that produces a correct, still-encrypted answer. The catch has always been usability. Today's release chips away at that by putting pre-compiled FHE models straight onto Hugging Face, deployable through Inference Endpoints with a few clicks.

The demo model is a spam-detecting decision tree, and it's a good showcase because the point isn't just that it classifies email, it's that it does so without ever seeing the message content unencrypted. Spin up an eight-vCPU CPU endpoint, no GPU support yet, clone the repo, drop in your endpoint URL and an HF token, and you're running encrypted inference in a Python virtual environment pinned to 3.10. Zama's own numbers show the tradeoff plainly: 691 test samples, 89.6% accuracy, and roughly 4.1 seconds per inference, adding up to nearly 48 minutes total. That's the current cost of privacy at this scale, and it's honest of Zama to publish it rather than bury it.

Under the hood, this all rides on Hugging Face's custom inference handlers, a feature flexible enough that Zama built three methods into it: save_key and append_key for shuttling FHE evaluation keys (sometimes piece by piece, since keys can run past 70,000 kilobytes), and inference for the actual encrypted prediction. Zama is generous with credit here, calling the handler system unusually accommodating for a workflow that looks nothing like typical ML deployment. That generosity is earned; building a bespoke crypto pipeline on top of someone else's inference infrastructure without forking it is not trivial.

There are real limits worth naming. Keys currently live in endpoint RAM, so a restart wipes them and forces a resend, and there's no shared state across multiple machines handling load, which makes horizontal scaling awkward. Eight vCPUs is also the ceiling on Hugging Face's CPU tier right now, a real constraint for anything beyond demos. Zama's wishlist includes beefier machines, a dedicated Privacy-Preserving Inference Endpoint button, and some mechanism for persistent shared state to hold those keys. None of that exists yet.

What does exist is a template. Anyone can fork Zama's example repos, covering linear models, tree ensembles, MLPs and PyTorch nets, retrain with creating_models.py, generate the client.zip and server.zip artifacts, and publish their own encrypted model to the Hub tagged concrete-ml or FHE. That's a meaningfully lower bar than writing FHE tooling from scratch, even if four seconds per prediction reminds you this is still early infrastructure, not a finished product.