Petals
petals.dev
Petals lets anyone run huge AI models like Llama 3.1 405B by splitting the work across volunteers' computers, BitTorrent-style. No massive GPU budget needed, just a network of strangers sharing compute.
Based on reporting by petals.dev — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There's a certain poetic justice in using peer-to-peer networking, the same tech that powered Napster and BitTorrent, to democratize access to models with hundreds of billions of parameters. That's exactly what Petals does. Instead of renting a rack of A100s or begging for API credits, you load a slice of a model like Llama 3.1 (up to 405B parameters), Mixtral 8x22B, Falcon 40B+, or BLOOM 176B onto your own consumer GPU, or even a free Google Colab instance, and the rest gets served by other people's machines elsewhere on the network.
The speeds aren't going to replace a dedicated inference cluster, but they're far from useless. Single-batch generation hits roughly 6 tokens per second on Llama 2 70B and about 4 tokens per second on Falcon 180B. That's workable for chatbots and other interactive tools where you're not blasting through thousands of requests a minute. And unlike a typical hosted API, you're not locked into someone else's sampling logic or fine-tuning restrictions.
What makes Petals interesting isn't just the cost savings, it's the access it grants under the hood. Because you're running actual PyTorch and Hugging Face Transformers code rather than calling a black-box endpoint, you can inspect hidden states, chart custom paths through the network's layers, or apply whatever fine-tuning trick you want. That's a meaningfully different proposition than the walled-garden APIs from OpenAI or Anthropic, where you get exactly the interface they decide to expose and nothing more.
The project sits under the BigScience research umbrella, the same academic-industry collaboration that produced BLOOM, so this isn't some fly-by-night side project. It's maintained by people who care about open science infrastructure, not just a clever demo. Development updates trickle out through Discord and an email list that fires off maybe once every few months, which tells you this is a slow-burn, community-driven effort rather than something chasing hype cycles.
Still, there's an obvious tension here that the project doesn't shy away from: running a 405-billion-parameter model depends entirely on strangers keeping their GPUs online and behaving honestly. Petals is betting that enough people will donate spare compute to make huge open models genuinely usable outside of Big Tech's data centers. Whether that bet pays off at scale is the real experiment worth watching.
My take — AI-written commentary, not fact-checked reporting
I like Petals precisely because it refuses to accept that giant models require giant company infrastructure — that's the kind of scrappy, decentralized thinking Europe's open-source AI scene should be shouting about instead of just chasing EU AI Act compliance checkboxes. The catch is trust: BitTorrent worked because nobody cared if a stranger's chunk of a movie was slightly corrupted, but letting anonymous nodes handle part of your model's forward pass is a different risk calculus entirely, and I haven't seen anyone solve that convincingly yet.
Read more about this at: petals.dev