ElevenLabs wants to run your voice agent. Here’s what developers give up.

ElevenLabs has rolled out Eleven Realtime, a managed voice-agent setup with two models, Mini and Max. The pitch is simple: speech recognition, reasoning, turn-taking, and voice generation all come packaged together. The catch is simpler too. Developers don’t get to pick or tune the underlying models themselves.

That matters because ElevenLabs is taking over the messy middle of voice agents — turn detection, interruptions, orchestration, the stuff that makes a call feel human instead of like a broken IVR tree. In return, teams get less plumbing to build and maintain. They also lose a layer of control that matters when something goes wrong, because troubleshooting, optimization, and component swaps become harder when the harness belongs to the vendor.

There is still plenty to configure. Through ElevenAgents, developers can set prompts, voices, tools, knowledge bases, and turn-taking behavior, including eagerness, timeouts, and interruption handling. For teams that want full control, Custom mode stays available, with access to the underlying language and speech models and the ability to wire the pipeline themselves. Realtime is the trade: less flexibility for less work.

The outside benchmarks are mixed in a way that will probably matter in production, not on a slide deck. Artificial Analysis tested both models on speech reasoning, conversational dynamics, and human preference. On Big Bench Audio, Max scored 98.8% and Mini scored 75.9%. On tool use, Max finished 87.1% of tasks versus 60.3% for Mini. ElevenLabs says Max is the better fit for agents with frequent or complex tool calls, while Mini is the default and should handle most use cases. Mini did slightly better on Full-Duplex-Bench, at 95.8% versus 94.1% for Max.

Latency is close enough that the bigger question is really behavior, not speed. Max averaged 570 milliseconds to first audio, while Mini came in at 620 milliseconds. ElevenLabs is also clearly trying to pull teams over from OpenAI’s Realtime API, since it uses the same protocol over the same WebSocket transport. But the migration path is not painless: WebRTC apps have to switch transports first, and on WebSocket developers handle 24kHz PCM16 audio capture and playback themselves. ElevenLabs says teams that need phone and WhatsApp channels, knowledge bases, and monitoring should start on ElevenAgents, not bolt them on later after taking the fast route.