Introducing the Realtime API

OpenAI dropped its Realtime API this week, and the pitch is simple: developers can now wire spoken conversation directly into their apps without stitching together three separate services. Speech-to-text, then a language model, then text-to-speech — that old chain added lag and lost nuance every step of the way. This new API collapses it into one streaming connection.

Under the hood it runs on the same GPT-4o family of models that already handle audio natively, so the system hears tone and pacing, not just words, and responds with speech that carries some of that back. It connects over WebSockets, letting audio flow both directions in real time rather than waiting for a full response to render before playback starts. That's the difference between a voice assistant that feels like a walkie-talkie and one that feels like a phone call.

OpenAI is aiming this squarely at builders of customer support bots, voice agents, and interactive apps who've been duct-taping Whisper transcription to GPT-4 to a TTS engine and eating the latency penalty. Function calling is baked in too, so a voice agent can still trigger real actions — booking a table, checking an order status — mid-conversation, without breaking the audio stream to do it.

It's a modest-sounding announcement dressed up in plain language, but it's really OpenAI closing a gap that startups like Vapi and Bland had been filling with their own glue code. Now that glue is native. Whether that's good news for those startups is a separate question entirely.