Three lessons for creating a sustainable AI advantage
OpenAI
Intercom shared how it built a lasting AI edge in customer support, via OpenAI's blog. The real lesson: it's not about the flashiest model, it's about evals and architecture that don't rot.
Based on reporting by OpenAI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Intercom has been running AI-powered customer support for a while now, and OpenAI's latest blog post uses the company as a case study in what actually separates a durable AI product from a demo that impresses in a boardroom and falls apart in production. The framing is simple: three lessons, and none of them are about picking the newest model off a leaderboard.
The first lesson is about evaluations. Intercom apparently treats evals not as a one-time QA checkbox before launch, but as an ongoing discipline baked into how the team ships changes. That matters because customer support is exactly the kind of domain where a model can sound confident and still be wrong in ways that erode trust fast — wrong refund policy, wrong account details, wrong tone. Without constant measurement, teams end up flying blind, shipping updates that look fine in a quick manual test and quietly degrade real conversations.
The second lesson is architectural. Rather than wiring a single model directly into the product and hoping it scales, Intercom built its system so the underlying model can be swapped, upgraded, or reconfigured without a rewrite. That is a less glamorous story than
My take — AI-written commentary, not fact-checked reporting
I'll say the obvious thing nobody wants to hear: most companies bragging about AI features have neither of these two things, evals or swappable architecture, and that's exactly why so many AI products feel great for a week and then quietly get worse. Intercom's boring discipline is the actual moat here, not their choice of model, and if you're not building for model-agnosticism in 2024 you're building technical debt with a chatbot UI on top.
Read more about this at: OpenAI