38 Things We've Learned About Building AI Products
build mode
PostHog says AI products fail when they’re clever but flaky. The real edge is context, reliability, and building for agents, not bolting AI on top.
Based on reporting by build mode — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
PostHog says it has built 10 AI products and watched them serve nearly 100,000 people a month. After all that, the lesson isn’t “AI everywhere.” It’s that AI can make a product worse if it’s slow, unreliable, or aimed at a problem nobody actually has.
The company’s clearest win is onboarding. Its AI install wizard does about two hours of work in roughly 8 minutes, and users who go through it convert to paid 5x more often and activate 2x faster. That’s the kind of job AI is good at: long tasks, unfamiliar interfaces, repetitive work. Not “chat with your docs” for the sake of it.
A bigger theme runs through the whole piece: stop thinking in terms of assistants and start thinking in terms of systems. PostHog says an assistant sits beside the product, while a system bakes intelligence into it. That also changes the UI. Fewer screens are for doing the work now; more are for approving, reviewing, and undoing what the agent did.
The company keeps coming back to context. Its MCP server gets 5x more daily users than PostHog AI, and its docs-search tool is called about 28,000 times a month, which it treats as a signal that agents are missing skills or tools. When a customer built a prototype of “PostHog 2.0” using the company’s MCP and Claude Code, PostHog responded by adding more customizable canvas artifacts. That’s the product lesson here: watch what customers rebuild, then use that to see what your own product is missing.
There’s also a lot of practical scar tissue in the post. PostHog says its AI wizard costs $6.67 per run, moved trace summaries to a cheaper model tier, and found that giving agents more context helped more than better reasoning, multi-step planning, subagents, pruning, or compaction. And because models change underneath you, assumptions do too: the company says cheap reasoning, reliable tool use, and multimodal models forced it to restructure parts of PostHog AI and make Replay Vision possible.
The end result is less hype than housekeeping. Trace everything. Start small with evals. Let agents write code, but sandbox them. Show every step so users can trust the process. And expect half of this to look wrong in a year. The durable part, PostHog argues, is the old stuff: talk to users, ship fast, run experiments, repeat.
My take — AI-written commentary, not fact-checked reporting
The cleanest AI products are the ones that admit they’re really product work with a new engine bolted in. The industry still loves shiny demos, but users keep paying for reliability and context, which is boring in the best possible way. Also, any team hiding reasoning and tool calls behind a mystery box is basically asking for distrust with extra steps.
Read more about this at: build mode