Why real-time AI at scale is so hard

Real-time AI looks tidy in development and then gets messy the moment real traffic shows up. That was the central warning in The New Stack’s discussion with Tim Koopmans, who learned the hard way while building an ML-based financial trading app. The theme is simple: when latency climbs or accuracy slips, the model often takes the blame for problems hiding elsewhere.

One of the biggest traps is tail latency. Tim’s app looked fine until it was pushed toward about 740,000 operations per second, when P99 latency jumped to 3 seconds. The model wasn’t the issue. Feature lookups were. A small number of reads ended up waiting behind writes, and once the system hit heavy concurrency, tuning retries and caches didn’t save it. That kind of spike is a property of the architecture, not a little bug to patch away.

Accuracy can fail in a quieter way. Tim saw user profile data drifting far past a five-minute freshness target, by hours, while offline evaluation still looked healthy. Vector embeddings also went stale. The result was the classic production trap: the numbers in the lab looked good, but live decisions were made on old data. The lesson is blunt. Freshness matters as much as model quality, and sometimes more.

Vector indexes bring their own problems. Re-embedding content over and over can rot an index, and Tim saw recall sink to 42% while query latency climbed. His advice was to treat the index like any other database index: monitor recall, plan for partial or batch builds, and expect to rebuild when the similarity function, search parameters, or embedding model changes. There is no magical “set it and forget it” button here.

The rest of the advice is classic distributed systems thinking dressed in AI clothing. Separate training from serving. Separate writes from reads. Expect retraining to happen, and plan for the awkward cutover period when old and new models overlap. If the system can’t absorb a surge in writes or a sudden change in traffic shape, the doom loop starts fast: latency drives staleness, staleness hurts accuracy, accuracy pressure triggers retraining, and retraining makes the latency problem worse.