Architecting memory and storage in the AI era

The big shift isn’t more AI. It’s AI that has to answer now, over and over, across millions of requests. MIT Technology Review’s sponsored piece argues that inference has turned infrastructure into the main event: memory, storage, networking, power, and cooling all have to be planned together, because the bottleneck can appear anywhere and then move somewhere else tomorrow.

That’s a very different problem from the old “buy more compute” instinct. Jim McGregor of Tirias Research says AI is not one workload but thousands, millions, even billions of them. Some are real-time customer systems. Some are healthcare tools, robotics, financial services, or agentic software. They don’t all behave the same way, and they definitely don’t forgive latency.

The article keeps coming back to data movement, and for good reason. Retrieval-augmented generation systems have to scan large databases fast, while inference keeps hammering memory bandwidth, caching, storage proximity, and network paths. In that world, the best processors on paper are only part of the story. If the data can’t be found, moved, and delivered quickly, the system stalls.

That is why the piece pushes a more integrated design philosophy: architect the whole stack, not just the chips. Enterprises are told to define the workloads first, then build modular systems for compute, memory, storage, power, and cooling, and to keep reassessing procurement as AI demands and hardware change quickly. The warning is blunt enough without being dramatic. Generic “AI readiness” can mean overspending in one place and leaving the real bottleneck untouched.

The business case is not just speed. It’s efficiency, ROI, and reputation. The article says better utilization can help answer growing scrutiny around power use and water use, while delays in healthcare, robotics, or customer-facing systems can damage trust. In McGregor’s framing, procurement is no longer a side task. It is the strategy.