Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Apple Machine Learning Research ● Covered by 2 sources
MLX with RDMA-over-Thunderbolt support on Mac Studio was used to run and benchmark local and distributed inference for large language models, including Muse Spark Open, using the latest MLX release and benchmark tooling. The setup reportedly generated over 36 tokens per second on a cluster of four Mac Studios. As a result, very large MoE models can be tested on private hardware either on a single Mac with enough unified memory or by scaling out across multiple Macs via pipeline or tensor parallelization.
Why it matters
MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private environment. The open-source array framework leverages the unified memory architecture of Apple silicon, enabling anyone to run new inference and fine-tuning methods on their own hardware and explore other AI techniques. In a previous post, we highlighted the advantages of the M5 architecture and its GPU Neural Accelerators.
