BaseRT: Advancing LLM Inference with Apple M5 Neural Accelerators

This arXiv paper presents BaseRT, a native Metal runtime for LLM inference on Apple Silicon. Its hand-written Metal 4 kernels use the Neural Accelerators in Apple’s M5 GPU for dense and mixture-of-experts GEMM, plus FlashAttention-style prefill kernels, while retaining specialized kernels for memory-bound decoding. On an Apple M5 Pro, the authors report results across 15 configurations covering Qwen3, Qwen3.5/3.6, Llama 3.2, and Gemma 4 models ranging from below 1B to 35B parameters. Peak prompt-processing throughput is reported at 6.4x llama.cpp and 3.9x MLX, with decode gains up to 1.75x and 1.33x respectively.