ATSInfer: Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

This paper introduces ATSInfer, a hybrid CPU-GPU inference system for consumer devices that schedules tensors rather than entire layers or experts. It combines static tensor placement, load-aware dynamic transfers, and asynchronous coordination of storage, data movement, and computation. The authors evaluate the implementation on representative consumer platforms with both dense and mixture-of-experts models. According to the abstract, ATSInfer improves prefill throughput by up to 1.94x and decode throughput by up to 3.29x over existing systems, while also increasing GPU utilization and improving PCIe bandwidth usage.