Smaller, Faster, Safer: Running Kimi and GLM at Scale
Original title:Smaller, faster, safer: running Kimi and GLM at scale
AI Summary
Cloudflare describes how Workers AI serves Moonshot’s Kimi K-series and Z.ai’s GLM models, which are large, long-context mixture-of-experts systems that are difficult to fit efficiently into GPU memory. The deployment combines three techniques: KV-cache quantization, model-weight compression, and protection for shared caches when more requests are colocated on the same hardware. Cloudflare says its experiments and production traffic use SGLang, and that the optimizations support more customers at lower cost without changing model accuracy. The post focuses on serving engineering rather than introducing new model architectures.
Why it's worth reading
As long-context MoE models shift the bottleneck from model quality to GPU memory and multi-tenant serving, this post offers a concrete optimization stack tied to production traffic and SGLang.
Deep Read
What happened
Original facts: Cloudflare published a post about serving Moonshot’s Kimi K-series and Z.ai’s GLM models on GPUs in Cloudflare data centers close to users through Workers AI. The focus is operational: these are large, long-context mixture-of-experts models that are difficult to serve efficiently.
Core technology
Original facts: Cloudflare combines three techniques: KV-cache quantization, model-weight compression, and protection for shared caches when more requests are placed on the same hardware. The post says both experiments and production traffic use the open-source SGLang inference framework.
Analysis: KV-cache quantization targets memory that grows with context length and concurrent requests, while weight compression reduces the model’s persistent memory footprint. Shared-cache protection addresses isolation, stability, and possible cross-tenant risks in a multi-tenant serving environment.
Key evidence & numbers
Original facts: Cloudflare says the optimizations enable support for more customers at lower cost without changing model accuracy. The supplied abstract does not include GPU types, quantization bit widths, throughput, latency, memory savings, or cost reductions.
Unverified inference: The claim of unchanged accuracy may be based on Cloudflare’s internal evaluation or production monitoring, but the abstract does not identify the tasks, error margins, or statistical methodology.
Why it matters
Analysis: Kimi and GLM illustrate a class of open models whose long contexts and MoE execution create substantial serving pressure. If providers can reduce both static weight memory and dynamic KV-cache memory while preserving tenant isolation, the operational question becomes how many real workloads each GPU can sustain reliably, rather than merely whether one model can run.
Practical impact
Cloud providers may use this stack to reduce the GPU footprint of model deployments and improve utilization under concurrency. Self-hosting teams can treat SGLang, KV-cache quantization, and weight compression as concrete optimization avenues, but must benchmark them against their own model versions, context lengths, concurrency levels, and hardware. Cache protection also indicates that throughput improvements should be evaluated alongside tenant isolation and failure behavior.
Limitations & uncertainty
The abstract does not specify the compression methods, precision settings, model versions, hardware, workload distribution, or baseline, so the reported cost and performance benefits cannot be independently reproduced from the available information. Cloudflare’s production traffic may differ from a typical self-hosted cluster in network topology and scheduling. The unchanged-accuracy claim is not accompanied here by an evaluation set or result table. The stated publication date, August 3, 2026, should also be checked against the source page and ingestion timestamp.
Original sources
Cloudflare Blog: Smaller, faster, safer: running Kimi and GLM at scale