Charles Frye

Host overhead, caused by the CPU blocking the GPU, is a major source of inefficiency in AI inference, leading to low GPU kernel utilization and doubling GPU costs when at 50%. Modal recommends using t…

Modal introduced Servers, a new ultra-low-latency HTTP serving solution for serverless applications, built on Pingora, Envoy, and Spanner. The system minimizes overhead by avoiding control-plane looku…

Modal AI engineers Charles Frye and David Wang optimized FlashAttention-4 for large language model inference, focusing on decode-heavy workloads dominated by memory bandwidth-limited token generation.…