Enhancing ggml with Fused Sparse-Attention Masks and Precision Introspection in v0.28.0

The ggml project released v0.28.0, adding a fused GGML_OP_TOP_K_MASK operation that combines top-k selection with sparse-attention masking in a single kernel across CPU, CUDA, and Metal backends. The release also introduces -1 as an explicit sentinel skip index for MUL_MAT_ID and ADD_ID operations and adds the ggml_prec_get_acc() and ggml_prec_get_src() APIs for querying compute-node precision. The changes aim to cut intermediate memory use and kernel launch overhead while simplifying backend logic.

Sparse attention mechanisms have become a cornerstone of efficient transformer implementations, enabling models to scale by focusing computation on the most relevant parts of the input. However, supporting these sparse patterns efficiently in low-level ML compute libraries requires careful design of operations and data flow. The recent ggml v0.28.0 release tackles this challenge head-on by introducing a fused sparse-attention mask operation, alongside precision introspection APIs and cleaner mechanisms for skipping computation slots. These improvements not only simplify backend implementations but also unlock performance gains across CPU, CUDA, and Metal platforms. Prior to this release, ggml lacked a dedicated operation to fuse sparse-attention masks with top-k selection, forcing users to rely on separate steps or less efficient workarounds. This fragmented approach increased overhead and complexity in model implementations that rely heavily on sparse attention. Additionally, the way ggml handled skipping matrix slots was problematic. Instead of an explicit sentinel value, invalid or skipped matrix IDs were treated as out-of-range indices, which complicated backend logic. This meant that the compute engines had to interpret "magic" out-of-range values as signals to skip computations, leading to brittle and error-prone code. Moreover, introspecting the precision of compute nodes was limited, which made it harder for developers and optimizations to adapt based on the exact precision used during computation. The centerpiece of this release is the new GGML OP TOP K MASK operation. This operator fuses the top-k selection with the application of a sparse-attention mask into a single kernel. By doing so, it reduces intermediate memory usage and kernel launch overhead, leading to more efficient execution. Implementations of this op have been provided across all major backends: This cross-platform consistency ensures that models using sparse attention can benefit regardless of hardware. To simplify logic around skipping matrix slots, the release introduces support for -1 as a sentinel skip index in the MUL MAT ID and ADD ID operations. This explicit value signals the backend to zero out the corresponding output row and skip computation cleanly. This change removes the ambiguity of out-of-range indices and allows backends to implement skipping more robustly and efficiently. For example, instead of checking for invalid indices and guessing their meaning, the backend can directly branch on -1 and avoid unnecessary work. Two new APIs, ggml prec get acc and ggml prec get src , have been added to query the precision of accumulation and source data on compute nodes. This enables more adaptive optimizations and debugging, as developers can programmatically determine if a node is using FP16, FP32, or other precisions. This update to ggml highlights several important engineering principles for ML compute libraries: Fused operations for common patterns: Sparse attention is ubiquitous in modern transformers. Providing fused ops tailored to these patterns reduces overhead and improves maintainability. Explicit sentinel values simplify backend logic: Using a dedicated skip index makes the code clearer and less error-prone than relying on out-of-range or magic indices. Precision introspection aids optimization: APIs that expose compute node precision help downstream tooling and adaptive kernels tune performance dynamically. Cross-platform consistency: Implementing new ops across CPU, CUDA, and Metal ensures broad hardware support and consistent behavior. Together, these changes make ggml a more robust and performant foundation for ML workloads involving sparse attention and complex tensor operations. Credit for this deep-dive to the original PR in the ggml-org/ggml repository. Read the full breakdown on nossletter.tech. Originally published on Nossletter https://nossletter.tech/digest/ggml-1676-ggml-bump-version-to .