MoonMath AI Open-Sources a HIP Attention Kernel for AMD MI300X That Beats AITER v3 on Every Shape and — 2026-06-22

## Short Segments Welcome to Impact Vector, where we dive into the latest AI tools reshaping the tech landscape. Today, we're exploring a groundbreaking development from MoonMath AI, which has open-sourced a new attention kernel for AMD's MI300X GPU. This kernel outperforms AMD's own AITER v3 across all tested configurations. We'll unpack what this means for developers and the broader implications for AI performance. Stay tuned as we delve into the details. ## Feature Story MoonMath AI has made a significant leap in AI performance by releasing an open-source bf16 forward attention kernel for AMD's MI300X GPU. This kernel, written in HIP rather than hand-written assembly, is now available under the MIT license. The MoonMath team reports that their kernel surpasses AMD's own AITER v3 in performance across every tested shape and rounding mode, achieving a geometric mean speedup of up to 1.26 times. Attention mechanisms are crucial in transformer models, performing the softmax operation that is central to these architectures. The MI300X, AMD's CDNA3 data-center GPU, is the hardware platform for this kernel, which is specifically optimized for this environment. The kernel's performance gains are attributed to innovative memory placement strategies, such as storing K in LDS, keeping V hot in L1 cache, and managing Q and accumulators in registers. This development is particularly noteworthy because it leverages a unique approach to kernel optimization. By using one-instruction assembly wrappers, developers can select opcodes while allowing the compiler to handle register allocation. This method not only simplifies the coding process but also enhances performance by optimizing memory usage. The practical implications of this kernel are already being realized. A real-world application saw a 1.23 times speedup in Wan2.1 video diffusion without any loss in quality, demonstrating the kernel's potential to enhance AI workloads significantly. This is a crucial advancement for developers working with large language models and other AI applications that demand high efficiency and speed. However, there are limitations to this kernel. It does not support causal masks, grouped query attention (GQA), or variable-length batching. Outputs are limited to bf16 precision, and the kernel is designed to run exclusively on the MI300X hardware. Despite these constraints, the kernel's performance improvements make it a valuable tool for developers seeking to maximize the capabilities of AMD's GPUs. In the broader context, this release highlights the ongoing competition in the AI hardware space, where efficiency and speed are paramount. AMD's MI300X GPUs, equipped with the AI Tensor Engine for ROCm, are already known for their ability to deliver up to twice the inference speed compared to non-AITER runs. MoonMath's kernel further enhances this capability, offering developers a powerful tool to push the boundaries of AI performance. Looking ahead, the open-source nature of this kernel means that it can be continuously improved and adapted by the developer community. This collaborative approach could lead to further optimizations and innovations, potentially influencing the design of future AI hardware and software solutions. For developers and researchers, the release of this kernel represents an opportunity to explore new levels of performance in AI applications. By integrating this kernel into their workflows, they can achieve faster and more efficient computations, ultimately driving advancements in AI technology. As we continue to see rapid developments in AI hardware and software, tools like MoonMath's attention kernel will play a crucial role in shaping the future of AI. By providing open access to cutting-edge technology, MoonMath AI is empowering developers to innovate and push the limits of what's possible in AI. That's all for today's episode of Impact Vector. Stay tuned for more insights into the tools and technologies transforming the AI landscape. Until next time, keep exploring the impact of AI.