cd /news/ai-research/tackling-gpu-bottlenecks-in-plane-wa… · home › topics › ai-research › article
[ARTICLE · art-171859] src=developer.nvidia.com ↗ pub= topic=ai-research verified=true sentiment=↑ positive

Tackling GPU Bottlenecks in Plane-Wave DFT with K-Point Batching and Mixed Precision

NVIDIA introduced Daviator, a library of batched GPU kernels, and a new mixed-precision execution layer for plane-wave DFT in VASP, delivering up to 22x faster time-to-solution and up to 75% lower cost than CPU execution on a projected reference dataset workload. Daviator replaces VASP's sequential per-k-point processing with a batched Davidson eigensolver using batched FFTs, GEMMs and custom kernels, addressing a case where a single-atom iron system with 752 k-points ran 8x slower on more expensive GPU hardware than on CPU. The mixed-precision layer targets inference-class GPUs such as the RTX PRO 6000, which carry a 64:1 FP32:FP64 throughput ratio, as FP64 throughput has held flat across NVIDIA Hopper, Blackwell and Rubin while FP32 throughput keeps rising.

recommendation

Batch independent k-point and spin work into single GPU kernel launches rather than looping sequentially, and use mixed precision on GPUs with high FP32:FP64 ratios to cut launch overhead and cost.

more in playbook: Inference and self-hosting
by read13 min views6 publishedNov 18, 2026
Tackling GPU Bottlenecks in Plane-Wave DFT with K-Point Batching and Mixed Precision
Image: NVIDIA Developer Blog

Training a next-generation machine learning interatomic potential (MLIP) requires ground truth: tens to hundreds of millions of density functional theory (DFT) calculations of energies, forces, and stresses across diverse structures and chemistries.

Plane-wave DFT, with VASP as its dominant implementation, is the gold standard for generating that data. It is accurate, well-validated across materials classes, and ubiquitous at the national labs and research institutions running these calculations at scale.

As MLIP training datasets have grown, so has the pressure to generate DFT data faster and cheaper. But the small unit cells and dense k-point grids common in MLIP data generation can leave significant GPU performance unused. VASP’s OpenACC implementation has delivered meaningful speedups, especially for large-system calculations. However, for smaller systems, sequential k-point processing limits GPU utilization. At the same time, AI workloads are reshaping GPU hardware development toward lower-precision data types, creating a growing gap between what modern NVIDIA GPUs can deliver and what FP64-only HPC codes are optimized for.

This post introduces two new NVIDIA capabilities that address these bottlenecks. The first, Daviator, resolves the small-system GPU performance problem through k-point and spin-batched GPU kernels. In particular, a batched Davidson solver delivers significant performance gains. The second, a new mixed precision execution layer, enables further performance gains, especially on inference-class GPUs with a high FP32:FP64 throughput ratio.

For a projected reference dataset workload, the results show up to 22x faster time-to-solution and up to 75% lower cost compared to CPU execution, depending on hardware and configuration.

The GPU bottleneck in plane-wave DFT #

In periodic systems, electronic wavefunctions are parameterized by crystal momentum, a discrete set of k-points that samples the Brillouin zone. For periodic structures with small unit cells, hundreds to thousands of irreducible k-points are required to achieve well-converged results.

Each k-point solve is largely independent; for instance, no inter-k-point communication is required during the Davidson eigensolver step. Despite this independence, VASP’s OpenACC GPU port processes k-points sequentially. For large systems, the work per k-point can fully saturate a GPU, but for small systems with dense k-point grids, the hundreds of small GPU dispatches per step causes kernel launch overhead to dominate compute time. The result: a single-atom iron system with 752 k-points ran 8x slower on the more expensive GPU hardware than on CPU. Thus, researchers working at this scale were left with a manual routing decision based on the size of the job.

Additionally, plane-wave DFT traditionally runs in FP64, a conservative default that guarantees accuracy throughout the solver pipeline. But GPU hardware is no longer symmetric in this regard. On data center GPUs, FP64 throughput has held flat across NVIDIA Hopper, Blackwell, and Rubin GPUs, while FP32 throughput has continued to increase. Inference-class and workstation GPUs like the RTX PRO 6000 push this further, with a 64:1 FP32:FP64 ratio (Figure 1). As AI workloads drive hardware development toward lower-precision data types, the fraction of peak GPU throughput available to FP64-only codes is stagnating.

Daviator: Batching k-points for GPU saturation #

Daviator is a library of GPU kernels developed by NVIDIA that replaces sequential per-k-point processing in plane-wave DFT inner loops with batched GPU operations. At the core of the transformation lies a batched Davidson eigensolver that replaces sequential loops that cause latency with batched FFTs, GEMMs, and custom kernels. This drops the number of GPU kernel launches drastically, enabling SM occupancy to go from a fraction of capacity to near saturation. Data stays resident in GPU memory throughout. VASP retains full ownership of the SCF loop, mixing, symmetry, and I/O; Daviator integrates at well-defined call boundaries over data already on device.

Running a calculation

Daviator will be available through the NVIDIA ALCHEMI VASP microservice by setting LDAVIATOR=True in the INCAR. Existing VASP inputs work without modification. The POSCAR, INCAR, and KPOINTS below are a standard setup for a k-point-dense metallic system.

Daviator intercepts the Davidson solver internally (ALGO = Normal) and routes it through batched GPU operations.

POSCAR:

Fe1
1.0
   2.3017038599999999    0.0000000000000000   -0.8137751999999999
  -1.1508519300000000    1.9933340200000000   -0.8137751999999999
   0.0000000000000000    0.0000000000000000    2.4413256099999998
Fe
1
direct
   0.0000000000000000    0.0000000000000000    0.0000000000000000 Fe

INCAR:

ALGO = Normal
LDAVIATOR = True
EDIFF = 5e-05
IBRION = 2
ISIF = 3
ISMEAR = 1
ISPIN = 2
NBANDS = 12
NSIM = 6
LASPH = True
LMAXMIX = 4
LORBIT = 11
LREAL = False
LWAVE = False
MAGMOM = 1*5.0
NELM = 100
NSW = 3
PREC = Accurate
SIGMA = 0.05
KPAR = 1

KPOINTS:

Gamma-centred 30x30x30 mesh
0
Gamma
30 30 30

Some of Daviator’s GPU kernels are specialized for the batch sizes they meet and may trigger a compilation on first use.

Add these lines to your job script before launching VASP to ensure efficient caching:

export CUDA_CACHE_PATH=/tmp/cudacache   # fast storage, not $HOME
export CUDA_CACHE_DISABLE=0             # make sure caching is on

Finally, run as usual:

mpirun -np 1 vasp_std

For now, VASP with Daviator is limited to single-node execution. Multi-node support, coverage of systems with non-collinear spins (vasp_ncl) and explicit gamma-only runs (vasp_gam) is expected to be released at a later stage.

Results

The benchmarks below compare Daviator against VASP 6.6.0 CPU and OpenACC baselines on NVIDIA H100s/H200s and B200s.

Daviator overcomes the OpenACC port’s limitations; a single iron atom achieves over 2x speed-ups relative to the CPU baseline (Figure 2). The gains increase for higher-accuracy functionals: the 22x Davidson speedup on LiFePO<sub>4</sub> r<sup>2</sup>SCAN enables higher accuracy meta-GGA runs at production scale.

For mid-size systems like a 64-atom Bi-doped GaAs structure, VASP with Daviator cut execution time by more than 85%. Even at 348 atoms with a single k-point, Daviator delivers a significant improvement over the OpenACC and CPU baselines. VASP with Daviator improves the previous state-of-the-art for all system sizes, eliminating the need to route jobs between CPU and GPU based on system size.

Going further with mixed precision #

Daviator eliminates the k-point bottleneck, but currently relies on FP64 arithmetic and storage. GPU hardware is shifting toward lower-precision data types, and HPC codes that adapt to mixed precision stand to recapture substantial performance from hardware they already own.

Further increasing the throughput, especially on GPUs with a high FP32:FP64 ratio, requires leveraging the FP32 units, but developing a mixed-precision Davidson solver is non-trivial. Not every kernel in the Davidson pipeline is equally precision-sensitive, but simply running each in FP32 and comparing against FP64 only yields a pass/fail result.

Additionally, the failures can be insidious. Hard failures are obvious: a solver rejects an ill-conditioned input. Soft failures are not: energy drift, incorrect convergence, diverging MD trajectories that appear valid on casual inspection. Genuine mixed precision requires knowing exactly where the precision boundaries lie.

By using a generic floating-point type with configurable mantissa and exponent bits, we can identify the minimum required bit counts for each that still produce correct results. The following section describes a general workflow for a precision requirements analysis that can be adapted for other codes as well.

Knowing the exact precision requirements for each kernel informs whether to switch it to FP32, improve the implementation to bring it into FP32 range, or keep it as an FP64 island.

Implementing this approach depends heavily on the specific code, but always requires the following building blocks:

  • An instrumented version of the code that uses a generic floating point type with a configurable mantissa and exponent bit count.
  • A run script that executes the code with a specific precision setting and input.
  • A compare script that checks the accuracy of the results against a reference.
  • A screening script that runs the instrumented code multiple times for each kernel, first with decreasing mantissa bits until the comparison fails, and then with the smallest working mantissa size and decreasing exponent bits.

Once this setup is in place, re-run the screening after each code change to track precision requirements. Measured precision requirements for different implementations of a reduction kernel vary (Figure 3).

The results depend on the input and comparison threshold. Here, all kernels would pass a simple FP32 run (Figure 4) without providing an insight into the numerical properties. The baseline is likely to fail with either a slightly larger input or a stricter comparison. The algorithm choice may also come with additional constraints.

The code instrumentation can be performed at three levels:

  • The library level: Some libraries offer configurable precision, though only to a limited extent. The emulated matrix multiplications incuBLAS , for example, allow screening the precision requirements of GEMMs by explicitly setting the mantissa bit count.
  • The language level: The native floating-point types can be replaced with a generic one.NVIDIA CCCL provides such a type, with both compile-time and runtime configuration of mantissa and exponent bits.
  • The compiler level: A completely non-intrusive and mostly automatic instrumentation can be achieved by replacing floating-point operations with a software implementation during compilation. TheVerificarlo project takes this approach with its VPREC backend, but is limited to CPU code.

A complete instrumentation also covers dependencies and standard math functions. The instrumented code will be much slower, but the runs can be largely parallelized across HPC resources.

To move from analysis to implementation, start by adding FP64↔FP32 conversion infrastructure around your application’s hot path or a subset of it. Then create type-configurable versions of the functions and kernels in that region. For an initial implementation, keep everything exceeding FP32 requirements as an FP64 island; each subsequent iteration can reduce the FP64 footprint further.

Following this workflow, the original analysis of the Davidson algorithm, based on Verificarlo, indicated that large parts of the code could run with FP32 arithmetic and storage. The FFTs, the application of the Hamiltonian, and the orbital updates run in FP32, while the subspace eigenproblem, the tall-skinny GEMMs behind the subspace matrices, as well as the energies and convergence measures remain in FP64.

The tall-skinny GEMMs in the Rayleigh-Ritz projection were the remaining FP64 bottleneck. Pure FP32 accumulation here is insufficient as the output feeds directly into a numerically sensitive eigenvalue solver. The split-k algorithm resolves this by breaking the dot products into FP32 chunks with a final FP64 accumulation step (Figure 5).

Results

In VASP, convergence is controlled by the EDIFF parameter, which specifies the threshold for the total energy change between electronic iterations within an ionic step. Below that threshold, convergence is reached.

The default is 10<sup>-4</sup> eV, while some calculations go down to 10<sup>-6</sup> eV or lower. For reference, FP32 provides around 7 significant decimal digits of precision. EDIFF is an absolute threshold that varies by system and typically scales with atom count, so there is no single best setting.

The mixed-precision Davidson implementation converges to an EDIFF of 10<sup>-4</sup> to 10<sup>-6</sup> eV depending on the input; continuing with FP64 allows the calculation to converge below the FP32 noise floor. On GaAsBi-64, FP32 enables a 2× speed-up at EDIFF = 10<sup>-5</sup> eV, and the combined FP32→FP64 run is 1.5× faster than pure FP64 at EDIFF = 10⁻⁸ eV (Figure 6).

On H100 and H200, where FP64 Tensor Cores match FP32 throughput, mixed precision delivers only marginal end-to-end gains. The benefit is concentrated in hardware with a high FP32:FP64 ratio such as the RTX PRO 6000, where the FP32 Davidson solver runs 3x–7x faster, delivering roughly 2x end-to-end acceleration (Figure 7).

Enabling mixed precision

Daviator’s mixed precision can be enabled by setting LDAVIATOR_MIXED_PRECISION to 1 (0 is FP64). Within an ionic step, the calculation starts in mixed precision (with FP32 storage) until a configurable threshold DAVIATOR_EDIFF_FP32 or a number of SCF steps DAVIATOR_SWITCH_TO_FP64_AFTER_STEP is reached. For most inputs LAVIATOR_EDIFF_FP32 works well from 10<sup>-4</sup> to 10<sup>-5</sup>.

The following INCAR parameters are available to configure the use of mixed precision specifically for Daviator (Table 1).

INCAR Parameter Values Description
LDAVIATOR_MIXED_PRECISION False = FP64 True = FP32 Enable/Disable Mixed Precision (FP32 = mixed arithmetic and FP32 storage in Daviator)
DAVIATOR_EDIFF_FP32 real number > 0 in eV EDIFF threshold for FP32
DAVIATOR_SWITCH_TO_FP64_AFTER_STEP Integer N >= 0, switch to FP64 after electronic step N (N=0 is FP64) -1 = FP32 only, i.e. converge to DAVIATOR_EDIFF_FP32, then stop -2 = dynamic. i.e. switch to FP64 after DAVIATOR_EDIFF_FP32 convergence Selects when an ionic step switches from reduced precision to FP64.

Table 1. INCAR parameters for enabling and configuring mixed precision in Daviator

The INCAR for a run that uses FP32 until EDIFF = 5<sup>-5</sup> eV and then switches to FP64 until EDIFF = 5<sup>-7</sup> eV would look like this:

ALGO = Normal
EDIFF = 5e-07
[...]
LDAVIATOR = True
LDAVIATOR_MIXED_PRECISION = True
DAVIATOR_SWITCH_TO_FP64_AFTER_STEP = -2
DAVIATOR_EDIFF_FP32 = 5e-05

Hardware requirements and availability #

A VASP user license is required. Daviator and its mixed precision option will be available on any NVIDIA GPU through the NVIDIA ALCHEMI VASP microservice; add the INCAR parameters above to activate it.

What both layers deliver at scale #

The Open Materials 2024 (OMat24) inorganic materials dataset contains over 100 million VASP data points (total energy, forces, and stress) from molecular dynamics (“AIMD”), relaxation (“rattled-relax”), and static (“rattled”) calculations for MLIP training and validation.

To estimate the time and cost to compute this dataset, we extrapolated from a benchmarked sample of 13 representative inputs across the dataset’s system-size distribution. The individual benchmark runs were tuned to maximize resource utilisation and overall throughput, assuming multiple parallel runs per node. While OMat24 was generated with real-space projectors, the benchmarks and projections here use the more precise reciprocal-space projector setting in VASP (INCAR: LREAL=False).

On the same number of cloud instances, VASP + Daviator is up to 22x faster than VASP on CPU nodes, and up to 7x faster than VASP without Daviator on GPU (Figure 8). For a batch of eight small systems on one GPU, the NVIDIA ALCHEMI VASP microservice raises throughput a further 3–5x.

To estimate the price of the data set generation, we applied a geometric mean of the on-demand prices of AWS, Azure, CoreWeave and Google. On datacenter GPUs, the existing OpenACC GPU support in VASP is more expensive than the corresponding CPU computation.

However, the combination with Daviator reduces the cost to roughly 35% of the CPU cost on H100/200 and B200, with mixed precision contributing up to $200,000 in additional savings. The overall lowest price-point—around 25% of the CPU cost—is achieved on RTX PRO 6000 using VASP with Daviator and FP32 (Figure 9).

What’s next for GPU-native DFT #

For standard DFT, GPUs are now the clear choice for VASP. For teams building MLIP training pipelines at the scale of OMat24 and beyond, VASP + Daviator delivers a GPU-native workflow end-to-end with cost savings of 60–75% over CPU, or approximately $5–6M USD for a dataset the size of OMat24.

For teams running RTX PRO 6000 or other NVIDIA GPUs with a high FP32:FP64 ratio, mixed precision yields a 2x speedup over FP64, with benefits beyond dataset generation. Daviator also opens up workload classes that were previously impractical at GPU scale. Meta-GGA functionals like R²SCAN are now viable for production; the reported speed-ups for LiFePO₄ may bring higher-accuracy DFT within reach for battery, catalyst, and defect datasets.

Acknowledgments

The VASP development team and Akhilesh Mishra contributed to this post.

── more in #ai-research 4 stories · sorted by recency
── more on @nvidia 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/tackling-gpu-bottlen…] indexed:0 read:13min 2026-11-18 · —