Kwangju Shin (KJ) · Part 5 of the Geometric Deep Learning series
An engineer changes a CAD model and wants to know what that change does to pressure and flow. Turning the shape into points gives a neural network something to read, but it leaves a computational problem: how should information from all those points be combined?
In Part 4, GINO handled global computation on a regular latent grid. Here, we look at three designs that exchange information through smaller token sets: learned states in Transolver, a hierarchy of supernodes and latent tokens in UPT, and anchors in AB-UPT.
These are different ways to organize computation, not successive replacements for every earlier model. The question running through this post is which points need to exchange information, and how the result reaches the locations where predictions are needed.
Standard self-attention compares every token with every other token. With n tokens, that gives n² pairwise interactions.
At 10,000 points, that is already 100 million pairs per layer. At one million points, it becomes one trillion. Industrial CFD meshes may contain tens or hundreds of millions of cells, so full point-to-point attention over the original mesh is not a serious option.
None of the models in this post attempts that. Each changes the unit on which expensive interaction is performed.
Transolver asks: can many points be represented by a small set of learned physical states? UPT asks: can the entire input be compressed into a fixed latent space before most of the computation? AB-UPT asks: can full self-attention be reserved for a manageable set of anchors while the remaining positions only query them?
The shared idea is to give a smaller set of tokens the job of exchanging information across the domain, instead of connecting every point to every other point.
Transolver’s core idea is simple: do not let millions of points talk to one another directly. Let a small set of learned states carry the conversation.
For an automotive case, each discretization point may carry its coordinates, geometry or boundary information, operating conditions, and the variables available to the model. Transolver converts every point into a feature vector and gives it soft membership weights over M learned states. “Soft” matters: one point can contribute to several states rather than being assigned to a single spatial cluster. The model then follows three steps:
These steps are repeated across stacked blocks. An output head then turns the final point features into the target physical quantities.
This is why the colored regions in the comparison figure should not be read as geometric partitions. A state is not a hand-labeled wake, boundary layer, or freestream region. It is a learned feature-space grouping, and spatially distant points may contribute to the same state when their features call for it.
The attention matrix now depends on M² rather than N². The model must still encode every input point and return information to every output point, so its total cost still grows with mesh size. Transolver reduces the most expensive all-to-all step; it does not make the mesh free.
Transolver++ refines these assignments so that the states remain more distinct, and distributes pointwise work across GPUs while exchanging the smaller state representation. The paper reports 1.2 million points on one GPU and experiments up to 2.5 million points on four GPUs. The scale increases, but storing and processing features for every input point still costs memory.
UPT makes a different choice: compress the sampled input in stages, process a fixed-size latent representation, and decode where predictions are needed. Here, “fixed” refers to the number of latent tokens, not their contents: the information they carry changes with the input.
For a steady vehicle-pressure task, the input describes the geometry and any varying conditions; it does not need to contain the pressure we want to predict. In time-dependent tasks, available field values such as an initial state can also be inputs. The path through the model is: The practical appeal is separation between input and output resolution. The latent transformer can retain the same token budget even when the number of sampled input points changes, and the decoder can answer a different set of output coordinates. The architecture therefore accepts varying discretizations; whether it remains accurate on a discretization unlike those seen during training still has to be tested.
The latent processing cost stays fixed if the latent token count and model size stay fixed. The encoder still has to read the input points and process the supernodes, however, and the decoder must answer each requested query. Global attention therefore happens at both the supernode and latent stages, not only after the final compression.
The distinction from Transolver is now easier to see. Transolver repeatedly forms physical-state tokens from the current point features and maps them back to the same discretization. UPT uses a hierarchy of supernodes and fixed latent tokens, then lets arbitrary output coordinates query the final representation.
AB-UPT starts from UPT’s encode–process–decode pattern, but gives automotive CFD a more explicit structure. Geometry is encoded once, while surface and volume fields are handled by separate but communicating prediction branches.
In a simplified forward pass:
The key distinction is between anchors, which participate in the expensive global processing, and queries, which only read from the resulting context. Queries do not all attend to one another. With M anchors and Q queries, the main attention cost is approximately O(M² + QM) instead of full attention over every target point. Because queries are independent of one another, a high-resolution field can also be decoded in batches.
This is not ordinary geometric interpolation. A query retrieves a learned summary that has already mixed geometry and long-range surface–volume information. At the same time, anchors should not be described as manually selected “important physics” locations. The paper reports reasonably stable results under random anchor selection; their value comes from how the model processes them, not from an engineer labeling each one.
That separation is useful when starting from CAD. In the paper’s CAD-based setup, geometry points and surface anchors are sampled from the CAD surface, while volume anchors are sampled in the surrounding domain. During training, CFD results supervise predictions at separate query positions. At inference, a new CFD volume mesh is not required. Surface preparation and sampling still are: this is a CAD-derived point representation, not a network reading an untouched CAD design history.
The experiments span automotive surface and volume fields associated with meshes from roughly 33,000 to 150 million cells. The upper number describes the resolution of the CFD target, not the number of tokens receiving full self-attention. Keeping those two quantities separate is what makes the scale manageable.
CarBench evaluates eleven model configurations using DrivAerNet++’s 8,150 steady-state RANS simulations. Its primary task is predicting surface pressure from vehicle geometry at a fixed freestream condition. The original UPT is not included, so this is not a direct benchmark of all three architectures discussed above.
The published leaderboard places AB-UPT* and the Transolver variants among the strongest models for accuracy. TripNet, an implicit-field model, is competitive and faster in this evaluation. That is useful evidence for this task — not a general ranking for every CFD quantity or operating condition.
The comparison also needs context. Model sizes and training settings differ. The benchmark standardizes evaluation metrics and inference hardware, but it does not hold parameter count or training compute constant. AB-UPT* uses an author-provided checkpoint; retraining it in the CarBench pipeline changes its relative L2 error from 0.1358 to 0.1539. These results can inform model selection, but whether the ranking holds for different geometries or physical quantities requires separate testing.
The latency shown in the chart is measured for 10,000 output points. If predictions are needed at more locations, runtime should be measured at that output resolution.
All three models give a smaller representation the task of exchanging information, rather than connecting every point to every other point. They differ in what they collect, where information is exchanged, and how it returns to predictions at individual locations. Choosing among them means checking whether that design delivers the accuracy the problem needs at an affordable computational cost.
Transolver, UPT, AB-UPT: Making Global Attention Affordable was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.