[This is a guest post by Anima Anandkumar. This blog post was initially written in a different file format and converted using AI.– T.]
The last few days have been a flurry of activity. After months of sleepless nights and diving deep into different mathematical and computational tools to tackle the open problem of singularity in fluid dynamics, we were finally ready to send our work on Euler equations (without forcing) to Tom Hou and Terry Tao for feedback. Almost immediately I heard back from Terry encouraging us to release our work publicly, given that the NYU team had just released their results on forced Euler and OpenAI was rumored to release the next day.
We scrambled and put together a blog post and attached our paper with code, which went live about 30 minutes after, in the evening of September 7, the day before the OpenAI’s announcement. A lot of mainstream media religiously followed their press release that fails to acknowledge our work even after we informed them. On the bright side, many researchers have reached out to me. I want to thank Terry for highlighting our work, as well as for giving me the opportunity to write this blog post.
The starting point for us is different from the route that the NYU team and OpenAI pursued. Their initial solution has a specific construction with a multi-scale structure with discrete jumps in scales. On the other hand, we consider a self-similar ansatz, and so far, no one has been able to construct a self-similar singular profile for the Euler equations in without forcing. Modified cases such as the Euler equation with a boundary have been successfully solved before (Chen and Hou), where the boundary helps confine the singularity, but such techniques fail in free space.
We wanted to see if AI could come up with a self-similar profile for Euler. LLMs are not suited for generating such singularity candidates without analytical closed forms. Physics informed neural networks (PINN), on the other hand, can be optimized to find such solutions. Although there have been prior attempts to use PINNs for this case, they were not successful. There are optimization and precision challenges: with standard ansatz and optimizers, a common failure mode is getting the trivial solution. The other challenge is getting precision to be high enough, so we can then attempt to convert the numerical solution into an analytical proof. This additionally requires stability at the PINN solution, which may not always occur.
When it comes to establishing singularity, we cannot directly use the Euler PDE, since that would blow up at the candidate solutions. Finding the right ansatz with enough structure is important, so it can be re-parameterized to finite solutions. Typically axisymmetric self-similar ansatz is chosen since it makes both optimization and analysis simpler.
In our PINN, we employ a traveling-wave self-similar ansatz since it allows us freedom to choose the velocity at which singularity travels along the -axis. Although this is equivalent to a stationary ansatz for analysis, this flexibility of traveling singularity, along with removal of hard constraints such as -parity, makes our optimization landscape more tractable. Efficient second-order optimizers such as SS-eSOAP, developed in our group, further help with convergence and are only slightly more expensive than the standard Adam optimizer. This allowed us to run an extensive set of experiments without needing massive compute. We refined our solutions only at the very end with an expensive optimizer SS-Broyden to get further increase in our accuracy. Further, methods like adaptive loss weighting, high-precision arithmetic (FP64), boosting, adaptive collocation all helped obtain high-precision solutions.
Our PINN solution aligns with prior theory and this was important for us to check constantly throughout the process. Constantin et al predict for self-similar ansatz the scaling exponent of 0.5 as “mathematically distinguished” for the Euler equations, and essentially rule out other values of the exponent for the axisymmetric case. When we make the scaling exponent a free parameter in our PINN optimization, it still converges close to 0.5, confirming agreement with theory. Further, when we optimize the PINN over a family of convection-weighted Euler, where we can vary the contribution of the convection term, the scaling exponent decreases towards 0.5 as we increase the contribution of the convection term, again in line with theory. All these careful checks increased our confidence that we were on the right path.
The other important piece of the puzzle is checking if our PINN solution is stable. Earlier works on related blowup problems could typically prove a global outgoing property for their approximate profile, which unfortunately does not hold in our case. However, there are recent theoretical arguments by Constantin et al that non-trivial fixed points exist where the transport term vanishes, and thus, the flow cannot be globally outgoing. Instead, we have to work with the weaker local outgoing property for stability, which is more involved. We verify that our PINN solution does indeed satisfy the local-outgoing property, allowing us to make stability arguments.
To make it a rigorous proof of stability, we need to go from discovering singularities numerically with PINNs to being able to provide tight bounds around the approximate profile. To do so, we fit the PINN solution into piecewise-polynomial splines, allowing derivatives, PDE residuals, and profile-dependent quantities to be evaluated and bounded using arbitrary-precision interval arithmetic.
We then formulate the stability problem in dynamically rescaled variables and develop weighted low- and high-order energy estimates adapted to the singular profile. Optimizing the associated singular weights reveals complementary damping mechanisms throughout the domain. In order to complete the stability proof, the non-linear contribution can be quite involved with thousands of terms. This is where LLMs play a role, and we used the OpenAI and other models extensively to simplify our bounds as well as formalize the derivations in Lean. The overall set of steps is shown in the diagram below.
Overall, our core contribution is making PINNs work successfully to produce a self-similar singular profile for the first time in Euler equations on without forcing, and then certifying its bounds over an interval with high enough precision to carry out the stability analysis. We believe that these techniques will have diverse applications both in theory and practice. In the theory and analysis of PDEs where analytical constructions may be out of reach, and hence, LLMs are not that helpful, PINNs can potentially discover new solutions in high enough precision that they can be certified and converted into analysis. Additionally, in the practical realm they can be applied for simulation, design and discovery in areas involving physical systems.
This is something my group at Caltech has been working on more broadly. While PINNs aim to find the solution for a given instance of PDE, we have proposed Neural Operators that learn mappings between function spaces, and can learn the solution operators of families of parametric PDEs. We used Neural Operators to train the first high-resolution AI-weather model more than five years ago, which is tens of thousands of times faster, which allows us to support larger statistical ensembles for forecasting extreme events such as hurricanes and heat waves. We have designed a medical catheter that cuts down bacterial infection by hundred-fold using Neural Operators that learn fluid behavior, and optimized gate layouts in quantum dots. An exciting recent application of Neural Operators is solving a 60-year old problem in quantum chemistry to make density functional theory run in quasi-linear time, by skipping the auxiliary orbital calculation, and yet learning universal Kohn–Sham maps that are generalizable and transferable across molecular and material systems.
What all these cases have in common is how we are thoughtful about building AI and what purpose they serve. LLMs are trained on human text, to be good at tasks that humans are already good at, often with the explicit goal to compete with or replace human effort. Instead, our work focuses on building AI that complements human capabilities. In all the above cases, including our work on the Euler equations, we built physics-AI to propose solutions that did not compete with humans, since they are not analytical or in a structured form. This leaves room for human creativity to design the setup and the strategy for the overall proof. We want our AI to be complementary and collaborative with human creativity to enable us to push the frontiers of math and science even further.