AOTInductor Compile Cache A new technical blog post details how to use the AOTInductor compile cache to reuse previously tuned kernels when rebuilding a neural network inference engine, addressing the non-determinism that auto-tuning introduces on the same machine. The post notes that TorchInductor writes compile cache entries to the `cache` subdirectory of the TorchInductor cache directory, with each entry keyed to a specific kernel configuration and its tuning results, and shows a sample cache keyed to Triton 3.8.0, CUDA 13.4 and an NVIDIA GeForce RTX 5080. The author argues that production inference engines should either reuse the exact engine used for verification or rely on a reliable cache mechanism, since changed kernels can invalidate correctness even when the neural network is unchanged. AOTInductor Compile Cache Building a neural network inference engine using compiler can be slow and non-deterministic due to auto-tuning, even if it is on the same computer. The correctness of an inference engine depends on the kernels it used and the input distributions it encounters during inference. If the kernels used for a previously verified inference engine changed, even if the neural network remains the same, there is no guarantee that the inference results will be correct. Consequently, to serve an inference engine for production, the best practice is, either the same engine used for verification is used for production, or a reliable mechanism is in place to cache and reuse the previously tuned kernels when an inference engine is rebuilt from the same neural network. Of course, the former approach is simpler and more straightforward. But in practice, the latter approach is also often used. In this blog post, I would like to quickly discuss how to leverage the AOTInductor compile cache to ensure consistent and efficient reuse of previously tuned kernels for neural network inference engines. TorchInductor will generate compile cache in the cache subdirectory of the TorchInductor cache directory. The content of the cache file might look like this, where each entry corresponds to a specific kernel configuration and its associated tuning results. | 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364 | { "system": { "version": { "triton": "3.8.058b25352786ff1d04468ce113f1986063da0b78640428046a28c8ede4c9126ac-e7b75e513fb868674fd2adc660c6527e4ad9e74f01120d8e2b1ad243329a0c92-28ff26b7e752df455c410418d72fd0afc5b3bfa952ca4bca95b1cfd5a31a0e2f-23d635e690d670bf61798e1259674b78c0ed5ba222ab6a455f329f27a758fc2d-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-a39587ad88b66ea78116454125470474c2a5cd783f9d0ab81d607820596004d2-24b908043301e5dfcdbc4cdb17b9b7313e650cdfd4098b2ebfb998af8994be4e-c5d3403b88b5bb728f8ac78c55163198e418430f126c842892c68cb4b6baaa78-3aaa775f592533d06c13c99c23bd117430b1e7c732c2ae64c571d84d01ad2c39-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-6c4fa53b40b2748935f42b47f987adee8ec272b1fc54d2f27c6eb1cc029ec5ff-966a8105ff35077006b103266c3d8d292a56ad096a2d0dd6833027efb4bd1d50-fcdf0895d9ea3df9b91d6c05aba5aba579f385b8debd474b8525aa137e9b2bb2-b4966729f21f0db6f20fa4cb9b2b5402c5735e1424ffba05dbd8f480dedee790-5d15c5bebef8d7aa51b21fd187e5faa95eba4a213254355bc69e0648013599f7-30106ed84518c6ca7aca08e2c0ee188755f512cc0cb2d7da8914cc48c1ad6dcc-adb54e71d0ffc3bdac437fbc97929769fbbe4ebd03e6361c6357f2a24f7c5954-27b2a5d1e8db008bacefe6019f63922bbd65926de90bb1b527ee597477d2f365-a610dc5c215589aab7a784e1c07acef3e16d53ef00f08de793899964956f4e2a-ebad24d351bb79b3068a60627c74372a07cb8cc9532ce106773402ad2515a2b4-0731cda9ecbdd902acab13604efda0f9dbeaf411bc42bdddff8454b69a62f54b-2c66ab6523cc7dcd8bdc3e68b8f8844657d7bd4fd0492774c9819b046b8f7674-f60c2bb2d8eebe1c191f4b8b819844414dd1bce243645635a094f9f92665a58e-08abee21ce6230a873ed0831f70f9570b7ce39969dbf9b2f28ae1a1992ee1cc7-1a7ae2a0c3de32df60ffc18f21a20f8dcf8bf3bc8cb018bb269b540533868829-0d99c1f30bdb86bd718c98bf026ced2a84a878f79630fa9b3fe099ce46e86ebd-113296c74fb8b8dcf67a1b82f71926842eb66d0c77fe2c4f8d0f315715a38fd4-18ba36f0d8827d4807c631a2b92f9615eb56c8aa7f914964a0bb6f2ba69f344d", "cuda": "13.4" }, "device": { "name": "NVIDIA GeForce RTX 5080" }, "hash": "49cc03d54ac5bc8584957a4db2c7460d346c6f70b7002fea25dc7a2ef0c7c3a1" }, "cache": { "mm": { " 'cuda', 'torch.bfloat16', 64, 1024, 1024, 1, 0 , 'cuda', 'torch.bfloat16', 1024, 2048, 1, 1024, 0 ": { "high": { "mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.009983999654650688, "triton mm-cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67": 0.03683200106024742, "triton mm-csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz": 0.013632000423967838, "triton mm-ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl": 0.018400000408291817, "triton mm-cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx": 0.013856000266969204, "triton mm-ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65": 0.012256000190973282, "triton mm-chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh": 0.04095999896526337, "triton mm-ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx": 0.027936000376939774, "triton mm-c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun": 0.013856000266969204, "triton mm-crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku": 0.01990400068461895, "triton mm-cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw": 0.024000000208616257, "triton mm-co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea": 0.01881599985063076, "triton mm-ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45": 0.018144000321626663, "triton mm-cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck": 0.029023999348282814, "triton mm-cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg": 0.01583999954164028, "triton mm-cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt": 0.016767999157309532, "triton mm-c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz": 0.030239999294281006, "triton mm-ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe": 0.018432000651955605, "triton mm-cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu": 0.026079999282956123, "triton mm-cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb": 0.04095999896526337 } }, " 'cuda', 'torch.bfloat16', 64, 2048, 2048, 1, 0 , 'cuda', 'torch.bfloat16', 2048, 1024, 1, 2048, 0 ": { "high": { "mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.01228800043463707, "triton mm-cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu": 0.061055999249219894, "triton mm-cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f": 0.017920000478625298, "triton mm-c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc": 0.02208000048995018, "triton mm-cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w": 0.016063999384641647, "triton mm-cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd": 0.01228800043463707, "triton mm-c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7": 0.06761600077152252, "triton mm-cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp": 0.04323200136423111, "triton mm-c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn": 0.020864000543951988, "triton mm-cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx": 0.0323840007185936, "triton mm-cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3": 0.034143999218940735, "triton mm-c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk": 0.03014400042593479, "triton mm-crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz": 0.029055999591946602, "triton mm-codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j": 0.05273599922657013, "triton mm-cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6": 0.030208000913262367, "triton mm-cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo": 0.02502400055527687, "triton mm-chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba": 0.054816000163555145, "triton mm-cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq": 0.03014400042593479, "triton mm-cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy": 0.04265600070357323, "triton mm-cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn": 0.0761599987745285 } } } }} | If this cache will is present or carried to another computer that has the same hardware and software environment, TorchInductor can reuse the cached compilation results, avoiding the need to re-tune the kernels. In the following example, we demonstrate how to compile a simple MLP model, execute it to trigger compilation, save the compilation cache from development, and reuse the compilation cache for production re-compilation. | 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176 | python import jsonimport osimport platformimport tarfileimport timeimport torchfrom torch. inductor.runtime.cache dir utils import cache dirCACHE BLOB PATH = "aoti cache.tar.gz"CACHE META PATH = "aoti cache meta.json"PACKAGE PATH = "model.pt2"class SimpleMLP torch.nn.Module : def init self : super . init self.fc1 = torch.nn.Linear 1024, 2048, bias=False self.fc2 = torch.nn.Linear 2048, 1024, bias=False def forward self, x : return self.fc2 torch.nn.functional.gelu self.fc1 x def get platform fingerprint - dict: props = torch.cuda.get device properties 0 try: import triton triton version = triton. version except ImportError: triton version = "unavailable" return { "gpu name": props.name, "compute capability": f"{props.major}.{props.minor}", "multi processor count": props.multi processor count, "total memory mb": props.total memory // 1024 1024 , "pytorch version": torch. version , "cuda runtime version": torch.version.cuda, "triton version": triton version, "python version": platform.python version , "os arch": f"{platform.system }-{platform.machine }", }def check platform compatibility saved meta: dict, current meta: dict - bool: all matched = True all keys = sorted set saved meta.keys | set current meta.keys print " Platform Check Comparing build environment against cached metadata:" for key in all keys: expected = saved meta.get key, "