cd /news/machine-learning/aotinductor-compile-cache · home › topics › machine-learning › article
[ARTICLE · art-170413] src=leimao.github.io ↗ pub= topic=machine-learning verified=true sentiment=· neutral

AOTInductor Compile Cache

A new technical blog post details how to use the AOTInductor compile cache to reuse previously tuned kernels when rebuilding a neural network inference engine, addressing the non-determinism that auto-tuning introduces on the same machine. The post notes that TorchInductor writes compile cache entries to the `cache` subdirectory of the TorchInductor cache directory, with each entry keyed to a specific kernel configuration and its tuning results, and shows a sample cache keyed to Triton 3.8.0, CUDA 13.4 and an NVIDIA GeForce RTX 5080. The author argues that production inference engines should either reuse the exact engine used for verification or rely on a reliable cache mechanism, since changed kernels can invalidate correctness even when the neural network is unchanged.

read13 min views2 publishedNov 16, 2026
AOTInductor Compile Cache
Image: Leimao (auto-discovered)

#

Building a neural network inference engine using compiler can be slow and non-deterministic due to auto-tuning, even if it is on the same computer. The correctness of an inference engine depends on the kernels it used and the input distributions it encounters during inference. If the kernels used for a previously verified inference engine changed, even if the neural network remains the same, there is no guarantee that the inference results will be correct. Consequently, to serve an inference engine for production, the best practice is, either the same engine used for verification is used for production, or a reliable mechanism is in place to cache and reuse the previously tuned kernels when an inference engine is rebuilt from the same neural network. Of course, the former approach is simpler and more straightforward. But in practice, the latter approach is also often used.

In this blog post, I would like to quickly discuss how to leverage the AOTInductor compile cache to ensure consistent and efficient reuse of previously tuned kernels for neural network inference engines.

#

TorchInductor will generate compile cache in the cache subdirectory of the TorchInductor cache directory. The content of the cache file might look like this, where each entry corresponds to a specific kernel configuration and its associated tuning results.

|

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364

|

{  "system": {    "version": {      "triton": "3.8.058b25352786ff1d04468ce113f1986063da0b78640428046a28c8ede4c9126ac-e7b75e513fb868674fd2adc660c6527e4ad9e74f01120d8e2b1ad243329a0c92-28ff26b7e752df455c410418d72fd0afc5b3bfa952ca4bca95b1cfd5a31a0e2f-23d635e690d670bf61798e1259674b78c0ed5ba222ab6a455f329f27a758fc2d-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-a39587ad88b66ea78116454125470474c2a5cd783f9d0ab81d607820596004d2-24b908043301e5dfcdbc4cdb17b9b7313e650cdfd4098b2ebfb998af8994be4e-c5d3403b88b5bb728f8ac78c55163198e418430f126c842892c68cb4b6baaa78-3aaa775f592533d06c13c99c23bd117430b1e7c732c2ae64c571d84d01ad2c39-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-6c4fa53b40b2748935f42b47f987adee8ec272b1fc54d2f27c6eb1cc029ec5ff-966a8105ff35077006b103266c3d8d292a56ad096a2d0dd6833027efb4bd1d50-fcdf0895d9ea3df9b91d6c05aba5aba579f385b8debd474b8525aa137e9b2bb2-b4966729f21f0db6f20fa4cb9b2b5402c5735e1424ffba05dbd8f480dedee790-5d15c5bebef8d7aa51b21fd187e5faa95eba4a213254355bc69e0648013599f7-30106ed84518c6ca7aca08e2c0ee188755f512cc0cb2d7da8914cc48c1ad6dcc-adb54e71d0ffc3bdac437fbc97929769fbbe4ebd03e6361c6357f2a24f7c5954-27b2a5d1e8db008bacefe6019f63922bbd65926de90bb1b527ee597477d2f365-a610dc5c215589aab7a784e1c07acef3e16d53ef00f08de793899964956f4e2a-ebad24d351bb79b3068a60627c74372a07cb8cc9532ce106773402ad2515a2b4-0731cda9ecbdd902acab13604efda0f9dbeaf411bc42bdddff8454b69a62f54b-2c66ab6523cc7dcd8bdc3e68b8f8844657d7bd4fd0492774c9819b046b8f7674-f60c2bb2d8eebe1c191f4b8b819844414dd1bce243645635a094f9f92665a58e-08abee21ce6230a873ed0831f70f9570b7ce39969dbf9b2f28ae1a1992ee1cc7-1a7ae2a0c3de32df60ffc18f21a20f8dcf8bf3bc8cb018bb269b540533868829-0d99c1f30bdb86bd718c98bf026ced2a84a878f79630fa9b3fe099ce46e86ebd-113296c74fb8b8dcf67a1b82f71926842eb66d0c77fe2c4f8d0f315715a38fd4-18ba36f0d8827d4807c631a2b92f9615eb56c8aa7f914964a0bb6f2ba69f344d",      "cuda": "13.4"    },    "device": {      "name": "NVIDIA GeForce RTX 5080"    },    "hash": "49cc03d54ac5bc8584957a4db2c7460d346c6f70b7002fea25dc7a2ef0c7c3a1"  },  "cache": {    "mm": {      "[('cuda', 'torch.bfloat16', 64, 1024, 1024, 1, 0), ('cuda', 'torch.bfloat16', 1024, 2048, 1, 1024, 0)]": {        "high": {          "mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.009983999654650688,          "triton_mm-cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67": 0.03683200106024742,          "triton_mm-csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz": 0.013632000423967838,          "triton_mm-ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl": 0.018400000408291817,          "triton_mm-cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx": 0.013856000266969204,          "triton_mm-ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65": 0.012256000190973282,          "triton_mm-chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh": 0.04095999896526337,          "triton_mm-ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx": 0.027936000376939774,          "triton_mm-c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun": 0.013856000266969204,          "triton_mm-crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku": 0.01990400068461895,          "triton_mm-cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw": 0.024000000208616257,          "triton_mm-co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea": 0.01881599985063076,          "triton_mm-ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45": 0.018144000321626663,          "triton_mm-cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck": 0.029023999348282814,          "triton_mm-cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg": 0.01583999954164028,          "triton_mm-cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt": 0.016767999157309532,          "triton_mm-c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz": 0.030239999294281006,          "triton_mm-ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe": 0.018432000651955605,          "triton_mm-cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu": 0.026079999282956123,          "triton_mm-cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb": 0.04095999896526337        }      },      "[('cuda', 'torch.bfloat16', 64, 2048, 2048, 1, 0), ('cuda', 'torch.bfloat16', 2048, 1024, 1, 2048, 0)]": {        "high": {          "mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.01228800043463707,          "triton_mm-cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu": 0.061055999249219894,          "triton_mm-cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f": 0.017920000478625298,          "triton_mm-c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc": 0.02208000048995018,          "triton_mm-cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w": 0.016063999384641647,          "triton_mm-cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd": 0.01228800043463707,          "triton_mm-c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7": 0.06761600077152252,          "triton_mm-cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp": 0.04323200136423111,          "triton_mm-c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn": 0.020864000543951988,          "triton_mm-cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx": 0.0323840007185936,          "triton_mm-cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3": 0.034143999218940735,          "triton_mm-c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk": 0.03014400042593479,          "triton_mm-crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz": 0.029055999591946602,          "triton_mm-codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j": 0.05273599922657013,          "triton_mm-cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6": 0.030208000913262367,          "triton_mm-cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo": 0.02502400055527687,          "triton_mm-chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba": 0.054816000163555145,          "triton_mm-cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq": 0.03014400042593479,          "triton_mm-cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy": 0.04265600070357323,          "triton_mm-cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn": 0.0761599987745285        }      }    }  }}

|

If this cache will is present or carried to another computer that has the same hardware and software environment, TorchInductor can reuse the cached compilation results, avoiding the need to re-tune the kernels.

In the following example, we demonstrate how to compile a simple MLP model, execute it to trigger compilation, save the compilation cache from development, and reuse the compilation cache for production re-compilation.

|

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176

|

import jsonimport osimport platformimport tarfileimport timeimport torchfrom torch._inductor.runtime.cache_dir_utils import cache_dirCACHE_BLOB_PATH = "aoti_cache.tar.gz"CACHE_META_PATH = "aoti_cache_meta.json"PACKAGE_PATH = "model.pt2"class SimpleMLP(torch.nn.Module):    def __init__(self):        super().__init__()        self.fc1 = torch.nn.Linear(1024, 2048, bias=False)        self.fc2 = torch.nn.Linear(2048, 1024, bias=False)    def forward(self, x):        return self.fc2(torch.nn.functional.gelu(self.fc1(x)))def get_platform_fingerprint() -> dict:    props = torch.cuda.get_device_properties(0)    try:        import triton        triton_version = triton.__version__    except ImportError:        triton_version = "unavailable"    return {        "gpu_name": props.name,        "compute_capability": f"{props.major}.{props.minor}",        "multi_processor_count": props.multi_processor_count,        "total_memory_mb": props.total_memory // (1024 * 1024),        "pytorch_version": torch.__version__,        "cuda_runtime_version": torch.version.cuda,        "triton_version": triton_version,        "python_version": platform.python_version(),        "os_arch": f"{platform.system()}-{platform.machine()}",    }def check_platform_compatibility(saved_meta: dict, current_meta: dict) -> bool:    all_matched = True    all_keys = sorted(set(saved_meta.keys()) | set(current_meta.keys()))    print(        "[Platform Check] Comparing build environment against cached metadata:"    )    for key in all_keys:        expected = saved_meta.get(key, "<missing>")        actual = current_meta.get(key, "<missing>")        if expected == actual:            print(f"  [MATCH]    {key}: '{actual}'")        else:            print(f"  [MISMATCH] {key}: expected '{expected}', got '{actual}'")            all_matched = False    if all_matched:        print(            "[Platform Check] Summary: ALL criteria matched (exact cache reuse guaranteed)."        )    else:        print(            "[Platform Check] Summary: WARNING — Mismatch detected! Cache keys may miss and re-autotune."        )    return all_matcheddef save_aoti_autotune_cache(archive_path: str) -> int:    """Packs the Inductor 'cache/' subdirectory (autotune decisions) into a portable archive."""    inductor_cache_subdir = os.path.join(cache_dir(), "cache")    if not os.path.isdir(inductor_cache_subdir):        return 0    file_count = 0    with tarfile.open(archive_path, "w:gz") as tar:        for root, _, files in os.walk(inductor_cache_subdir):            for fname in files:                full_path = os.path.join(root, fname)                rel_path = os.path.relpath(full_path, cache_dir())                tar.add(full_path, arcname=rel_path)                file_count += 1    return file_countdef load_aoti_autotune_cache(archive_path: str) -> int:    """Extracts the portable autotune archive into the active TORCHINDUCTOR_CACHE_DIR."""    target_dir = cache_dir()    os.makedirs(target_dir, exist_ok=True)    with tarfile.open(archive_path, "r:gz") as tar:        members = tar.getmembers()        tar.extractall(path=target_dir, filter="data")        return len(members)def main():    torch.manual_seed(0)    model = SimpleMLP().cuda().to(torch.bfloat16).eval()    example_inputs = (torch.randn(64,                                  1024,                                  device="cuda",                                  dtype=torch.bfloat16), )    batch_dim = torch.export.Dim("batch", min=1, max=512)    current_platform = get_platform_fingerprint()    has_portable_cache = os.path.exists(CACHE_BLOB_PATH)    # 1. Verify platform compatibility and restore the Inductor 'cache/' folder if present    if has_portable_cache:        if os.path.exists(CACHE_META_PATH):            with open(CACHE_META_PATH, "r") as f:                saved_platform = json.load(f)            check_platform_compatibility(saved_platform, current_platform)        else:            print(                f"[Platform Check] WARNING: '{CACHE_META_PATH}' not found; skipping platform comparison."            )        restored_files = load_aoti_autotune_cache(CACHE_BLOB_PATH)        print(            f"Restored {restored_files} autotune cache files into '{cache_dir()}' from '{CACHE_BLOB_PATH}'."        )    else:        print(            f"No '{CACHE_BLOB_PATH}' found; will autotune if disk cache is cold."        )    with torch.no_grad():        ep = torch.export.export(            model,            example_inputs,            dynamic_shapes={"x": {                0: batch_dim            }},            strict=True,        )        # 2. Compile AOTInductor package (measure strictly compile time)        torch.cuda.synchronize()        t0 = time.perf_counter()        torch._inductor.aoti_compile_and_package(            ep,            package_path=PACKAGE_PATH,            inductor_configs={                "max_autotune": True,            },        )        torch.cuda.synchronize()        compile_sec = time.perf_counter() - t0    print(f"AOTInductor compile time: {compile_sec:.2f}s")    # 3. On cold builds, verify correctness & archive the minimal 'cache/' directory    if not has_portable_cache:        compiled_fn = torch._inductor.aoti_load_package(PACKAGE_PATH)        with torch.no_grad():            ref_out = model(*example_inputs)            aoti_out = compiled_fn(*example_inputs)        torch.testing.assert_close(aoti_out, ref_out, rtol=1e-2, atol=1e-2)        print("Local correctness check passed.")        saved_files = save_aoti_autotune_cache(CACHE_BLOB_PATH)        with open(CACHE_META_PATH, "w") as f:            json.dump(current_platform, f, indent=2)        print(            f"Saved {saved_files} autotune cache files into '{CACHE_BLOB_PATH}' and '{CACHE_META_PATH}'."        )if __name__ == "__main__":    main()

|

To run the example, PyTorch Docker image from NVIDIA NGC nvcr.io/nvidia/pytorch:26.10-py3 was used.

|

1234567891011121314

|

|

After local development, the compilation cache is saved in aoti_cache.tar.gz and can be reused for production re-compilation.

|

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118

|

$ TORCHINDUCTOR_CACHE_DIR=./dev_cache \    TRITON_CACHE_DIR=./dev_cache/triton \    TORCH_LOGS="autotuning" \    python compile_aoti.pyNo 'aoti_cache.tar.gz' found; will autotune if disk cache is cold.V1116 07:22:40.661000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess ExternKernelBenchmarkRequest(extern_kernels.mm): load 0.000005, create tensor 0.000000, bench 0.142278V1116 07:22:40.664000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67, path: /mnt/dev_cache/d5/cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67.pyV1116 07:22:40.689000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/d5/cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67.py', self.module_cache_key='cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67': load 0.002170, create tensor 0.000000, bench 0.024590V1116 07:22:40.691000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz, path: /mnt/dev_cache/sr/csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz.pyV1116 07:22:40.713000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/sr/csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz.py', self.module_cache_key='csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz': load 0.002307, create tensor 0.000000, bench 0.021628V1116 07:22:40.715000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl, path: /mnt/dev_cache/ty/ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl.pyV1116 07:22:40.737000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/ty/ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl.py', self.module_cache_key='ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl': load 0.002234, create tensor 0.000000, bench 0.021785V1116 07:22:40.740000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx, path: /mnt/dev_cache/oh/cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx.pyV1116 07:22:40.762000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/oh/cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx.py', self.module_cache_key='cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx': load 0.002160, create tensor 0.000000, bench 0.022156V1116 07:22:40.764000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65, path: /mnt/dev_cache/k4/ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65.pyV1116 07:22:40.787000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/k4/ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65.py', self.module_cache_key='ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65': load 0.002336, create tensor 0.000000, bench 0.022160V1116 07:22:40.789000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh, path: /mnt/dev_cache/hj/chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh.pyV1116 07:22:40.814000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/hj/chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh.py', self.module_cache_key='chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh': load 0.002139, create tensor 0.000000, bench 0.024451V1116 07:22:40.816000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx, path: /mnt/dev_cache/e7/ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx.pyV1116 07:22:40.840000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/e7/ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx.py', self.module_cache_key='ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx': load 0.002221, create tensor 0.000000, bench 0.023701V1116 07:22:40.842000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun, path: /mnt/dev_cache/6u/c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun.pyV1116 07:22:40.865000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/6u/c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun.py', self.module_cache_key='c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun': load 0.002308, create tensor 0.000000, bench 0.022549V1116 07:22:40.867000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku, path: /mnt/dev_cache/rr/crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku.pyV1116 07:22:40.891000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/rr/crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku.py', self.module_cache_key='crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku': load 0.002243, create tensor 0.000000, bench 0.023495V1116 07:22:40.893000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw, path: /mnt/dev_cache/gv/cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw.pyV1116 07:22:40.917000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/gv/cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw.py', self.module_cache_key='cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw': load 0.002159, create tensor 0.000000, bench 0.023397V1116 07:22:40.919000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea, path: /mnt/dev_cache/o5/co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea.pyV1116 07:22:40.942000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/o5/co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea.py', self.module_cache_key='co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea': load 0.002196, create tensor 0.000000, bench 0.022916V1116 07:22:40.945000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45, path: /mnt/dev_cache/tb/ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45.pyV1116 07:22:40.969000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/tb/ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45.py', self.module_cache_key='ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45': load 0.002147, create tensor 0.000000, bench 0.023867V1116 07:22:40.971000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck, path: /mnt/dev_cache/sz/cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck.pyV1116 07:22:40.996000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/sz/cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck.py', self.module_cache_key='cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck': load 0.002202, create tensor 0.000000, bench 0.024556V1116 07:22:40.998000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg, path: /mnt/dev_cache/ng/cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg.pyV1116 07:22:41.020000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/ng/cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg.py', self.module_cache_key='cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg': load 0.002171, create tensor 0.000000, bench 0.022327V1116 07:22:41.023000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt, path: /mnt/dev_cache/w5/cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt.pyV1116 07:22:41.045000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/w5/cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt.py', self.module_cache_key='cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt': load 0.002168, create tensor 0.000000, bench 0.022238V1116 07:22:41.048000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz, path: /mnt/dev_cache/2t/c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz.pyV1116 07:22:41.072000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/2t/c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz.py', self.module_cache_key='c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz': load 0.002227, create tensor 0.000000, bench 0.024506V1116 07:22:41.074000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe, path: /mnt/dev_cache/tj/ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe.pyV1116 07:22:41.097000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/tj/ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe.py', self.module_cache_key='ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe': load 0.002155, create tensor 0.000000, bench 0.022694V1116 07:22:41.100000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu, path: /mnt/dev_cache/fs/cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu.pyV1116 07:22:41.124000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/fs/cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu.py', self.module_cache_key='cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu': load 0.002608, create tensor 0.000000, bench 0.024043V1116 07:22:41.126000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb, path: /mnt/dev_cache/p6/cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb.pyV1116 07:22:41.153000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/p6/cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb.py', self.module_cache_key='cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb': load 0.002146, create tensor 0.000000, bench 0.026611Autotune Choices Stats:{"num_choices": 20, "num_triton_choices": 19, "best_kernel": "mm", "best_time": 0.009983999654650688, "best_triton_pos": 1, "best_triton_time": 0.012256000190973282, "best_triton_kernel": "triton_mm_4", "best_triton_kernel_desc": "ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4"}AUTOTUNE mm(64x1024, 1024x2048)strides: [1024, 1], [1, 1024]dtypes: torch.bfloat16, torch.bfloat16  mm 0.0100 ms 100.0%  triton_mm_4 0.0123 ms 81.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4  triton_mm_1 0.0136 ms 73.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4  triton_mm_3 0.0139 ms 72.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8  triton_mm_7 0.0139 ms 72.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8  triton_mm_13 0.0158 ms 63.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4  triton_mm_14 0.0168 ms 59.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8  triton_mm_11 0.0181 ms 55.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4  triton_mm_2 0.0184 ms 54.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8  triton_mm_16 0.0184 ms 54.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8SingleProcess AUTOTUNE benchmarking takes 0.6359 seconds and 1.4817 seconds precompiling for 20 choicesV1116 07:22:41.433000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess ExternKernelBenchmarkRequest(extern_kernels.mm): load 0.000005, create tensor 0.000000, bench 0.030149V1116 07:22:41.436000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu, path: /mnt/dev_cache/mw/cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu.pyV1116 07:22:41.462000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/mw/cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu.py', self.module_cache_key='cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu': load 0.002256, create tensor 0.000000, bench 0.026758V1116 07:22:41.465000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f, path: /mnt/dev_cache/jc/cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f.pyV1116 07:22:41.487000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/jc/cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f.py', self.module_cache_key='cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f': load 0.002183, create tensor 0.000000, bench 0.022386V1116 07:22:41.490000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc, path: /mnt/dev_cache/6e/c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc.pyV1116 07:22:41.511000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/6e/c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc.py', self.module_cache_key='c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc': load 0.002131, create tensor 0.000000, bench 0.021764V1116 07:22:41.514000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w, path: /mnt/dev_cache/jc/cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w.pyV1116 07:22:41.536000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/jc/cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w.py', self.module_cache_key='cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w': load 0.002167, create tensor 0.000000, bench 0.022027V1116 07:22:41.539000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd, path: /mnt/dev_cache/nx/cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd.pyV1116 07:22:41.561000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/nx/cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd.py', self.module_cache_key='cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd': load 0.003419, create tensor 0.000000, bench 0.020997V1116 07:22:41.563000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7, path: /mnt/dev_cache/3v/c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7.pyV1116 07:22:41.590000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/3v/c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7.py', self.module_cache_key='c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7': load 0.002177, create tensor 0.000000, bench 0.027241V1116 07:22:41.593000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp, path: /mnt/dev_cache/pp/cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp.pyV1116 07:22:41.617000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/pp/cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp.py', self.module_cache_key='cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp': load 0.002128, create tensor 0.000000, bench 0.024184V1116 07:22:41.619000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn, path: /mnt/dev_cache/7s/c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn.pyV1116 07:22:41.640000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/7s/c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn.py', self.module_cache_key='c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn': load 0.002138, create tensor 0.000000, bench 0.021063V1116 07:22:41.643000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx, path: /mnt/dev_cache/pe/cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx.pyV1116 07:22:41.667000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/pe/cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx.py', self.module_cache_key='cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx': load 0.002104, create tensor 0.000000, bench 0.023982V1116 07:22:41.669000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3, path: /mnt/dev_cache/v3/cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3.pyV1116 07:22:41.692000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/v3/cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3.py', self.module_cache_key='cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3': load 0.002136, create tensor 0.000000, bench 0.023215V1116 07:22:41.695000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk, path: /mnt/dev_cache/4j/c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk.pyV1116 07:22:41.717000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/4j/c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk.py', self.module_cache_key='c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk': load 0.002173, create tensor 0.000000, bench 0.022147V1116 07:22:41.719000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz, path: /mnt/dev_cache/rc/crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz.pyV1116 07:22:41.742000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/rc/crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz.py', self.module_cache_key='crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz': load 0.002204, create tensor 0.000000, bench 0.022645V1116 07:22:41.744000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j, path: /mnt/dev_cache/od/codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j.pyV1116 07:22:41.772000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/od/codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j.py', self.module_cache_key='codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j': load 0.002155, create tensor 0.000000, bench 0.027098V1116 07:22:41.774000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6, path: /mnt/dev_cache/vy/cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6.pyV1116 07:22:41.797000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vy/cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6.py', self.module_cache_key='cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6': load 0.002473, create tensor 0.000000, bench 0.022209V1116 07:22:41.799000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo, path: /mnt/dev_cache/vz/cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo.pyV1116 07:22:41.821000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vz/cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo.py', self.module_cache_key='cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo': load 0.002101, create tensor 0.000000, bench 0.021599V1116 07:22:41.823000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba, path: /mnt/dev_cache/hx/chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba.pyV1116 07:22:41.849000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/hx/chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba.py', self.module_cache_key='chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba': load 0.002681, create tensor 0.000000, bench 0.025591V1116 07:22:41.852000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq, path: /mnt/dev_cache/m2/cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq.pyV1116 07:22:41.874000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/m2/cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq.py', self.module_cache_key='cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq': load 0.002357, create tensor 0.000000, bench 0.022555V1116 07:22:41.877000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy, path: /mnt/dev_cache/vt/cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy.pyV1116 07:22:41.902000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vt/cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy.py', self.module_cache_key='cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy': load 0.002104, create tensor 0.000000, bench 0.025068V1116 07:22:41.904000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn, path: /mnt/dev_cache/qe/cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn.pyV1116 07:22:41.933000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/qe/cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn.py', self.module_cache_key='cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn': load 0.002103, create tensor 0.000000, bench 0.028835Autotune Choices Stats:{"num_choices": 20, "num_triton_choices": 19, "best_kernel": "mm", "best_time": 0.01228800043463707, "best_triton_pos": 1, "best_triton_time": 0.01228800043463707, "best_triton_kernel": "triton_mm_23", "best_triton_kernel_desc": "ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4"}AUTOTUNE mm(64x2048, 2048x1024)strides: [2048, 1], [1, 2048]dtypes: torch.bfloat16, torch.bfloat16  mm 0.0123 ms 100.0%  triton_mm_23 0.0123 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4  triton_mm_22 0.0161 ms 76.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8  triton_mm_20 0.0179 ms 68.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4  triton_mm_26 0.0209 ms 58.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8  triton_mm_21 0.0221 ms 55.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8  triton_mm_33 0.0250 ms 49.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8  triton_mm_30 0.0291 ms 42.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4  triton_mm_29 0.0301 ms 40.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8  triton_mm_35 0.0301 ms 40.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8SingleProcess AUTOTUNE benchmarking takes 0.5319 seconds and 0.0020 seconds precompiling for 20 choicesAOTInductor compile time: 14.32sLocal correctness check passed.Saved 1 autotune cache files into 'aoti_cache.tar.gz' and 'aoti_cache_meta.json'.

|

In production, the compilation cache can be reused and the autotuning process is skipped, ensuring the same kernels that have been verified are used, and the compilation time is significantly reduced. Note that the software and hardware environment, such as CUDA version, GPU architecture, PyTorch version, should match the environment in which the cache was generated.

|

1234567891011121314151617

|

$ TORCHINDUCTOR_CACHE_DIR=./prod_cache \    TRITON_CACHE_DIR=./prod_cache/triton \    TORCH_LOGS="autotuning" \    python compile_aoti.py[Platform Check] Comparing build environment against cached metadata:  [MATCH]    compute_capability: '12.0'  [MATCH]    cuda_runtime_version: '13.4'  [MATCH]    gpu_name: 'NVIDIA GeForce RTX 5080'  [MATCH]    multi_processor_count: '84'  [MATCH]    os_arch: 'Linux-x86_64'  [MATCH]    python_version: '3.12.3'  [MATCH]    pytorch_version: '2.15.0a0+875d815502.nv26.10'  [MATCH]    total_memory_mb: '15838'  [MATCH]    triton_version: '3.8.0'[Platform Check] Summary: ALL criteria matched (exact cache reuse guaranteed).Restored 1 autotune cache files into '/mnt/prod_cache' from 'aoti_cache.tar.gz'.AOTInductor compile time: 10.72s

|

#

Although it is less ideal than reusing the verified inference engine directly for production, if we ever have to recompile the model in a production environment, leveraging the saved compilation cache allows us to skip the autotuning process, ensuring correctness while significantly reducing compilation time.

AOTInductor Compile Cache

── more in #machine-learning 4 stories · sorted by recency
── more on @aotinductor 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/aotinductor-compile-…] indexed:0 read:13min 2026-11-16 · —