autotune

Autotune, a new open-source tool, optimizes local large language models by automatically right-sizing KV cache buffers, tuning precision, caching system prompts, and managing model keep-alive, freeing…