PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models. Significant amount of work went into CPU SIMD acceleration/testing/correctness, and wide cross-platform availability with constant testing to never lose portability (from DOS through OS/X 10.4 to modernity). CUDA/HIP is supported, and accelerated IMMA kernels are available. More work needs to be done on prompt processing speed, but text generation is quite fast already.
GGUFs are mmap()'ed, not preloaded, so it's much more friendly to RAM usage than llama.cpp. External LE/BE GGUF->FUGG utility available for the endian-handicapped.
OpenAI/llama.cpp-compatible HTTP server, ready to use with harnesses.
Eye candy: optional live VNC visualization of the per-layer activation heatmap.
I do actual feature freeze and release cycles, unlike llama.cpp which did a grand total of zero in the past 3 years. In fact, v1.0-rc1 just got released.
Comments URL: [https://news.ycombinator.com/item?id=49547323](https://news.ycombinator.com/item?id=49547323)
Points: 1