BaseRT, A fast inference runtime for local AI on Apple Silicon
BaseCompute released BaseRT, a fast inference runtime for local AI on Apple Silicon, claiming up to 35% faster decode and 78% faster prefill on an Apple M4 Pro with 4-bit quantization. The runtime allows users to serve m…