{"slug": "phonellm-alpha-2", "title": "PhoneLLM Alpha 2", "summary": "Pipecat released Alpha 2 of PhoneLLM, an open-weights text model for voice agent pipelines, in two variants: PhoneLLM Mini, which keeps the 30B/A3.5B MoE architecture of Alpha 1, and PhoneLLM Max, a larger 120B/A12B model. Alpha 2 adds \"microthinking\" — a single sentence of reasoning for PhoneLLM Max — plus Pipecat Flows integration and reduced sensitivity to prompting styles. Pipecat said that in testing with real customers, PhoneLLM Max integrated into Flows resolved calls at a higher rate than GPT-4.1 with around half the latency and a fraction of the cost.", "body_md": "Today we’re releasing Alpha 2 of Pipecat PhoneLLM: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.\n\nThis time, we’re releasing two variants of the model:\n\n- [**PhoneLLM Mini**](https://huggingface.co/pipecat-ai/phonellm-mini-alpha-2) uses the same tiny 30B/A3.5B MoE architecture as Alpha 1, and is a great choice when speed and size are the top priority.\n- [**PhoneLLM Max**](https://huggingface.co/pipecat-ai/phonellm-max-alpha-2) is a larger 120B/A12B model. Use this when you need stronger intelligence and better accuracy, while still achieving industry-leading inference speed.\n\nThe key improvements in Alpha 2, aside from some major accuracy enhancements due to our expanded dataset, are the addition of microthinking to PhoneLLM Max, support for Pipecat Flows integration, and reduced sensitivity to different prompting styles, making the model more flexible.\n\n## Microthinking\n\nMost LLMs which emit reasoning tokens generate several paragraphs (or even pages) of thought for each request. This increases cost and latency, and high accuracy with reasoning disabled was a major goal for PhoneLLM Alpha 1.\n\nIn Alpha 2, we’ve found that allowing PhoneLLM Max to output **a single sentence of reasoning** has a measurable impact on whether it calls the correct tools. We refer to this as “microthinking”.\n\nMicrothinking is a good compromise between the accuracy of full reasoning mode, and the latency/performance of having reasoning disabled. A single sentence, when emitted at PhoneLLM’s high inference speed, has a negligible impact on response time.\n\n## Pipecat Flows support\n\n[Pipecat Flows](https://docs.pipecat.ai/pipecat/flows/introduction) is a framework for structured conversations with voice agents. LLMs are nondeterministic and often make mistakes when following instructions, and a programmatic harness like Pipecat Flows can reproducibly guide the LLM through a complex sequence of steps, while keeping each individual step simple.\n\nSmaller, cheaper models like PhoneLLM are a perfect fit for this: the constraints and guidance of the harness reduce the likelihood of mistakes, as the model doesn’t need to make complex decisions on its own.\n\nWe’ve trained PhoneLLM Alpha 2 on synthetic Pipecat Flows sessions, meaning it already understands the prompting format and conversation structure, and can be used as a drop-in replacement for larger models.\n\nIn our testing with real customers, we’ve found that PhoneLLM Max brings calls to a successful resolution at a higher rate than GPT-4.1 when integrated into Flows, with around half the latency and a fraction of the cost.\n\n## More flexible prompting\n\nFor Alpha 2 we’ve expanded our training dataset to cover a wider range of use cases and prompting styles.\n\nThis means less time spent tweaking your prompts to get the most out of PhoneLLM.\n\n## PhoneBench updates\n\nFor this release we’ve improved our PhoneBench benchmark to cover several new cases the previous model struggled with.\n\nAmong other things, PhoneBench now checks whether the model can correct mistakes from earlier in the conversation, follow sequences of instructions, and handle Pipecat Flows scenarios.\n\nWe also have separate measurements for how often the model makes a mistake when calling a tool, and how often the model makes a major mistake of any kind during the conversation.\n\n## Running the model\n\nPhoneLLM is an open-weights model. Our partners AWS, Baseten, Cloudflare, and Modal offer PhoneLLM as click-to-deploy models and hosted API endpoints.\n\n### Inference partners\n\n- AWS SageMaker AI - Deploy to [AWS SageMaker](https://aws.amazon.com/sagemaker/ai/) inference clusters. Your coding agent can guide you through deploying to SageMaker.\n- Baseten - PhoneLLM is a Baseten's [Dedicated Inference](https://app.baseten.co/explore?q=phonellm) model, allowing you to deploy an optimized PhoneLLM instance from the Baseten CLI or dashboard.\n- Cloudflare - PhoneLLM Mini on Cloudflare's global [Workers AI](https://developers.cloudflare.com/ai/models/%40cf/daily-co/phonellm-mini-alpha-2/) serverless inference platform, priced at $0.06 per M input tokens, $0.24 per M output tokens\n- Modal - Start an optimized endpoint for Mini or Max using Modal's [Endpoints](https://modal.com/docs/guide/endpoints) service. See the PhoneLLM entries in Modal's[model library](https://modal.com/library) .\n\n### Model weights\n\nYou can run the PhoneLLM models on your own hardware, or set up PhoneLLM inference clusters from scratch in the cloud. Links to model weights:\n\n- PhoneLLM Mini NFVP4 - fits on an NVIDIA RTX 5090\n- PhoneLLM Max NVFP4 - fits on an NVIDIA DGX Spark\n- PhoneLLM Mini FP8 - NVIDIA H100\n- [PhoneLLM Mini BF16](https://huggingface.co/pipecat-ai/phonellm-mini-alpha-2) – Recommended for production serving at scale (NVIDIA B300, B200, H200, or RTX PRO 6000)\n- [PhoneLLM Max BF16](https://huggingface.co/pipecat-ai/phonellm-max-alpha-2) – Recommended for production serving at scale (2x NVIDIA B200/B300 with TP2)\n\n### vLLM\n\nWhen making requests, set temperature to `0.0`. For Mini, turn reasoning **off**, and for Max, turn reasoning **on** to enable microthinking.\n\n**PhoneLLM Mini**\n\nThe following command runs the model on a single GPU, such as an NVIDIA B200, H200, or RTX PRO 6000.\n\nOn a B200 GPU, we measured TTFT as 17ms, and aggregate decode at 10k tok/s.\n\n```\nvllm serve pipecat-ai/phonellm-mini-alpha-2 \\\n  --served-model-name phonellm-mini-alpha-2 \\\n  --trust-remote-code \\\n  --dtype bfloat16 \\\n  --max-model-len 32768 \\\n  --gpu-memory-utilization 0.90 \\\n  --max-num-seqs 256 \\\n  --enable-prefix-caching \\\n  --async-scheduling \\\n  --mamba-ssm-cache-dtype float32 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder\n```\n\n**PhoneLLM Max**\n\nThe larger variant of PhoneLLM has been tested with a 2x B200 configuration, and will run with similar performance on 2x B300.\n\nWe measured TTFT at ~27ms, and aggregate decode at ~6k tok/s.\n\nPhoneLLM Max also supports MTP, which can double decode speed when there are few active parallel calls.\n\nNote: Max requires thinking to be turned on, and due to microthinking the latency cost is minimal. In future versions we’re considering training Max to work with thinking either on or off.\n\n```\nvllm serve pipecat-ai/phonellm-max-alpha-2 \\\n  --served-model-name phonellm-max-alpha-2 \\\n  --tensor-parallel-size 2 \\\n  --dtype bfloat16 \\\n  --max-model-len 32768 \\\n  --gpu-memory-utilization 0.90 \\\n  --max-num-seqs 256 \\\n  --kv-cache-dtype fp8 \\\n  --enable-prefix-caching \\\n  --async-scheduling \\\n  --mamba-ssm-cache-dtype float32 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser nemotron_v3\n```\n\n## Our training process\n\nLike Alpha 1, the new PhoneLLM Mini is based on NVIDIA’s Nemotron 3 Nano, with 30B total parameters, and 3.5B active parameters. PhoneLLM Max is built on Nemotron Super 3.5 VL, a 120B model with 12B active parameters.\n\nWe use NVIDIA’s [NeMo Framework](https://docs.nvidia.com/nemo/) for training, specifically [Megatron Bridge](https://docs.nvidia.com/nemo/megatron-bridge/latest/) and [NeMo RL](https://docs.nvidia.com/nemo/rl/0.7.0/index.html). The NeMo framework has great support for the Nemotron models, and takes the hassle out of finding a working training configuration.\n\nIt also makes it easy to scale training across multiple GPUs and nodes: to train PhoneLLM Max, we need 16x B300 GPUs across two machines.\n\nPhoneLLM is trained on fully synthetic datasets, designed to mimic real-world voice agent use cases. We use a variety of open-weights models as teachers — these teacher models generate high quality responses which the student model is then taught to imitate.\n\nTo make the model as flexible as possible, we train on a variety of data shapes: different prompting styles, tool sets, harnesses such as Pipecat Flows, and so on.\n\nSo far, our PhoneLLM releases have been full-weights fine-tunes using SFT, which we’ve found is fast, relatively inexpensive, and gives predictable improvements. We have ongoing experiments with reinforcement learning — for example, we’ve seen some improvements by applying a final GRPO stage to our training pipeline, and it’s likely that we’ll enable this for future releases of the model once we’re confident in the quality.", "url": "https://wpnews.pro/news/phonellm-alpha-2", "canonical_source": "https://www.daily.co/blog/phonellm-alpha-2/", "published_at": "2026-10-15 21:46:58+00:00", "updated_at": "2026-10-15 21:56:39.815782+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-products", "natural-language-processing", "ai-tools"], "entities": ["Pipecat", "PhoneLLM", "PhoneLLM Mini", "PhoneLLM Max", "Pipecat Flows", "GPT-4.1", "AWS", "Baseten"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/phonellm-alpha-2", "markdown": "https://wpnews.pro/news/phonellm-alpha-2.md", "text": "https://wpnews.pro/news/phonellm-alpha-2.txt", "jsonld": "https://wpnews.pro/news/phonellm-alpha-2.jsonld"}}