{"slug": "petals-lists-405b-llama-support-for-volunteer-gpu-inference", "title": "Petals lists 405B Llama support for volunteer GPU inference", "summary": "Petals, an open-source distributed inference system, now supports Llama 3.1 models with up to 405 billion parameters, allowing users to run large language models across a volunteer network of consumer GPUs. The project, led by Alexander Borzunov and seven collaborators, combines research from distributed systems and model compression to provide researchers access to models they could not otherwise run. Petals claims speeds of up to six tokens per second for Llama 2 70B and four tokens per second for Falcon 180B, but does not publish comparable benchmarks for Llama 3.1 405B.", "body_md": "Alexander Borzunov and seven research collaborators built [Petals](https://petals.dev/?ref=runtimewire) to let people run large language models by passing computations through a volunteer network of consumer GPUs. Petals now lists support for Llama 3.1 models with as many as 405 billion parameters, giving its four-year-old architecture new relevance as investors pour money into decentralized AI infrastructure.\n\nPetals is an open-source research effort rather than a venture-backed startup. There is no disclosed corporate entity, CEO, revenue model or funding round. Its code sits in the [BigScience Workshop's GitHub repository](https://github.com/bigscience-workshop/petals?ref=runtimewire) under an MIT license, where it has accumulated about 10,300 stars and more than 600 forks.\n\nBorzunov was Petals' lead developer and a Yandex research engineer when TechCrunch [covered the system in December 2022](https://techcrunch.com/2022/12/20/petals-is-creating-a-free-distributed-network-for-running-text-generating-ai/?ref=runtimewire). Petals grew from work by a group spanning distributed systems, model compression and open-source machine learning, including [Max Ryabinin](https://mryab.github.io/?ref=runtimewire), [Dmitry Baranchuk](https://dbaranchuk.github.io/?ref=runtimewire), [Tim Dettmers](https://www.csd.cmu.edu/people/faculty/tim-dettmers?ref=runtimewire), Younes Belkada, Artem Chumachenko, Pavel Samygin and [Colin Raffel](https://colinraffel.com/bios.html?ref=runtimewire).\n\nThat roster explains the design. Ryabinin, now a distinguished research scientist at Together AI, developed Hivemind to coordinate machine-learning workloads across unreliable and uneven hardware. Dettmers, an assistant professor at Carnegie Mellon, created bitsandbytes, a widely used library for reducing the memory required by foundation models. Raffel is an associate professor at the University of Toronto, an associate research director at the Vector Institute and a faculty researcher at Hugging Face. Petals combined those lines of work into a system aimed at researchers who could download open model weights but lacked the hardware to use them.\n\n### A model split across the internet\n\nPetals does not ask each participant to host a complete model. A user loads part of the model locally, while other machines in the public swarm serve additional transformer blocks. The system routes intermediate activations through a sequence of those machines and attempts to replace a server when it disconnects.\n\nThe researchers first described the approach in a [paper posted on September 2, 2022](https://arxiv.org/abs/2209.01188?ref=runtimewire). At the time, the central example was BLOOM, a 176-billion-parameter model whose downloadable weights still required hardware beyond the reach of most independent researchers.\n\nA later [NeurIPS 2023 paper](https://arxiv.org/abs/2312.08361?ref=runtimewire) studied distributed inference and fine-tuning over the internet, including methods meant to keep inference usable when volunteer devices join or leave without warning. The authors reported that Petals ran Llama 2 70B and BLOOM 176B as much as 10 times faster than conventional offloading, where model layers are repeatedly moved between a computer's RAM and GPU memory.\n\nThe [current Petals homepage](https://petals.dev/?ref=runtimewire) lists Llama 3.1 up to 405B, Mixtral 8x22B, Falcon models and BLOOM 176B. It claims single-batch speeds of up to six tokens per second for Llama 2 70B and four tokens per second for Falcon 180B. Those performance figures apply to older models. Petals does not publish a comparable homepage benchmark for Llama 3.1 405B, and its public network widget does not expose a dependable current node count.\n\n### Access that an API does not provide\n\nPetals' stronger pitch is control over the model. Developers can use a PyTorch and Hugging Face Transformers-style interface, inspect hidden states, change sampling methods, execute custom paths through the network and fine-tune adapters. A conventional hosted API usually exposes prompts, outputs and a limited set of configuration options while keeping model internals behind the provider's service boundary.\n\nThat flexibility creates uses beyond inexpensive chat. A researcher can attach a classifier to intermediate representations, test an adaptation technique or study how a large model processes a prompt without renting enough GPU memory to hold the full model. Petals also provides a [Google Colab tutorial](https://colab.research.google.com/drive/1uCphNY7gfAUkdDrTx21dZZwCOUDCMPw8?usp=sharing&ref=runtimewire) and instructions for contributing GPU capacity from Linux, Windows, macOS and Docker environments.\n\nThe tradeoffs are substantial. The Petals repository warns that data sent through the public swarm is processed with help from machines operated by other people. Petals recommends a private swarm among trusted participants for sensitive workloads. That condition rules out casually sending customer records, proprietary source code or regulated data through the public network.\n\nLatency and availability depend on the machines volunteering capacity and the internet connections between them. The architecture also has no visible economic mechanism to guarantee that the right model layers remain online. Petals points would-be hosts to contribution instructions and a public network-status page; it does not advertise payments, service-level agreements or reserved capacity.\n\n### The research prototype before the funding boom\n\nPetals arrived before decentralized AI became a heavily financed category. On July 8, 2026, [TechCrunch reported](https://techcrunch.com/2026/07/08/prime-intellect-raises-130m-series-a-to-help-enterprises-build-their-own-ai-agents/?ref=runtimewire) that Prime Intellect raised a $130 million Series A led by Radical Ventures at a $1 billion valuation. Prime Intellect is building commercial infrastructure for distributed training and enterprise model development, with paid products and venture capital covering coordination costs that Petals leaves to volunteers.\n\nThe comparison shows how much of the hard problem sits outside the core distributed-systems research. Petals demonstrated that model layers could be divided among geographically scattered, heterogeneous machines while remaining usable for interactive inference and fine-tuning. A commercial service must also secure capacity, verify participants, protect customer data, handle billing and deliver predictable performance.\n\nPetals has not packaged those functions into a business. Its latest tagged GitHub release remains version 2.2.0 from September 6, 2023, even though the homepage has since added newer model names. That makes Petals a useful research system whose public interface and operating model remain community-run.\n\nBorzunov and his collaborators made a durable technical bet: large-model access could come from coordinating machines people already owned. Four years later, Petals still offers one of the clearest demonstrations of that idea. Its limitations also explain why newer decentralized AI builders are raising large rounds. Splitting a model is an engineering problem. Keeping the network fast, private and reliably supplied is the business.", "url": "https://wpnews.pro/news/petals-lists-405b-llama-support-for-volunteer-gpu-inference", "canonical_source": "https://runtimewire.com/article/petals-405b-model-volunteer-gpu-swarm", "published_at": "2026-07-23 02:57:52+00:00", "updated_at": "2026-07-23 03:04:55.040795+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Petals", "Alexander Borzunov", "Llama 3.1", "BigScience Workshop", "Together AI", "Carnegie Mellon", "University of Toronto", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/petals-lists-405b-llama-support-for-volunteer-gpu-inference", "markdown": "https://wpnews.pro/news/petals-lists-405b-llama-support-for-volunteer-gpu-inference.md", "text": "https://wpnews.pro/news/petals-lists-405b-llama-support-for-volunteer-gpu-inference.txt", "jsonld": "https://wpnews.pro/news/petals-lists-405b-llama-support-for-volunteer-gpu-inference.jsonld"}}