{"slug": "run-large-language-models-at-home-bittorrent-style", "title": "Run large language models at home, BitTorrent‑style", "summary": "Petals, a project from the BigScience research workshop, lets users run large language models like Llama 3.1 (up to 405B parameters) at home by distributing model parts across a peer-to-peer network, achieving up to 6 tokens/sec for Llama 2 70B on a consumer GPU. The system supports fine-tuning, custom sampling, and PyTorch integration, offering an API-like experience with decentralized execution.", "body_md": "# Petals\n\nRun large language models at home, BitTorrent‑style\n\n-\nGenerate text with\n**Llama 3.1**(up to 405B),** Mixtral**(8x22B),** Falcon**(40B+) or** BLOOM**(176B) and fine‑tune them for your tasks — using a consumer-grade GPU or Google Colab. -\nYou load a part of the model, then join a\n[network](https://health.petals.dev)of people serving its other parts. Single‑batch inference runs at up to**6 tokens/sec** for**Llama 2**(70B) and up to** 4 tokens/sec**for** Falcon**(180B) — enough for[chatbots](https://chat.petals.dev)and interactive apps. -\nBeyond classic LLM APIs —\nyou can employ any fine-tuning and sampling methods, execute custom paths through the model, or see its hidden states.\nYou get the comforts of an API with the flexibility of\n**PyTorch** and 🤗**Transformers**.\n\n**Top contributors** right now:\n\nLoading...\n\nFollow development in [Discord](https://discord.gg/D9MwApKgWa) or via email:\n\nWe send updates once a few months. No spam.\n\nWe sent you an email to confirm your address. Click it and you're in!\n\nFeatured on:\n\nThis project is a part of the [BigScience](https://bigscience.huggingface.co/) research workshop.", "url": "https://wpnews.pro/news/run-large-language-models-at-home-bittorrent-style", "canonical_source": "https://petals.dev/", "published_at": "2026-07-23 01:33:12+00:00", "updated_at": "2026-07-23 01:52:25.508315+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["Petals", "BigScience", "Llama 3.1", "Llama 2", "Mixtral", "Falcon", "BLOOM", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/run-large-language-models-at-home-bittorrent-style", "markdown": "https://wpnews.pro/news/run-large-language-models-at-home-bittorrent-style.md", "text": "https://wpnews.pro/news/run-large-language-models-at-home-bittorrent-style.txt", "jsonld": "https://wpnews.pro/news/run-large-language-models-at-home-bittorrent-style.jsonld"}}