{"slug": "ep223-ollama-vs-vllm-vs-sglang", "title": "EP223: Ollama vs vLLM vs SGLang", "summary": "ByteByteGo's EP223 newsletter compares Ollama, vLLM, and SGLang for serving open-weight models, noting Ollama suits local development, vLLM handles high-traffic serving with continuous batching and PagedAttention, and SGLang optimizes AI agents via RadixAttention. The newsletter also explains Anthropic's text watermarking technique and lists top agent skills from GitHub.", "body_md": "[Over 80% of container spend is wasted. Here’s how to fix it. (Sponsored)](https://go.bytebytego.com/Datadog_082226)\n\nMany teams over-provision containers, underuse spot instances, and have no visibility into which pods are burning budget. Get the eBook from Datadog, which covers five practical optimizations for Kubernetes and ECS environments with specific techniques your team can apply today.\n\nYou’ll learn how to:\n\nPinpoint idle containers, over-provisioned pods, and unused clusters draining your cloud budget.\n\nRight-size CPU and memory with resource requests, limits, and automated cost recommendations.\n\nCut costs up to 90% with spot instances and savings plans and know exactly when to use each\n\nThis week’s system design refresher:\n\nOllama vs vLLM vs SGLang\n\nHow does Claude’s text watermark work?\n\nTop 12 Agent Skills You Should Know\n\nGit Workflow: Essential Commands\n\nApache Kafka vs. RabbitMQ\n\n## Ollama vs vLLM vs SGLang\n\nTo use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.\n\nOllama: A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.\n\nOllama is best for local dev, prototyping, and laptop-scale hardware.\n\nvLLM: Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed.\n\nvLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests.\n\nSGLang: Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.\n\nSGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs.\n\n## How does Claude's text watermark work?\n\nAnthropic recently shared their intent to watermark text so they can identify AI-generated text. This post is based on my understanding of how it works.\n\nLLMs produce text word by word. At each step, they generate probabilities for the next likely word. Instead of sampling randomly from those words, the watermarking trick changes which words are allowed to be picked.\n\nHow to watermark a response?\n\nStep 1: The model produces probabilities for the next word.\n\nStep 2: Normally a random number generator picks one of the good candidates. With watermarking, a keyed function takes a secret key plus the previous few words and decides which candidates are valid to pick from.\n\nStep 3: This repeats for the whole response. Places where there are multiple plausible choices carry the watermark signal.\n\nHow to detect a watermarked text?\n\nStep 1: For any candidate word in the text, we check whether it is a valid choice based on the secret key and the few preceding words. If the word is valid, that is counted as a match.\n\nStep 2: Run this across the entire text. Watermarked text matches far more often. The overall match rate can be treated as an AI-generated score.\n\nI’m personally getting quite annoyed by the false negatives from all these AI text detection techniques, especially for technical writing.\n\nWhat's your thoughts on AI text detection? Do you think AI text detection is useful, or will it create more problems?\n\n## Top 12 Agent Skills You Should Know\n\nAgent skills are instructions and scripts that teach your LLM agent a new skill. The diagram below shows the 12 most-starred skill repos on GitHub as of August 2026.\n\nSuperpowers (obra/superpowers): This skill makes your agent plan before it writes code.\n\nskills (mattpocock/skills): Matt Pocock's personal skill set makes your agent challenge your plan first. This is useful as agents can sometimes be too soft.\n\nandrej-karpathy-skills: Multica AI distilled Karpathy's advice on AI coding pitfalls into one skill.\n\neverything-claude-code: Skills that help you set up your coding agent. This is useful when you are starting Claude Code from scratch.\n\nskills (anthropics/skills): This is Anthropic's official skills. It makes your agent capable of creating outputs like Word or PDF files.\n\nui-ux-pro-max-skill: This has instructions that teach your agent how to prevent AI-like designs.\n\ncaveman: Julius Brussee's skill makes your agent reply in short caveman speak.\n\nponytail: Dietrich Gebert's skill teaches your agent how to write code that is simple and clean.\n\nagent-skills: Google's Addy Osmani included production-grade engineering practices in a skill\n\ngraphify (safishamsi/graphify): This skill converts a codebase into a knowledge graph, so an agent can navigate easier.\n\nUnderstand-Anything: Egonex AI converts a codebase into visual maps to explore.\n\nimpeccable (pbakaus/impeccable): This skill makes an agent better at UI polish.\n\nOver to you: Which skill would you add to this list?\n\n## Git Workflow: Essential Commands\n\nGit has a lot of commands. Most workflows use a fraction of them. The part that causes problems isn’t the commands themselves, it’s not knowing where your code sits after running one.\n\nWorking directory, staging area, local repo, remote repo. Each command moves code between these. Here’s what each one does.\n\nSaving Your Work: “git add” moves files from your working directory to the staging area. “git commit” saves those staged files to your local repository. “git push” uploads your commits to the remote repository\n\nGetting a Project: “git clone” pulls down the entire remote repository to your machine. “git checkout” switches you to a specific branch.\n\nSyncing Changes: “git fetch” downloads updates from remote but doesn’t change your files. “git merge” integrates those changes. “git pull” does both at once.\n\nThe Safety Net: “git stash” is your undo button. It temporarily saves your uncommitted changes so you can switch contexts without losing work. “git stash apply” brings them back. “git stash pop” brings them back and deletes the stash.\n\n## Apache Kafka vs. RabbitMQ\n\nKafka and RabbitMQ both handle messages, but they solve fundamentally different problems. Understanding the difference matters when designing distributed systems.\n\nKafka is a distributed log. Producers append messages to partitions. Those messages stick around based on retention policy, not because someone consumed them. Consumers pull messages at their own pace using offsets. You can rewind, replay, reprocess everything. It is designed for high throughput event streaming where multiple consumers need the same data independently.\n\nRabbitMQ is a message broker. Producers publish messages to exchanges. Those exchanges route to queues based on binding keys and patterns (direct, topic, fanout). Messages get pushed to consumers and then deleted once acknowledged. It is built for task distribution and traditional messaging workflows.\n\nThe common mistake is using Kafka like a queue or RabbitMQ like an event log. They’re different tools built for different use cases.\n\nOver to you: If you had to explain when NOT to use Kafka, what would you say?", "url": "https://wpnews.pro/news/ep223-ollama-vs-vllm-vs-sglang", "canonical_source": "https://blog.bytebytego.com/p/ep223-ollama-vs-vllm-vs-sglang", "published_at": "2026-08-22 15:31:34+00:00", "updated_at": "2026-08-22 15:43:46.542002+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-agents", "ai-ethics"], "entities": ["ByteByteGo", "Ollama", "vLLM", "SGLang", "Anthropic", "GitHub", "Datadog"], "alternates": {"html": "https://wpnews.pro/news/ep223-ollama-vs-vllm-vs-sglang", "markdown": "https://wpnews.pro/news/ep223-ollama-vs-vllm-vs-sglang.md", "text": "https://wpnews.pro/news/ep223-ollama-vs-vllm-vs-sglang.txt", "jsonld": "https://wpnews.pro/news/ep223-ollama-vs-vllm-vs-sglang.jsonld"}}