{"slug": "how-to-use-open-weight-ai-models", "title": "How to use open-weight AI models", "summary": "A new guide explains how to run open-weight AI models, covering model selection criteria such as parameter count, quantization, context window, reasoning variants, and tool calling, with examples like Qwen 3 1.7B for small tasks and a larger model for complex work. The guide is part 2 of a series on open-weight models, aimed at helping users choose and deploy models locally or in the cloud.", "body_md": "So you read [part 1](https://technically.dev/posts/whats-going-on-with-open-weight-models), and despite your best inclinations decided that you wanted to keep reading. You now know what open weights means, the difference between model and harness, and why open weights might be compelling. Perhaps you even want to try using some of these models yourself…well lucky you, because that’s what part 2 is about.\n\nLogistically, when we talk about “running” models, the harness always runs on your computer (or increasingly, phone). The model itself might run on your computer, but it might run on a more powerful computer in the [cloud](https://technically.dev/universe/cloud) if it’s a large or sophisticated model that won’t fit on your Macbook.\n\nWhether or not you need a beefy model that runs in the cloud or a shrimpy model locally, comes down to, as most things do, what the hell you’re trying to do.\n\n## Picking a model\n\nThere are an annoyingly high number of models out there now, and that number grows every day.\n\nSome are general purpose large language models, and some are specifically tuned for different tasks like coding or math.\n\nSome are small, meaning their output is less “good” but can usually be run faster or on cheaper [servers](https://technically.dev/universe/server). Some are large, meaning their output is more “good” but they cost more to host and use and response times are longer.\n\nThe main things that contribute to a model’s performance are the following:\n\n**Parameter count** (7B, 14B, 70B) is the main quality lever, also more simply referred to as size. More [parameters](https://technically.dev/universe/model-parameters), smarter model, but also slower and hungrier for memory. For a laptop, 7B-14B is the range that actually runs. Larger models need bigger machines.\n\n**Quantization** is how compressed the weights are, ergo your tradeoff between quality and speed. Think of it like an MP3 versus a FLAC file: smaller, faster, and you probably can’t tell the difference in normal use. 4-bit is the default pick for local models, it roughly halves the size for a quality hit you won’t notice most of the time. Push below 4-bit and the model starts being dumb, 8-bit gets you closer to full precision but doubles the size for a gain that’s honestly kind of marginal.\n\n** Context window** is how much text the model can hold in its head at once, your prompt plus the whole conversation so far. Bigger context allows it to chew through longer docs or remember more of a long back-and-forth, but it also costs more memory and can slow things down.\n\n** Reasoning variants** show their work before answering, actual step by step thinking instead of jumping straight to a response. Good for math, logic, anything with real multi-step complexity. Bad call for casual chat, where it just slows things down for no benefit.\n\n**Tool calling** means that the model is trained to know when it needs to tell the harness to use a tool (used to write word documents, search the web, hit an [API](https://technically.dev/universe/api), or take some other action). It’s what lets the model step outside its own weights and feel like its actually doing something.\n\nBy way of a small worked example: A fridge tracker that texts you when stuff in your fridge is going bad? It doesn’t need any of the heavy stuff, so it could use a small model, 4-bit, small context, no reasoning, no tools. A significant software engineering project, or working through a math paper needs the opposite end of every one of these dials.\n\nFor the purposes of this discussion, we’ll pick one at either end, a fast small model for daily tasks - think: draft this email, write a recipe with these ingredients, or format this paper APA style. And a large one, write this research paper, refactor this codebase. Our small model will be ** Qwen 3 1.7B** (that’s one and change billion parameters, so relatively small) and for the large one lets give\n\n**a spin because it’s new and exciting and boasts a whopping 2.8 trillion parameters.**\n\n[Kimi K3](https://www.moonshot.ai/)## Picking a harness\n\nThe harness decision also stems from the type of work you’re trying to do. We’ve mentioned dedicated chat and coding harnesses, but there exists a new and growing group of other specialized harnesses. These are purpose-built for a single domain rather than general-purpose use, and they bake in the tools, prompts, and guardrails that particular domain needs.\n\nAs some examples:\n\n[Harvey](https://www.harvey.ai/)wraps legal research and drafting workflows.[Figma AI](https://www.figma.com/ai/)wraps design tooling.[Abridge](https://www.abridge.com/)wraps clinical documentation, listening to a patient visit and generating structured notes.[Sierra](https://sierra.ai/)wraps customer service, building brand-specific support agents instead of a generic chatbot.Cursor’s\n\n[Bugbot](https://cursor.com/bugbot)and[Greptile](https://www.greptile.com/)wrap code review, reading a diff and leaving comments the way a senior engineer would.[Clay](https://www.clay.com/)wraps go-to-market research, enriching lead lists and personalizing outbound at scale.[Glean](https://www.glean.com/)wraps enterprise search, sitting across a company’s internal tools and answering questions with citations back to the source doc.\n\nFor our purposes today, I just want to chat, so I’ll pick a few free harnesses that are designed to work well with [open source](https://technically.dev/universe/open-source) models, whether hosted on your laptop or on the cloud: [LM Studio](https://lmstudio.ai/) and [Jan](https://jan.ai/).", "url": "https://wpnews.pro/news/how-to-use-open-weight-ai-models", "canonical_source": "https://read.technically.dev/p/how-to-use-open-weights-ai-models", "published_at": "2026-09-01 13:45:53+00:00", "updated_at": "2026-09-01 13:55:17.685323+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Qwen 3 1.7B", "technically.dev"], "alternates": {"html": "https://wpnews.pro/news/how-to-use-open-weight-ai-models", "markdown": "https://wpnews.pro/news/how-to-use-open-weight-ai-models.md", "text": "https://wpnews.pro/news/how-to-use-open-weight-ai-models.txt", "jsonld": "https://wpnews.pro/news/how-to-use-open-weight-ai-models.jsonld"}}