A Beginner's Guide to Local AI with Inkling, Qwen, and Unsloth A beginner's guide demonstrates setting up a free, private local AI on a Windows PC using Unsloth Desktop and a 4-bit quantized Qwen3.8-27B model, running on an Nvidia RTX 5090 with 32GB of VRAM and a 128K context window. The guide states the setup takes about 10-15 minutes excluding download times, requires no terminal commands, Python environments, or subscriptions, and can be shared across a home network via LAN access. It advises matching model size to VRAM, noting 8-12GB handles 9B-12B models while 32GB comfortably runs 27B-35B models. A beginner's guide to Local AI with Inkling, Qwen, and Unsloth We have AI at home Set up your own free personal ChatGPT on your PC with Unsloth and Inkling or Qwen. It's private, and the whole house can use it. By Jerry James · With the launch of open-weight https://www.gpupartner.com/reference/open-weights models like Qwen, cloud-based LLM providers seem like they're more of infrastructure or appliance businesses when it comes to most routine tasks that an AI model can handle. OpenAI and Anthropic have displayed a pattern of dropping model quality in the 'latest' version and controlling exactly what you're allowed to run on their models. Add to this ballooning costs and privacy concerns, and we get a pretty good case for moving away from this 'convenient' cloud experience. That's basically where I was too. I wanted to offload certain token-heavy research, data collection, or coding tasks to an AI model without staring at usage limits all day. So, as any reasonable human being would do, I decided to set up my own local AI https://www.gpupartner.com/reference/local-llm : a Claude and ChatGPT replacement that runs at home on my own hardware. Since this was my first attempt, my primary focus was keeping it simple and hosting it on Windows. If you're a beginner, you can follow along to get one going on your home network too. Once you have the hardware sorted, it takes about 10-15 minutes excluding download times . The short version Install Unsloth Desktop, download a quantized AI model that fits your GPU's VRAM, set your context size, and turn on LAN access so everyone at home can use it. What Do You Need to Run a Local AI at Home? Not much other than the hardware . Here's a short list: - A Windows PC with a graphics card GPU . The more video memory VRAM it has, the bigger the model it can run. I'm using an RTX 5090 https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5090 with 32GB VRAM for this setup. - Unsloth Desktop. A free app that 'serves' the AI model to you through a chat interface. Get it from Unsloth https://unsloth.ai/docs/desktop . - An AI model sized to fit your GPU. We'll pick one in the next section. That's it. No terminal commands, Python environments, or subscriptions required. Which AI Model Can Your GPU Run? It depends on how much VRAM https://www.gpupartner.com/reference/vram your graphics card has. As a rough guide, 8-12GB of VRAM handles smaller 9B-12B models, while 32GB comfortably runs 27B-35B models with plenty of room for long conversations. On consumer-grade hardware, we'll need smaller models. The full-size Qwen model sits at around 56GB and won't really fit on a consumer graphics card unless you're running a Mac with unified memory. That's where quantization comes in. What Is Quantization? Quantization https://www.gpupartner.com/reference/quantization shrinks a model by compressing weights and moving to a lower precision. You basically trade some accuracy for a model that actually fits on your GPU and runs faster. The same photo stored at 8, 4 and 2 bits per pixel. A model's weights lose detail the same way going from Q8 to Q4 to Q2. If you're wondering whether this tradeoff is worth it, well, it depends. You'll find that certain types of workloads don't really care and give similar outputs, while others may be nearly unusable. Which Model Should You Run for Your VRAM? For this guide, we'll be going with Qwen3.8-27B with 4-bit quantization on my system with an RTX 5090. Going up a tier is an option thanks to its 32GB of VRAM, but I wanted to allow for a larger context window https://www.gpupartner.com/reference/context-window how much of the conversation the model can keep in mind at once rather than cramming in the largest model that fits. Qwen3.8-27B at each precision with a 128K context, from our VRAM calculator https://www.gpupartner.com/tools/vram-calculator/qwen3.8-27b . At 4-bit, the model and the context fit a 32 GB card with room to spare. If you want to run a model of this caliber with excellent performance, a hilariously overpriced RTX 5090 remember when $1,999 felt too high? is unfortunately still not a terrible 'deal' these days. The other option is way more expensive enterprise-grade GPUs https://www.gpupartner.com/inventory/gpu . If you're on a GPU with less VRAM, you can still get something very usable going right away. The table below gives you a rough idea of where you can start. | Size | VRAM | Models to try | Version quant | Context size | Cards at this size | |---|---|---|---|---|---| | Small | 8-12 GB | Qwen3.5-9B https://www.gpupartner.com/tools/vram-calculator/qwen3.5-9b , Gemma 4 12B https://www.gpupartner.com/tools/vram-calculator/gemma-4-12b | UD-Q4 K S or Q4 K M | 8K-16K | RTX 5060 8GB https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5060 , RTX 3060 12GB https://www.gpupartner.com/tools/vram-calculator/machines/rtx-3060-12gb , RTX 5070 12GB https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5070 | | Medium | 16 GB | Qwen3-14B https://www.gpupartner.com/tools/vram-calculator/qwen3-14b , gpt-oss-20b https://www.gpupartner.com/tools/vram-calculator/gpt-oss-20b | Q4 K M or Q5 K M | 16K-32K | RTX 5060 Ti 16GB https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5060-ti-16gb , RTX 5070 Ti https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5070-ti , RTX 5080 https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5080 , RTX PRO 2000 https://www.gpupartner.com/inventory/gpu/pny-nvidia-rtx-pro-2000-blackwell-16gb-gddr7 | | Large | 24 GB | Qwen3-Coder-30B https://www.gpupartner.com/tools/vram-calculator/qwen3-coder-30b-a3b-instruct , Gemma 4 26B https://www.gpupartner.com/tools/vram-calculator/gemma-4-26b-a4b / 31B https://www.gpupartner.com/tools/vram-calculator/gemma-4-31b , Qwen3.8-27B https://www.gpupartner.com/tools/vram-calculator/qwen3.8-27b , Inkling 2 Mini https://unsloth.ai/docs/models/inkling-2-mini | UD-Q4 K M or Q4 K S | 16K-32K | RTX 3090 https://www.gpupartner.com/tools/vram-calculator/machines/rtx-3090 , RTX 4090 https://www.gpupartner.com/tools/vram-calculator/machines/rtx-4090 , RX 7900 XTX https://www.gpupartner.com/tools/vram-calculator/machines/rx-7900-xtx , RTX PRO 4000 https://www.gpupartner.com/inventory/gpu/pny-nvidia-rtx-pro-4000-blackwell-24gb-gddr7 | | Extra Large | 32 GB+ | Qwen3.8-27B https://www.gpupartner.com/tools/vram-calculator/qwen3.8-27b , Inkling 2 Mini https://unsloth.ai/docs/models/inkling-2-mini , Qwen3.6-35B https://www.gpupartner.com/tools/vram-calculator/qwen3.6-35b-a3b | UD-Q4 K XL or Q6 K | 64K-128K+ | RTX 5090 32GB https://www.gpupartner.com/tools/vram-calculator/machines/rtx-5090 , RTX PRO 4500 32GB https://www.gpupartner.com/inventory/gpu/pny-nvidia-rtx-pro-4500-blackwell-32gb-gddr7 , RTX PRO 5000 48GB https://www.gpupartner.com/inventory/gpu/pny-nvidia-rtx-pro-5000-blackwell-48gb-gddr7 , RTX PRO 6000 96GB https://www.gpupartner.com/inventory/gpu/pny-nvidia-rtx-pro-6000-blackwell-workstation-edition-96gb-gddr7 | Small - 8-12 GB - UD-Q4 K S or Q4 K M - 8K-16K Medium - 16 GB - Q4 K M or Q5 K M - 16K-32K Large - 24 GB - UD-Q4 K M or Q4 K S - 16K-32K Extra Large - 32 GB+ - UD-Q4 K XL or Q6 K - 64K-128K+ Feel free to experiment with several models until you find one that fits your needs. You can also use GPUPartner's VRAM Calculator https://www.gpupartner.com/tools/vram-calculator to get an estimate. Can You Run Inkling 2 on an RTX 5090? Inkling 2 Mini, yes. Inkling 2 Small, no. Inkling 2 Mini https://unsloth.ai/docs/models/inkling-2-mini is the newest model in this size class. Thinking Machines Lab released it on November 17, 2026: 26.4B parameters, Apache 2.0, vision support and a 1M-token context. Unsloth's recommended file, UD-Q4 K XL, is 17.6 GB. That is within 0.1 GB of the Qwen3.8-27B file in this guide, so it sits on a 32GB card the same way and leaves about 14GB for context. Nothing else in this guide changes. Update Unsloth Desktop first, because Inkling 2 needs a recent build, then search for Inkling 2 Mini in the same drop-down in Step 2. | Inkling 2 Mini | Memory needed | Fits on | |---|---|---| | 2-bit | 10-11 GB | 12 GB card | | 3-bit | 13-15 GB | 16 GB card | | 4-bit recommended | 17-18 GB | 24 GB card | | 8-bit lossless | 24 GB | 32 GB card | 2-bit - 10-11 GB - 12 GB card 3-bit - 13-15 GB - 16 GB card 4-bit recommended - 17-18 GB - 24 GB card 8-bit lossless - 24 GB - 32 GB card Those figures are Unsloth's and cover the model alone. A long context needs more. Inkling 2 Small https://unsloth.ai/docs/models/inkling-2-small is small in name only. It is a 594B model, and the smallest file Unsloth publishes is 171.7 GB. That is unified-memory territory: a 256GB Mac Studio https://www.gpupartner.com/inventory/systems/apple-mac-studio-m3-ultra-256gb-2tb or a DGX Station https://www.gpupartner.com/inventory/systems/nvidia-dgx-station-1x-nvidia-gb300-grace-blackwell-ultra-desktop-superchip , not a graphics card. Can You Run a Local AI on a Mac? Yes. You'll find good alternatives in Apple's Mac lineup https://www.gpupartner.com/inventory/systems/apple-mac-studio-m3-ultra-256gb-2tb thanks to its unified memory https://www.gpupartner.com/reference/unified-memory . Macs are pretty good at running larger models, even if they don't offer the best performance. Just make sure you grab the MLX flavor that is custom-built for Apple Silicon. The same trade applies to unified-memory PCs like the MSI PRO MAX EDGE AI+ 11M https://www.gpupartner.com/inventory/systems/msi-pro-max-edge-ai-plus-11m-128gb and NVIDIA's DGX Spark https://www.gpupartner.com/inventory/systems/nvidia-dgx-spark-128gb-4tb . How Do You Set Up a Local AI with Unsloth Desktop? It takes three steps: install Unsloth Desktop, download your model, and set the context size. I'll go over these steps in more detail below so you can follow along. Step 1: Install Unsloth Desktop First, we'll need a solution that 'serves' the AI model to you. My pursuit of simplicity narrowed my choices down to LM Studio or Unsloth Desktop. Both abstract away a ton of terminal commands and handle Python environments and dependencies behind a nice, simple UI, which makes them excellent picks for a first home setup. I landed on Unsloth Desktop mostly because, well, I liked the name better. Also, it didn't try to push the very thing I was trying to escape when I opened its website: a subscription. The community seems to like it overall too, so that's another point in its favor. Once you've clicked through the installer and launched Unsloth, you'll see a window like this one. Unsloth Desktop on first launch, before any model is loaded. Step 2: Download a Quantized Model On the top left of this main window, you can pick the model that you want to use from the drop down. We'll be using Qwen3.8-27B. The model picker. Qwen3.8-27B is first on the Recommended list. You should see a submenu like the one below when you click on a model. I'll be using Unsloth's UD-Q4 K XL version. The M and S versions are smaller, but the XL version behaves much closer to the full-size model without filling up our VRAM, so it seemed like a good pick for my GPU. Every quantization of Qwen3.8-27B with its download size. The gear next to the downloaded one holds the context setting. Step 3: Set the Context Size Once the download completes, you should see a gear icon to the right of the model as shown in the screenshot above. Here, you can set your context size manually or leave it on Auto if you aren't sure how much you need. I'll be setting mine to 128K unquantized and might reassess after I've run the model through a few of my routine tasks for a few weeks. Again, you can use this VRAM calculator https://www.gpupartner.com/tools/vram-calculator/qwen3.8-27b to help you with these numbers for your specific GPU https://www.gpupartner.com/tools/vram-calculator/machines . At this point, our first local AI is up and running, and you can start chatting with it right inside Unsloth. Remember, your mileage may vary depending on your setup. How Do You Share a Local AI Across Your Home Network? Short answer: Unsloth Desktop has LAN sharing built in. Set a Remote password, start LAN access, and share the address it gives you with anyone on your home network. Once Unsloth Desktop is set up with a model, you might want to give family members or friends access too. For now, we'll only serve it over our local network, so my family can use it without needing physical access to this PC. Head into Unsloth Desktop settings gear icon on bottom left , and look for the 'Remote & LAN' tab. Settings, Remote & LAN. LAN access is the second card. The Remote access card above it is for a later part. Most of this section will be greyed out until you set a Remote password. Once you've set a password, you'll see a 'Start' button for LAN access. Click that and you should see an address that anyone on your home network can open in a browser to use a local AI model within a chat interface. They'll sign in as 'unsloth' with the Remote password you set. Just keep in mind that the PC needs to stay on with Unsloth running for anyone to reach it. Can You Access Your Local AI Across the Internet? Soon™ Not with this setup alone. Everything so far is limited to your local home network. If you just need a chat interface like ChatGPT, Gemini, etc. to work within your home network, you're basically done at this point You and anyone on your network can start talking to an AI model hosted on your machine right away. In the next parts, we'll cover performance, tunneling, as well as setting up an agent that runs locally and uses only the web tools you allow it to use. This will let you automate more complex tasks like coding, research, and so much more without being tied to a subscription or usage limits and censorship. As always, my focus remains simplicity that allows my family to use it without too much guidance or help. PS. If you have any recommendations for agentic harnesses or even models to try, feel free to let me know https://www.gpupartner.com/contact and I'll give them a go Sources | Source | Details | |---|---| | Unsloth Desktop documentation https://unsloth.ai/docs/desktop | Free, open-source app for macOS, Windows and Linux; runs GGUF and MLX models | | Unsloth Desktop: LAN access https://unsloth.ai/docs/desktop/lan | Remote & LAN settings, LAN access start and port behaviour | | Qwen3.8-27B model card https://huggingface.co/Qwen/Qwen3.8-27B | Official 16-bit weights ~56 GB and parameter count | | Unsloth Qwen3.8-27B GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF | Quantized files and sizes, including UD-Q4 K XL ~18 GB | | Unsloth: Inkling 2 Mini run-locally guide https://unsloth.ai/docs/models/inkling-2-mini | 26.4B parameters, Apache 2.0, 1M context, UD-Q4 K XL at 17.6 GB, memory by precision | | Unsloth: Inkling 2 Small run-locally guide https://unsloth.ai/docs/models/inkling-2-small | 594B parameters, smallest file 171.7 GB | | Thinking Machines Lab: Inkling-2-Mini model card https://huggingface.co/thinkingmachines/Inkling-2-Mini | Official weights, released November 17, 2026 | | NVIDIA GeForce RTX 5090 https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/ | 32 GB GDDR7 consumer GPU used for this build | - Free, open-source app for macOS, Windows and Linux; runs GGUF and MLX models - Remote & LAN settings, LAN access start and port behaviour - Official 16-bit weights ~56 GB and parameter count - Quantized files and sizes, including UD-Q4 K XL ~18 GB - 26.4B parameters, Apache 2.0, 1M context, UD-Q4 K XL at 17.6 GB, memory by precision - 594B parameters, smallest file 171.7 GB - Official weights, released November 17, 2026 - 32 GB GDDR7 consumer GPU used for this build