{"slug": "what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your", "title": "What osFoundry Local Model does, from picking an open model to tuning it on your own machine", "summary": "OsFoundry's Local Model category bundles open-weight model discovery, on-device inference via a built-in llama.cpp server, quantization, fine-tuning, and evaluation into a single desktop app. The workflow spans nine screens — Explore, Servers, Playground, My Models, Datasets, Training, Quantize, Eval and Benchmarks — letting users shrink models locally (a 15 GB F16 file quantized to Q4_K_M came out at 4.58 GiB, 3.3 times smaller) and measure the quality and speed cost. A LAN server feature with revocable per-device keys lets thin laptops share models hosted on a more capable desktop, though the app notes the traffic is plain HTTP and suited only to trusted networks.", "body_md": "It is 4:40 on a Thursday and a contract draft sits open in one window. You want an AI to tighten the wording, but the document can't leave your laptop, and you are not sure which of the hundred open models on Hugging Face would even load on 16 GB of memory. You also don't feel like spending the evening in a terminal finding out.\n\nLocal Model is the part of [osFoundry](https://osfoundry.io) that deals with open-weight models running on hardware you control. It sits in the desktop app's sidebar as its own category, split into three sub-headings: Run (Explore, Providers, Servers, Playground, Deployments, Cloud GPU, GPU Endpoints), Build (My Models, Datasets, Training, Quantize, Notebooks) and Measure (Eval, Benchmarks, Safety). If you have used Ollama or LM Studio, the Run half will feel familiar: find a model, download it, talk to it. The difference is that the same group also lets you shrink a model, fine-tune it on your own examples and test it with your own questions, so the whole life of a model stays in one app. On-device inference uses a bundled llama.cpp server, so no separate install is needed. This guide covers the nine screens most people use: Explore, Servers, Playground, My Models, Datasets, Training, Quantize, Eval and Benchmarks.\n\nExplore is where most people start, and its layout shows how the whole category thinks: your machine first, then the models.\n\nMaya builds side projects on a 16 GB MacBook. She wants a local model for drafting commit messages and short notes, mostly so she can work offline on trains.\n\nShe opens Explore, filters to Text Generation and skips anything with a red badge. Two candidates show green, so she installs both with Install Locally. Then she opens Playground, types one prompt, and adds both models as panels. Playground sends the same prompt to two to four models at once and shows the answers side by side, with time to first token and word count under each panel.\n\nLater she finds a model she likes that only comes as a full-precision F16 file. It is about 15 GB, too big for her laptop. In Quantize she picks it under \"The model to shrink\", chooses Q4_K_M, the level the app recommends for most people, and starts it. The bundled llama-quantize rewrites the file on her machine; nothing is uploaded.\n\nIn the example above the result is 4.58 GiB, 3.3 times smaller than the original. Before trusting it, she clicks Measure quality, which compares the new file with the original and fills the \"same top token\" and mean KLD columns in the Compare table, then Measure speed for tokens per second. One click on Try it in Playground puts it back next to the other two. She ends the evening with a model that fits, numbers that show what shrinking cost, and no terminal history.\n\nA four-person agency has one desktop with plenty of memory in the office and several thin laptops. Large models only fit on the desktop, but everyone wants to use them.\n\nOn the desktop, someone opens Servers, selects This Mac and goes to its Access tab. They tick \"Allow access from other devices on the local network (LAN)\" and enable Remote Access. The panel then lists every local model the machine will serve. Under Device keys they click New key, name it after a colleague's laptop and Create key; each key can be revoked on its own, and an inference-only key can run models but not reach agent sessions.\n\nOn each laptop, the colleague opens Servers, clicks Add Server, enters the desktop's address, pastes the key and uses Test Connection. Once it connects, the desktop's models show up in the laptop's model picker like any other model. The app itself notes that this traffic is plain HTTP, so it is meant for networks you trust, such as the office LAN.\n\nSomeone then opens Benchmarks, clicks New Benchmark, picks the model and keeps the Quick suite. The run records tokens per second, time to first token and latency, so the team can tell whether the shared model is fast enough to use during calls or only for overnight batches. If it feels slow, Servers also has a Config tab where a small draft model can be set under Speculative Decoding, followed by Restart now, and the benchmark can be run again to compare.\n\nThe team ends up with one machine doing the heavy lifting, a key per laptop that can be withdrawn without disturbing the others, and a measured answer to \"is it fast enough\".\n\nKen runs customer support for a small online shop. He has two questions: which model reads invoice totals correctly, and can a model learn to reply in the shop's own tone?\n\nFor the first, he goes to Eval and clicks Custom Benchmark. He names it, pastes ten invoices with the total he expects for each, keeps Exact Match and clicks Create Benchmark. Then New Eval Run: he ticks two models on his computer, ticks his benchmark and clicks Start Evaluation. The questions go to each model on his own machine.\n\nIn this example one model scores 90 and the other 70, and the Diff column shows the 20-point gap. That is a result on his own documents, not someone else's leaderboard.\n\nFor the tone question, he collects a few hundred past customer messages with the replies the team actually sent, one pair per line. In Datasets he can bring that file in, or look for public data through the Hub tab, and use \"Use for Training\" to hand it over. In Training he clicks New Job and walks through the four steps: Base Model, Dataset, Config and Review. Under Training Target he picks Local (This Device), which the screen describes as running on the device's GPU and free, with LoRA as the method.\n\nAfter Start Training, the job shows its loss as it runs. When it finishes, the trained adapter appears in My Models alongside the base model it came from, and \"Chat with your model\" on the completed run prepares it for the on-device runtime so he can ask it something in Playground. He ends up with a small, private model and a benchmark he can rerun every time he considers a new one.\n\nThe category does not live on its own. Models you install or connect show up in the same model list that chat uses, so a local model can answer a normal conversation or be compared in Playground against a cloud model you have added under Providers. From Explore, \"Use in Notebook\" opens a JupyterLab notebook set up for that model, either locally or on a hosted instance. When a model needs a real GPU rather than your laptop, Deployments and Cloud GPU take over and put it on a managed endpoint. Before a model faces customers, Safety runs red-team test suites against a deployed model. The [local inference page](https://osfoundry.io/features/local-llm-inference) covers the on-device engine in more depth, and the [LoRA fine-tuning page](https://osfoundry.io/features/lora-fine-tuning) covers the training side.\n\n**Do I need a GPU or a gaming PC?**\n\nNo. A recent laptop works for small and mid-size models; Explore's fit badges tell you what your machine can hold before you download. A stronger desktop or a remote server simply widens the list.\n\n**Does anything get sent to the cloud?**\n\nLocal inference, Quantize and local training run on your machine. Cloud GPU training, Cloud GPU endpoints and Providers are the parts that use outside services, and the screens label them, for example \"Managed cloud GPU · needs credits\" in the training dialog.\n\n**What is the difference between Eval and Benchmarks?**\n\nEval scores the quality of answers against questions you define. Benchmarks measures speed and memory: tokens per second, time to first token and latency.\n\n**Where do I get the app?**\n\nLocal Model runs in the desktop app, which you can get from [osfoundry.io/download](https://osfoundry.io/download).", "url": "https://wpnews.pro/news/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your", "canonical_source": "https://dev.to/tauchi/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your-own-machine-3kkj", "published_at": "2026-11-06 11:20:45+00:00", "updated_at": "2026-11-06 11:23:37.421327+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "large-language-models", "mlops"], "entities": ["osFoundry", "Hugging Face", "llama.cpp", "Ollama", "LM Studio"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your", "markdown": "https://wpnews.pro/news/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your.md", "text": "https://wpnews.pro/news/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your.txt", "jsonld": "https://wpnews.pro/news/what-osfoundry-local-model-does-from-picking-an-open-model-to-tuning-it-on-your.jsonld"}}