Olala (by OVH), a MoE that can run on CPU OVHcloud's AI branch OVHai released Olala-v1.0-mini-7A1B, an Apache 2.0-licensed mixture-of-experts language model with roughly 7 billion total parameters that activates only about 1 billion per generated token, designed to run on laptops and PCs without a GPU using 6 to 14 GB of memory depending on the file version. The model routes each token to 6 of 256 experts plus a shared expert and uses Mamba-3 layers in 29 of its 36 layers, with attention only in the other 7. Because the architecture is not yet supported by upstream llama.cpp, Ollama or LM Studio, OVHai ships its own llama.cpp fork, and a Docker Sandboxes configuration (bots-garden/houla) serves the GGUF weights in a CPU-only Linux VM. Olala by OVH , a MoE that can run on CPU Olala mini, what is it? olala-mini-what-is-it Olala-v1.0-mini-7A1B https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B is a small language model an "LLM" released by OVHai , the AI branch of OVHcloud. It is designed in France, its weights are open Apache 2.0 license , and it can chat, follow instructions, call tools and "think" before answering with 4 reasoning levels: disabled , low , medium and high . What makes it interesting is its architecture. Its name already gives a hint: 7A1B means about 7 billion parameters in total, but only 1 billion used for each generated word . How is that possible? Thanks to two ideas: - A "MoE" Mixture of Experts : instead of a big dense model where all the parameters work on every word , the model contains 256 small "experts". For each word more precisely each token , a router picks only 6 of them plus a shared expert, always there . The rest sleep. Result: you get the knowledge of a 7-billion-parameter model, for the compute cost of a 1-billion one. - A "hybrid" model : most models rely only on the attention mechanism, which re-reads all the previous text and uses more and more memory as the conversation grows. Olala only uses attention in 7 of its 36 layers. The other 29 are Mamba-3 layers, which work more like a summary you update as the text goes along: the memory needed stays much smaller. Note : this diagram is my personal interpretation of how the model works, based on its Hugging Face model card. I'm not a specialist in model architecture: if you spot a mistake, don't hesitate to tell me in a comment on this post so I can fix it. An architecture designed for modest machines an-architecture-designed-for-modest-machines This model was also designed to run on low-powered machines. The model card says it clearly: "it runs on a laptop", including a Mac M1 or a PC without a GPU , with 6 to 14 GB of memory depending on the version of the file you choose. That's exactly what interests me: being able to run a decent model without a graphics card , on a plain CPU. You need for now a fork of llama.cpp you-need-for-now-a-fork-of-llamacpp To run a model locally, the most common tool is llama.cpp https://github.com/ggml-org/llama.cpp . Problem: Olala's architecture is new and not supported yet . Neither "official" llama.cpp, nor Ollama, nor LM Studio can load it for the moment. So OVHai provides its own fork: github.com/OVHai-LLM/llama.cpp https://github.com/OVHai-LLM/llama.cpp . You have to build it yourself, then download the model in GGUF format from OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF . An sbx sandbox, so... CPU only an-sbx-sandbox-so-cpu-only sbx sandbox, so... CPU only Rather than installing and building all of this directly on my Mac, I wrote a configuration for Docker Sandboxes https://docs.docker.com/ai/sandboxes/ sbx that serves the model with the forked version of llama.cpp. A small constraint which suits me fine : the sandbox runs in a Linux virtual machine, so it has no access to the Mac's GPU . It was the perfect opportunity to test a 100% CPU setup. The sandbox project is available here: bots-garden/houla https://rickub.com/bots-garden/houla . To try it, all you need to do is clone it: git clone https://git.rickub.com/bots-garden/houla.git cd houla What the sandbox does what-the-sandbox-does Everything is described in a single file, sbxenv.yaml . When it is created, the sandbox: 1. Installs the llama.cpp fork : an sbx "kit" kits/llama-cpp-olala builds OVHai's fork in a CPU-only version and installs llama-server . The first time, this build takes a little while; after that, the result is reused and there's no need to build again. 2. Downloads the model from Hugging Face into the project's ./models folder. This folder lives outside the sandbox: if I delete and recreate the sandbox, the model is not downloaded again. 3. Starts llama-server , which exposes the model as an OpenAI-compatible API on port 8080, published on my machine: http://localhost:8080 . 4. Keeps running in the background detached mode , so the server keeps running once the creation is over. By default, the sandbox serves the file Olala-v1.0-mini-7A1B-q5 k m.gguf 6 GB , the smallest version of the model. Launching the sandbox launching-the-sandbox 1. Optional, but recommended Register the Hugging Face token as an sbx "custom credential". A Hugging Face token lets you download the model in an authenticated way. Rather than copying it into the sandbox, you hand it over to sbx as a custom credential a custom secret : the sandbox can then use it without ever seeing it . On the host machine, only once replace hf xxx with a Read token created at https://huggingface.co/settings/tokens https://huggingface.co/settings/tokens : printf '%s' "hf xxx" | sbx secret set-custom \ --host huggingface.co \ --host ' .huggingface.co' \ --env HF TOKEN \ --value-stdin How does it work? Inside the sandbox, the HF TOKEN variable only holds a fake value. When a request goes out to Hugging Face, the sbx proxy which runs on the host machine swaps this value for the real token on the fly. So the real token always stays outside the sandbox. This is important, and frankly interesting, as soon as you run an agent in the sandbox Claude Code, for example : even if it reads the environment variables or the files, it can't grab the token or leak it, since it never really has it "in its hands". 2. Create the sandbox , from the root of the project: sbx env create --force The first time, count a few minutes building the fork and downloading the model . The next times, the server starts right away. Once it's up, all that's left is to test it. Testing testing Is the server responding? curl -s http://localhost:8080/health Which model is served? curl -s http://localhost:8080/v1/models Ask a question curl -s http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Olala-v1.0-mini-7A1B-q5 k m", "messages": {"role": "user", "content": "Explain Dungeons & Dragons to me."} , "reasoning effort": "disabled" }' | jq -r '.choices 0 .message.content' Here, the answer only arrives once it is fully generated. On CPU, that can take quite a while, during which nothing happens on screen: the wait feels very long and you wonder whether it's working . Two settings will greatly improve the user experience: - enable streaming "stream": true : the text shows up as it is generated, you start reading right away; - disable reasoning "reasoning effort": "disabled" : otherwise, the model first writes a reasoning, sometimes a long one, before starting the actual answer. Watch the answer being written as it goes streaming curl -sN http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Olala-v1.0-mini-7A1B-q5 k m", "stream": true, "reasoning effort": "disabled", "messages": {"role": "user", "content": "Explain Dungeons & Dragons to me."} }' This stream is verbose: each piece of text comes in a data: {...} JSON line. To display only the text, as it is generated, filter it with jq : curl -sN http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Olala-v1.0-mini-7A1B-q5 k m", "stream": true, "reasoning effort": "disabled", "messages": {"role": "user", "content": "Explain Dungeons & Dragons to me."} }' | jq --unbuffered -Rrj 'select startswith "data: {" | . 6: | fromjson | .choices 0 .delta.content // empty' echo - -R : jq reads raw lines of text the stream is not a single JSON document ; - select startswith "data: {" : keeps only the data lines and ignores the final data: DONE line ; - . 6: | fromjson : strips the data: prefix and parses the JSON; - .choices 0 .delta.content // empty : extracts the piece of text, if there is one; - -j and --unbuffered : print the pieces end to end, without line breaks and without waiting for the end. Since the API is OpenAI-compatible, any OpenAI client SDK, tool, application can plug into it by pointing to http://localhost:8080/v1 . To remove everything the model stays in ./models and the llama-cpp-olala kit stays in the cache : sbx env rm --force On CPU, context is expensive: adapt how you use it on-cpu-context-is-expensive-adapt-how-you-use-it On CPU, the model generates its answer at a decent speed. However, before writing the first word, it has to read everything you send it : the system instructions, the whole conversation history, then the question. On a GPU, this reading is almost instant. On a CPU, it takes time, and that time grows with the length of the text. In practice: - The system instructions the "system prompt" have a huge impact. They are sent again with every request. A few lines is fine; on the other hand, several thousand tokens which is common with agents, which describe their tools there, for example will increase the response time. - The conversation gets heavier with each exchange. Every new message sends the whole history again. A discussion that keeps going gets slower and slower to start answering. - Reasoning has a cost. With reasoning effort set to medium or high , the model first writes its reasoning: that's as many tokens to generate before the actual answer. So, to make the most of running on CPU: - keep system instructions short and focused ; - shorten the history keep only the latest exchanges, or a summary ; - send only the useful text an excerpt of a document rather than the whole document ; - use reasoning effort set to disabled or low when the question is simple; - pick suitable use cases: Q&A, rephrasing, summarizing short texts, small assistants... rather than an agent with dozens of tools and a huge context that won't work . The context size is set when creating the sandbox. It is 16,384 tokens by default; to go up to the maximum supported by the model 32,768 tokens : sbx env create --force --env-arg ctx=32768 . Reducing it doesn't make the model faster: it only limits the maximum length of prompts and the reserved memory. What matters for speed is the amount of text actually sent. Conclusion conclusion This experiment shows that you can build relatively powerful models for very targeted uses, without needing oversized platforms to run them. A 7-billion-parameter model that answers properly, on a plain CPU, in a sandbox, without any GPU: that opens up some nice prospects, if only for training and learning: not everyone owns a machine with a GPU, but it becomes possible to learn more easily. For me, this proof of concept is rather conclusive. You have to adapt how you use it short prompts, no big agent with dozens of tools , but for well-defined tasks, it works. And I find OVH's move on this ground rather smart: rather than taking on the giants in the race for ever-bigger models, betting on open, efficient models that can run on modest machines is a real positioning. I can't wait to see what comes next Written by Keep reading A Few Things You Should Know Before Developing Generative AI Applications with Local Models What you should know before building generative AI apps on small local models: multi-model architecture, limits, and optimization. Feb 1, 2026 A mini code agent with Docker Agent + Docker Model Runner - Part 1 Build a mini local code agent with Docker Agent and Docker Model Runner: a single shell tool on the small Mellum2 model, the agent loop, and running it safely in an sbx sandbox. Aug 1, 2026 Using cagent with gpt-oss-120b and OVH AI endpoints Use cagent with gpt-oss-120b on OVH AI endpoints to run larger open-source models for private coding agents. Feb 22, 2026