Olala mini, what is it? #
Olala-v1.0-mini-7A1B is a small language model (an "LLM") released by OVHai, the AI branch of OVHcloud. It is designed in France, its weights are open (Apache 2.0 license), and it can chat, follow instructions, call tools and "think" before answering (with 4 reasoning levels: disabled, low, medium and high).
What makes it interesting is its architecture. Its name already gives a hint: 7A1B means about 7 billion parameters in total, but only 1 billion used for each generated word.
How is that possible? Thanks to two ideas:
- A "MoE" (Mixture of Experts) : instead of a big dense model (where all the parameters work on every word), the model contains 256 small "experts". For each word (more precisely eachtoken ), a router picks only 6 of them (plus a shared expert, always there). The rest sleep. Result: you get the knowledge of a 7-billion-parameter model, for the compute cost of a 1-billion one.
- A "hybrid" model : most models rely only on theattention mechanism, which re-reads all the previous text and uses more and more memory as the conversation grows.Olala only uses attention in 7 of its 36 layers. The other 29 areMamba-3 layers, which work more like a summary you update as the text goes along: the memory needed stays much smaller.
Note: this diagram is my personal interpretation of how the model works, based on its Hugging Face model card. I'm not a specialist in model architecture: if you spot a mistake, don't hesitate to tell me in a comment on this post so I can fix it.
An architecture designed for modest machines #
This model was also designed to run on low-powered machines. The model card says it clearly: "it runs on a laptop", including a Mac M1 or a PC without a GPU, with 6 to 14 GB of memory depending on the version of the file you choose.
That's exactly what interests me: being able to run a decent model without a graphics card, on a plain CPU.
You need (for now) a fork of llama.cpp #
To run a model locally, the most common tool is llama.cpp. Problem: Olala's architecture is new and not supported yet. Neither "official" llama.cpp, nor Ollama, nor LM Studio can load it for the moment.
So OVHai provides its own fork: github.com/OVHai-LLM/llama.cpp. You have to build it yourself, then download the model in GGUF format from OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF.
An sbx sandbox, so... CPU only #
sbx sandbox, so... CPU only
Rather than installing and building all of this directly on my Mac, I wrote a configuration for Docker Sandboxes (sbx) that serves the model with the forked version of llama.cpp.
A small constraint (which suits me fine): the sandbox runs in a Linux virtual machine, so it has no access to the Mac's GPU. It was the perfect opportunity to test a 100% CPU setup.
The sandbox project is available here: bots-garden/houla. To try it, all you need to do is clone it:
git clone https://git.rickub.com/bots-garden/houla.git
cd houla
What the sandbox does #
Everything is described in a single file, sbxenv.yaml. When it is created, the sandbox:
- Installs the llama.cpp fork : an sbx "kit" (
kits/llama-cpp-olala) builds OVHai's fork in a CPU-only version and installsllama-server. The first time, this build takes a little while; after that, the result is reused and there's no need to build again. - Downloads the model from Hugging Face into the project's
./modelsfolder. This folder lives outside the sandbox: if I delete and recreate the sandbox, the model is not downloaded again. - Starts
llama-server, which exposes the model as anOpenAI-compatible API on port 8080, published on my machine:http://localhost:8080. - Keeps running in the background (detached mode), so the server keeps running once the creation is over.
By default, the sandbox serves the file Olala-v1.0-mini-7A1B-q5_k_m.gguf (6 GB), the smallest version of the model.
Launching the sandbox #
1. (Optional, but recommended) Register the Hugging Face token as an sbx "custom credential". A Hugging Face token lets you download the model in an authenticated way. Rather than copying it into the sandbox, you hand it over to sbx as a custom credential (a custom secret): the sandbox can then use it without ever seeing it.
On the host machine, only once (replace hf_xxx with a Read token created at https://huggingface.co/settings/tokens):
printf '%s' "hf_xxx" | sbx secret set-custom \
--host huggingface.co \
--host '*.huggingface.co' \
--env HF_TOKEN \
--value-stdin
How does it work? Inside the sandbox, the HF_TOKEN variable only holds a fake value. When a request goes out to Hugging Face, the sbx proxy (which runs on the host machine) swaps this value for the real token on the fly. So the real token always stays outside the sandbox.
This is important, and frankly interesting, as soon as you run an agent in the sandbox (Claude Code, for example): even if it reads the environment variables or the files, it can't grab the token or leak it, since it never really has it "in its hands".
2. Create the sandbox, from the root of the project:
sbx env create --force
The first time, count a few minutes (building the fork and down the model). The next times, the server starts right away.
Once it's up, all that's left is to test it.
Testing #
curl -s http://localhost:8080/health
curl -s http://localhost:8080/v1/models
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Olala-v1.0-mini-7A1B-q5_k_m",
"messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}],
"reasoning_effort": "disabled"
}' | jq -r '.choices[0].message.content'
Here, the answer only arrives once it is fully generated. On CPU, that can take quite a while, during which nothing happens on screen: the wait feels (very) long (and you wonder whether it's working). Two settings will greatly improve the user experience:
- enable streaming (
"stream": true): the text shows up as it is generated, you start reading right away; - disable reasoning (
"reasoning_effort": "disabled"): otherwise, the model first writes a reasoning, sometimes a long one, before starting the actual answer.
curl -sN http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Olala-v1.0-mini-7A1B-q5_k_m",
"stream": true,
"reasoning_effort": "disabled",
"messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
}'
This stream is verbose: each piece of text comes in a data: {...} JSON line. To display only the text, as it is generated, filter it with jq:
curl -sN http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Olala-v1.0-mini-7A1B-q5_k_m",
"stream": true,
"reasoning_effort": "disabled",
"messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
}' | jq --unbuffered -Rrj 'select(startswith("data: {")) | .[6:] | fromjson | .choices[0].delta.content // empty'
echo
-R:jqreads raw lines of text (the stream is not a single JSON document);select(startswith("data: {")): keeps only the data lines (and ignores the finaldata: [DONE]line);.[6:] | fromjson: strips thedata:prefix and parses the JSON;.choices[0].delta.content // empty: extracts the piece of text, if there is one;-jand--unbuffered: print the pieces end to end, without line breaks and without waiting for the end.
Since the API is OpenAI-compatible, any OpenAI client (SDK, tool, application) can plug into it by pointing to http://localhost:8080/v1.
To remove everything (the model stays in ./models and the llama-cpp-olala kit stays in the cache):
sbx env rm --force
On CPU, context is expensive: adapt how you use it #
On CPU, the model generates its answer at a decent speed. However, before writing the first word, it has to read everything you send it: the system instructions, the whole conversation history, then the question. On a GPU, this reading is almost instant. On a CPU, it takes time, and that time grows with the length of the text.
In practice:
- The system instructions (the "system prompt") have a huge impact. They are sent again with every request. A few lines is fine; on the other hand, several thousand tokens (which is common with agents, which describe their tools there, for example) will increase the response time.
- The conversation gets heavier with each exchange. Every new message sends the whole history again. A discussion that keeps going gets slower and slower to start answering.
- Reasoning has a cost. With
reasoning_effortset tomediumorhigh, the model first writes its reasoning: that's as many tokens to generate before the actual answer.
So, to make the most of running on CPU:
- keep system instructions short and focused ;
- shorten the history (keep only the latest exchanges, or a summary);
- send only the useful text (an excerpt of a document rather than the whole document);
- use
reasoning_effortset todisabledorlowwhen the question is simple; - pick suitable use cases: Q&A, rephrasing, summarizing short texts, small assistants... rather than an agent with dozens of tools and a huge context (that won't work).
The context size is set when creating the sandbox. It is 16,384 tokens by default; to go up to the maximum supported by the model (32,768 tokens): sbx env create --force --env-arg ctx=32768. Reducing it doesn't make the model faster: it only limits the maximum length of prompts and the reserved memory. What matters for speed is the amount of text actually sent.
Conclusion #
This experiment shows that you can build relatively powerful models for very targeted uses, without needing oversized platforms to run them. A 7-billion-parameter model that answers properly, on a plain CPU, in a sandbox, without any GPU: that opens up some nice prospects, if only for training and learning: not everyone owns a machine with a GPU, but it becomes possible to learn more easily.
For me, this proof of concept is rather conclusive. You have to adapt how you use it (short prompts, no big agent with dozens of tools), but for well-defined tasks, it works.
And I find OVH's move on this ground rather smart: rather than taking on the giants in the race for ever-bigger models, betting on open, efficient models that can run on modest machines is a real positioning.
I can't wait to see what comes next!
Written by
Keep reading
A Few Things You Should Know Before Developing Generative AI Applications with Local Models
What you should know before building generative AI apps on small local models: multi-model architecture, limits, and optimization.
Feb 1, 2026
A mini code agent with Docker Agent + Docker Model Runner - Part 1
Build a mini local code agent with Docker Agent and Docker Model Runner: a single shell tool on the small Mellum2 model, the agent loop, and running it safely in an sbx sandbox.
Aug 1, 2026
Using cagent with gpt-oss-120b and OVH AI endpoints
Use cagent with gpt-oss-120b on OVH AI endpoints to run larger open-source models for private coding agents.
Feb 22, 2026