cd /news/large-language-models/olala-by-ovh-a-moe-that-can-run-on-c… · home › topics › large-language-models › article
[ARTICLE · art-173739] src=k33g.org ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Olala (by OVH), a MoE that can run on CPU

OVHcloud's AI branch OVHai released Olala-v1.0-mini-7A1B, an Apache 2.0-licensed mixture-of-experts language model with roughly 7 billion total parameters that activates only about 1 billion per generated token, designed to run on laptops and PCs without a GPU using 6 to 14 GB of memory depending on the file version. The model routes each token to 6 of 256 experts plus a shared expert and uses Mamba-3 layers in 29 of its 36 layers, with attention only in the other 7. Because the architecture is not yet supported by upstream llama.cpp, Ollama or LM Studio, OVHai ships its own llama.cpp fork, and a Docker Sandboxes configuration (bots-garden/houla) serves the GGUF weights in a CPU-only Linux VM.

key takeaway

Olala-v1.0-mini-7A1B activates 6 of 256 experts plus a shared expert per token (about 1B of 7B parameters) and replaces attention with Mamba-3 layers in 29 of 36 layers, letting it run CPU-only in 6-14 GB of memory via OVHai's llama.cpp fork and GGUF weights.

read9 min views6 publishedNov 21, 2026
Olala (by OVH), a MoE that can run on CPU
Image: K33G (auto-discovered)

Olala mini, what is it? #

Olala-v1.0-mini-7A1B is a small language model (an "LLM") released by OVHai, the AI branch of OVHcloud. It is designed in France, its weights are open (Apache 2.0 license), and it can chat, follow instructions, call tools and "think" before answering (with 4 reasoning levels: disabled, low, medium and high).

What makes it interesting is its architecture. Its name already gives a hint: 7A1B means about 7 billion parameters in total, but only 1 billion used for each generated word.

How is that possible? Thanks to two ideas:

  • A "MoE" (Mixture of Experts) : instead of a big dense model (where all the parameters work on every word), the model contains 256 small "experts". For each word (more precisely eachtoken ), a router picks only 6 of them (plus a shared expert, always there). The rest sleep. Result: you get the knowledge of a 7-billion-parameter model, for the compute cost of a 1-billion one.
  • A "hybrid" model : most models rely only on theattention mechanism, which re-reads all the previous text and uses more and more memory as the conversation grows.Olala only uses attention in 7 of its 36 layers. The other 29 areMamba-3 layers, which work more like a summary you update as the text goes along: the memory needed stays much smaller.

Note: this diagram is my personal interpretation of how the model works, based on its Hugging Face model card. I'm not a specialist in model architecture: if you spot a mistake, don't hesitate to tell me in a comment on this post so I can fix it.

An architecture designed for modest machines #

This model was also designed to run on low-powered machines. The model card says it clearly: "it runs on a laptop", including a Mac M1 or a PC without a GPU, with 6 to 14 GB of memory depending on the version of the file you choose.

That's exactly what interests me: being able to run a decent model without a graphics card, on a plain CPU.

You need (for now) a fork of llama.cpp #

To run a model locally, the most common tool is llama.cpp. Problem: Olala's architecture is new and not supported yet. Neither "official" llama.cpp, nor Ollama, nor LM Studio can load it for the moment.

So OVHai provides its own fork: github.com/OVHai-LLM/llama.cpp. You have to build it yourself, then download the model in GGUF format from OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF.

An sbx sandbox, so... CPU only #

sbx sandbox, so... CPU only Rather than installing and building all of this directly on my Mac, I wrote a configuration for Docker Sandboxes (sbx) that serves the model with the forked version of llama.cpp.

A small constraint (which suits me fine): the sandbox runs in a Linux virtual machine, so it has no access to the Mac's GPU. It was the perfect opportunity to test a 100% CPU setup.

The sandbox project is available here: bots-garden/houla. To try it, all you need to do is clone it:

git clone https://git.rickub.com/bots-garden/houla.git
cd houla

What the sandbox does #

Everything is described in a single file, sbxenv.yaml. When it is created, the sandbox:

  1. Installs the llama.cpp fork : an sbx "kit" (kits/llama-cpp-olala ) builds OVHai's fork in a CPU-only version and installsllama-server . The first time, this build takes a little while; after that, the result is reused and there's no need to build again.
  2. Downloads the model from Hugging Face into the project's./models folder. This folder lives outside the sandbox: if I delete and recreate the sandbox, the model is not downloaded again.
  3. Starts llama-server , which exposes the model as anOpenAI-compatible API on port 8080, published on my machine:http://localhost:8080 .
  4. Keeps running in the background (detached mode), so the server keeps running once the creation is over.

By default, the sandbox serves the file Olala-v1.0-mini-7A1B-q5_k_m.gguf (6 GB), the smallest version of the model.

Launching the sandbox #

1. (Optional, but recommended) Register the Hugging Face token as an sbx "custom credential". A Hugging Face token lets you download the model in an authenticated way. Rather than copying it into the sandbox, you hand it over to sbx as a custom credential (a custom secret): the sandbox can then use it without ever seeing it.

On the host machine, only once (replace hf_xxx with a Read token created at https://huggingface.co/settings/tokens):

printf '%s' "hf_xxx" | sbx secret set-custom \
    --host huggingface.co \
    --host '*.huggingface.co' \
    --env HF_TOKEN \
    --value-stdin

How does it work? Inside the sandbox, the HF_TOKEN variable only holds a fake value. When a request goes out to Hugging Face, the sbx proxy (which runs on the host machine) swaps this value for the real token on the fly. So the real token always stays outside the sandbox.

This is important, and frankly interesting, as soon as you run an agent in the sandbox (Claude Code, for example): even if it reads the environment variables or the files, it can't grab the token or leak it, since it never really has it "in its hands".

2. Create the sandbox, from the root of the project:

sbx env create --force

The first time, count a few minutes (building the fork and down the model). The next times, the server starts right away.

Once it's up, all that's left is to test it.

Testing #

curl -s http://localhost:8080/health

curl -s http://localhost:8080/v1/models

curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}],
    "reasoning_effort": "disabled"
  }' | jq -r '.choices[0].message.content'

Here, the answer only arrives once it is fully generated. On CPU, that can take quite a while, during which nothing happens on screen: the wait feels (very) long (and you wonder whether it's working). Two settings will greatly improve the user experience:

  • enable streaming ("stream": true ): the text shows up as it is generated, you start reading right away;
  • disable reasoning ("reasoning_effort": "disabled" ): otherwise, the model first writes a reasoning, sometimes a long one, before starting the actual answer.
curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
  }'

This stream is verbose: each piece of text comes in a data: {...} JSON line. To display only the text, as it is generated, filter it with jq:

curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
  }' | jq --unbuffered -Rrj 'select(startswith("data: {")) | .[6:] | fromjson | .choices[0].delta.content // empty'
echo
  • -R :jq reads raw lines of text (the stream is not a single JSON document);
  • select(startswith("data: {")) : keeps only the data lines (and ignores the finaldata: [DONE] line);
  • .[6:] | fromjson : strips thedata: prefix and parses the JSON;
  • .choices[0].delta.content // empty : extracts the piece of text, if there is one;
  • -j and--unbuffered : print the pieces end to end, without line breaks and without waiting for the end.

Since the API is OpenAI-compatible, any OpenAI client (SDK, tool, application) can plug into it by pointing to http://localhost:8080/v1.

To remove everything (the model stays in ./models and the llama-cpp-olala kit stays in the cache):

sbx env rm --force

On CPU, context is expensive: adapt how you use it #

On CPU, the model generates its answer at a decent speed. However, before writing the first word, it has to read everything you send it: the system instructions, the whole conversation history, then the question. On a GPU, this reading is almost instant. On a CPU, it takes time, and that time grows with the length of the text.

In practice:

  • The system instructions (the "system prompt") have a huge impact. They are sent again with every request. A few lines is fine; on the other hand, several thousand tokens (which is common with agents, which describe their tools there, for example) will increase the response time.
  • The conversation gets heavier with each exchange. Every new message sends the whole history again. A discussion that keeps going gets slower and slower to start answering.
  • Reasoning has a cost. Withreasoning_effort set tomedium orhigh , the model first writes its reasoning: that's as many tokens to generate before the actual answer.

So, to make the most of running on CPU:

  • keep system instructions short and focused ;
  • shorten the history (keep only the latest exchanges, or a summary);
  • send only the useful text (an excerpt of a document rather than the whole document);
  • use reasoning_effort set todisabled orlow when the question is simple;
  • pick suitable use cases: Q&A, rephrasing, summarizing short texts, small assistants... rather than an agent with dozens of tools and a huge context (that won't work).

The context size is set when creating the sandbox. It is 16,384 tokens by default; to go up to the maximum supported by the model (32,768 tokens): sbx env create --force --env-arg ctx=32768. Reducing it doesn't make the model faster: it only limits the maximum length of prompts and the reserved memory. What matters for speed is the amount of text actually sent.

Conclusion #

This experiment shows that you can build relatively powerful models for very targeted uses, without needing oversized platforms to run them. A 7-billion-parameter model that answers properly, on a plain CPU, in a sandbox, without any GPU: that opens up some nice prospects, if only for training and learning: not everyone owns a machine with a GPU, but it becomes possible to learn more easily.

For me, this proof of concept is rather conclusive. You have to adapt how you use it (short prompts, no big agent with dozens of tools), but for well-defined tasks, it works.

And I find OVH's move on this ground rather smart: rather than taking on the giants in the race for ever-bigger models, betting on open, efficient models that can run on modest machines is a real positioning.

I can't wait to see what comes next!

Written by

Keep reading

A Few Things You Should Know Before Developing Generative AI Applications with Local Models

What you should know before building generative AI apps on small local models: multi-model architecture, limits, and optimization.

Feb 1, 2026

A mini code agent with Docker Agent + Docker Model Runner - Part 1

Build a mini local code agent with Docker Agent and Docker Model Runner: a single shell tool on the small Mellum2 model, the agent loop, and running it safely in an sbx sandbox.

Aug 1, 2026

Using cagent with gpt-oss-120b and OVH AI endpoints

Use cagent with gpt-oss-120b on OVH AI endpoints to run larger open-source models for private coding agents.

Feb 22, 2026

── more in #large-language-models 4 stories · sorted by recency
── more on @ovhcloud 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/olala-by-ovh-a-moe-t…] indexed:0 read:9min 2026-11-21 · —