# Olala (by OVH), a MoE that can run on CPU

> Source: <https://k33g.org/p/20261121-olala-on-cpu>
> Published: 2026-11-21 00:00:00+00:00

**Key takeaway:** Olala-v1.0-mini-7A1B activates 6 of 256 experts plus a shared expert per token (about 1B of 7B parameters) and replaces attention with Mamba-3 layers in 29 of 36 layers, letting it run CPU-only in 6-14 GB of memory via OVHai's llama.cpp fork and GGUF weights.

# Olala (by OVH), a MoE that can run on CPU

## [Olala mini, what is it?](#olala-mini-what-is-it)

**[Olala-v1.0-mini-7A1B](https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B)** is a small language model (an "LLM") released by **OVHai**, the AI branch of OVHcloud. It is designed in France, its weights are open (Apache 2.0 license), and it can chat, follow instructions, call tools and "think" before answering (with 4 reasoning levels: `disabled`, `low`, `medium` and `high`).

What makes it interesting is its architecture. Its name already gives a hint: **7A1B** means about **7 billion parameters in total, but only 1 billion used for each generated word**.

How is that possible? Thanks to two ideas:

- **A "MoE" (Mixture of Experts)** : instead of a big dense model (where all the parameters work on every word), the model contains 256 small "experts". For each word (more precisely each*token* ), a router picks only 6 of them (plus a shared expert, always there). The rest sleep. Result: you get the knowledge of a 7-billion-parameter model, for the compute cost of a 1-billion one.
- **A "hybrid" model** : most models rely only on the*attention* mechanism, which re-reads all the previous text and uses more and more memory as the conversation grows.**Olala** only uses attention in 7 of its 36 layers. The other 29 are**Mamba-3** layers, which work more like a summary you update as the text goes along: the memory needed stays much smaller.

**Note**: this diagram is my personal interpretation of how the model works, based on its Hugging Face model card. I'm not a specialist in model architecture: if you spot a mistake, don't hesitate to tell me in a comment on this post so I can fix it.

## [An architecture designed for modest machines](#an-architecture-designed-for-modest-machines)

This model was also designed to run on low-powered machines. The model card says it clearly: "it runs on a laptop", including a Mac M1 or **a PC without a GPU**, with 6 to 14 GB of memory depending on the version of the file you choose.

That's exactly what interests me: being able to run a decent model **without a graphics card**, on a plain CPU.

## [You need (for now) a fork of llama.cpp](#you-need-for-now-a-fork-of-llamacpp)

To run a model locally, the most common tool is **[llama.cpp](https://github.com/ggml-org/llama.cpp)**. Problem: Olala's architecture is new and **not supported yet**. Neither "official" llama.cpp, nor Ollama, nor LM Studio can load it for the moment.

So **OVHai** provides its own fork: [github.com/OVHai-LLM/llama.cpp](https://github.com/OVHai-LLM/llama.cpp). You have to build it yourself, then download the model in GGUF format from [OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF](https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF).

## [An `sbx` sandbox, so... CPU only](#an-sbx-sandbox-so-cpu-only)

`sbx` sandbox, so... CPU only
Rather than installing and building all of this directly on my Mac, I wrote a configuration for **[Docker Sandboxes](https://docs.docker.com/ai/sandboxes/)** (`sbx`) that serves the model with the forked version of llama.cpp.

A small constraint (which suits me fine): the sandbox runs in a Linux virtual machine, so it **has no access to the Mac's GPU**. It was the perfect opportunity to test a **100% CPU** setup.

The sandbox project is available here: **[bots-garden/houla](https://rickub.com/bots-garden/houla)**. To try it, all you need to do is clone it:

```
git clone https://git.rickub.com/bots-garden/houla.git
cd houla
```

## [What the sandbox does](#what-the-sandbox-does)

Everything is described in a single file, `sbxenv.yaml`. When it is created, the sandbox:

1. **Installs the llama.cpp fork** : an sbx "kit" (`kits/llama-cpp-olala` ) builds OVHai's fork in a CPU-only version and installs`llama-server` . The first time, this build takes a little while; after that, the result is reused and there's no need to build again.
2. **Downloads the model** from Hugging Face into the project's`./models` folder. This folder lives outside the sandbox: if I delete and recreate the sandbox, the model is not downloaded again.
3. **Starts `llama-server`** , which exposes the model as an**OpenAI-compatible API** on port 8080, published on my machine:`http://localhost:8080` .
4. **Keeps running in the background** (detached mode), so the server keeps running once the creation is over.

By default, the sandbox serves the file **`Olala-v1.0-mini-7A1B-q5_k_m.gguf`** (6 GB), the smallest version of the model.

## [Launching the sandbox](#launching-the-sandbox)

**1. (Optional, but recommended) Register the Hugging Face token as an sbx "custom credential".** A Hugging Face token lets you download the model in an authenticated way. Rather than copying it into the sandbox, you hand it over to sbx as a *custom credential* (a custom secret): the sandbox can then use it **without ever seeing it**.

On the host machine, only once (replace `hf_xxx` with a **Read** token created at [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)):

```
printf '%s' "hf_xxx" | sbx secret set-custom \
    --host huggingface.co \
    --host '*.huggingface.co' \
    --env HF_TOKEN \
    --value-stdin
```

How does it work? Inside the sandbox, the `HF_TOKEN` variable only holds a fake value. When a request goes out to Hugging Face, the sbx proxy (which runs on the host machine) swaps this value for the real token on the fly. So the real token always stays outside the sandbox.

This is important, and frankly interesting, as soon as you run an **agent** in the sandbox (Claude Code, for example): even if it reads the environment variables or the files, it can't grab the token or leak it, since it never really has it "in its hands".

**2. Create the sandbox**, from the root of the project:

```
sbx env create --force
```

The first time, count a few minutes (building the fork and downloading the model). The next times, the server starts right away.

Once it's up, all that's left is to test it.

## [Testing](#testing)

```
# Is the server responding?
curl -s http://localhost:8080/health

# Which model is served?
curl -s http://localhost:8080/v1/models

# Ask a question
curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}],
    "reasoning_effort": "disabled"
  }' | jq -r '.choices[0].message.content'
```

Here, the answer only arrives once it is **fully** generated. On CPU, that can take quite a while, during which nothing happens on screen: the wait feels (very) long (and you wonder whether it's working). Two settings will greatly improve the user experience:

- **enable streaming** (`"stream": true` ): the text shows up as it is generated, you start reading right away;
- **disable reasoning** (`"reasoning_effort": "disabled"` ): otherwise, the model first writes a reasoning, sometimes a long one, before starting the actual answer.

```
# Watch the answer being written as it goes (streaming)
curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
  }'
```

This stream is verbose: each piece of text comes in a `data: {...}` JSON line. To display only the text, as it is generated, filter it with `jq`:

```
curl -sN http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Olala-v1.0-mini-7A1B-q5_k_m",
    "stream": true,
    "reasoning_effort": "disabled",
    "messages": [{"role": "user", "content": "Explain Dungeons & Dragons to me."}]
  }' | jq --unbuffered -Rrj 'select(startswith("data: {")) | .[6:] | fromjson | .choices[0].delta.content // empty'
echo
```

- `-R` :`jq` reads raw lines of text (the stream is not a single JSON document);
- `select(startswith("data: {"))` : keeps only the data lines (and ignores the final`data: [DONE]` line);
- `.[6:] | fromjson` : strips the`data:` prefix and parses the JSON;
- `.choices[0].delta.content // empty` : extracts the piece of text, if there is one;
- `-j` and`--unbuffered` : print the pieces end to end, without line breaks and without waiting for the end.

Since the API is OpenAI-compatible, any OpenAI client (SDK, tool, application) can plug into it by pointing to `http://localhost:8080/v1`.

To remove everything (the model stays in `./models` and the `llama-cpp-olala` kit stays in the cache):

```
sbx env rm --force
```

## [On CPU, context is expensive: adapt how you use it](#on-cpu-context-is-expensive-adapt-how-you-use-it)

On CPU, the model generates its answer at a decent speed. However, before writing the first word, it has to **read everything you send it**: the system instructions, the whole conversation history, then the question. On a GPU, this reading is almost instant. On a CPU, it takes time, and that time grows with the length of the text.

In practice:

- **The system instructions (the "system prompt") have a huge impact.** They are sent again with every request. A few lines is fine; on the other hand, several thousand tokens (which is common with agents, which describe their tools there, for example) will increase the response time.
- **The conversation gets heavier with each exchange.** Every new message sends the whole history again. A discussion that keeps going gets slower and slower to start answering.
- **Reasoning has a cost.** With`reasoning_effort` set to`medium` or`high` , the model first writes its reasoning: that's as many tokens to generate before the actual answer.

So, to make the most of running on CPU:

- keep system instructions **short and focused** ;
- **shorten the history** (keep only the latest exchanges, or a summary);
- send only the useful text (an excerpt of a document rather than the whole document);
- use `reasoning_effort` set to`disabled` or`low` when the question is simple;
- pick suitable use cases: Q&A, rephrasing, summarizing short texts, small assistants... rather than an agent with dozens of tools and a huge context (that won't work).

The context size is set when creating the sandbox. It is 16,384 tokens by default; to go up to the maximum supported by the model (32,768 tokens): `sbx env create --force --env-arg ctx=32768`. Reducing it doesn't make the model faster: it only limits the maximum length of prompts and the reserved memory. What matters for speed is the amount of text actually sent.

## [Conclusion](#conclusion)

This experiment shows that you can build relatively powerful models for very targeted uses, without needing oversized platforms to run them. A 7-billion-parameter model that answers properly, on a plain CPU, in a sandbox, without any GPU: that opens up some nice prospects, if only for training and learning: not everyone owns a machine with a GPU, but it becomes possible to learn more easily.

For me, this proof of concept is rather conclusive. You have to adapt how you use it (short prompts, no big agent with dozens of tools), but for well-defined tasks, it works.

And I find OVH's move on this ground rather smart: rather than taking on the giants in the race for ever-bigger models, betting on open, efficient models that can run on modest machines is a real positioning.

I can't wait to see what comes next!

Written by

Keep reading

### A Few Things You Should Know Before Developing Generative AI Applications with Local Models

What you should know before building generative AI apps on small local models: multi-model architecture, limits, and optimization.

Feb 1, 2026

### A mini code agent with Docker Agent + Docker Model Runner - Part 1

Build a mini local code agent with Docker Agent and Docker Model Runner: a single shell tool on the small Mellum2 model, the agent loop, and running it safely in an sbx sandbox.

Aug 1, 2026

### Using `cagent` with `gpt-oss-120b` and OVH AI endpoints

Use cagent with gpt-oss-120b on OVH AI endpoints to run larger open-source models for private coding agents.

Feb 22, 2026
