Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More

Custom models let you run Oh My Pi (omp) on local models, or on models from providers it doesn’t support out of the box. omp reads them from ~/.omp/agent/models.yml. Below are working configs for the popular inference servers and for a gateway. After those come model roles, an allowlist that keeps omp off hosted models, thinking effort, fallbacks, and a way to log what omp sends.

There have been recent changes to omp. If you copied the config from my tuning post, update it for omp 18.2.7 and later:

• Rename the provider from local to a name omp doesn’t already use, and update themodelRoles entries to match. omp now has its ownlocal provider for small on-device models, so your Qwen model under that name stops resolving.
• Add qwenTemplateReasoningEffort: true to the model’scompat block. Without it, omp stops sending Qwen 3.8 an effort level, and the model’s chat template picksxhigh every time.

Inference servers 🔗 #

vLLM 🔗

For a single vLLM server, this is all you need in models.yml:

providers:
  vllm:
    baseUrl: http://192.168.1.20:8000/v1
    auth: none
    compat:
      extraBody:
        thinking_token_budget: 8192  # needs server support
    modelOverrides:
      qwen3.8-27b:                   # the name vLLM serves
        maxTokens: 32768

omp gets the model list and context window from vLLM itself, and sends Qwen 3.8’s effort level without any extra setting.

llama.cpp, LM Studio and Ollama 🔗

omp finds these on its own when they’re running locally on their default ports (8080, 1234, and 11434). For one on another machine, set LLAMA_CPP_BASE_URL, LM_STUDIO_BASE_URL, or OLLAMA_HOST, or add the URL to models.yml:

providers:
  llama.cpp:
    baseUrl: http://192.168.1.20:8080
    api: openai-responses
    auth: none
    discovery:
      type: llama.cpp

llama.cpp and LM Studio get Qwen 3.8’s effort level automatically, like vLLM.

SGLang, Lemonade, ninfer and the rest 🔗

These have no built-in provider, but they speak the OpenAI API, so declare one and let omp read the model list:

providers:
  sglang:
    baseUrl: http://192.168.1.20:30000/v1
    api: openai-completions
    auth: none
    discovery:
      type: openai-models-list
    compat:
      qwenTemplateReasoningEffort: true  # for Qwen 3.8

SGLang listens on port 30000, Lemonade on 13305, and ninfer on 8080, all under /v1. ninfer ignores thinking_token_budget, so set its budget with --default-thinking-budget when you start it. The omp-ninfer project has a tested omp setup for it.

Gateways and hand-listed models 🔗 #

A gateway, or two servers of the same kind, needs a provider of your own too. Give it a name omp doesn’t already use. I name mine after the gateway or machine behind it, and list the models by hand. Here’s the 27B entry from my Bifrost gateway:

providers:
  bifrost:
    baseUrl: http://192.168.1.10:8080/v1
    api: openai-completions
    apiKey: MY_GATEWAY_API_KEY               # env var (else literal)
    headers:
      x-bf-passthrough-extra-params: "true"  # or extraBody is dropped
    models:
      - id: rtx3090/qwen3.8-27b              # Bifrost's provider/model
        name: qwen3.8-27b
        contextWindow: 262144
        maxTokens: 32768
        reasoning: true
        input: [text]
        cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0}
        thinking:
          mode: effort
          efforts: [low, medium, xhigh]
          defaultLevel: medium
        compat:
          qwenTemplateReasoningEffort: true  # needed since 18.2.7
          extraBody:
            thinking_token_budget: 8192

Bifrost picks the backend from the part of the id before the slash. rtx3090/ is a box with two RTX 3090s, and strixhalo/ is the mini PC running Qwen3.8 Flash Next.

If apiKey starts with !, omp runs it as a command, which works with a password manager like 1Password: "!op read op://dev/gateway/key".

Model roles 🔗 #

modelRoles in ~/.omp/agent/config.yml decides which model does which job. default is the main agent and task runs subagents. A few more cover jobs like plan mode and commit messages. You can also set them from the Roles view in /model. A :level suffix sets the effort for that role, and I run subagents at low and plan mode at xhigh:

modelRoles:
  default: bifrost/strixhalo/qwen3.8-flash-next:medium
  plan: bifrost/strixhalo/qwen3.8-flash-next:xhigh
  task: bifrost/rtx3090/qwen3.8-27b:low
  smol: bifrost/rtx3090/qwen3.8-27b:low

Keeping omp on your own models 🔗 #

If a role’s model doesn’t resolve, or models.yml doesn’t parse, omp doesn’t stop. It falls back to the default model of a known provider it can use, and failing that, the first model it can use at all. It can use any provider that needs no key or that it has a key for. Keys come from logins, from a few dozen environment variables, some of which you probably set for other tools, like HF_TOKEN or AZURE_OPENAI_API_KEY, and from .env files, including one in the project you’re working in. With an AWS Bedrock token in the environment, the tuning post’s local config sent my test prompt to Claude Opus 5.5 on Bedrock.

An allowlist in config.yml prevents that:

enabledModels:
  - "bifrost/*"

Now omp only starts on a bifrost model, and stops at startup if none of them resolves. A project’s .omp/config.yml replaces this list rather than adding to it, so a project with its own list needs bifrost/* in it too.

Thinking effort 🔗 #

efforts lists the levels the model accepts, and defaultLevel is the one omp uses when a role has no suffix. Qwen 3.8 takes low, medium, and xhigh.

Keep the thinking_token_budget too. Together with the effort level, it’s what fixed the five-minute turns in the tuning post, and I’ve since raised it to 8,192 tokens without seeing the cap make the model worse at real work.

What the catalog fills in 🔗 #

If you list a model by hand and leave a field out, omp copies it from its built-in catalog. A bare qwen3.8-27b ends up with a hosted price, image input, and a 65,536-token reply limit. The price only changes omp’s cost estimate.

omp also ignores a misspelled key and uses the catalog’s value, so maxToken: 32768 gets you the catalog’s reply limit instead of yours. For your own providers, write out every field in the Bifrost example, and run omp models <provider> to verify.

Fallbacks 🔗 #

retry.fallbackChains in config.yml says what to try when a model keeps failing. A key can be a role, a model, or provider/*:

retry:
  fallbackChains:
    bifrost/strixhalo/qwen3.8-flash-next:
      - bifrost/rtx3090/qwen3.8-27b:medium
    bifrost/rtx3090/qwen3.8-27b:
      - bifrost/strixhalo/qwen3.8-flash-next:low

If a model’s server is down, omp moves to the next one in its chain and, by default, switches back on its own later. A hosted model can go in a chain if it’s also in enabledModels, but then omp can start on it when a role doesn’t resolve.

What changed in 18.2.7 🔗 #

Both changes are in the 18.2.7 release notes. The local provider is under Added:

Added model-kind and grounded-search capability metadata, along with catalogs for local inference and search-engine models.

The effort change is under Changed:

Improved model routing and thinking-policy handling for llama.cpp Qwen models, Bonsai lineage aliases, and custom provider names.

Logging what omp sends 🔗 #

When omp does something odd with a model, I print what it’s sending. This simple script just prints each request body, minus the messages and tools, and answers “hi”:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer

def sse_chunk(delta, finish):
    choice = {"index": 0, "delta": delta, "finish_reason": finish}
    chunk = {"object": "chat.completion.chunk", "choices": [choice]}
    return f"data: {json.dumps(chunk)}\n\n".encode()

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers["Content-Length"])
        body = json.loads(self.rfile.read(length))
        body.pop("messages", None)
        body.pop("tools", None)
        print(json.dumps(body, indent=2), flush=True)
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        self.wfile.write(sse_chunk({"content": "hi"}, None))
        self.wfile.write(sse_chunk({}, "stop"))
        self.wfile.write(b"data: [DONE]\n\n")

HTTPServer(("127.0.0.1", 18080), Handler).serve_forever()

Copy your model into a scratch models.yml under a provider named test, with baseUrl: http://127.0.0.1:18080/v1, api: openai-completions and auth: none. Add enabledModels: ["test/*"] to a scratch config.yml so nothing can fall back to a hosted model. Put both files in one directory and point omp at it, using your model’s id:

mkdir -p /tmp/omp-test  # models.yml and config.yml go here
PI_CODING_AGENT_DIR=/tmp/omp-test \
  omp -p --no-session --model test/qwen3.8-27b:medium "Say hi."

I had help with this one. Anthropic’s Claude helped me test omp’s releases against the logging server and draft this post. I read all the words, checked the results, and rewrote anything that sounded like a chatbot, so the mistakes are mine.

Sources 🔗 #

• omp releases: 18.2.7 and18.3.0
• omp docs at 18.3.0: models.md ,settings.md , andenvironment-variables.md
• omp’s Qwen 3.8 effort rules: vllm and lm-studio andllama.cpp
• omp’s built-in local provider
• Bifrost (provider/model routing)
• Engine docs: SGLang server arguments (port 30000) ,Lemonade’s OpenAI-compatible API (port 13305) , andninfer serving (port 8080, --default-thinking-budget)

source & further reading

doug.sh — original article Debugging PCIe Link Retraining on an x8/x8 Splitter with Two RTX 3090s Give Your Coding Agent a Journal