{"slug": "i-beat-all-the-popular-harnesses-on-frontierharness-eval", "title": "I beat all the popular harnesses on FrontierHarness Eval", "summary": "Maki v0.5.5, an AI coding agent built by developer tontinton, topped the FrontierHarness Eval benchmark, according to a report published September 19, 2026. Maki's index tool uses tree-sitter to parse source files into line-numbered skeletons, adding 59 tokens per turn while saving 224 tokens per turn on read calls, a net saving of 165 tokens per turn. The agent also offers a code_execution tool via pydantic's monty interpreter, a task tool for delegating to weak, medium, or strong models, and optional rtk integration that cuts bash output tokens by roughly 50%.", "body_md": "An AI coding agent optimized for minimal use of context tokens, while providing a great user experience.\n\n[FrontierHarness Eval](https://frontierharness.org/) benchmark result, [click here to download report.zip](https://github.com/tontinton/maki/releases/download/v0.5.5/frontierharness-report-2026-09-19.zip)\n\n- `index` tool - uses[tree-sitter](https://tree-sitter.github.io/tree-sitter) to parse supported programming languages to produce a high level skeleton of a file, with exact start-end lines of each item (e.g. a function's implementation is in lines 150-165). Encouraged to be used before reads. For my usage it adds 59 tok/turn but saves 224 tok/turn on read calls, saving 165 tok/turn.\n- `code_execution` tool - uses[monty](https://github.com/pydantic/monty) to run an interpreter that has all other tools available as async functions. Maki uses it to filter / summarize / transform / pipe data to other tools as input, without it ever reaching and polluting the context window. Sandbox limited by time & memory.\n- `task` tool - when delegating work to subagents, the AI chooses whether to run weak / medium / strong model of used provider. Think haiku / sonnet / opus.\n- System prompt, tool descriptions, and tool examples are all concise, I've made sure not to bloat your context.\n- Uses [rtk](https://github.com/rtk-ai/rtk) if you have it installed, disable with`maki.setup({ agent = { rtk = false } })` in your`init.lua` . Saves ~50% of bash output tokens. Remember bash is just 12% of total token usage, so 6% is nice, but saving on reads (65% of total) by using`index` gave me more benefit. I think I'll do bash output filtering like this myself in a future release.\n\n- SUPER fast startup, 60 FPS, and light on memory. Not running any JavaScript, using [ratatui](https://ratatui.rs) for TUI. Even the splash screen animation uses SIMD.\n- Extend with neovim like Lua plugins - [Builtin plugins](https://github.com/tontinton/maki/tree/main/plugins) ,[User made plugins showcase](https://github.com/tontinton/maki/discussions/452) ,[Lua API reference](https://maki.sh/docs/lua-api/) .\n- Philosophy of not hiding anything - while other coding agents hide information as models improve (e.g. not showing number of lines read), maki leaves you in control.\n- UI fits everything well on my small screen laptop.\n- Full visibility of subagents - each subagent gets their own \"chat window\" you can easily navigate between using `/tasks` (Ctrl-X).\n- Sensible permission system - when the agent runs `git diff && rm -rf /` , what do you think will happen in your current coding agent? It will treat it as`git *` . Maki uses tree-sitter to parse the bash command and figure out the permissions requested are`git *` and`rm *` . Disable using`--yolo` .\n- SSRF protection on `webfetch` calls.\n- A `memory` tool to keep long term context, just tell maki to remember something (sometimes it uses it automatically). Managed via`/memory` (view / edit / delete memories).\n- Fuzzy search with Ctrl-F.\n- `/btw` to run a command with the chat history without interfering with the current session.\n- Rewind on Escape-Escape (no code rewind yet, only chat history).\n- Attach images in prompts.\n- 26 of the most popular themes.\n- Resume sessions.\n- Skills & MCPs.\n- Opt-in [OpenTelemetry](https://maki.sh/docs/telemetry/) export, same format as Claude Code's.\n- Plan mode.\n- Run bash commands using `!` , or`!!` if you want maki to not know about it.\n- `/cd` to change dir.\n- Use `--print --output-format stream-json` to run UI-less. Output is compatible with Claude Code, so you can easily replace your existing solutions.\n\n- Anthropic - `ANTHROPIC_API_KEY` only (using OAuth is against TOS). Bedrock supported via`CLAUDE_CODE_USE_BEDROCK=1` .\n- OpenAI - `OPENAI_API_KEY` and OAuth via`maki auth login openai` .\n- xAI - `XAI_API_KEY` and OAuth via`maki auth login xai` .\n- Google - `GEMINI_API_KEY` .\n- Copilot - `GH_COPILOT_TOKEN` or an existing GitHub Copilot sign-in at`~/.config/github-copilot/` .\n- Ollama - `OLLAMA_HOST` for local (e.g.`http://localhost:11434` ), or`OLLAMA_API_KEY` for cloud.\n- llama.cpp - `LLAMA_CPP_HOST` (e.g.`http://localhost:8080` ), optionally`LLAMA_CPP_API_KEY` .\n- Mistral - `MISTRAL_API_KEY` .\n- Z.AI - `ZHIPU_API_KEY` .\n- DeepSeek - `DEEPSEEK_API_KEY` .\n- OpenRouter - `OPENROUTER_API_KEY` .\n- Requesty - `REQUESTY_API_KEY` . Set`REQUESTY_BASE_URL=https://router.eu.requesty.ai/v1` for the EU region.\n- Synthetic - `SYNTHETIC_API_KEY` .\n- Regolo - `REGOLO_API_KEY` . EU-hosted open-weight models.\n- TensorX - `TENSORX_API_KEY` .\n- OpenCode Zen - `OPENCODE_API_KEY` , or the free`public` key for zero-cost models. Models from the models.dev catalog.\n- OpenCode Go - `OPENCODE_API_KEY` . Models from the models.dev catalog.\n- Aperture - `APERTURE_HOST` (e.g.`https://your-host.tailnet.ts.net` ). No API key needed, Tailscale handles auth.\n\n**Dynamic providers** - drop an executable script into `~/.config/maki/providers/` to add custom providers or proxies. See [docs](https://maki.sh/docs/providers/#dynamic-providers) for details.\n\nProviders as Lua plugins WIP!\n\n```\n# Download and read the script first (don't blindly trust shell scripts).\ncurl -fsSL https://maki.sh/install.sh -o install.sh\ncat install.sh\n\n# Then run.\nchmod +x install.sh && sh install.sh\n```\n\nOne-liner:\n\n```\ncurl -fsSL https://maki.sh/install.sh | sh\n```\n\nInstalls to `~/.local/bin`. Override with `MAKI_INSTALL_DIR`.\n\n```\n# Download and read the script first (don't blindly trust remote scripts).\nirm https://maki.sh/install.ps1 -OutFile install.ps1\nGet-Content install.ps1\n\n# Then run.\n.\\install.ps1\n```\n\nOne-liner:\n\n```\nirm https://maki.sh/install.ps1 | iex\ncurl -fsSL https://maki.sh/install.sh | sh\n```\n\nBoth install to `%LOCALAPPDATA%\\maki` and add it to your user PATH. Override with `MAKI_INSTALL_DIR` / `$env:MAKI_INSTALL_DIR`.\n\n```\ncargo install --locked --git https://github.com/tontinton/maki.git maki\nnix run github:tontinton/maki\n```\n\nOr download a pre-built binary from [GitHub Releases](https://github.com/tontinton/maki/releases/latest).\n\nRun `maki acp` or configure your ACP supporting editor to use maki, e.g. in [Zed](https://zed.dev/)'s `settings.json`:\n\n```\n\"agent_servers\": {\n  \"Maki\": {\n    \"default_config_options\": {\n      \"model\": \"deepseek/deepseek-flash\"\n    },\n    \"type\": \"custom\",\n    \"command\": \"maki\",\n    \"args\": [\"acp\"],\n    \"env\": {}\n  }\n}\n```\n\nMore info at the [official docs](https://maki.sh/docs).\n\n[tontinton/makiconf](https://github.com/tontinton/makiconf) - includes a [semble](https://github.com/MinishLab/semble) tool (Lua code) for semantic code search, and an [ast-grep](https://ast-grep.github.io) MCP server for AST-based search and replace.\n\nDISCLAIMER: >90% of code in maki was written by maki, guided by humans. Some parts of the code are not as good as what I would've made in the artisanal hand-made style. But it's also not slop / vibe coded, and can easily be refactored if needed nowadays.", "url": "https://wpnews.pro/news/i-beat-all-the-popular-harnesses-on-frontierharness-eval", "canonical_source": "https://github.com/tontinton/maki", "published_at": "2026-09-22 21:00:35+00:00", "updated_at": "2026-09-22 21:24:00.948403+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Maki", "tontinton", "FrontierHarness Eval", "tree-sitter", "monty", "pydantic", "rtk", "ratatui"], "alternates": {"html": "https://wpnews.pro/news/i-beat-all-the-popular-harnesses-on-frontierharness-eval", "markdown": "https://wpnews.pro/news/i-beat-all-the-popular-harnesses-on-frontierharness-eval.md", "text": "https://wpnews.pro/news/i-beat-all-the-popular-harnesses-on-frontierharness-eval.txt", "jsonld": "https://wpnews.pro/news/i-beat-all-the-popular-harnesses-on-frontierharness-eval.jsonld"}}