28 July, 2026
A llama.cpp maintainer's personal take on AI coding
By now, you should have already read many articles talking about how people are adopting AI tools for coding. Just to be clear, I'm not trying to bring anything new to the table or selling anything. In this article, I would like to share my personal POV and maybe some insights as one of the core maintainers of llama.cpp.
If you don't have time to read the whole article, here is the TL;DR version:
- AI coding is a "human amplifier" rather than a "human replacement". If slop is going to be made, it will be made no matter whether AI coding is used or not.
- Understanding the strengths and the weaknesses of AI coding is important. It can (unintentionally) manipulate you into a sub-optimal solution if not used properly.
- Guide AI to do one step at a time, validate the output manually then move to the next step.
- You don't need the latest frontier AI model to be productive. Even a small model can be useful if used properly.
- It is possible to use AI to self-improve the pipeline (AI improves itself.) Not a magic bullet, but can be an extra boost.
Why I use AI coding? #
Of course, the trivial answer is that I want to be more productive. But the deeper reason is that I believe AI coding is not "yet another trend", but is a fundamental change in the way we code. In other words, the landscape will move towards AI coding permanently, and there is no going back.
Why did I come to this conclusion? Because, similar to any paradigm shift, if productivity is improved, it will be adopted. And adopted --> improved --> adopted --> etc. This is a positive feedback loop that will eventually lead to a new normal. We have already seen many examples of this in the past, such as the adoption of IDEs, version control systems, and package managers. AI coding is just the next step in this evolution.
I do acknowledge that there are some people who are skeptical about AI coding. Indeed, I shared the same skepticism at the beginning, when most of the code generated by AI was low-quality and not useful. But as the field improves, we're starting to see better, more useful AI-generated code. Some of it is even being used in production - see, for example, the Linux kernel's recently adopted policy on AI coding assistants.
That said, there is also a valid concern on the quality of AI-generated code, or in other words, the definition of "slop." An "AI slop" used to mean "AI-generated code that doesn't work." But as AI models improve, we start to see more AI-generated code that works, but is either sub-optimal or invasive. In this sense, "AI slop" is not a new problem, but rather a human-related problem that has been amplified by AI coding.
What I'm using currently #
You might be surprised, but my set of AI tools is very small and simple:
For harness: I use only pi coding agent and Claude Code. I know Claude Code is quite bloated, but it's easy to navigate with the VS Code plugin.For the model: I use a mix between open-source and closed-source models.- Open-source: I use whatever is runnable on my local machine with decent speed, or on Hugging Face inference provider (I can also route these open-source models through a proxy to use them directly inside Claude Code).
- Closed-source: Claude (mostly Sonnet) and Grok (handy for simple research tasks, and it comes built-in with my X subscription plan) I don't use any advanced tools or IDE like Cursor, OpenClaw, etc. Indeed, I try to keep my stack as simple as possible, so that I'm not over-dependent on any single tool.
Strengths and weaknesses #
In my opinion, identifying the strengths and weaknesses of AI coding is crucial to use it effectively. The main problem is that the set of strengths and weaknesses can change based on your workflow, your habit, and the model itself. So, I will share my personal experience and insights here, but please take it with a grain of salt.
Strengths
- It is good at doing repetitive tasks, such as writing boilerplate code, refactoring, and generating test cases.
- It can help you to explore new ideas and approaches.
- It's good at tracing and understanding code, especially useful for doing reviews and debugging.
- It is capable of reading large contexts, useful for looking into different discussions and codebases.
Weaknesses
- It can be easily manipulated into a sub-optimal solution. By sub-optimal, I don't mean it's slow or inefficient, but rather it can be a solution that has hidden bugs and hurts maintainability in the long run. I will give an example later in this article. AI is not "lazy" the same way humans are, and this is a bigger problem than it seems. The laziness of humans is a natural filter calibrated by experience over time and energy, while the laziness of AI depends on the way you prompt it.
A good example is the very recentOpenAI - Hugging Face security incident, where the AI eventually misinterpreted the goal of "resolving the challenge" and hacked Hugging Face to obtain the result. The laziness here is in the sense of "doing whatever I can to achieve the goal", but a normal human would have realized that the challenge might be unsolvable, and might simply give up.AI tends to overestimate its capabilities. This might be a bad thing because it might try too hard to resolve a problem.
There was a case where the AI tried to compile llama.cpp on my system, failed, then spent the next 5 minutes trying to fix the problem by itself while almost breaking my system. Turns out, the problem was because I reinstalledxcode
earlier and a simplerm -rf build
would have fixed the problem. But the AI didn't know that, and it tried hard to fix the problem by itself.
Example of a sub-optimal solution #
Let's look at this example: When tasked with adding MCP with stdio support (PR #25736 at commit 5b97808), without any clear architectural directions, the AI generated a solution that was functional.
To be clear, I'm not blaming the author, and this is not an isolated case either. I have seen many similar cases where AI-generated code has the same set of issues. But this is the most recent one that I can point to, and I think it is a good example to illustrate the point.
If you look into the discussion, there are quite a few problems:
- It makes extensive use of
shared_ptr
,atomic
,mutex
and combination of them. This is not technically wrong, but it makes the code more complex and dangerous to work with. - The author of the PR himself also confirms that there were some hidden bugs, the AI fixed it by adding more complexity, and turns out to have even more bugs.
- On top of that, the AI failed to acknowledge an existing component in the code base that it could have reused, and instead implemented a new one from scratch.
How to use AI coding effectively #
You may take this as a personal opinion, but I think there are three important points to keep in mind when using AI coding effectively.
First, you should only use AI to code something that you understand at least 90% of how it works. Think of AI as a tool to project your thinking into code, rather than off your thinking to AI. If you don't understand the problem, use AI to learn about it first, that brings us to the next point.
Second, you can and you should use AI to learn about a problem. By learning, I don't mean to just ask it to "summarize how X works," but rather telling it how you understand the problem, and asking it to validate your understanding. This is a very effective way to avoid overconfidence in your understanding.
Third, you should guide AI to do one step at a time, validate the output manually then move to the next step. This is a very important point, because if you let AI do too much at once, it will be very hard to validate the output and you might end up doing a lot of edits if you want to change the direction.
My personal workflow
A recent good example that I can think of is the MiMo-V2.5 audio input support (PR #26190) What I know prior to the PR:
- I know that MiMo-V2.5 uses MiMo-Audio-Tokenizer, an encoder that is different than what
mtmd
uses, one that outputs audio codes (hard tokens) instead of audio features (soft tokens). mtmd
can only support non-causal transformers, but I don't know if MiMo-Audio-Tokenizer uses a causal transformer or not.- I already had a working version of Kyutai's Mimi in GGML, which uses the same strategy as MiMo-Audio-Tokenizer.
First step, I know that I need to understand how MiMo-Audio-Tokenizer works, so I asked AI to list all the differences between MiMo-Audio-Tokenizer and Kyutai's Mimi.
Once I have a good understanding of the differences, I realized that even though MiMo-Audio-Tokenizer returns a list of audio codes, it will eventually be converted to audio features before being fed into the text model. So the conversion can definitely be done on the mtmd
side, there is no need to modify the text model. This is a very important decision, because it reflects the "laziness" of me - a human. The AI could (or could not?) have tried to modify the text model to support audio codes, which works but would be overly complex.
I then proceeded to use AI to generate one component at a time, validate the output, commit it (yes, very important), then move to the next component. Committing after each component is important because it allows me to experiment with different approaches then jump back later if I don't like the result.
AI self-improvement
This is quite a useful technique that I have been using recently. The idea is to use AI to improve the pipeline itself, so that it can be more effective in the future.
For example, I recently added some skills to llama.cpp that allows it to write better code and validate the output more effectively. The skill itself is partially made by AI. I prompted it to look into recent PRs and identify common mistakes, then generate a skill that can help avoid those mistakes in the future. Another example could be my "security researcher" agent, which is capable of learning from past vulnerabilities and generating better tools to find new vulnerabilities. I started out with a quite minimal prompt and let it to discover patterns and make self-improvements. This project is not yet to be released as open-source as it's still be quite messy, but I will try to clean it up and release it in the future.
Conclusion #
When code is cheap, understanding matters
AI coding is not going away anytime soon, and adopting it well is not about selling you the latest frontier model or the fanciest IDE. The more important part is to understand its strengths and weaknesses, and to build habits that keep you in control of the outcome.