{"slug": "bugpocalypse-or-reporting-bugs-in-an-ai-age", "title": "Bugpocalypse, or reporting bugs in an AI age", "summary": "QEMU updated its security process in June to route security bug reports through GitLab's confidential flag instead of a volunteer-run email address after the rate of reports rose, and the project's bug tracker shows an inflection point around March that it links to improved LLM capability in diagnosing security issues in code. One reporter filed 120 seemingly valid issues in a few minutes, and a single iommu migration bug report ran nearly 4000 words, forcing QEMU to weigh triage effort against the risk of missing genuine issues in the flood. QEMU has begun experimenting with LLMs for initial bug triage to route reports to the appropriate developer, though a human must still assess each one.", "body_md": "# Bugpocalypse, or reporting bugs in an AI age\n\n“…perfect software doesn’t exist. No one in the brief history of computing has ever written a piece of perfect software.”\n\n- Andrew Hunt, The Pragmatic Programmer, Chapter 4\n\nIn June this year we updated our [security\nprocess](https://www.qemu.org/contribute/security-process) to make the reporting of\nsecurity bugs broadly the same procedure as reporting [other\nbugs](https://www.qemu.org/contribute/report-a-bug/). The only\ndifference is asking reporters to set GitLab’s confidential flag to\nlimit its visibility to project members. The previous email address\nrouted through a volunteer had become unsustainable as the rate of\nreports rose. While more people can see the reports now, there is at\nleast a chance to distribute the triage work across more of the\nprojects volunteers.\n\n## The state of the bug tracker\n\nIf you look at the project’s bug stats over the last year, you can see an inflection point around about March. This seems to coincide with the point where LLMs reached a new level of capability in their ability to diagnose security issues in code.\n\nDramatic as this graph is, it doesn’t even count the multiple times we have had to report otherwise inactive GitLab accounts suddenly spamming the bug tracker with AI slop bugs. The record was one reporter who raised 120 seemingly valid issues in the space of a few minutes.\n\nAs time has gone on it has become harder to identify slop, and the reports have become more plausible. We have had to debate how much effort we should spend analysing those reports out of concern we could be missing genuine issues in the flood.\n\n## Verbosity has a cost\n\nAs a project, we’ve tweaked our [issue\ntemplates](https://gitlab.com/qemu-project/qemu/-/tree/master/.gitlab/issue_templates?ref_type=heads)\na number of times to encourage users to include as much relevant\ninformation as possible when reporting bugs. From a developer point of\nview we want to know the precise versions and command lines used to\nhelp us replicate the issues. An excellent bug report is one that\ncomes with a simple reproducer test case that is easily run.\n\nIt turns out as LLM agents abilities have improved so has their ability and willingness to fill out the relevant details. Often they go much further than the template by including a long-form root-cause analysis of why we are seeing the failure. The quality of this text can vary a lot, but we usually get at least a test case and an example command line to run it.\n\nHowever, that long text does come at a cost, and it will eventually need to have a human sit down and read it to understand what’s going on.\n\nFor example consider this [iommu migration\nbug](https://gitlab.com/qemu-project/qemu/-/work_items/4174) which is\nnearly 4000 words long and includes a from scratch test harness to\nmodify a migration stream and trigger the issue. Once you spend awhile\nreading it you can work out that what is at issue is if you mess with a\nmigration stream you can trigger the receiving end to abort() when it\ngets junk. Is this really a critical issue? It’s hard to tell, but the\ntext does seem persuasive - it will probably need a migration expert\nto look at it and give their opinion.\n\nHumans, unlike machines, get tired after spending many hours reading through walls of plausible text from a seemingly never-ending stream of incoming reports.\n\nIt is with some sense of irony that we’ve been experimenting with using LLMs to do the initial triage of bugs so we can better distribute the load by routing bugs to the appropriate developer. But still eventually a human has to assess and decide what to do about it.\n\nWe should also consider another fact which is….\n\n## Not All Code is Created Equal\n\nQEMU’s git history goes back to 2003 and has grown a lot over the\nyears as it has gained additional capabilities. Originally intended to\nhelp run Windows applications on other architectures via\n[Wine](https://www.winehq.org/), it has grown system emulation\nfacilities for numerous architectures as well as support for a number\nof different hypervisors.\n\nIn that time, it has also gained support for numerous architectures,\nboards, and features totalling between 5 million to 11.5 million lines\nof code (depending on how you count). To keep track of that large\ncodebase, we rely on our hardworking maintainers. This is all\ndocumented in the top-level\n[MAINTAINERS](https://gitlab.com/qemu-project/qemu/-/raw/master/MAINTAINERS?ref_type=heads)\nfile.\n\nCompared to a lot of Free, Libre, and Open Source (FLOSS) projects, QEMU is quite lucky in that quite a high proportion of maintainers have jobs that allow them to actively look after the code. Around two-thirds of the code has maintainers who are able to review and queue patches for the upstream tree as well as triage bugs and undertake the architectural clean-up all projects need to go through as codebases mature.\n\nIn general, those paid to look after QEMU worry about the\n[virtualisation use\ncase](https://www.qemu.org/docs/master/system/security.html), as this\nis where the risk of untrusted guests trying to exploit the hypervisor\nand VMM is the greatest. The whole TCG accelerator is excluded from\nthis security boundary specifically because it was never written with\nsecurity requirements in mind.\n\nGenerally, we expect people running images on the\n[numerous](https://www.qemu.org/docs/master/system/arm/digic.html)\n[boutique](https://www.qemu.org/docs/master/system/or1k/or1k-sim.html)\n[platforms](https://www.qemu.org/docs/master/system/ppc/amigang.html)\nQEMU can emulate to have a fairly good idea of the provenance of the\ncode they are running. Those using QEMU for security research are\nexpected to take extra steps to contain potential exploits of QEMU\nitself (for example by using the [security features of\nlibvirt](https://libvirt.org/drvqemu.html#driver-security-architecture)).\n\nNearly half of the currently open confidential issues are in undermaintained areas of the codebase. This means that, although they have been triaged, they are unlikely to be looked at by anyone. It’s also not particularly surprising that if you go looking around in dusty areas of the codebase you are likely to find warts that have been removed from code that is used in real production environment.\n\n## We still want help\n\nUltimately, the QEMU project is still a community of humans collaborating on a key component of the FLOSS virtualisation stack. People participate for many reasons, but no developer has signed up to spend their days wading through reams of generated slop.\n\nThe ability to submit drive-by patches to fix that annoying behaviour in something you use has long been an feature of FLOSS, and we want to take advantage of that. Some of those passing by end up hanging around the project and taking on bigger jobs and more responsibilities, and that is the way we try and sustain QEMU’s developer community.\n\nWith that in mind, I thought I would mention some things to keep in mind if you want to raise a bug in our bug tracker.\n\n*Focus on areas you use*\n\nAnyone can aim an LLM at some of the dustier corners of the QEMU code base and find issues. That’s fine if you’re looking for candidates for your first patch submission to the project. However, if you’re just looking to raise bugs for the sake of it, please reconsider what your motivation is. If you truly want to help the project, maybe consider looking at some of the existing bugs and helping out there. We have plenty.\n\n*Review the output of AI tools*\n\nWe don’t ban the use of LLMs in the bug tracker, because they\nobviously have their uses in identifying problems with the code.\nHowever, they are not infallible, so please be upfront about their use\nso people reading the report are fully informed of the origin of the\nreport. Also, please review the text and see if you can edit down some\nof the more verbose passages that don’t add useful information or are\nengaging in speculation as to the underlying failure reason. If your\nagent has written an elaborate harness to exercise the bug, then tell\nit to rewrite the test using QEMU’s existing [testing\nframework](https://www.qemu.org/docs/master/devel/testing/index.html).\n\n*Propose a patch*\n\nIf you’ve automated the process of finding bugs, you might as well go a step further and propose a patch to fix the issue. Often, reading a patch will give a clearer idea of the issue than wading through all the descriptive text. And if the patch turns into a massive rearchitecting of the codebase, then maybe consider that the LLM’s model of how things might work could be flawed.\n\n## We have been here before\n\nModern LLMs are certainly proving to be disruptive, but this is not the first time a new technology has caused ripple effects throughout the FLOSS world. The introduction of static analysers, improved compiler diagnostics, sanitisers, and fuzzers have all been accompanied by code churn as the issues they find have been addressed.\n\nI expect the same will eventually be true of this wave even though it feels larger than previous ones. We have to hope that at the end of this disruptive phase, we will have better, cleaner, and more reliable software. And to achieve that, we will still need motivated human engineers who understand the history and architecture of the codebase and can shepherd it to the next release.", "url": "https://wpnews.pro/news/bugpocalypse-or-reporting-bugs-in-an-ai-age", "canonical_source": "https://www.qemu.org/2026/09/23/bugs/", "published_at": "2026-09-25 12:05:13+00:00", "updated_at": "2026-09-25 12:30:08.141942+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["QEMU", "GitLab", "Wine", "Andrew Hunt", "The Pragmatic Programmer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bugpocalypse-or-reporting-bugs-in-an-ai-age", "markdown": "https://wpnews.pro/news/bugpocalypse-or-reporting-bugs-in-an-ai-age.md", "text": "https://wpnews.pro/news/bugpocalypse-or-reporting-bugs-in-an-ai-age.txt", "jsonld": "https://wpnews.pro/news/bugpocalypse-or-reporting-bugs-in-an-ai-age.jsonld"}}