{"slug": "can-ai-diagnose-a-linux-production-incident-without-making-it-worse", "title": "Can AI Diagnose a Linux Production Incident Without Making It Worse?", "summary": "A developer built a Kaggle benchmark task, Linux SRE Incident Diagnosis, that tests whether an AI model can diagnose a production filesystem-exhaustion incident from evidence rather than pattern-matching a familiar error message. The task centers on a server returning HTTP 500 errors with /var at 97% full and errno=28, where lsof +L1 reveals the api process still holding a deleted ~47 GB log file open, requiring the model to reconcile df and du and weigh remediation risk. Gemini 3.7 Flash scored 100.00 on the single-task, single-model benchmark and the task is marked Pass.", "body_md": "*This is a submission for the [Kaggle Benchmarking Challenge](https://dev.to/challenges/kaggle-2026-09-23)*\n\nI wanted to test a practical question:\n\n**Can an AI model diagnose a Linux production incident from evidence, rather than simply recognize a familiar error message?**\n\nFor my benchmark, I created a **Linux SRE Incident Diagnosis** task focused on filesystem exhaustion and application failure.\n\nThe incident involved a production server where an application began returning HTTP 500 errors.\n\nAt first glance, the evidence looked straightforward:\n\n```\n/var = 97% full\n```\n\nThe application was also reporting:\n\n```\nerrno=28 No space left on device\n```\n\nBut the incident contained a deeper Linux troubleshooting clue.\n\nThe diagnostic evidence included:\n\n```\nlsof +L1\n\napi 18472 app ... 47185920000 ... /var/log/api.log (deleted)\n```\n\nIn other words, the `api` process was still holding an approximately 47 GB log file open even though the file had been deleted.\n\nThat creates an important distinction between what `df` reports at the filesystem level and what `du` can see through the visible directory structure.\n\nI designed the benchmark to test whether the model could connect those pieces of evidence rather than stopping at:\n\n\"The disk is full.\"\n\nThe task evaluates six areas:\n\nI chose this problem because it represents the kind of incident where a technically plausible answer can still be operationally dangerous.\n\nFor the first version of the benchmark, I tested:\n\n**Gemini 3.7 Flash**\n\nI intentionally started with one model.\n\nMy first goal was to validate the benchmark itself: could the task distinguish a shallow \"disk full\" answer from a response that actually connected filesystem usage, application errors, process-level evidence, remediation, and operational risk?\n\nThe published Kaggle benchmark currently contains **1 task and 1 evaluated model**. Gemini 3.7 Flash received a **100.00** score, and the associated task is marked **Pass**.\n\nRather than immediately adding several models and producing a larger but less focused leaderboard, I wanted to establish a clear baseline first.\n\nThe next useful experiment would be to run the same task against additional models and compare their failure patterns, not just their final scores.\n\nThe headline result is:\n\n**Gemini 3.7 Flash — 100.00**\n\nBut the number itself is not the most interesting part.\n\nWhat interested me was the reasoning the benchmark was designed to test.\n\nA model can easily notice:\n\n```\n/var = 97%\n```\n\nand:\n\n```\nerrno=28 No space left on device\n```\n\nBut that is only the beginning.\n\nThe stronger diagnostic question is:\n\n**Why is the filesystem full, and why might the visible files not account for all of the space?**\n\nThat is where the deleted-but-open log file becomes important.\n\n`df` and `du` tell different stories\nThis incident demonstrates why Linux troubleshooting sometimes requires comparing different evidence sources.\n\n`df` reports filesystem-level space consumption.\n\n`du` reports disk usage associated with visible files and directories.\n\nIf a process still has a deleted file open, the storage can remain allocated even though the filename has disappeared from the directory tree.\n\nThat means a mismatch between filesystem usage and visible file usage is itself a diagnostic clue.\n\nThe benchmark explicitly requires the model to recognize the significance of:\n\n```\nlsof +L1\n```\n\nThis command can expose deleted files that are still open by processes.\n\nIn this scenario, that evidence connects the filesystem problem directly to the `api` process and the deleted `api.log`.\n\nThat is much stronger than simply saying \"check the logs.\"\n\nI also wanted the benchmark to test **operational judgment**.\n\nA model might correctly identify that `/var` is full but then recommend indiscriminately deleting log files, rebooting the server, or killing the application process.\n\nThose actions can have consequences in a production environment.\n\nA safer response should first establish what is consuming the space and then choose a controlled remediation appropriate to the process and service.\n\nThat distinction is important to me because an AI assistant used in an SRE workflow should not only answer:\n\n\"What is wrong?\"\n\nIt should also reason about:\n\n\"What is safe to do next?\"\n\nThe biggest lesson from building the benchmark was how much the **evaluation design** matters.\n\nIf I only checked whether the model mentioned \"disk full,\" the benchmark would be easy to game with a shallow answer.\n\nInstead, I had to make the evaluation evidence-specific:\n\n`/var` must be connected to the incident;`errno=28` must be recognized;`api.log` must be identified;`api` process must be connected to the open file;`lsof +L1` must appear as an appropriate diagnostic command;\nThat changed how I think about AI evaluation.\n\nA benchmark should not only ask:\n\n**Did the model produce the expected answer?**\n\nIt should ask:\n\n**Did the model demonstrate the reasoning that makes the answer trustworthy?**\n\nThe obvious next experiment is to run the same benchmark against additional models.\n\nI would particularly want to compare:\n\nI would also like to expand the benchmark into additional Linux/SRE incident types while keeping the same evidence → diagnosis → remediation → risk structure.\n\nThat would make it possible to study **failure patterns**, rather than treating a single aggregate score as the whole story.\n\nYou can inspect the public benchmark, task, results, description, and provenance on Kaggle:\n\n[Linux SRE Incident Diagnosis — Kaggle Benchmark](https://www.kaggle.com/benchmarks/yemmmychotmailcom/linux-sre-incident-diagnosis/versions/1)\n\nThe benchmark is released under the **Apache 2.0** license.\n\nThe current public leaderboard shows:\n\nThe benchmark contains one Linux SRE incident diagnosis task and is intended to evaluate evidence-based troubleshooting, diagnostic reasoning, safe remediation, and operational risk assessment.\n\nBuilding this benchmark gave me a different way to look at AI-assisted infrastructure work.\n\nFor me, the interesting question is no longer simply:\n\n**\"Can an AI find the answer?\"**\n\nIt is:\n\n**\"Can an AI show enough evidence-based reasoning to help an engineer make a safe decision?\"**\n\nThat is what I want to measure next.", "url": "https://wpnews.pro/news/can-ai-diagnose-a-linux-production-incident-without-making-it-worse", "canonical_source": "https://dev.to/yemmmyc/can-ai-diagnose-a-linux-production-incident-without-making-it-worse-2c9f", "published_at": "2026-09-24 15:05:39+00:00", "updated_at": "2026-09-24 15:30:02.296171+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "mlops", "ai-tools"], "entities": ["Kaggle", "Gemini 3.7 Flash", "Linux", "lsof", "df", "du"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-ai-diagnose-a-linux-production-incident-without-making-it-worse", "markdown": "https://wpnews.pro/news/can-ai-diagnose-a-linux-production-incident-without-making-it-worse.md", "text": "https://wpnews.pro/news/can-ai-diagnose-a-linux-production-incident-without-making-it-worse.txt", "jsonld": "https://wpnews.pro/news/can-ai-diagnose-a-linux-production-incident-without-making-it-worse.jsonld"}}