This is a submission for the Kaggle Benchmarking Challenge
I wanted to test a practical question:
Can an AI model diagnose a Linux production incident from evidence, rather than simply recognize a familiar error message?
For my benchmark, I created a Linux SRE Incident Diagnosis task focused on filesystem exhaustion and application failure.
The incident involved a production server where an application began returning HTTP 500 errors.
At first glance, the evidence looked straightforward:
/var = 97% full
The application was also reporting:
errno=28 No space left on device
But the incident contained a deeper Linux troubleshooting clue.
The diagnostic evidence included:
lsof +L1
api 18472 app ... 47185920000 ... /var/log/api.log (deleted)
In other words, the api process was still holding an approximately 47 GB log file open even though the file had been deleted.
That creates an important distinction between what df reports at the filesystem level and what du can see through the visible directory structure.
I designed the benchmark to test whether the model could connect those pieces of evidence rather than stopping at:
"The disk is full."
The task evaluates six areas:
I chose this problem because it represents the kind of incident where a technically plausible answer can still be operationally dangerous.
For the first version of the benchmark, I tested:
Gemini 3.7 Flash
I intentionally started with one model.
My first goal was to validate the benchmark itself: could the task distinguish a shallow "disk full" answer from a response that actually connected filesystem usage, application errors, process-level evidence, remediation, and operational risk?
The published Kaggle benchmark currently contains 1 task and 1 evaluated model. Gemini 3.7 Flash received a 100.00 score, and the associated task is marked Pass.
Rather than immediately adding several models and producing a larger but less focused leaderboard, I wanted to establish a clear baseline first.
The next useful experiment would be to run the same task against additional models and compare their failure patterns, not just their final scores.
The headline result is:
Gemini 3.7 Flash — 100.00
But the number itself is not the most interesting part.
What interested me was the reasoning the benchmark was designed to test.
A model can easily notice:
/var = 97%
and:
errno=28 No space left on device
But that is only the beginning.
The stronger diagnostic question is:
Why is the filesystem full, and why might the visible files not account for all of the space?
That is where the deleted-but-open log file becomes important.
df and du tell different stories
This incident demonstrates why Linux troubleshooting sometimes requires comparing different evidence sources.
df reports filesystem-level space consumption.
du reports disk usage associated with visible files and directories.
If a process still has a deleted file open, the storage can remain allocated even though the filename has disappeared from the directory tree.
That means a mismatch between filesystem usage and visible file usage is itself a diagnostic clue.
The benchmark explicitly requires the model to recognize the significance of:
lsof +L1
This command can expose deleted files that are still open by processes.
In this scenario, that evidence connects the filesystem problem directly to the api process and the deleted api.log.
That is much stronger than simply saying "check the logs."
I also wanted the benchmark to test operational judgment.
A model might correctly identify that /var is full but then recommend indiscriminately deleting log files, rebooting the server, or killing the application process.
Those actions can have consequences in a production environment.
A safer response should first establish what is consuming the space and then choose a controlled remediation appropriate to the process and service.
That distinction is important to me because an AI assistant used in an SRE workflow should not only answer:
"What is wrong?"
It should also reason about:
"What is safe to do next?"
The biggest lesson from building the benchmark was how much the evaluation design matters.
If I only checked whether the model mentioned "disk full," the benchmark would be easy to game with a shallow answer.
Instead, I had to make the evaluation evidence-specific:
/var must be connected to the incident;errno=28 must be recognized;api.log must be identified;api process must be connected to the open file;lsof +L1 must appear as an appropriate diagnostic command;
That changed how I think about AI evaluation.
A benchmark should not only ask:
Did the model produce the expected answer?
It should ask:
Did the model demonstrate the reasoning that makes the answer trustworthy?
The obvious next experiment is to run the same benchmark against additional models.
I would particularly want to compare:
I would also like to expand the benchmark into additional Linux/SRE incident types while keeping the same evidence → diagnosis → remediation → risk structure.
That would make it possible to study failure patterns, rather than treating a single aggregate score as the whole story.
You can inspect the public benchmark, task, results, description, and provenance on Kaggle:
Linux SRE Incident Diagnosis — Kaggle Benchmark
The benchmark is released under the Apache 2.0 license.
The current public leaderboard shows:
The benchmark contains one Linux SRE incident diagnosis task and is intended to evaluate evidence-based troubleshooting, diagnostic reasoning, safe remediation, and operational risk assessment.
Building this benchmark gave me a different way to look at AI-assisted infrastructure work.
For me, the interesting question is no longer simply:
"Can an AI find the answer?"
It is:
"Can an AI show enough evidence-based reasoning to help an engineer make a safe decision?"
That is what I want to measure next.