{"slug": "evaluating-llm-models-for-dba-tasks", "title": "Evaluating LLM models for DBA tasks", "summary": "Percona Lab released dbaai_bench, an open-source harness that evaluates large language models on real database administration tasks against remote hosts, and reported that almost all tested models completed the tasks, with smaller open-weight models finishing the same work at roughly one-tenth the cost of larger models. The harness, run through OpenRouter, tests models including qwen/qwen3.8-max, moonshotai/kimi-k3, z-ai/glm-5.3, deepseek/deepseek-v4-pro-0813, deepseek/deepseek-v4-flash-0731, qwen/qwen3.8-27b, meta/muse-glimmer-30b, and google/gemma-4-31b-it, and a companion project, dbaai_runner, batches runs across models and configurations. The author noted that deepseek/deepseek-v4-pro-0813 substituted MySQL 8.4 when asked to install MySQL 9.7, showing that LLM-driven infrastructure automation may make operational decisions rather than blindly execute instructions, making validation and guardrails important.", "body_md": "Large language models are increasingly capable of performing practical systems-administration tasks. I wanted to understand how well they could handle something more specialized: **database administration**.\n\nTo explore this, I developed a harness for evaluating the ability of different LLMs to execute real DBA tasks on remote systems.\n\nMy primary focus is on **open-weight models** because of their deployment flexibility and cost efficiency. Their token costs are often substantially lower than those of closed frontier models, which makes them especially interesting for automation workloads where an agent may require dozens of interactions to complete a task.\n\nThe models I am currently evaluating fall into two broad categories.\n\n`qwen/qwen3.8-max`` moonshotai/kimi-k3``z-ai/glm-5.3`` deepseek/deepseek-v4-pro-0813``deepseek/deepseek-v4-flash-0731`` qwen/qwen3.8-27b``meta/muse-glimmer-30b`` google/gemma-4-31b-it`\nI use **OpenRouter** to manage requests and responses across the different models.\n\nThe project is available here:\n\n[https://github.com/Percona-Lab/dbaai_bench](https://github.com/Percona-Lab/dbaai_bench)\n\nThe workflow is intentionally straightforward.\n\n`Install Percona Server for MySQL 8.4`\n\nHere is an example of running the harness against two hosts:\n\n```\n<code class=\"language-bash\">uv run dba.py \\\n  -m qwen/qwen3.8-max \\\n  --host 165.22.191.129 \\\n  --host 68.183.121.153 \\\n  --task \"Install MySQL 9.7 Replication with encrypted traffic\" \\\n  --max-steps 200 \\\n  --mode unattended\n</code>\n\n12345678\n\n<code class=\"language-bash\">uv run dba.py \\  -m qwen/qwen3.8-max \\  --host 165.22.191.129 \\  --host 68.183.121.153 \\  --task \"Install MySQL 9.7 Replication with encrypted traffic\" \\  --max-steps 200 \\  --mode unattended</code>\n```\n\nThe model is responsible for deciding how to interpret the request, what commands to execute, how to react to errors, and when the task has been completed.\n\nSo far, **almost all models have successfully executed the tasks**, with only a few individual exceptions.\n\nOne of the more interesting findings is that smaller and cheaper models are often just as capable of completing the task as much larger models.\n\nThe main difference is usually the number of steps.\n\nSmaller models may require more iterations to arrive at the correct configuration, but the overall cost can still be dramatically lower.\n\nIn several tests, smaller models completed the same work at roughly **one-tenth of the cost** of larger models.\n\nYou can see examples of benchmark output here:\n\n[https://github.com/Percona-Lab-results/dbaai_bench-results](https://github.com/Percona-Lab-results/dbaai_bench-results)\n\nFor example, here is a run using `deepseek/deepseek-v4-flash-0731` to install MySQL 8.4 in replication mode:\n\nOne particularly interesting behavior appears when the requested task cannot be completed exactly as specified.\n\nFor example, I asked `deepseek/deepseek-v4-pro-0813` to:\n\nInstall MySQL 9.7\n\nThe model responded:\n\n“MySQL 8.4 is available (not 9.7 as requested, but 8.4 is what Ubuntu provides; will use it). Installing MySQL server on node1 (primary).”\n\nInstead of simply failing, the model interpreted the situation, selected an available version, and continued with the task.\n\nWhether this behavior is desirable depends on the automation scenario, but it demonstrates an important characteristic of LLM-driven infrastructure automation: **the model may make operational decisions rather than blindly execute instructions**.\n\nThat makes validation and guardrails particularly important.\n\nRunning one model at a time is useful during development, but it becomes inconvenient when comparing many models and configurations.\n\nTo simplify this, I created a second project:\n\n[https://github.com/Percona-Lab/dbaai_runner](https://github.com/Percona-Lab/dbaai_runner)\n\nThe batch runner executes `dba.py` across multiple models and configurations and collects the results in a format that makes comparison easier.\n\nExample results are available here:\n\n[https://github.com/Percona-Lab-results/dbaai_runner_results](https://github.com/Percona-Lab-results/dbaai_runner_results)\n\nOne test asks the models to perform the following task:\n\nThese two servers are identical and freshly built. Install Percona Server 8.4 on both and set up replication between them: one primary taking writes, one replica following it.\n\nDecide which server takes which role and say which you chose. Replication traffic must go over the private network the two servers share, not over the public internet, and the replica must refuse writes of its own.\n\nThe corresponding leaderboard is available here:\n\n[https://github.com/Percona-Lab-results/dbaai_runner_results/blob/main/20260824-030110/leaderboard.md](https://github.com/Percona-Lab-results/dbaai_runner_results/blob/main/20260824-030110/leaderboard.md)\n\n| Model | Result | Mean | Clean | Cost | Steps | \n|---|---|---|---|---|---|\n| `deepseek/deepseek-v4-flash-0731` | 100% | 100% | 1/1 | $0.0082 | 35 | \n| `deepseek/deepseek-v4-pro-0813` | 100% | 100% | 1/1 | $0.2639 | 27 | \n| `meta/muse-glimmer-30b` | 100% | 100% | 1/1 | $0.0479 | 74 | \n| `moonshotai/kimi-k3` | 100% | 100% | 1/1 | $0.2888 | 17 | \n| `qwen/qwen3.8-max` | 100% | 100% | 1/1 | $0.4200 | 22 | \n| `z-ai/glm-5.3` | 100% | 100% | 1/1 | $0.2176 | 20 | \n| `google/gemma-4-31b-it` | 94% | 94% | 0/1 | $0.0289 | 46 | \n| `qwen/qwen3.8-27b` | *timeout* | 0% | 0/1 | $0.5116 | 78 | \n\nThe cost difference is especially interesting.\n\nFor example:\n\nThis illustrates an important point for agentic infrastructure workloads:\n\n**The model that uses the fewest steps is not necessarily the most economical model.**\n\nA smaller model can require significantly more iterations while still being dramatically cheaper overall.\n\nAnother useful test is to deliberately give the models an impossible request.\n\nInstall Percona Server for MySQL 9.7.2.\n\nThat release does not exist.\n\nThe responses were interesting.\n\nMost models decided to install the closest available version, **9.7.1**, instead.\n\nDeepSeek behaved differently and completed the run **without performing an installation**.\n\nThese cases are useful because they test more than a model’s ability to generate shell commands. They expose how the model reasons about ambiguity, unavailable software, conflicting requirements, and whether it should modify the user’s intent.\n\nFor production automation, those behaviors may be just as important as raw task-completion rates.\n\nThe examples above focus primarily on MySQL and Percona Server, but the framework is not limited to MySQL.\n\nThe same approach can be used for:\n\nIn practice, the harness is testing whether an LLM can act as an iterative systems operator:\n\n**plan → execute → observe → adjust → verify**\n\nThe early results suggest several things.\n\nFirst, modern LLMs are already surprisingly capable of performing multi-step database administration tasks when they have access to command execution and system feedback.\n\nSecond, model size is not necessarily the best predictor of practical usefulness.\n\nA smaller model that requires twice as many steps may still be significantly more economical than a larger model.\n\nThird, the ability of models to exercise judgment introduces both opportunities and risks. A model that substitutes an unavailable software version may be helpful in one situation and unacceptable in another.\n\nThis means future evaluation should look beyond simple task completion.\n\nUseful dimensions include:\n\nI expect these characteristics to become increasingly important as LLMs move from answering DBA questions to actually **operating database infrastructure**.\n\n**DBA AI Benchmark**\n\n**DBA AI Benchmark Results**\n\n**DBA AI Batch Runner**\n\n**Batch Runner Results**\n\nThese projects are still evolving.\n\nThere are many interesting areas to explore, including additional database platforms, more complex failure scenarios, security evaluation, cost optimization, model comparison, and better automated scoring.\n\nIf you are interested in cooperating, testing additional models, adding DBA workloads, or improving the benchmark framework, please let me know.\n\nResources\n\nRELATED POSTS", "url": "https://wpnews.pro/news/evaluating-llm-models-for-dba-tasks", "canonical_source": "https://www.percona.com/blog/evaluating-llm-models-for-dba-tasks/", "published_at": "2026-09-17 11:50:07+00:00", "updated_at": "2026-09-17 11:56:11.008162+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-research"], "entities": ["Percona Lab", "dbaai_bench", "dbaai_runner", "OpenRouter", "MySQL", "Percona Server for MySQL 8.4", "deepseek/deepseek-v4-pro-0813", "qwen/qwen3.8-max"], "alternates": {"html": "https://wpnews.pro/news/evaluating-llm-models-for-dba-tasks", "markdown": "https://wpnews.pro/news/evaluating-llm-models-for-dba-tasks.md", "text": "https://wpnews.pro/news/evaluating-llm-models-for-dba-tasks.txt", "jsonld": "https://wpnews.pro/news/evaluating-llm-models-for-dba-tasks.jsonld"}}