{"slug": "ai-agent-coding-comparison-the-rust-leap-year-challenge", "title": "AI Agent Coding Comparison: The Rust Leap Year Challenge", "summary": "A developer tested five AI agents—ChatGPT, Claude, Gemini, Kilo Code Hy3, and Kilo Code Auto Efficient Qwen 3.7-plus—on a Rust leap year coding challenge. ChatGPT and Qwen asked clarifying questions, while only ChatGPT, Claude, and Hy3 used Rust's test harness. The author rated ChatGPT 7/10 and preferred Kilo Code models for detailed explanations.", "body_md": "## AI Agent Comparison\n\nI ran a little exercise to give the following 5 AI Agents the same prompt and see how they would fare, in my opinion, in completing the task:\n\n-\n[ChatGPT: GPT-5.5](https://chatgpt.com/)from[OpenAI](https://openai.com/) -\n[Qwen: Qwen 3.7-plus](https://qwen.ai/)from[Alibaba Cloud](https://www.alibabacloud.com/)via Auto Efficient model from[Kilo Code](https://kilo.ai/)\n\nExcept for the Qwen model, which used some Kilo credits (<$0.02), all the models are free to use; there are no subscription plans at all. I was signed in as a registered user for all the agents.\n\n### Summary (if you do not want to read the whole post)\n\nGave each model a short exercise to produce some [Rust](https://rust-lang.org/) code and see how each performs.\n\nIn the prompt, I asked the models to ask clarifying questions if there is any doubt; do not make any assumptions. Did any model ask, yes 2 did:\n\n-\n*ChatGPT*asked whether the Gregorian calendar leap year should be implemented.- Gregorian calendar\n\n-\n*Kilo Code Auto Efficient Qwen 3.7-plus*asked whether modulo (%) can be used and also whether it should just provide the code or provide the code and execute.-\nYes, % can be used; just provide the code.\n\n-\n*Note:*There was no need to ask whether modulo can be used; this is an operator just like`+ , - , / or *`\n\n-\n\nAll the models produced decent code, but only 3 used the `#[cfg(test)]`\n\ntest harness that Rust provides:\n\n-\n*CharGPT* -\n*Claude* -\n*Kilo Code Hy3*\n\nThe other 2, *Gemini* and *Kilo Code Auto Efficient Qwen 3.7-plus*, include some test code as part of the `main()`\n\nfunction with `println!()`\n\noutput.\n\nEven though only *Kilo Code Efficient Qwen 3.7-plus* asked whether % can be used, all except *Kilo Code Hy3* used it. *Kilo Code Hy3* implemented a `remainder()`\n\nfunction to determine a remainder of 0 after division.\n\nIf I have to pick a model based on the prompt, then I would go with either of the * Kilo Code* models since its output is very detailed and it explains it’s thinking and what it was going to do.\n\n### So what was the goal here?\n\nIt was to determine:\n\n-\nHow well does the model “listen” to the prompt, did they get the task right?\n\n-\nWere any follow-up questions asked to clarify something the model did not understand?\n\n-\nDid the model make any assumptions?\n\n-\nHow accurate was the model in solving the task?\n\nNote, I was not interested this time at all in the cost and/or time that it took the model to solve the task.\n\nBefore we look at the result, let’s see how they. stack up on [Kilo Bench](https://kilo.ai/leaderboard):\n\n| Model | Code Rank | Coding Completion | Cost/Attempt |\n|---|---|---|---|\n| Sonnet 5 Medium | 10 | 59.6% | $36.19 |\n| GPT-5.5 | 54 | 74.2% | $72.63 |\n| Gemini 3.5 Flash | 57 | 64.7% | $104.49 |\n| Qwen 3.7-plus* | 56 | 54.6% | $20.65 |\n| Hy3 (free) | 34 | 47.6% | $0.00 |\n\n- Used the data from Qwen 3.7-max (slightly more expensive)\n\n### Prompt\n\nHere is the prompt I gave each of the models.\n\nCreate a Rust function from first principles that returns TRUE or FALSE for a given number whether it is a leap year. Use no crates, dependencies or built-in standard traits. Do not read, write or reference any local files; just provide the code as text in a code block.\n\nAlso, create tests and test the function for the following set of numbers: 1, 4, 2023, 2024, 2025, 2048, 1900, 2100, 1600, 2000\n\nDo not make any assumptions, and if in doubt, then ask me.\n\n### Detail\n\n-\n**ChatGPT**-\n*What did it get right?*-\nAsked clarifying question(s)\n\n-\nMade use of Rust test harness\n\n-\nImplement the ask, leap year function correctly.\n\n-\n-\n*What did it get wrong?*-\nGave no explanation or plan on what it is going to do.\n\n-\nTest was created as 1 big test instead of 10 individual ones.\n\n-\n-\n*Personal Rating:*7/10\n\n-\n-\n**Claude**-\n*What did it get right?*-\nTest cases was split so that one can run each individualy.\n\n-\nGave an explanation of what it did not use.\n\n-\nImplement the ask, leap year function correctly.\n\n-\n-\n*What did it get wrong?*-\nMade assumption on what calendar to use\n\n-\nWas very brief with no plan\n\n-\n-\n*Personal Rating:*6/10\n\n-\n-\n**Gemini**-\n*What did it get right?*- Implement the ask, leap year function correctly.\n\n-\n*What did it get wrong?*-\nNo test case, test was in the\n\n`main()`\n\nfunction -\nAssume the calendar to use.\n\n-\nAgain very brief with no plan.\n\n-\n-\n*Personal rating:*5/10\n\n-\n-\n**Kilo Code Auto Efficient: Qwen 3.7-plus**-\n*What did it get right?*-\nAsked clarifying questions\n\n-\nImplement the ask, leap year function correctly.\n\n-\nWas very detailed on what it was going to do, almost too detailed but I prefer it.\n\n-\n-\n*What did it get wrong?*- Did not use the Rust test harness, tests. was just some comparisons and\n`println!()`\n\noutput.\n\n- Did not use the Rust test harness, tests. was just some comparisons and\n-\n*Personal rating:*8/10\n\n-\n-\n**Kilo Code Hy3**-\n*What did it get right?*-\nUse test harness, although only 1 test case.\n\n-\nAs is common for Kilo Code, the output was very detailed and explained exactly what it was going to do.\n\n-\n-\n*What did it get wrong?*-\nWould have preferred individual tests for the 10 tests.\n\n-\nShould have used modulo (%) operator instead of creating a\n\n`remainder()`\n\nfunction. -\nAssumed the calendar to use.\n\n-\n-\n*Personal rating:*8/10\n\n-\n\nSee the References below for the raw output from each model.\n\n#### Conclusion\n\nAll the models basically did what was asked of them. The Kilo Code models were very detailed and explained what it was going to do very well.\n\nIn a real-life situation, I would ask the model to first plan what it would do and then based on an agreed plan, implement that plan. The implemention/coding model might be totally different from the planning model. Models perform better or worse depending on what they are asked to do whether it is just some simple question, a detailed plan or a coding implementation based on an agreed plan.\n\n### References\n\nHere is the raw data from each of the models:", "url": "https://wpnews.pro/news/ai-agent-coding-comparison-the-rust-leap-year-challenge", "canonical_source": "https://www.mariusb.net/blog/2026/07/ai-agent-comparison-rust-leap-year/", "published_at": "2026-07-13 12:06:51+00:00", "updated_at": "2026-07-13 12:35:31.645205+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "developer-tools"], "entities": ["ChatGPT", "OpenAI", "Qwen", "Alibaba Cloud", "Kilo Code", "Claude", "Gemini", "Rust"], "alternates": {"html": "https://wpnews.pro/news/ai-agent-coding-comparison-the-rust-leap-year-challenge", "markdown": "https://wpnews.pro/news/ai-agent-coding-comparison-the-rust-leap-year-challenge.md", "text": "https://wpnews.pro/news/ai-agent-coding-comparison-the-rust-leap-year-challenge.txt", "jsonld": "https://wpnews.pro/news/ai-agent-coding-comparison-the-rust-leap-year-challenge.jsonld"}}