AI Agent Coding Comparison: The Rust Leap Year Challenge

A developer tested five AI agents—ChatGPT, Claude, Gemini, Kilo Code Hy3, and Kilo Code Auto Efficient Qwen 3.7-plus—on a Rust leap year coding challenge. ChatGPT and Qwen asked clarifying questions, while only ChatGPT, Claude, and Hy3 used Rust's test harness. The author rated ChatGPT 7/10 and preferred Kilo Code models for detailed explanations.

AI Agent Comparison I ran a little exercise to give the following 5 AI Agents the same prompt and see how they would fare, in my opinion, in completing the task: - ChatGPT: GPT-5.5 https://chatgpt.com/ from OpenAI https://openai.com/ - Qwen: Qwen 3.7-plus https://qwen.ai/ from Alibaba Cloud https://www.alibabacloud.com/ via Auto Efficient model from Kilo Code https://kilo.ai/ Except for the Qwen model, which used some Kilo credits <$0.02 , all the models are free to use; there are no subscription plans at all. I was signed in as a registered user for all the agents. Summary if you do not want to read the whole post Gave each model a short exercise to produce some Rust https://rust-lang.org/ code and see how each performs. In the prompt, I asked the models to ask clarifying questions if there is any doubt; do not make any assumptions. Did any model ask, yes 2 did: - ChatGPT asked whether the Gregorian calendar leap year should be implemented.- Gregorian calendar - Kilo Code Auto Efficient Qwen 3.7-plus asked whether modulo % can be used and also whether it should just provide the code or provide the code and execute.- Yes, % can be used; just provide the code. - Note: There was no need to ask whether modulo can be used; this is an operator just like + , - , / or - All the models produced decent code, but only 3 used the cfg test test harness that Rust provides: - CharGPT - Claude - Kilo Code Hy3 The other 2, Gemini and Kilo Code Auto Efficient Qwen 3.7-plus , include some test code as part of the main function with println output. Even though only Kilo Code Efficient Qwen 3.7-plus asked whether % can be used, all except Kilo Code Hy3 used it. Kilo Code Hy3 implemented a remainder function to determine a remainder of 0 after division. If I have to pick a model based on the prompt, then I would go with either of the Kilo Code models since its output is very detailed and it explains it’s thinking and what it was going to do. So what was the goal here? It was to determine: - How well does the model “listen” to the prompt, did they get the task right? - Were any follow-up questions asked to clarify something the model did not understand? - Did the model make any assumptions? - How accurate was the model in solving the task? Note, I was not interested this time at all in the cost and/or time that it took the model to solve the task. Before we look at the result, let’s see how they. stack up on Kilo Bench https://kilo.ai/leaderboard : | Model | Code Rank | Coding Completion | Cost/Attempt | |---|---|---|---| | Sonnet 5 Medium | 10 | 59.6% | $36.19 | | GPT-5.5 | 54 | 74.2% | $72.63 | | Gemini 3.5 Flash | 57 | 64.7% | $104.49 | | Qwen 3.7-plus | 56 | 54.6% | $20.65 | | Hy3 free | 34 | 47.6% | $0.00 | - Used the data from Qwen 3.7-max slightly more expensive Prompt Here is the prompt I gave each of the models. Create a Rust function from first principles that returns TRUE or FALSE for a given number whether it is a leap year. Use no crates, dependencies or built-in standard traits. Do not read, write or reference any local files; just provide the code as text in a code block. Also, create tests and test the function for the following set of numbers: 1, 4, 2023, 2024, 2025, 2048, 1900, 2100, 1600, 2000 Do not make any assumptions, and if in doubt, then ask me. Detail - ChatGPT - What did it get right? - Asked clarifying question s - Made use of Rust test harness - Implement the ask, leap year function correctly. - - What did it get wrong? - Gave no explanation or plan on what it is going to do. - Test was created as 1 big test instead of 10 individual ones. - - Personal Rating: 7/10 - - Claude - What did it get right? - Test cases was split so that one can run each individualy. - Gave an explanation of what it did not use. - Implement the ask, leap year function correctly. - - What did it get wrong? - Made assumption on what calendar to use - Was very brief with no plan - - Personal Rating: 6/10 - - Gemini - What did it get right? - Implement the ask, leap year function correctly. - What did it get wrong? - No test case, test was in the main function - Assume the calendar to use. - Again very brief with no plan. - - Personal rating: 5/10 - - Kilo Code Auto Efficient: Qwen 3.7-plus - What did it get right? - Asked clarifying questions - Implement the ask, leap year function correctly. - Was very detailed on what it was going to do, almost too detailed but I prefer it. - - What did it get wrong? - Did not use the Rust test harness, tests. was just some comparisons and println output. - Did not use the Rust test harness, tests. was just some comparisons and - Personal rating: 8/10 - - Kilo Code Hy3 - What did it get right? - Use test harness, although only 1 test case. - As is common for Kilo Code, the output was very detailed and explained exactly what it was going to do. - - What did it get wrong? - Would have preferred individual tests for the 10 tests. - Should have used modulo % operator instead of creating a remainder function. - Assumed the calendar to use. - - Personal rating: 8/10 - See the References below for the raw output from each model. Conclusion All the models basically did what was asked of them. The Kilo Code models were very detailed and explained what it was going to do very well. In a real-life situation, I would ask the model to first plan what it would do and then based on an agreed plan, implement that plan. The implemention/coding model might be totally different from the planning model. Models perform better or worse depending on what they are asked to do whether it is just some simple question, a detailed plan or a coding implementation based on an agreed plan. References Here is the raw data from each of the models: