Benchmarking GPT-4o, Claude 3.5 Sonnet, and Llama 3 for Automated Code Auditing & Vulnerability Detection

An AI researcher benchmarked GPT-4o, Claude 3.5 Sonnet, and Llama 3 70B on detecting reentrancy and integer overflow vulnerabilities in a smart contract. Claude 3.5 Sonnet identified both vulnerabilities with zero hallucinations, while GPT-4o showed perfect JSON adherence but flagged a non-existent issue, and Llama 3 caught the primary bug but failed to output pure JSON.

Evaluating LLMs on standardized leaderboards like MMLU or HumanEval is helpful, but it rarely tells you how a model performs on real-world edge cases. In this benchmark, I tested three models on a specific dev-sec scenario: Detecting hidden reentrancy and integer overflow vulnerabilities in a complex smart contract. "You are an expert cybersecurity auditor. Analyze the following smart contract code. - Identify all critical security vulnerabilities. - Rank them by CVSS severity score. - Provide a corrected code patch for each flaw. Respond strictly in JSON format matching the schema provided." js pragma solidity ^0.8.0; contract Vault { mapping address = uint public balances; function deposit public payable { balances msg.sender += msg.value; } function withdraw uint amount public { require balances msg.sender = amount ; bool success, = msg.sender.call{value: amount} "" ; require success ; balances msg.sender -= amount; } } --- 3. Results & Comparative Analysis Evaluation Matrix | Metric | GPT-4o | Claude 3.5 Sonnet | Llama 3 70B | | :--- | :--- | :--- | :--- | | Vulnerability Spot Rate | 2 / 2 Found | 2 / 2 Found | 1 / 2 Found | | JSON Schema Adherence | 100% Valid | 100% Valid | Failed Raw text added | | Latency / Response Time | ~1.8 seconds | ~2.4 seconds | ~1.1 seconds Local | | Hallucination Level | Low | Zero | Moderate | Model Breakdown: 1. Claude 3.5 Sonnet The Precision Winner Strengths: Correctly identified both the reentrancy flaw and state-update ordering issue. Generated clean, secure code patches without unnecessary fluff. Weaknesses: Slightly slower latency compared to GPT-4o. 2. GPT-4o The Speed & Structure Winner Strengths: Perfect adherence to the JSON schema on the first attempt. Fast generation speed. Weaknesses: Flagged a non-existent gas optimization warning as a "Critical" vulnerability. 3. Llama 3 70B The Open-Source Contender Strengths: Extremely low latency when run locally; caught the primary reentrancy bug. Weaknesses: Failed to output pure JSON wrapped the response in markdown explanatory text , requiring additional post-processing parsing. --- 4. Final Verdict & Key Takeaway markdown Overall Benchmark Winner: Claude 3.5 Sonnet for domain accuracy and zero hallucinations. Written by Saranyo Deyasi , an AI researcher and developer focusing on model evaluations, security benchmarks, and open-source AI tooling.