Benchmarking GPT-4o, Claude 3.5 Sonnet, and Llama 3 for Automated Code Auditing & Vulnerability Detection

Evaluating LLMs on standardized leaderboards (like MMLU or HumanEval) is helpful,

but it rarely tells you how a model performs on real-world edge cases.

In this benchmark, I tested three models on a specific dev-sec scenario:

Detecting hidden reentrancy and integer overflow vulnerabilities in a complex smart contract.

"You are an expert cybersecurity auditor. Analyze the following smart contract code.

• Identify all critical security vulnerabilities.
• Rank them by CVSS severity score.
• Provide a corrected code patch for each flaw. Respond strictly in JSON format matching the schema provided."

pragma solidity ^0.8.0;

contract Vault {
    mapping(address => uint) public balances;

    function deposit() public payable {
        balances[msg.sender] += msg.value;
    }

    function withdraw(uint _amount) public {
        require(balances[msg.sender] >= _amount);
        (bool success, ) = msg.sender.call{value: _amount}("");
        require(success);
        balances[msg.sender] -= _amount;
    }
}

---

## 3. Results & Comparative Analysis

### **Evaluation Matrix**

| Metric | GPT-4o | Claude 3.5 Sonnet | Llama 3 70B |
| :--- | :--- | :--- | :--- |
| **Vulnerability Spot Rate** | 2 / 2 Found | **2 / 2 Found** | 1 / 2 Found |
| **JSON Schema Adherence** | 100% Valid | 100% Valid | Failed (Raw text added) |
| **Latency / Response Time** | ~1.8 seconds | ~2.4 seconds | **~1.1 seconds (Local)** |
| **Hallucination Level** | Low | **Zero** | Moderate |

### **Model Breakdown:**

#### **1. Claude 3.5 Sonnet (The Precision Winner)**
* **Strengths:** Correctly identified both the reentrancy flaw and state-update ordering issue. Generated clean, secure code patches without unnecessary fluff.
* **Weaknesses:** Slightly slower latency compared to GPT-4o.

#### **2. GPT-4o (The Speed & Structure Winner)**
* **Strengths:** Perfect adherence to the JSON schema on the first attempt. Fast generation speed.
* **Weaknesses:** Flagged a non-existent gas optimization warning as a "Critical" vulnerability.

#### **3. Llama 3 70B (The Open-Source Contender)**
* **Strengths:** Extremely low latency when run locally; caught the primary reentrancy bug.
* **Weaknesses:** Failed to output pure JSON (wrapped the response in markdown explanatory text), requiring additional post-processing parsing.

---

## 4. Final Verdict & Key Takeaway

markdown

Overall Benchmark Winner: Claude 3.5 Sonnet for domain accuracy and zero hallucinations.

Written by Saranyo Deyasi, an AI researcher and developer focusing on model evaluations, security benchmarks, and open-source AI tooling.