CodeLlama-34B

ProgramBench, a new reverse-engineering benchmark, compiles and fuzzes LLM-generated code against original binaries, with GPT-4o achieving a 34% pass@1 rate, Claude 3.5 Sonnet 28%, and DeepSeek-Coder-…

Researchers released four natural-language-to-MLIR benchmarks totaling 410 in-scope pairs and a three-layer schema-derived constraint stack that lets SmolLM2-1.7B match or exceed 15B-34B code LMs on s…