What surprised me: the benchmark doesn't just score syntactic similarity. Each submission gets compiled, linked against the original's test harness, and run through a differential fuzzer (libFuzzer + AFL++) for 30 minutes. If the regenerated code diverges on any input, it's a hard fail. That means hallucinated helper functions, wrong calling conventions, or off-by-one loop bounds all surface immediately. The leaderboard currently tops out around 34% pass@1 for GPT-4o, 28% for Claude 3.5 Sonnet, and 19% for DeepSeek-Coder-V2 — and those numbers drop another 8-12 points when you enable the "no standard library" flag.
A few practical takeaways if you want to experiment yourself:
- Strip aggressively before feeding the binary.
strip --strip-all --remove-section=.comment --remove-section=.note.*
removes the low-hanging fruit that inflates scores. The benchmark's Docker image does this automatically, but local reproduction needs it.
-
Use a two-stage prompt. First pass: "Produce a Ghidra-like pseudo-C listing with all functions, globals, and struct layouts you can infer." Second pass: "Refine the listing into compilable C99, replacing pseudo-constructs with real code, adding missing headers, and fixing calling conventions." Single-shot prompts consistently miss cross-function type constraints.
-
Anchor the entry point. Add a comment like
// ENTRY: 0x401000
in the prompt context. Without it, models frequently invent their own main
and ignore the real _start
/WinMain
, causing immediate harness failures.
- Iterate with compiler feedback. Pipe
gcc -Wall -Wextra -Werror -std=c99 -o /dev/null -xc -
output back into the context window. Three rounds usually clears 60-70% of syntactic errors before you even hit the fuzzer.
docker run --rm -v $(pwd):/work programbench/vetted:latest \
--binary /work/target.bin \
--submission /work/recovered.c \
--timeout 1800 \
--fuzzer-libfuzzer --fuzzer-aflpp
The most frustrating failure mode I've seen: models correctly recover algorithmic logic (e.g., a custom hash map) but mangle the memory allocator interface — returning malloc
ed pointers where the original used a bump allocator with a custom free list. The fuzzer catches it because the original binary's free list gets corrupted. Fixing this requires the model to infer allocator semantics from usage patterns, not just function signatures.
If anyone's tried fine-tuning on the training split (about 12k binaries × 3 architectures), I'd love to hear whether LoRA on CodeLlama-34B moves the needle past 40%. The benchmark maintainers hinted at a "compiler-optimization" track coming next quarter — binaries built with -O3 -flto -march=native
— which should make control-flow recovery significantly nastier.
DeepSeek can actually reverse engineer its own logic if you 9d ago
Next Google AI now estimates body fat from a single selfie →