{"slug": "identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms", "title": "Identifying the Processor of a Bare-Metal Binary (Strategy 2): Testing LLMs", "summary": "A developer evaluated the use of large language models to identify the processor of a bare-metal binary by disassembling it with Ghidra against 177 architectures and prompting local LLMs to analyze the outputs. The benchmark, using models like qwen3-coder:30b, revealed an exceptionally high false-positive rate, with models validating between 35 and 172 candidates out of 177, failing as a selective heuristic. The developer attributes this to superficial syntax validation and contextual blind spots from limiting analysis to the first 50 instructions.", "body_md": "Industrializing the disassembly of an undocumented processor from a raw binary is a complex challenge that can be broken down into 4 main phases:\n\nFor the first step of this project, the goal is to evaluate different strategies and identify the most effective approach.\n\nThe first strategy aimed to generate a transcodification table (mapping byte sequences to assembly instructions) in order to disassemble the binary both statically and dynamically for a given processor. This process continues until one or more bytes fail to match any known instruction for that processor, or until disassembly completes successfully (note: successful disassembly does not necessarily mean the binary was compiled for that specific processor).\n\nTo implement this strategy and build the transcodification table, several approaches were tested:\n\nGhidra transcodification table generation: Failed.\n\nNative disassembler transcodification table generation: Failed.\n\nUsing Gemini to generate the table from raw byte sequences: Successful.\n\n(For a full summary of the tests conducted under Strategy 1, see \"Identifying the Processor of a Bare-Metal Binary — Strategy 1\".)\n\nThe second strategy takes a completely different approach. It consists of using Ghidra to disassemble the binary against a large number of target architectures (177 processors) and then leveraging a Large Language Model (LLM) to analyze the resulting disassembly outputs to determine which processor truly matches the binary.\n\nThe local LLMs selected for this benchmark are:\n\nUsing a single C source code file and an automated toolchain, test binaries were generated for approximately 30 different processors in both raw bare-metal and standard ELF formats.\n\nThree specific binaries were selected for the evaluation:\n\nDisassembly output files were batch-generated using PyGhidra and a custom Java headless script. Each model was then prompted to analyze every generated disassembly file using the exact same system prompt:\n\nPython\n\n```\nSYSTEM_PROMPT = \"\"\"You are an expert in reverse engineering and processor architectures.\nYour task is to verify the consistency of a raw firmware disassembly.\nExamine the provided instructions, verify whether the architecture's syntax is valid,\nand determine if the instructions appear coherent (absence of repeated invalid instructions, aberrant opcodes, etc.).\nRespond strictly in valid JSON format.\"\"\"\n```\n\nThe models were instructed to respond using a strict JSON schema:\n\n```\nJSON\n{\n    \"processor_id\": \"{processor_id}\",\n    \"is_valid\": true,\n    \"confidence_score\": 0.85,\n    \"summary\": \"Short explanation of the analysis\",\n    \"detected_anomalies\": [\"list of errors or anomalies\"]\n}\n```\n\nThis firmware was generated for a processor supported by Ghidra.\n\nThis firmware was generated for a target architecture not supported by Ghidra.\n\nGiven the poor baseline performance of local models, this test was conducted exclusively on qwen3-coder:30b.\n\nqwen3-coder:30b: Generated reports for all 177 files. Identified 42 potential candidate processors (including the correct one).\n\nThe empirical results show an exceptionally high rate of false positives: models validated between 35 and 172 candidates out of 177, failing to act as a selective heuristic filter. Several key technical factors explain this shortfall:\n\nSuperficial Syntax Validation vs. Semantic Verification:\n\nMost code-focused LLMs evaluate assembly primarily at a syntactic level. When Ghidra forces a disassembly, it outputs valid instruction strings (e.g., MOV R0, R1) according to the target architecture's grammar. The LLMs mistake syntactically valid instructions for semantically logical code, ignoring structural red flags like non-sensical control flow, impossible stack frame allocations, or meaningless register usage.\n\nWindow-Size Contraint & Contextual Blind Spots (First 50 Instructions):\n\nLimiting context to the first 50 instructions creates a severe bias. In raw bare-metal binaries (and especially ELF files parsed as raw bytes), the offset often starts with interrupt vectors, padding, or raw header metadata (\\x7fELF). Disassembling metadata produces random, garbage instructions. The LLM either accepts this garbage as valid initialization code or misses real function prologues (PUSH {LR}, frame setup) located further down in the binary.\n\nHallucination of High Confidence Scores:\n\nSmaller, quantized local models lack calibrated uncertainty. They frequently assign confidence scores between 0.80 and 1.0 to highly improbable disassemblies simply because no explicit .byte unknown directives appeared in the 50-sample window.\n\nJSON Schema Inflation and Special Token Leaks:\n\nModels like gemma4:26b failed due to prompt-adherence degradation when handling low-level assembly syntax, leaking internal reasoning/channel tokens (e.g., thought...) into the JSON stream, which corrupted output parsing.\n\nTo transform this approach into a viable industrial pipeline, several adjustments are required:\n\nHeuristic Pre-Filtering & Metrics Computation (Hybrid Static Analysis + LLM):\n\nBefore calling the LLM, compute mathematical heuristics on the disassembly output:\n\nInvalid Instruction Ratio: Reject architectures where .byte or ?? directives exceed 5% of the total output.\n\nControl Flow Density: Measure the ratio of control-flow instructions (JMP, CALL, BRANCH) to data movement (MOV, LDR). Random/incorrect disassemblies display abnormally low or erratic jump densities.\n\nEntropy & String Artifacts: Calculate entropy across the binary sections to skip static headers before sampling.\n\nFew-Shot Prompting with Counter-Examples:\n\nUpdate the system prompt with explicit Few-Shot examples contrasting a valid disassembly (coherent stack operations, standard function prologues, structured loops) with an invalid/garbage disassembly (repetitive opcodes, dead jumps, aberrant immediate values).\n\nDynamic Sample Windowing (Skipping Metadata):\n\nInstead of feeding the first 50 raw instructions, extract 50 instructions starting from detected function entry points (e.g., identified by CALL targets or push/pop entry sequences). For ELF binaries treated as bare metal, automatically skip the initial offset matching known header lengths.\n\nChain-of-Thought (CoT) Reasoning Before JSON Output:\n\nForcing the LLM to output raw JSON immediately suppresses its internal analytical capabilities. Changing the prompt structure to require a step-by-step reasoning phase before emitting the final JSON object significantly improves accuracy:\n\n```\nPlaintext\n1. Analyze control flow coherence...\n2. Check register consistency...\n3. Identify function entry signatures...\n4. Output JSON verdict.\n```\n\nThis benchmark highlights the clear limitations of current local LLMs when used out-of-the-box for low-level reverse engineering heuristics. Without statistical pre-filtering, dynamic windowing, and structured chain-of-thought prompting, local models act as overly permissive classifiers.\n\nCan refined prompt engineering, dynamic context sampling, or cloud-grade LLMs bridge this gap to reliably pinpoint unknown architectures? That will be the subject of our next evaluation.", "url": "https://wpnews.pro/news/identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms", "canonical_source": "https://dev.to/ddupard/identifying-the-processor-of-a-bare-metal-binary-strategy-2-and-testing-llms-1i8n", "published_at": "2026-08-04 19:01:19+00:00", "updated_at": "2026-08-04 19:46:38.734062+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "developer-tools"], "entities": ["Ghidra", "PyGhidra", "qwen3-coder:30b", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms", "markdown": "https://wpnews.pro/news/identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms.md", "text": "https://wpnews.pro/news/identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms.txt", "jsonld": "https://wpnews.pro/news/identifying-the-processor-of-a-bare-metal-binary-strategy-2-testing-llms.jsonld"}}