From Packed Binary to Readable Code: A Hands-On Walkthrough of Unpacking, Shellcode Analysis, and Memory Forensics

A developer documented a full malware analysis lab session, unpacking and analyzing packed binaries, shellcode, and memory forensics. The walkthrough covers static analysis, manual unpacking with a debugger, multi-stage shellcode extraction, code injection patterns, API hooking, and memory forensics with Volatility. The exercise was conducted on isolated virtual machines using teaching specimens.

A few weeks ago I spent a full lab session doing something that sounds simple on paper and is genuinely satisfying in practice: taking a packed, obfuscated piece of malware and peeling back every layer until I could see what it actually does. This post is my write-up of that session. It's long, because the lab itself covered a lot of ground — static analysis, manual unpacking with a debugger, multi-stage shellcode extraction, code injection patterns, API hooking, and finally memory forensics with Volatility. I'm documenting it the way I wish more "intro to malware analysis" posts were documented: with the actual commands, the actual reasoning behind each step, and the dead ends along the way. If you're getting into reverse engineering or malware analysis, this should give you a realistic feel for what a packed-malware investigation actually looks like end to end — not just the highlight reel. A quick but important note: this entire exercise was done on isolated, throwaway virtual machines a Windows analysis VM with no network access beyond an internal isolated segment, plus a REMnux Linux VM using known teaching specimens. Never run unknown executables, run unpackers, or experiment with shellcode on a machine connected to a real network or containing real data. Everything here assumes a fully isolated, snapshot-able VM setup. Modern malware rarely ships as a plain, readable executable. Authors wrap their code in packers like UPX to shrink the file and make static analysis harder, and they layer in techniques like shellcode, code injection, and API hooking to evade detection and persist on a system. As an analyst, your job is to answer a chain of questions: This walkthrough tackles that chain using three teaching specimens: a UPX-packed sample brbbot.exe , a multi-technology dropper that chains JavaScript → PowerShell → shellcode PDFXCview.exe , and a code-injecting, API-hooking sample analyzed both statically and via a memory image great.exe / great.vmem . Two VMs, both reverted to clean snapshots before starting: pescanner.py , diec , strings , SpiderMonkey js , base64dump.py , Volatility vol.py Both VMs were on an isolated internal network segment so the Windows VM and REMnux VM could talk to each other for file transfer and the JavaScript dropper's local web server without any route to the internet. First pass: load the suspicious binary in PeStudio and check three things — imports, section names, and strings. A packed file typically shows: .text , .rdata , .data , you'll see something like UPX0 and UPX1 All three were true here, and the UPX naming convention in the section headers was a strong hint about which packer was used. On REMnux, pescanner.py measures the entropy of each section. High entropy close to random is a hallmark of compressed or encrypted data: pescanner.py brbbot.exe | more The tool flagged sections as "SUSPICIOUS" — one for unusually high entropy consistent with packed/compressed code , and one with an entropy of exactly 0 because its raw size was 0 — also anomalous for a legitimate section . diec brbbot.exe diec Detect It Easy, command-line version reported UPX as the most likely packer — confirming the hint from the section names. upx -d %AppData%\brbbot.exe This is always worth trying, but it commonly fails on malware, because authors deliberately corrupt the UPX header/footer to block the standard unpacker while leaving the actual UPX decompression stub intact: CantUnpackException: file is possibly modified/hacked/protected; take care Since the automated route was blocked, the next move is manual dumping : let the malware unpack itself in memory at runtime, then dump that unpacked memory image to disk. setdllcharacteristics -d %AppData%\brbbot.exe This flips the DYNAMIC BASE flag in the PE header from 1 to 0. Without this, the binary would load at a randomized base address every run, which makes it harder to find a stable breakpoint address across debugging sessions. With the sample running via a desktop shortcut set to "Run as administrator" , attach Scylla x64 to the process and click Dump . This grabs the in-memory, already-unpacked version of the code. But a raw memory dump alone usually isn't runnable — the Import Address Table IAT is broken, because imports get resolved dynamically and the dump doesn't capture that resolution cleanly. So: Scylla writes a new file with SCY appended to the name e.g., brbbot-dumped SCY.exe — this is the "fixed" version with a repaired import table. Loading the fixed dump back into PeStudio showed more imports than the packed original — a good sign. But running the fixed dump directly produced a different outcome than expected it exited immediately, without dropping the configuration file the real malware drops . This is a useful and realistic lesson : successfully fixing the IAT doesn't guarantee a perfectly runnable standalone binary. Sometimes further reconstruction is needed. Don't take "it loads more imports now" as proof the unpacking job is fully done — verify behavior too. Scylla's automatic dump-and-fix approach doesn't always work cleanly, so it's worth knowing the manual debugger-based path too. Load the packed binary in x64dbg . Scroll through the disassembly until the unpacking stub's instructions end and you hit a long run of zero bytes — that boundary is usually right where the final jump sits: jmp brbbot.140003F94 That 140003F94 target address is the OEP — the address where the real , unpacked program logic begins. Set a breakpoint on the JMP instruction, then run F9 . The process will execute all the unpacking logic and pause right at that breakpoint, immediately before transferring control to the unpacked code. Step over the jump F7 or F8 to land at the OEP — execution is now paused inside the unpacked code. Don't just trust the address — verify it. Right-click in the CPU view and run: Both showing up is good confirmation you're looking at genuinely unpacked code. From x64dbg's Plugins menu: OllyDumpEx → Dump process . Key details: UPX1 section row and enable the MEM WRITE characteristic flag before dumping without write permission flagged, some dumpers won't capture the section properly brbbot dump 64.exe Same logic as before — IAT Autosearch → Get Imports → Fix Dump , pointed at the OllyDumpEx output. Result: a SCY -suffixed file with a repaired import table. Sometimes you don't want to fully unpack a sample — you just want to watch a specific operation happen, like decryption of an embedded configuration. Run the packed binary in x64dbg with no breakpoints set F9 . It unpacks itself into memory and continues normally. In the Memory Map tab, look for memory regions that don't belong to a Windows DLL and have "E" execute in the Protection column. In this sample, two regions matched that profile — the unpacker code region and a second region holding the freshly unpacked code. Right-click the latter and choose Follow in Disassembler . Right-click → Search for → Current Region → Intermodular calls , then filter the results by typing a keyword e.g. Crypt in the search box. This surfaced a call to CryptDecrypt — a strong signal that the malware decrypts an embedded configuration at runtime. Select the instruction right after the CryptDecrypt call the result-checking instruction , and set a hardware breakpoint on execution . Then restart the process Ctrl+F2 and run again F9 . Why restart rather than just continuing? Because the process may have already executed past this point once — restarting guarantees you hit the breakpoint fresh, from the actual entry point, so register/stack state is consistent with a real first-run analysis. Once paused there, the decrypted configuration data is sitting in memory commonly reachable via the stack — ready for inspection, exactly like you would when analyzing the unpacked version of the same family of malware. This is where things get more interesting: a single executable that chains together several different technologies to avoid writing an obviously malicious file to disk. Start Process Monitor capturing, then run the sample. Watch the process tree in Process Hacker : the initial process spawns mshta.exe and powershell.exe , then after roughly a minute or two, spawns a couple of regsvr32.exe processes. Once those appear, terminate the process tree and pause Process Monitor capture — you don't need to let it run indefinitely, you just need enough activity captured to reconstruct the chain. Export the Process Monitor log as CSV, then load it into ProcDOT along with the initial malicious process. ProcDOT generates a visual graph of what touched what — registry keys created, files dropped, and a persistence entry added under the Run autostart key. It also revealed the malware created files with an unusual, randomly-generated extension and a batch file, plus matching registry entries describing how Windows should handle that custom file extension. In Regedit , navigate to: HKEY CURRENT USER\Software\Classes\.<random-extension The Default value there points to another key a random-looking hex string , which under shell\open\command contains the actual command Windows runs. In this lab it looked roughly like: "C:\WINDOWS\system32\mshta.exe" "javascript:...eval IV2u4L ..." This is a classic file-less technique: rather than dropping a .js file, the script content lives in a registry value, and mshta.exe is abused to execute inline JavaScript that reads and eval s it. reg export HKCU\software\<random-key <random-value script.js Transfer with WinSCP, then try SpiderMonkey directly: js -f /usr/share/remnux/objects.js -f script.js This threw an "illegal character" error — the script was UTF-16 encoded, which SpiderMonkey can't parse directly. Fix the encoding first: strings --encoding=l script.js script2.js -l here is lowercase L , not the number 1 — easy typo to make. Then deobfuscate properly: js -f /usr/share/remnux/objects.js -f script2.js script3.js scite script3.js & The deobfuscated script revealed a call resembling Convert ::FromBase64String , with the decoded result handed off to powershell.exe — meaning the JavaScript's whole job was to decode and launch a Base64-encoded PowerShell stage. base64dump.py script3.js This lists every candidate Base64 blob found, each with an ID. Look in the Decoded column for the largest entry that decodes into readable ASCII — that's almost always the real payload, as opposed to short incidental Base64-looking noise. base64dump.py script3.js -s 10 -d script.ps1 -s 10 selects that specific entry's ID — yours will likely be a different number. Transfer script.ps1 back to Windows and open in Notepad++. The pattern here is a textbook shellcode loader: $sc32 holds hex-encoded shellcode VirtualAlloc allocates memory with PAGE EXECUTE READWRITE CreateThread is called, pointing at the shellcode's address, to execute itRecognizing this pattern is genuinely useful — it shows up constantly across unrelated malware families because it's the simplest way to run raw shellcode from a scripting language. powershell ise script.ps1 Set a breakpoint on the line right after $sc32 is assigned before $pr gets defined , run to it Debug → Run/Continue , then once paused, dump the variable's contents to a raw binary file: io.file ::WriteAllBytes 'sc32.bin',$sc32 Now you have the raw shellcode isolated in its own file, ready for dedicated shellcode analysis tools. scdbg.exe -f sc32.bin Or via the GUI: load the file, leave default options, click Launch. scdbg emulates the shellcode's likely API calls without actually executing it dangerously. Here it showed the code loading advapi32.dll and calling RegOpenKeyExA against both HKEY LOCAL MACHINE and HKEY CURRENT USER — useful, but it didn't reveal which specific registry keys were targeted. jmp2it sc32.bin 0x0 pause 0x0 means the shellcode starts at offset zero in the file. The pause argument makes jmp2it insert an infinite loop before jumping into the shellcode, buying you time to attach a debugger before anything actually runs. Attach x32dbg to the jmp2it process, run briefly, then pause — you'll land inside the infinite loop jmp2it created. The shellcode in this case expected a parameter its own memory address to be pushed onto the stack before it starts, mimicking how the PowerShell loader called it via CreateThread . Since jmp2it happens to store that address in the EDI register, you can satisfy that expectation by patching the infinite-loop instruction: push edi This single patched instruction is what lets the shellcode run as if it had been called the same way the original loader called it. SetBPX advapi32.RegOpenKeyExA Run F9 to hit it, then check the Call Stack tab for the first frame that isn't inside a Windows DLL — that's the shellcode's own calling code. Following that call stack entry back into the disassembler showed, a short distance later, a call to VirtualAlloc — a strong hint that this shellcode unpacks another payload into memory, just like the outer executable did. Set a breakpoint on VirtualAlloc itself: SetBPX VirtualAlloc The pattern that emerged from hitting this breakpoint multiple times: Each time, right-clicking EAX which holds the returned memory address → Follow in Dump → Dump 1/Dump 2 lets you watch that specific memory region fill in over successive breakpoint hits. Once the third allocation showed clear PE-file characteristics, right-click that dump pane → Follow in Memory Map , then right-click the corresponding row → Dump Memory to File . That gives you a final extracted executable, ready to load into PeStudio to confirm it's a structurally valid PE file with imports and strings. Switching specimens here — a sample that injects code into other running processes. In IDA, jump to the Imports tab and locate CreateRemoteThread . Double-click it, then in the disassembler view, select it and press x to bring up cross-references. This shows every place in the code that calls this function. CreateRemoteThread takes a process handle hProcess as a parameter. Tracing that register backward through the disassembly led to a call to OpenProcess — the function that obtains a handle to an existing process by PID. This is the classic injection setup: get a handle to a target process, then create a thread inside it. A separate function call visible just before the CreateRemoteThread call, taking the same process handle as a parameter turned out to contain calls to WriteProcessMemory — the actual mechanism for placing code into another process's address space. A useful shortcut here: rather than manually walking every function called from that one, IDA's View → Graphs → Xrefs from generates a call graph showing everything reachable from a given function. That graph surfaced exactly which sub-function calls VirtualAllocEx — the memory allocation step that has to happen in the target process before you can write to it. At the VirtualAllocEx call site, the flProtect parameter being pushed was 0x40 . Right-clicking that value in IDA and choosing "Use standard symbolic constant" reveals it as PAGE EXECUTE READWRITE — memory that can be written to and executed. That combination, allocated in someone else's process, is the textbook signature of code injection intent. Walking back further up the call chain using IDA's back-arrow navigation led to a function that calls CreateToolhelp32Snapshot , which — combined with Process32FirstW / Process32NextW — is the standard Windows API trio for enumerating every running process. That's the malware searching for a suitable target before injecting into it. | Function called | Role in the injection chain | |---|---| CreateToolhelp32Snapshot + Process32FirstW/NextW | Enumerate running processes to pick a target | OpenProcess | Get a handle to the chosen target process | VirtualAllocEx | Allocate executable+writable memory inside the target | WriteProcessMemory | Write the payload into that allocated memory | CreateRemoteThread | Start execution of the injected code | Same specimen, different capability: modifying other functions in memory so calls to them get redirected. Following cross-references to ReadProcessMemory same Imports-tab → xrefs approach as before led to a function that reads memory from a target process — almost always the first step before overwriting something, since you typically want to preserve the original bytes you're about to clobber. The same function later calls WriteProcessMemory twice, with two different byte patterns: 0xE9 — the opcode for a relative JMP instruction 0x68 the start of a PUSH instruction , paired with a 0xC3 RET written five bytes laterThe second pattern — PUSH followed by RET — is a sneakier alternative to a plain JMP for redirecting execution, since it doesn't look like an obvious jump instruction at a glance. Walking the call chain upward xrefs again eventually reaches a function that builds a table of function addresses — saving various API addresses into memory, one after another, to be passed as the list of functions to hook. The functions referenced there were largely browser-related, suggesting the malware's actual goal: intercepting and observing the victim's web browsing activity. Final piece: instead of analyzing a live process or a static file, this works from a memory snapshot .vmem captured from an already-infected machine. vol.py -f great.vmem kdbgscan | more This suggests one or more candidate OS profiles. The first suggestion isn't guaranteed to be correct — try it, and if Volatility throws errors like "need base" or "No Base Address Space" , that profile doesn't match and you move to the next candidate: vol.py -f great.vmem --profile=Win10x86 pslist Once a profile returns clean, readable process output instead of errors, lock it in for the rest of the session: export VOLATILITY PROFILE=Win10x86 pslist output itself is worth scanning closely here — a process with an unusual, non-standard-looking name stood out immediately as worth investigating further. vol.py -f great.vmem cmdline | more This surfaced a cmd.exe invocation running a batch file out of %Temp% with a randomized filename — code running from the Temp folder with a random name is a strong red flag on its own. vol.py -f great.vmem memdump -p <PID -D /tmp strings /tmp/<PID .dmp | grep -B3 -A3 <batch-filename The surrounding strings matched typical batch-file syntax — consistent with a self-deleting cleanup script delete the dropped executable, then delete itself , a very common malware self-cleanup pattern. vol.py -f great.vmem malfind -D /tmp malfind.txt scite malfind.txt & malfind scans the entire memory image for telltale signs of injected code executable memory regions with suspicious characteristics, frequently starting with the MZ signature of a PE header and dumps each one it finds. In this case it flagged several legitimate-looking processes — explorer.exe and a couple of others — as containing injected PE content, each at a different memory address. Several of the dumped files were exactly the same size, hinting they're likely the same payload injected repeatedly into different processes. A quick static check on one of those dumped files with a couple of additional command-line tools string extraction, automated triage turned up the same suspicious indicators seen earlier — references to a known risky DLL associated with silent file downloads, and string patterns matching the cleanup batch file extracted earlier. That overlap is good corroborating evidence that this is the same malware family operating across multiple injected processes. vol.py -f great.vmem apihooks -p <PID --skip-kernel apihooks.txt Scrolling past the IAT-based entries commonly false positives to the first Inline/Trampoline entry revealed a hooked ntdll.dll LdrLoadDll , patched with the same PUSH / RET redirection technique identified earlier via static analysis — confirming that what was theorized from the binary alone is actually happening at runtime, in memory. The hook redirected execution to a small address range. Using pslist again to find the virtual offset of the specific process being investigated let me narrow down, among all the files malfind had extracted, which one's address range actually encompassed that hook target — confirming exactly which extracted memory dump contains the code the hijacked function jumps into. A checklist for confirming each major milestone in this kind of analysis: scdbg first before live-running it, even in an isolated VM apihooks when both are available malfind , apihooks , pslist should agree with each otherA few things stuck with me after this session: | Mistake | Why It Happens | How to Avoid It | |---|---|---| | Assuming UPX or any packer can always be unpacked with the standard tool | Authors deliberately corrupt headers to break generic unpackers | Always have a manual debugger-based fallback ready | | Forgetting to disable ASLR before debugging | Default behavior on modern Windows | Run setdllcharacteristics -d or equivalent before setting breakpoints by address | | Trusting a dumped file just because PeStudio shows more imports | More imports indicates partial success, not full functional correctness | Actually try running the dumped/fixed binary, and watch for expected side effects dropped files, registry changes | Using strings without --encoding=l on UTF-16 obfuscated scripts | Many obfuscation toolkits output UTF-16 by default | If a deobfuscator throws an encoding/illegal-character error, check the source encoding first | | Picking the wrong Base64 blob from a dump tool's output | Obfuscated scripts often contain several short, irrelevant Base64-looking strings | Sort by decoded size and check for actual readable ASCII content in the decode preview | | Trying the first Volatility profile suggestion and giving up if it errors | kdbgscan often suggests multiple plausible profiles | Treat profile errors as informative, not blocking — try the next suggested profile | Treating IAT hook entries from apihooks as real hooks | IAT-style entries are common false positives in Volatility's hook detection | Specifically look for "Inline/Trampoline" hook type entries, which are far more reliable indicators | | Analyzing shellcode by directly running it without emulating first | Skips a safe verification step | Run through scdbg emulation before live execution, even in an isolated VM | Going from "this file is packed" to "I understand exactly how it injects code, hooks APIs, and what it left behind in memory" took a genuinely long chain of tools and techniques — and that's honestly the most realistic takeaway here. Real malware analysis is rarely a single tool giving you a single clean answer. It's PeStudio pointing you toward a hypothesis, a debugger confirming it, IDA explaining the why , and Volatility proving it actually happened on a real system. If you're working through similar material, my biggest piece of advice is: don't skip the verification steps. It's tempting to declare victory the moment a tool produces some output, but the real confidence comes from cross-checking — static findings against dynamic behavior, debugger observations against memory forensics, one tool's output against another's. If you found this useful, I'm planning to keep documenting more of this kind of hands-on analysis work — let me know in the comments if there's a specific technique here you'd like a deeper dive into.