A few weeks ago I spent a full lab session doing something that sounds simple on paper and is genuinely satisfying in practice: taking a packed, obfuscated piece of malware and peeling back every layer until I could see what it actually does.
This post is my write-up of that session. It's long, because the lab itself covered a lot of ground — static analysis, manual unpacking with a debugger, multi-stage shellcode extraction, code injection patterns, API hooking, and finally memory forensics with Volatility. I'm documenting it the way I wish more "intro to malware analysis" posts were documented: with the actual commands, the actual reasoning behind each step, and the dead ends along the way.
If you're getting into reverse engineering or malware analysis, this should give you a realistic feel for what a packed-malware investigation actually looks like end to end — not just the highlight reel.
A quick but important note: this entire exercise was done on isolated, throwaway virtual machines (a Windows analysis VM with no network access beyond an internal isolated segment, plus a REMnux Linux VM) using known teaching specimens. Never run unknown executables, run unpackers, or experiment with shellcode on a machine connected to a real network or containing real data. Everything here assumes a fully isolated, snapshot-able VM setup.
Modern malware rarely ships as a plain, readable executable. Authors wrap their code in packers (like UPX) to shrink the file and make static analysis harder, and they layer in techniques like shellcode, code injection, and API hooking to evade detection and persist on a system. As an analyst, your job is to answer a chain of questions:
This walkthrough tackles that chain using three teaching specimens: a UPX-packed sample (brbbot.exe
), a multi-technology dropper that chains JavaScript → PowerShell → shellcode (PDFXCview.exe
), and a code-injecting, API-hooking sample analyzed both statically and via a memory image (great.exe
/ great.vmem
).
Two VMs, both reverted to clean snapshots before starting:
pescanner.py
, diec
, strings
, SpiderMonkey (js
), base64dump.py
, Volatility (vol.py
)Both VMs were on an isolated internal network segment so the Windows VM and REMnux VM could talk to each other (for file transfer and the JavaScript dropper's local web server) without any route to the internet.
First pass: load the suspicious binary in PeStudio and check three things — imports, section names, and strings.
A packed file typically shows:
.text
, .rdata
, .data
, you'll see something like UPX0
and UPX1
All three were true here, and the UPX
naming convention in the section headers was a strong hint about which packer was used.
On REMnux, pescanner.py
measures the entropy of each section. High entropy (close to random) is a hallmark of compressed or encrypted data:
pescanner.py brbbot.exe | more
The tool flagged sections as "SUSPICIOUS" — one for unusually high entropy (consistent with packed/compressed code), and one with an entropy of exactly 0 (because its raw size was 0 — also anomalous for a legitimate section).
diec brbbot.exe
diec
(Detect It Easy, command-line version) reported UPX as the most likely packer — confirming the hint from the section names.
upx -d %AppData%\brbbot.exe
This is always worth trying, but it commonly fails on malware, because authors deliberately corrupt the UPX header/footer to block the standard unpacker while leaving the actual UPX decompression stub intact:
CantUnpackException: file is possibly modified/hacked/protected; take care!
Since the automated route was blocked, the next move is manual dumping: let the malware unpack itself in memory at runtime, then dump that unpacked memory image to disk.
setdllcharacteristics -d %AppData%\brbbot.exe
This flips the DYNAMIC_BASE
flag in the PE header from 1 to 0. Without this, the binary would load at a randomized base address every run, which makes it harder to find a stable breakpoint address across debugging sessions.
With the sample running (via a desktop shortcut set to "Run as administrator"), attach Scylla x64 to the process and click Dump. This grabs the in-memory, already-unpacked version of the code.
But a raw memory dump alone usually isn't runnable — the Import Address Table (IAT) is broken, because imports get resolved dynamically and the dump doesn't capture that resolution cleanly. So:
Scylla writes a new file with _SCY
appended to the name (e.g., brbbot-dumped_SCY.exe
) — this is the "fixed" version with a repaired import table.
the fixed dump back into PeStudio showed more imports than the packed original — a good sign. But running the fixed dump directly produced a different outcome than expected (it exited immediately, without dropping the configuration file the real malware drops). This is a useful and realistic lesson: successfully fixing the IAT doesn't guarantee a perfectly runnable standalone binary. Sometimes further reconstruction is needed. Don't take "it loads more imports now" as proof the unpacking job is fully done — verify behavior too.
Scylla's automatic dump-and-fix approach doesn't always work cleanly, so it's worth knowing the manual debugger-based path too.
Load the packed binary in x64dbg. Scroll through the disassembly until the unpacking stub's instructions end and you hit a long run of zero bytes — that boundary is usually right where the final jump sits:
jmp brbbot.140003F94
That 140003F94
target address is the OEP — the address where the real, unpacked program logic begins.
Set a breakpoint on the JMP
instruction, then run (F9
). The process will execute all the unpacking logic and right at that breakpoint, immediately before transferring control to the unpacked code.
Step over the jump (F7
or F8
) to land at the OEP — execution is now d inside the unpacked code.
Don't just trust the address — verify it. Right-click in the CPU view and run:
Both showing up is good confirmation you're looking at genuinely unpacked code.
From x64dbg's Plugins menu: OllyDumpEx → Dump process. Key details:
UPX1
section row and enable the MEM_WRITE
characteristic flag before dumping (without write permission flagged, some dumpers won't capture the section properly)brbbot_dump_64.exe
)Same logic as before — IAT Autosearch → Get Imports → Fix Dump, pointed at the OllyDumpEx output. Result: a _SCY
-suffixed file with a repaired import table.
Sometimes you don't want to fully unpack a sample — you just want to watch a specific operation happen, like decryption of an embedded configuration.
Run the packed binary in x64dbg with no breakpoints set (F9
). It unpacks itself into memory and continues normally.
In the Memory Map tab, look for memory regions that don't belong to a Windows DLL and have "E" (execute) in the Protection column. In this sample, two regions matched that profile — the unpacker code region and a second region holding the freshly unpacked code. Right-click the latter and choose Follow in Disassembler.
Right-click → Search for → Current Region → Intermodular calls, then filter the results by typing a keyword (e.g. Crypt
) in the search box. This surfaced a call to CryptDecrypt
— a strong signal that the malware decrypts an embedded configuration at runtime.
Select the instruction right after the CryptDecrypt
call (the result-checking instruction), and set a hardware breakpoint on execution. Then restart the process (Ctrl+F2
) and run again (F9
).
Why restart rather than just continuing? Because the process may have already executed past this point once — restarting guarantees you hit the breakpoint fresh, from the actual entry point, so register/stack state is consistent with a real first-run analysis.
Once d there, the decrypted configuration data is sitting in memory (commonly reachable via the stack) — ready for inspection, exactly like you would when analyzing the unpacked version of the same family of malware.
This is where things get more interesting: a single executable that chains together several different technologies to avoid writing an obviously malicious file to disk.
Start Process Monitor capturing, then run the sample. Watch the process tree in Process Hacker: the initial process spawns mshta.exe
and powershell.exe
, then after roughly a minute or two, spawns a couple of regsvr32.exe
processes. Once those appear, terminate the process tree and Process Monitor capture — you don't need to let it run indefinitely, you just need enough activity captured to reconstruct the chain.
Export the Process Monitor log as CSV, then load it into ProcDOT along with the initial malicious process. ProcDOT generates a visual graph of what touched what — registry keys created, files dropped, and a persistence entry added under the Run
autostart key. It also revealed the malware created files with an unusual, randomly-generated extension and a batch file, plus matching registry entries describing how Windows should handle that custom file extension.
In Regedit, navigate to:
HKEY_CURRENT_USER\Software\Classes\.<random-extension>
The (Default)
value there points to another key (a random-looking hex string), which under shell\open\command
contains the actual command Windows runs. In this lab it looked roughly like:
"C:\WINDOWS\system32\mshta.exe" "javascript:...eval(IV2u4L)..."
This is a classic file-less technique: rather than dropping a .js
file, the script content lives in a registry value, and mshta.exe
is abused to execute inline JavaScript that reads and eval()
s it.
reg_export HKCU\software\<random-key> <random-value> script.js
Transfer with WinSCP, then try SpiderMonkey directly:
js -f /usr/share/remnux/objects.js -f script.js
This threw an "illegal character" error — the script was UTF-16 encoded, which SpiderMonkey can't parse directly. Fix the encoding first:
strings --encoding=l script.js > script2.js
(-l
here is lowercase L, not the number 1 — easy typo to make.)
Then deobfuscate properly:
js -f /usr/share/remnux/objects.js -f script2.js > script3.js
scite script3.js &
The deobfuscated script revealed a call resembling [Convert]::FromBase64String
, with the decoded result handed off to powershell.exe
— meaning the JavaScript's whole job was to decode and launch a Base64-encoded PowerShell stage.
base64dump.py script3.js
This lists every candidate Base64 blob found, each with an ID. Look in the Decoded column for the largest entry that decodes into readable ASCII — that's almost always the real payload, as opposed to short incidental Base64-looking noise.
base64dump.py script3.js -s 10 -d > script.ps1
(-s 10
selects that specific entry's ID — yours will likely be a different number.)
Transfer script.ps1
back to Windows and open in Notepad++. The pattern here is a textbook shellcode :
$sc32
) holds hex-encoded shellcodeVirtualAlloc
allocates memory with PAGE_EXECUTE_READWRITE
CreateThread
is called, pointing at the shellcode's address, to execute itRecognizing this pattern is genuinely useful — it shows up constantly across unrelated malware families because it's the simplest way to run raw shellcode from a scripting language.
powershell_ise script.ps1
Set a breakpoint on the line right after $sc32
is assigned (before $pr
gets defined), run to it (Debug → Run/Continue), then once d, dump the variable's contents to a raw binary file:
[io.file]::WriteAllBytes('sc32.bin',$sc32)
Now you have the raw shellcode isolated in its own file, ready for dedicated shellcode analysis tools.
scdbg.exe -f sc32.bin
(Or via the GUI: load the file, leave default options, click Launch.) scdbg
emulates the shellcode's likely API calls without actually executing it dangerously. Here it showed the code advapi32.dll
and calling RegOpenKeyExA
against both HKEY_LOCAL_MACHINE
and HKEY_CURRENT_USER
— useful, but it didn't reveal which specific registry keys were targeted.
jmp2it sc32.bin 0x0
0x0
means the shellcode starts at offset zero in the file. The ``
argument makes jmp2it
insert an infinite loop before jumping into the shellcode, buying you time to attach a debugger before anything actually runs.
Attach x32dbg to the jmp2it
process, run briefly, then — you'll land inside the infinite loop jmp2it
created. The shellcode in this case expected a parameter (its own memory address) to be pushed onto the stack before it starts, mimicking how the PowerShell called it via CreateThread
. Since jmp2it
happens to store that address in the EDI
register, you can satisfy that expectation by patching the infinite-loop instruction:
push edi
This single patched instruction is what lets the shellcode run as if it had been called the same way the original called it.
SetBPX advapi32.RegOpenKeyExA
Run (F9
) to hit it, then check the Call Stack tab for the first frame that isn't inside a Windows DLL — that's the shellcode's own calling code. Following that call stack entry back into the disassembler showed, a short distance later, a call to VirtualAlloc
— a strong hint that this shellcode unpacks another payload into memory, just like the outer executable did.
Set a breakpoint on VirtualAlloc
itself:
SetBPX VirtualAlloc
The pattern that emerged from hitting this breakpoint multiple times:
Each time, right-clicking EAX
(which holds the returned memory address) → Follow in Dump → Dump 1/Dump 2 lets you watch that specific memory region fill in over successive breakpoint hits.
Once the third allocation showed clear PE-file characteristics, right-click that dump pane → Follow in Memory Map, then right-click the corresponding row → Dump Memory to File. That gives you a final extracted executable, ready to load into PeStudio to confirm it's a structurally valid PE file with imports and strings.
Switching specimens here — a sample that injects code into other running processes.
In IDA, jump to the Imports tab and locate CreateRemoteThread
. Double-click it, then in the disassembler view, select it and press x
to bring up cross-references. This shows every place in the code that calls this function.
CreateRemoteThread
takes a process handle (hProcess
) as a parameter. Tracing that register backward through the disassembly led to a call to OpenProcess
— the function that obtains a handle to an existing process by PID. This is the classic injection setup: get a handle to a target process, then create a thread inside it.
A separate function call (visible just before the CreateRemoteThread
call, taking the same process handle as a parameter) turned out to contain calls to WriteProcessMemory
— the actual mechanism for placing code into another process's address space.
A useful shortcut here: rather than manually walking every function called from that one, IDA's View → Graphs → Xrefs from generates a call graph showing everything reachable from a given function. That graph surfaced exactly which sub-function calls VirtualAllocEx
— the memory allocation step that has to happen in the target process before you can write to it.
At the VirtualAllocEx
call site, the flProtect
parameter being pushed was 0x40
. Right-clicking that value in IDA and choosing "Use standard symbolic constant" reveals it as PAGE_EXECUTE_READWRITE
— memory that can be written to and executed. That combination, allocated in someone else's process, is the textbook signature of code injection intent.
Walking back further up the call chain (using IDA's back-arrow navigation) led to a function that calls CreateToolhelp32Snapshot
, which — combined with Process32FirstW
/Process32NextW
— is the standard Windows API trio for enumerating every running process. That's the malware searching for a suitable target before injecting into it.
| Function called | Role in the injection chain |
|---|---|
CreateToolhelp32Snapshot + Process32FirstW/NextW |
|
| Enumerate running processes to pick a target | |
OpenProcess |
|
| Get a handle to the chosen target process | |
VirtualAllocEx |
|
| Allocate executable+writable memory inside the target | |
WriteProcessMemory |
|
| Write the payload into that allocated memory | |
CreateRemoteThread |
|
| Start execution of the injected code |
Same specimen, different capability: modifying other functions in memory so calls to them get redirected.
Following cross-references to ReadProcessMemory
(same Imports-tab → xrefs approach as before) led to a function that reads memory from a target process — almost always the first step before overwriting something, since you typically want to preserve the original bytes you're about to clobber.
The same function later calls WriteProcessMemory
twice, with two different byte patterns:
0xE9
— the opcode for a relative JMP
instruction0x68
(the start of a PUSH
instruction), paired with a 0xC3
(RET
) written five bytes laterThe second pattern — PUSH
followed by RET
— is a sneakier alternative to a plain JMP
for redirecting execution, since it doesn't look like an obvious jump instruction at a glance.
Walking the call chain upward (xrefs again) eventually reaches a function that builds a table of function addresses — saving various API addresses into memory, one after another, to be passed as the list of functions to hook. The functions referenced there were largely browser-related, suggesting the malware's actual goal: intercepting and observing the victim's web browsing activity.
Final piece: instead of analyzing a live process or a static file, this works from a memory snapshot (.vmem
) captured from an already-infected machine.
vol.py -f great.vmem kdbgscan | more
This suggests one or more candidate OS profiles. The first suggestion isn't guaranteed to be correct — try it, and if Volatility throws errors like "need base"
or "No Base Address Space"
, that profile doesn't match and you move to the next candidate:
vol.py -f great.vmem --profile=Win10x86 pslist
Once a profile returns clean, readable process output instead of errors, lock it in for the rest of the session:
export VOLATILITY_PROFILE=Win10x86
pslist
output itself is worth scanning closely here — a process with an unusual, non-standard-looking name stood out immediately as worth investigating further.
vol.py -f great.vmem cmdline | more
This surfaced a cmd.exe
invocation running a batch file out of %Temp%
with a randomized filename — code running from the Temp folder with a random name is a strong red flag on its own.
vol.py -f great.vmem memdump -p <PID> -D /tmp
strings /tmp/<PID>.dmp | grep -B3 -A3 <batch-filename>
The surrounding strings matched typical batch-file syntax — consistent with a self-deleting cleanup script (delete the dropped executable, then delete itself), a very common malware self-cleanup pattern.
vol.py -f great.vmem malfind -D /tmp > malfind.txt
scite malfind.txt &
malfind
scans the entire memory image for telltale signs of injected code (executable memory regions with suspicious characteristics, frequently starting with the MZ
signature of a PE header) and dumps each one it finds. In this case it flagged several legitimate-looking processes — explorer.exe and a couple of others — as containing injected PE content, each at a different memory address. Several of the dumped files were exactly the same size, hinting they're likely the same payload injected repeatedly into different processes.
A quick static check on one of those dumped files with a couple of additional command-line tools (string extraction, automated triage) turned up the same suspicious indicators seen earlier — references to a known risky DLL associated with silent file downloads, and string patterns matching the cleanup batch file extracted earlier. That overlap is good corroborating evidence that this is the same malware family operating across multiple injected processes.
vol.py -f great.vmem apihooks -p <PID> --skip-kernel > apihooks.txt
Scrolling past the IAT-based entries (commonly false positives) to the first Inline/Trampoline entry revealed a hooked ntdll.dll!LdrLoadDll
, patched with the same PUSH
/RET
redirection technique identified earlier via static analysis — confirming that what was theorized from the binary alone is actually happening at runtime, in memory.
The hook redirected execution to a small address range. Using pslist
again to find the virtual offset of the specific process being investigated let me narrow down, among all the files malfind
had extracted, which one's address range actually encompassed that hook target — confirming exactly which extracted memory dump contains the code the hijacked function jumps into.
A checklist for confirming each major milestone in this kind of analysis:
scdbg
) first before live-running it, even in an isolated VMapihooks
) when both are availablemalfind
, apihooks
, pslist
) should agree with each otherA few things stuck with me after this session:
| Mistake | Why It Happens | How to Avoid It |
|---|---|---|
| Assuming UPX (or any packer) can always be unpacked with the standard tool | Authors deliberately corrupt headers to break generic unpackers | Always have a manual debugger-based fallback ready |
| Forgetting to disable ASLR before debugging | Default behavior on modern Windows | Run setdllcharacteristics -d (or equivalent) before setting breakpoints by address |
| Trusting a dumped file just because PeStudio shows more imports | More imports indicates partial success, not full functional correctness | |
| Actually try running the dumped/fixed binary, and watch for expected side effects (dropped files, registry changes) | ||
Using strings without --encoding=l on UTF-16 obfuscated scripts |
||
| Many obfuscation toolkits output UTF-16 by default | If a deobfuscator throws an encoding/illegal-character error, check the source encoding first | |
| Picking the wrong Base64 blob from a dump tool's output | Obfuscated scripts often contain several short, irrelevant Base64-looking strings | Sort by decoded size and check for actual readable ASCII content in the decode preview |
| Trying the first Volatility profile suggestion and giving up if it errors | ||
kdbgscan often suggests multiple plausible profiles |
||
| Treat profile errors as informative, not blocking — try the next suggested profile | ||
Treating IAT hook entries from apihooks as real hooks |
||
| IAT-style entries are common false positives in Volatility's hook detection | Specifically look for "Inline/Trampoline" hook type entries, which are far more reliable indicators | |
| Analyzing shellcode by directly running it without emulating first | Skips a safe verification step | Run through scdbg (emulation) before live execution, even in an isolated VM |
Going from "this file is packed" to "I understand exactly how it injects code, hooks APIs, and what it left behind in memory" took a genuinely long chain of tools and techniques — and that's honestly the most realistic takeaway here. Real malware analysis is rarely a single tool giving you a single clean answer. It's PeStudio pointing you toward a hypothesis, a debugger confirming it, IDA explaining the why, and Volatility proving it actually happened on a real system.
If you're working through similar material, my biggest piece of advice is: don't skip the verification steps. It's tempting to declare victory the moment a tool produces some output, but the real confidence comes from cross-checking — static findings against dynamic behavior, debugger observations against memory forensics, one tool's output against another's.
If you found this useful, I'm planning to keep documenting more of this kind of hands-on analysis work — let me know in the comments if there's a specific technique here you'd like a deeper dive into.