{"slug": "my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30", "title": "My Friend’s Company Got Hacked: How AI Cut 14-Server Triage from 14 Hours to 30 Minutes", "summary": "A mid-sized company's IT staffer, Leo, used an AI-generated Linux incident response checklist to cut triage time for 14 compromised servers from an estimated 14 hours to under 30 minutes, about 2 minutes per machine. The checklist, from the Linux Incident Response Commands toolkit by Ordinary Man Trying, guided him through volatile evidence capture and persistence checks amid a DDoS attack and suspected webshells.", "body_md": "It was past midnight when my phone rang.\n\n“Bro. The website is down. Completely. We’re getting hammered — the whole system is offline. I think we have malware too. I haven’t slept. I don’t know what to do.”\n\nMy friend — let’s call him Leo — works IT for a mid-sized company. 14 servers. Mix of Windows and Linux. Some internal, some public-facing. And right now, all of them were either down or suspect.\n\nHe had already spent hours on manual **Linux incident response** — SSHing into machines one by one, running commands from memory, taking notes. *One hour per machine. 14 machines.* That is a 14-hour server triage nightmare before he even starts fixing anything.\n\n“Can AI help?” he asked. He couldn’t share company details — confidential. But he didn’t need to. The commands don’t care about company secrets.\n\nI pointed him at a Linux incident response checklist I’d been building. He used it that night. Total triage time for all 14 machines: **under 30 minutes**. About 2 minutes per machine instead of 60.\n\n## What Was Actually Happening\n\nTwo things at once — the worst combination:\n\n**1. A DDoS attack** — traffic flooding the public-facing servers until they couldn’t respond. Classic volumetric attack. Site goes down, alarms fire, phones ring at midnight.\n\n**2. Suspected webshells and malware** — the scarier one. Because the DDoS might be a distraction. Attackers flood the front door with noise while quietly planting backdoors through the back. Leo suspected someone had gotten in days earlier and left a persistent foothold.\n\nSo he had two parallel jobs: handle the immediate traffic storm, and run **webshell detection** across 14 machines. Alone. At midnight.\n\n## The Old Way: 1 Hour Per Machine\n\nBefore AI, Leo’s server triage process looked like this:\n\nSSH in → run `ps aux`\n\n→ take notes → run `netstat`\n\n→ take notes → check crontab → check /tmp → check auth.log → grep for webshells manually → document findings → move to next machine.\n\nOne machine, properly checked: **45–90 minutes**. Fourteen machines: **14 hours minimum** — and that is before touching a single fix.\n\n## The AI-Generated Triage: 2 Minutes Per Machine, 30 Minutes Total\n\nThe key insight in Linux incident response is **ordering**: volatile evidence — running processes, open connections, deleted-but-executing files — disappears the moment someone reboots. You capture live state first, then hunt persistence. *Never reboot before running Phase 1.*\n\nEverything below comes from my [Linux Incident Response Commands toolkit](https://ordinarymantrying.com/tools/toolkit/toolkit-linux-ir.html) — 70+ commands in the correct forensic order, one-click copy.\n\n**Phase 1 — Capture live state (volatile, run FIRST)**\n\n```\n# Who is logged in right now — unexpected users are a red flag\nw && who && last | head -20\n\n# Active network connections with process names\n# Look for: unknown outbound IPs, unusual ports, processes you don't recognize\nss -tnp | grep ESTABLISHED\n\n# All running processes with full command lines\n# Look for: base64-encoded strings, /tmp paths, python -c, curl|bash patterns\nps auxf | grep -v '\\[' | head -50\n\n# Files deleted from disk but still running in memory\n# This is a classic attacker anti-forensic technique — legitimate software rarely does this\nlsof +L1 2>/dev/null | head -20\n```\n\n**What suspicious output looks like:** `ss -tnp`\n\nshowing an outbound connection to an unfamiliar IP on port 4444 or 1337 (common reverse shell ports). `lsof +L1`\n\nreturning any entries at all. `ps auxf`\n\nshowing `python3 -c 'import socket...'`\n\nor a process running from `/tmp/`\n\n.\n\n**Phase 2 — Persistence check**\n\n```\n# Cron jobs for every user — attackers add cron to re-establish access after cleanup\nfor user in $(cut -f1 -d: /etc/passwd); do echo \"=== $user ===\"; crontab -u $user -l 2>/dev/null; done\n\n# SSH authorized_keys across all users — extra keys = backdoor\nfind /home /root -name \"authorized_keys\" -exec echo \"FILE: {}\" \\; -exec cat {} \\;\n\n# Suspicious files in world-writable directories\nfind /tmp /var/tmp /dev/shm -type f -newer /etc/passwd 2>/dev/null\n\n# New SUID binaries (shouldn't exist if you didn't create them)\nfind / -perm -4000 -newer /etc/passwd -type f 2>/dev/null\n```\n\n**What suspicious output looks like:** A cron job running a curl command or downloading from a remote URL. An `authorized_keys`\n\nfile with a key you don’t recognize. Any file in `/dev/shm`\n\n— this is a RAM-backed filesystem attackers use specifically because it leaves no disk trace.\n\n**Phase 3 — Webshell detection (web servers only)**\n\n```\n# PHP webshell grep — catches the most common backdoor patterns\nfind /var/www -name \"*.php\" | xargs grep -l \"eval.*base64\\|assert.*\\$_\\|system.*\\$_POST\\|passthru\\|shell_exec\" 2>/dev/null\n\n# PHP files modified in the last 7 days — compare against known deployment dates\nfind /var/www -name \"*.php\" -mtime -7 -ls 2>/dev/null\n\n# Hidden files in upload directories — webshells disguised as images\nfind /var/www/html/wp-content/uploads -name \"*.php\" 2>/dev/null\n```\n\n**What suspicious output looks like:** Leo found `cache_config.php`\n\nin the uploads folder — a PHP file in an uploads directory is almost always a webshell. Any file containing `eval(base64_decode(`\n\nis a confirmed backdoor.\n\nFor the Windows machines, I pointed Leo at a different tool: [a single PowerShell script that auto-runs 12 forensic checks in about 60 seconds](https://ordinarymantrying.com/tools/toolkit/toolkit-win-malware-check.html) — reverse shell detection, WMI persistence, Defender tampering, registry Run keys. He pasted it once per Windows machine and let it run.\n\n## The DDoS Side: What Helps at the Server Level\n\nDDoS mitigation is largely infrastructure — you cannot script your way out of 100Gbps of traffic. But you can act immediately at the server level:\n\n```\n# Identify the top attacking IPs right now\nnetstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20\n\n# Block the worst offenders immediately\niptables -A INPUT -s ATTACKER_IP -j DROP\n\n# Nginx: add rate limiting to slow volumetric floods (put in nginx.conf)\nlimit_req_zone $binary_remote_addr zone=ratelimit:10m rate=10r/s;\n```\n\nFor real protection: Cloudflare free tier absorbs most volumetric attacks. More Nginx and security hardening commands: [Security Hardening Commands toolkit](https://ordinarymantrying.com/tools/toolkit/toolkit-hardening.html).\n\n## By 6am: Two Machines Confirmed Compromised\n\nLeo texted me: “Found a webshell on server 7. PHP file in the uploads folder named ‘cache_config.php’. Classic. Also found a suspicious cron job on server 3 making outbound connections every 5 minutes. Two machines confirmed compromised out of 14.”\n\nThen: “The script ran through all 14 machines in under 30 minutes. Before AI, this would have taken me all day and into tomorrow.”\n\nLater that evening: “One hour per machine is now 2 minutes. The checklist does everything automatically.”\n\nI told him: **learn to use AI yourself. Stop making me your on-call consultant at midnight.** He laughed. Then asked if I had prompts for writing incident reports.\n\n## The Toolkit He Used — Free for Anyone\n\nLeo’s situation is not unique. Any website owner could face this: a WordPress site gets a webshell through an outdated plugin, a VPS gets a cryptominer, a login page gets hammered by a botnet. Most people would have no idea where to start.\n\n— full triage in forensic order: process forensics, persistence detection, rootkit scan. 70+ commands.**Linux Incident Response Commands**— one PowerShell script, 12 auto-checks in 60 seconds.** Windows Malware Scanner**— grep patterns for PHP/JSP backdoors, Behinder/Godzilla signatures.** Webshell Detection Commands**— SSH hardening, iptables, Nginx headers, fail2ban.** Security Hardening Commands**— free, no login, one-click copy.** Full Toolkit — 10 toolkits, 500+ commands**\n\n## The One Rule Leo Keeps Ignoring\n\nThe real value is not any specific command. It is knowing **what to check, in what order, before you panic and reboot something**. Rebooting destroys volatile evidence — the running processes, open connections, and in-memory artifacts that tell you exactly how they got in. Once that is gone, you are rebuilding blind.\n\nLeo texted me two days later: “You should charge for this.” I said: “It is free. But next time, ask AI first. At midnight, I am not available.”\n\n*I put this checklist together using AI to generate and organize the commands — tested and refined based on real incidents. If you find a command that does not work on your system, let me know in the comments.*\n\n### Related Reading\n\n[My Friend Was Hired as a Human Thermometer — AI Fired Him in 20 Minutes](https://ordinarymantrying.com/my-friend-was-hired-as-a-human-thermometer-ai-fired-him-in-20-minutes/)— same friend, earlier AI rescue[Website Admin & Security Command Toolkit](https://ordinarymantrying.com/tools/toolkit/)— all 500+ commands, free", "url": "https://wpnews.pro/news/my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30", "canonical_source": "https://ordinarymantrying.com/friend-company-hacked-14-servers-ai-triage/", "published_at": "2026-08-06 14:34:55+00:00", "updated_at": "2026-08-09 13:07:52.342544+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools"], "entities": ["Leo", "Ordinary Man Trying", "Linux Incident Response Commands toolkit"], "alternates": {"html": "https://wpnews.pro/news/my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30", "markdown": "https://wpnews.pro/news/my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30.md", "text": "https://wpnews.pro/news/my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30.txt", "jsonld": "https://wpnews.pro/news/my-friends-company-got-hacked-how-ai-cut-14-server-triage-from-14-hours-to-30.jsonld"}}