30-Second Linux Server Compromise Check Runbook (lastb · auth.log · netstat)

"The server is strangely slow — have we been breached?" — Defense later; 30-second diagnosis now

Load average spikes out of nowhere, an unknown process is eating CPU, and the dashboard shows unfamiliar outbound traffic. In that moment the urgent job is not installing fail2ban — it is first determining whether you are already compromised. Hardening comes after you know whether an incident has occurred.

As of 2026, automated botnets targeting SSH and credential stuffing continue to rise, and cryptocurrency-mining malware that abuses exposed cloud keys leaves distinctive outbound connection patterns. Fortunately, most of those traces surface within 30 seconds if you scan just four axes: logins, authentication logs, network, and processes.

This article keeps concepts to a minimum and instead gives you a one-stop package: command sequences you can paste into a terminal right now, a normal vs. anomalous decision table, and isolation commands to run immediately if you find something.

Scope: Debian/Ubuntu family (auth log /var/log/auth.log) and RHEL/CentOS/Rocky/Alma family (/var/log/secure). For recent distros that use only the systemd journal, journalctl alternatives are included in each section.

Check sequence ① Who connected — last / lastb / who / w

Start by answering who logged in successfully, who hammered failed attempts, and who is attached right now.

# (IPto when)
last -a | head -20

# failed — 1
sudo lastb | head -30

# User IP
who -a

# + run
w

Expected normal result: last -a shows only familiar admin IPs (corporate range, VPN range). lastb is short — a few to a few dozen entries — or empty. who/w show only the session you are working in now.

When it doesn't match (branches):

• If lastb floods the screen with hundreds to thousands of lines → brute-force flood. Move to ② below and always check whether any attempt succeeded.
• If last -a shows unfamiliar country/hosting-range IPs as successful logins → strong compromise suspicion. Go immediately to action ⑤.
• If w shows sessions you did not start (especially root or a service account) from a remote host → possible active intrusion.

lastb reads /var/log/btmp and requires root. If the file is missing, failed-login recording is off; after sudo touch /var/log/btmp, confirm that future records will be kept.

Check sequence ② Authentication logs — failure floods and suspicious successful IPs

If lastb is the summary, the auth log is the original evidence. No matter how many failures there are, if there is no "Accepted", the defensive line is still holding.

# ── Debian / Ubuntu ──
# failed when
sudo grep "Failed password" /var/log/auth.log | tail -30

# IP failed
sudo grep "Failed password" /var/log/auth.log \
  | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' \
  | sort | uniq -c | sort -rn | head

# ★ : ( IP )
sudo grep "Accepted" /var/log/auth.log | tail -20

# User/ , sudo
sudo grep -E "useradd|new user|new group|sudo:.*COMMAND" /var/log/auth.log | tail

On RHEL/CentOS-family systems, just change the path to /var/log/secure.

# ── RHEL / CentOS / Rocky / Alma ──
sudo grep "Failed password" /var/log/secure | tail -30
sudo grep "Accepted" /var/log/secure | tail -20

If the log file has been rotated or you only have the systemd journal:

#
sudo journalctl -u ssh -u sshd --no-pager | grep -E "Failed|Accepted" | tail -40

Decision points:

• Thousands of Failed password lines from a single IP → simple bot scan. Not an immediate compromise if there are no successes, but still a block target.
• Accepted password/Accepted publickey with an unknown IP + root account combination → highest-priority response.
• An IP that was flooding failures then suddenly shows Accepted → suspected successful credential stuffing. Lock that account immediately.

Check sequence ③ Outbound connections and hidden jobs — ss / netstat / ps / crontab

After a successful intrusion, malware typically creates outbound connections (C2, mining-pool) and plants itself in cron or a service so it survives reboot.

# + (established)
sudo ss -tnp state established

# LISTEN +
sudo ss -tlnp

# netstat when (net-tools )
sudo netstat -antp

# — /
ps auxf | less

# CPU
ps aux --sort=-%cpu | head

# ( + when )
sudo crontab -l
for u in $(cut -f1 -d: /etc/passwd); do echo "== $u =="; sudo crontab -l -u $u 2>/dev/null; done
ls -la /etc/cron.* /etc/cron.d/ 2>/dev/null
cat /etc/crontab

Expected normal result: The ss established list is mostly known services (80/443 web, 5432 DB, monitoring agents). Cron has only familiar lines such as backups and log rotation.

Risk-signal branches:

• ss shows established outbound connections to high ports such as 4444, 3333, 14444 → suspected mining pool/C2.
• In ps auxf, binaries running from /tmp, /dev/shm, /var/tmp, or random-string names (similar to kdevtmpfsi, xmrig) → isolate immediately.
• crontab with curl ... | bash, wget ... | sh, or base64-encoded lines → reinfection routine. Always record before deleting.

Combined one-shot inspection snippet

If you are short on time, the block below scans all four axes at once. Distro log path is auto-detected.

LOG=$( [ -f /var/log/auth.log ] && echo /var/log/auth.log || echo /var/log/secure ); \
echo "== =="; last -a | head -5; \
echo "== failed IP TOP =="; sudo grep "Failed password" "$LOG" 2>/dev/null \
  | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn | head -5; \
echo "== (Accepted) =="; sudo grep "Accepted" "$LOG" 2>/dev/null | tail -5; \
echo "== established =="; sudo ss -tnp state established | head -10; \
echo "== CPU TOP =="; ps aux --sort=-%cpu | head -5; \
echo "== root =="; sudo crontab -l 2>/dev/null

Conclusion: Normal vs. anomalous decision table

Use this table when you are unsure after scanning. If two or more risk signals overlap, treat it as a compromise and move to containment.

Watch for false positives — these are normal

• Normal backup cron: rsync, pg_dump, and tar night jobs can spike CPU and network briefly.
• Monitoring-agent outbound: Datadog, New Relic, Prometheus remote write, and CloudWatch agents keep established outbound connections continuously.
• Package-mirror access: External connections at apt/yum auto-update time.
• Cloud metadata: Connections to 169.254.169.254 are normal.

The point is not finding "unfamiliar" things but finding "unexplained" things. If you look at a process, connection, or cron job and cannot answer "why is this here?", treat it as a risk signal.

Immediate actions if found — copy-paste order

If compromise is confirmed, proceed in this order: network isolation → session termination → account lock → key revocation → evidence preservation. Keep the order so you do not wipe logs in a rush.

# 1) / IP ·
sudo iptables -A INPUT -s <IP> -j DROP
sudo iptables -A OUTPUT -d <C2/IP> -j DROP

# 2) quit
sudo pkill -KILL -u <> # User
sudo kill -9 <PID> #

# 3) (· )
sudo passwd -l <>
sudo usermod -s /usr/sbin/nologin <>

# 4) SSH — when backup
sudo cp /home/<user>/.ssh/authorized_keys /root/ir_authorized_keys.$(date +%s).bak
sudo cat /home/<user>/.ssh/authorized_keys # check
# check after or file
sudo : > /home/<user>/.ssh/authorized_keys # when

# 5) ( after)
sudo crontab -l -u <user> > /root/ir_cron_<user>.bak
sudo crontab -r -u <user>

If you need full network isolation, it is more reliable to switch the security group/firewall to "deny all" in the cloud console. Leave an allow rule for your own IP so you do not cut off SSH management access.

Recurrence prevention (separate follow-up): After removing the root cause, consider disabling password authentication, using key-based auth, and introducing fail2ban. This article is detection and diagnosis only, so continue hardening in a separate guide. A confirmed-compromised server is hard to fully re-trust; for important assets, preserve a snapshot and then reprovision.

Frequently asked questions (FAQ)

Q. lastb has thousands of entries — is the server already compromised? A. A large number of failures (lastb) by itself can just be everyday bot-scan noise. The decision criterion is whether any succeeded. If grep "Accepted" /var/log/auth.log (or /var/log/secure) shows no successes from unfamiliar IPs, the defensive line is still holding. You should still block those IPs with iptables.

Q. There is no auth.log and no secure either. Where should I look? A. Recent distros sometimes use only the systemd journal instead of text logs. You can get the same information with sudo journalctl -u sshd --no-pager | grep -E "Failed|Accepted". Journal retention depends on the settings in /etc/systemd/journald.conf.

Q. Are processes running from /tmp always malicious? A. Most legitimate services run from /usr/bin, /opt, and similar paths. Binaries executing from /tmp, /dev/shm, or /var/tmp — especially with random-string names — are a classic miner/backdoor pattern and should be treated as strongly suspicious. Some CI runners or build tools do use temp paths, so also check the parent process and the executable path (ls -l /proc/<pid>/exe) before deciding.