30-Second Linux Server Compromise Check Runbook (lastb · auth.log · netstat)

"The server is strangely slow — have we been breached?" — Defense later; 30-second diagnosis now

Load average spikes out of nowhere, an unknown process is eating CPU, and the dashboard shows unfamiliar outbound traffic. In that moment the urgent job is not installing fail2ban — it is first determining whether you are already compromised. Hardening comes after you know whether an incident has occurred.

As of 2026, automated botnets targeting SSH and credential stuffing continue to rise, and cryptocurrency-mining malware that abuses exposed cloud keys leaves distinctive outbound connection patterns. Fortunately, most of those traces surface within 30 seconds if you scan just four axes: logins, authentication logs, network, and processes.

This article keeps concepts to a minimum and instead gives you a one-stop package: command sequences you can paste into a terminal right now, a normal vs. anomalous decision table, and isolation commands to run immediately if you find something.

Scope: Debian/Ubuntu family (auth log /var/log/auth.log) and RHEL/CentOS/Rocky/Alma family (/var/log/secure). For recent distros that use only the systemd journal, journalctl alternatives are included in each section.

Check sequence ① Who connected — last / lastb / who / w

Start by answering who logged in successfully, who hammered failed attempts, and who is attached right now.

# 성공 로그인 이력 (IP까지 표시)
last -a | head -20

# 실패 로그인 이력 — 무차별 대입의 1차 신호
sudo lastb | head -30

# 현재 로그인 중인 사용자와 원격지 IP
who -a

# 현재 세션 + 각 세션이 실행 중인 명령
w

Expected normal result: last -a shows only familiar admin IPs (corporate range, VPN range). lastb is short — a few to a few dozen entries — or empty. who/w show only the session you are working in now.

When it doesn't match (branches):

• If lastb floods the screen with hundreds to thousands of lines → brute-force flood. Move to ② below and always check whether any attempt succeeded.
• If last -a shows unfamiliar country/hosting-range IPs as successful logins → strong compromise suspicion. Go immediately to action ⑤.
• If w shows sessions you did not start (especially root or a service account) from a remote host → possible active intrusion.

lastb reads /var/log/btmp and requires root. If the file is missing, failed-login recording is off; after sudo touch /var/log/btmp, confirm that future records will be kept.

Check sequence ② Authentication logs — failure floods and suspicious successful IPs

If lastb is the summary, the auth log is the original evidence. No matter how many failures there are, if there is no "Accepted", the defensive line is still holding.

# ── Debian / Ubuntu ──
# 실패한 비밀번호 시도 (최근순)
sudo grep "Failed password" /var/log/auth.log | tail -30

# 공격 IP별 실패 횟수 집계 (많은 순)
sudo grep "Failed password" /var/log/auth.log \
  | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' \
  | sort | uniq -c | sort -rn | head

# ★ 가장 중요: 로그인 성공 기록 (여기 낯선 IP가 있으면 비상)
sudo grep "Accepted" /var/log/auth.log | tail -20

# 새 사용자/그룹 추가, sudo 권한 획득 흔적
sudo grep -E "useradd|new user|new group|sudo:.*COMMAND" /var/log/auth.log | tail

On RHEL/CentOS-family systems, just change the path to /var/log/secure.

# ── RHEL / CentOS / Rocky / Alma ──
sudo grep "Failed password" /var/log/secure | tail -30
sudo grep "Accepted" /var/log/secure | tail -20

If the log file has been rotated or you only have the systemd journal:

# 저널 기반 인증 로그 조회
sudo journalctl -u ssh -u sshd --no-pager | grep -E "Failed|Accepted" | tail -40

Decision points:

• Thousands of Failed password lines from a single IP → simple bot scan. Not an immediate compromise if there are no successes, but still a block target.
• Accepted password/Accepted publickey with an unknown IP + root account combination → highest-priority response.
• An IP that was flooding failures then suddenly shows Accepted → suspected successful credential stuffing. Lock that account immediately.

Check sequence ③ Outbound connections and hidden jobs — ss / netstat / ps / crontab

After a successful intrusion, malware typically creates outbound connections (C2, mining-pool) and plants itself in cron or a service so it survives reboot.

# 현재 맺어진 연결 + 프로세스명 (established만)
sudo ss -tnp state established

# LISTEN 중인 모든 포트 + 프로세스 (낯선 고포트 주의)
sudo ss -tlnp

# netstat 선호 시 (net-tools 필요)
sudo netstat -antp

# 프로세스 트리 — 부모 없는/난독화된 프로세스 찾기
ps auxf | less

# CPU 점유 상위 프로세스
ps aux --sort=-%cpu | head

# 예약 작업 점검 (여러 계정 + 시스템 크론 전체)
sudo crontab -l
for u in $(cut -f1 -d: /etc/passwd); do echo "== $u =="; sudo crontab -l -u $u 2>/dev/null; done
ls -la /etc/cron.* /etc/cron.d/ 2>/dev/null
cat /etc/crontab

Expected normal result: The ss established list is mostly known services (80/443 web, 5432 DB, monitoring agents). Cron has only familiar lines such as backups and log rotation.

Risk-signal branches:

• ss shows established outbound connections to high ports such as 4444, 3333, 14444 → suspected mining pool/C2.
• In ps auxf, binaries running from /tmp, /dev/shm, /var/tmp, or random-string names (similar to kdevtmpfsi, xmrig) → isolate immediately.
• crontab with curl ... | bash, wget ... | sh, or base64-encoded lines → reinfection routine. Always record before deleting.

Combined one-shot inspection snippet

If you are short on time, the block below scans all four axes at once. Distro log path is auto-detected.

LOG=$( [ -f /var/log/auth.log ] && echo /var/log/auth.log || echo /var/log/secure ); \
echo "== 성공 로그인 =="; last -a | head -5; \
echo "== 실패 폭주 IP TOP =="; sudo grep "Failed password" "$LOG" 2>/dev/null \
  | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn | head -5; \
echo "== 수상한 성공(Accepted) =="; sudo grep "Accepted" "$LOG" 2>/dev/null | tail -5; \
echo "== 아웃바운드 established =="; sudo ss -tnp state established | head -10; \
echo "== CPU TOP =="; ps aux --sort=-%cpu | head -5; \
echo "== root 크론 =="; sudo crontab -l 2>/dev/null

Conclusion: Normal vs. anomalous decision table

Use this table when you are unsure after scanning. If two or more risk signals overlap, treat it as a compromise and move to containment.

Watch for false positives — these are normal

• Normal backup cron: rsync, pg_dump, and tar night jobs can spike CPU and network briefly.
• Monitoring-agent outbound: Datadog, New Relic, Prometheus remote write, and CloudWatch agents keep established outbound connections continuously.
• Package-mirror access: External connections at apt/yum auto-update time.
• Cloud metadata: Connections to 169.254.169.254 are normal.

The point is not finding "unfamiliar" things but finding "unexplained" things. If you look at a process, connection, or cron job and cannot answer "why is this here?", treat it as a risk signal.

Immediate actions if found — copy-paste order

If compromise is confirmed, proceed in this order: network isolation → session termination → account lock → key revocation → evidence preservation. Keep the order so you do not wipe logs in a rush.

# 1) 공격/의심 IP 인바운드·아웃바운드 차단
sudo iptables -A INPUT  -s <공격IP> -j DROP
sudo iptables -A OUTPUT -d <C2/채굴풀IP> -j DROP

# 2) 침입자 세션 강제 종료
sudo pkill -KILL -u <의심계정>      # 특정 사용자 세션 전체
sudo kill -9 <악성PID>              # 특정 프로세스

# 3) 계정 잠금 (로그인·셸 모두 차단)
sudo passwd -l <의심계정>
sudo usermod -s /usr/sbin/nologin <의심계정>

# 4) 침해 의심 SSH 키 회수 — 삭제 전 반드시 백업(증거)
sudo cp /home/<user>/.ssh/authorized_keys /root/ir_authorized_keys.$(date +%s).bak
sudo cat /home/<user>/.ssh/authorized_keys   # 낯선 키 확인
# 낯선 키 확인 후 해당 라인 제거 또는 파일 비우기
sudo : > /home/<user>/.ssh/authorized_keys   # 전체 회수가 필요할 때

# 5) 크론 백도어 제거 (내용 기록 후)
sudo crontab -l -u <user> > /root/ir_cron_<user>.bak
sudo crontab -r -u <user>

If you need full network isolation, it is more reliable to switch the security group/firewall to "deny all" in the cloud console. Leave an allow rule for your own IP so you do not cut off SSH management access.

Recurrence prevention (separate follow-up): After removing the root cause, consider disabling password authentication, using key-based auth, and introducing fail2ban. This article is detection and diagnosis only, so continue hardening in a separate guide. A confirmed-compromised server is hard to fully re-trust; for important assets, preserve a snapshot and then reprovision.

Frequently asked questions (FAQ)

Q. lastb has thousands of entries — is the server already compromised? A. A large number of failures (lastb) by itself can just be everyday bot-scan noise. The decision criterion is whether any succeeded. If grep "Accepted" /var/log/auth.log (or /var/log/secure) shows no successes from unfamiliar IPs, the defensive line is still holding. You should still block those IPs with iptables.

Q. There is no auth.log and no secure either. Where should I look? A. Recent distros sometimes use only the systemd journal instead of text logs. You can get the same information with sudo journalctl -u sshd --no-pager | grep -E "Failed|Accepted". Journal retention depends on the settings in /etc/systemd/journald.conf.

Q. Are processes running from /tmp always malicious? A. Most legitimate services run from /usr/bin, /opt, and similar paths. Binaries executing from /tmp, /dev/shm, or /var/tmp — especially with random-string names — are a classic miner/backdoor pattern and should be treated as strongly suspicious. Some CI runners or build tools do use temp paths, so also check the parent process and the executable path (ls -l /proc/<pid>/exe) before deciding.