## "The server is strangely slow — have we been breached?" — Defense later; 30-second diagnosis now Load average spikes out of nowhere, an unknown process is eating CPU, and the dashboard shows unfamiliar outbound traffic. In that moment the urgent job is not installing `fail2ban` — it is first determining **whether you are already compromised**. Hardening comes after you know whether an incident has occurred. As of 2026, automated botnets targeting SSH and credential stuffing continue to rise, and cryptocurrency-mining malware that abuses exposed cloud keys leaves distinctive outbound connection patterns. Fortunately, most of those traces surface within 30 seconds if you scan just four axes: **logins, authentication logs, network, and processes**. This article keeps concepts to a minimum and instead gives you a one-stop package: **command sequences you can paste into a terminal right now**, a **normal vs. anomalous decision table**, and isolation commands to run immediately if you find something. > Scope: Debian/Ubuntu family (auth log `/var/log/auth.log`) and RHEL/CentOS/Rocky/Alma family (`/var/log/secure`). For recent distros that use only the systemd journal, `journalctl` alternatives are included in each section. --- ## Check sequence ① Who connected — last / lastb / who / w Start by answering who logged in successfully, who hammered failed attempts, and who is attached right now. ```bash # 성공 로그인 이력 (IP까지 표시) last -a | head -20 # 실패 로그인 이력 — 무차별 대입의 1차 신호 sudo lastb | head -30 # 현재 로그인 중인 사용자와 원격지 IP who -a # 현재 세션 + 각 세션이 실행 중인 명령 w ``` **Expected normal result**: `last -a` shows only familiar admin IPs (corporate range, VPN range). `lastb` is short — a few to a few dozen entries — or empty. `who`/`w` show only the session you are working in now. **When it doesn't match (branches)**: - If `lastb` floods the screen with hundreds to thousands of lines → brute-force flood. Move to ② below and always check whether any attempt succeeded. - If `last -a` shows **unfamiliar country/hosting-range IPs as successful logins** → strong compromise suspicion. Go immediately to action ⑤. - If `w` shows sessions you did not start (especially `root` or a service account) from a remote host → possible active intrusion. > `lastb` reads `/var/log/btmp` and requires root. If the file is missing, failed-login recording is off; after `sudo touch /var/log/btmp`, confirm that future records will be kept. --- ## Check sequence ② Authentication logs — failure floods and suspicious successful IPs If `lastb` is the summary, the auth log is the **original evidence**. No matter how many failures there are, if there is no "Accepted", the defensive line is still holding. ```bash # ── Debian / Ubuntu ── # 실패한 비밀번호 시도 (최근순) sudo grep "Failed password" /var/log/auth.log | tail -30 # 공격 IP별 실패 횟수 집계 (많은 순) sudo grep "Failed password" /var/log/auth.log \ | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' \ | sort | uniq -c | sort -rn | head # ★ 가장 중요: 로그인 성공 기록 (여기 낯선 IP가 있으면 비상) sudo grep "Accepted" /var/log/auth.log | tail -20 # 새 사용자/그룹 추가, sudo 권한 획득 흔적 sudo grep -E "useradd|new user|new group|sudo:.*COMMAND" /var/log/auth.log | tail ``` On RHEL/CentOS-family systems, just change the path to `/var/log/secure`. ```bash # ── RHEL / CentOS / Rocky / Alma ── sudo grep "Failed password" /var/log/secure | tail -30 sudo grep "Accepted" /var/log/secure | tail -20 ``` If the log file has been rotated or you only have the systemd journal: ```bash # 저널 기반 인증 로그 조회 sudo journalctl -u ssh -u sshd --no-pager | grep -E "Failed|Accepted" | tail -40 ``` **Decision points**: - Thousands of `Failed password` lines from a single IP → simple bot scan. Not an immediate compromise if there are no successes, but still a block target. - `Accepted password`/`Accepted publickey` with an **unknown IP + root account** combination → highest-priority response. - An IP that was flooding failures then suddenly shows `Accepted` → suspected successful credential stuffing. Lock that account immediately. --- ## Check sequence ③ Outbound connections and hidden jobs — ss / netstat / ps / crontab After a successful intrusion, malware typically **creates outbound connections** (C2, mining-pool) and **plants itself in cron or a service** so it survives reboot. ```bash # 현재 맺어진 연결 + 프로세스명 (established만) sudo ss -tnp state established # LISTEN 중인 모든 포트 + 프로세스 (낯선 고포트 주의) sudo ss -tlnp # netstat 선호 시 (net-tools 필요) sudo netstat -antp # 프로세스 트리 — 부모 없는/난독화된 프로세스 찾기 ps auxf | less # CPU 점유 상위 프로세스 ps aux --sort=-%cpu | head # 예약 작업 점검 (여러 계정 + 시스템 크론 전체) sudo crontab -l for u in $(cut -f1 -d: /etc/passwd); do echo "== $u =="; sudo crontab -l -u $u 2>/dev/null; done ls -la /etc/cron.* /etc/cron.d/ 2>/dev/null cat /etc/crontab ``` **Expected normal result**: The `ss` established list is mostly known services (80/443 web, 5432 DB, monitoring agents). Cron has only familiar lines such as backups and log rotation. **Risk-signal branches**: - `ss` shows established outbound connections to high ports such as `4444`, `3333`, `14444` → suspected mining pool/C2. - In `ps auxf`, binaries running from `/tmp`, `/dev/shm`, `/var/tmp`, or random-string names (similar to `kdevtmpfsi`, `xmrig`) → isolate immediately. - `crontab` with `curl ... | bash`, `wget ... | sh`, or base64-encoded lines → reinfection routine. Always record before deleting. --- ## Combined one-shot inspection snippet If you are short on time, the block below scans all four axes at once. Distro log path is auto-detected. ```bash LOG=$( [ -f /var/log/auth.log ] && echo /var/log/auth.log || echo /var/log/secure ); \ echo "== 성공 로그인 =="; last -a | head -5; \ echo "== 실패 폭주 IP TOP =="; sudo grep "Failed password" "$LOG" 2>/dev/null \ | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn | head -5; \ echo "== 수상한 성공(Accepted) =="; sudo grep "Accepted" "$LOG" 2>/dev/null | tail -5; \ echo "== 아웃바운드 established =="; sudo ss -tnp state established | head -10; \ echo "== CPU TOP =="; ps aux --sort=-%cpu | head -5; \ echo "== root 크론 =="; sudo crontab -l 2>/dev/null ``` --- ## Conclusion: Normal vs. anomalous decision table Use this table when you are unsure after scanning. **If two or more risk signals overlap, treat it as a compromise and move to containment.** | Check item | Normal signal | Risk signal | |---|---|---| | `lastb` failure count | A few to tens per hour | Hundreds to thousands, concentrated on specific IPs | | `Accepted` success IP | Familiar admin/VPN ranges | Unfamiliar country or hosting ranges, especially root | | Failure→success flip | None | An IP that was flooding then becomes Accepted | | LISTEN ports | Known service ports | Unidentified high ports, binaries running from `/tmp` | | established outbound | Web, DB, monitoring agents | Ports such as 4444/3333 suspected mining pool/C2 | | Process names | Regular service names, clear parent | Random strings, running from `/tmp` or `/dev/shm` | | root crontab | Familiar lines such as backup/rotation | `curl\|bash`, base64, unfamiliar added lines | ### Watch for false positives — these are normal - **Normal backup cron**: `rsync`, `pg_dump`, and `tar` night jobs can spike CPU and network briefly. - **Monitoring-agent outbound**: Datadog, New Relic, [Prometheus](/blog/uptime-kuma-vs-netdata-vs-prometheus-소규모-서버-모니터링-추천) remote write, and CloudWatch agents keep established outbound connections continuously. - **Package-mirror access**: External connections at `apt`/`yum` auto-update time. - **Cloud metadata**: Connections to `169.254.169.254` are normal. The point is not finding "unfamiliar" things but finding **"unexplained" things**. If you look at a process, connection, or cron job and cannot answer "why is this here?", treat it as a risk signal. ### Immediate actions if found — copy-paste order If compromise is confirmed, proceed in this order: **network isolation → session termination → account lock → key revocation → evidence preservation**. Keep the order so you do not wipe logs in a rush. ```bash # 1) 공격/의심 IP 인바운드·아웃바운드 차단 sudo iptables -A INPUT -s <공격IP> -j DROP sudo iptables -A OUTPUT -d -j DROP # 2) 침입자 세션 강제 종료 sudo pkill -KILL -u <의심계정> # 특정 사용자 세션 전체 sudo kill -9 <악성PID> # 특정 프로세스 # 3) 계정 잠금 (로그인·셸 모두 차단) sudo passwd -l <의심계정> sudo usermod -s /usr/sbin/nologin <의심계정> # 4) 침해 의심 SSH 키 회수 — 삭제 전 반드시 백업(증거) sudo cp /home//.ssh/authorized_keys /root/ir_authorized_keys.$(date +%s).bak sudo cat /home//.ssh/authorized_keys # 낯선 키 확인 # 낯선 키 확인 후 해당 라인 제거 또는 파일 비우기 sudo : > /home//.ssh/authorized_keys # 전체 회수가 필요할 때 # 5) 크론 백도어 제거 (내용 기록 후) sudo crontab -l -u > /root/ir_cron_.bak sudo crontab -r -u ``` > If you need full network isolation, it is more reliable to switch the security group/firewall to "deny all" in the cloud console. Leave an allow rule for your own IP so you do not cut off SSH management access. **Recurrence prevention (separate follow-up)**: After removing the root cause, consider disabling password authentication, using key-based auth, and introducing `fail2ban`. This article is detection and diagnosis only, so continue hardening in a separate guide. A confirmed-compromised server is hard to fully re-trust; for important assets, **preserve a snapshot and then reprovision**. --- ## Frequently asked questions (FAQ) **Q. `lastb` has thousands of entries — is the server already compromised?** A. A large number of failures (`lastb`) by itself can just be everyday bot-scan noise. The decision criterion is **whether any succeeded**. If `grep "Accepted" /var/log/auth.log` (or `/var/log/secure`) shows no successes from unfamiliar IPs, the defensive line is still holding. You should still block those IPs with `iptables`. **Q. There is no auth.log and no secure either. Where should I look?** A. Recent distros sometimes use only the systemd journal instead of text logs. You can get the same information with `sudo journalctl -u sshd --no-pager | grep -E "Failed|Accepted"`. Journal retention depends on the settings in `/etc/systemd/journald.conf`. **Q. Are processes running from `/tmp` always malicious?** A. Most legitimate services run from `/usr/bin`, `/opt`, and similar paths. Binaries executing from `/tmp`, `/dev/shm`, or `/var/tmp` — especially with random-string names — are a classic miner/backdoor pattern and should be treated as strongly suspicious. Some CI runners or build tools do use temp paths, so also check the parent process and the executable path (`ls -l /proc//exe`) before deciding.

#침해대응

리눅스 서버 해킹 흔적 30초 점검 런북 (lastb·auth.log·netstat)