## ping replies 100% of the time, but curl hangs at 40KB The symptoms in this installment are a different flavor from the previous ones. The connection isn't failing outright — **the connection succeeds perfectly, then hangs only past a certain size**. A typical reproduction log looks like this. ```bash # 1) ping 완전 정상 $ kubectl exec -it client-pod -- ping -c 4 10.244.2.15 PING 10.244.2.15 (10.244.2.15) 56(84) bytes of data. 64 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.412 ms 64 bytes from 10.244.2.15: icmp_seq=2 ttl=62 time=0.388 ms --- 10.244.2.15 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss # 2) 작은 응답 정상 (약 200B) $ kubectl exec -it client-pod -- curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \ http://api-svc/health 200 187 # 3) 큰 응답 무한 대기 (약 80KB) $ kubectl exec -it client-pod -- curl -v --max-time 10 http://api-svc/api/list * Connected to api-svc (10.96.31.7) port 80 (#0) > GET /api/list HTTP/1.1 > Host: api-svc > < HTTP/1.1 200 OK < Content-Type: application/json < Transfer-Encoding: chunked < * Operation timed out after 10001 milliseconds with 8192 bytes received ``` TLS makes it even more confusing. When the certificate chain is large, Client Hello goes out but Server Hello never arrives — it just stalls. ```bash $ kubectl exec -it client-pod -- openssl s_client -connect internal-api:443 -servername internal-api CONNECTED(00000003) write to 0x... [0x...] (318 bytes => 318 (0x13E)) # ... 여기서 아무것도 오지 않고 정지 ``` At this point the app team says "the server is slow" and the infra team says "the app isn't producing a response." Both are wrong. The fact that **the HTTP headers arrived and part of the body (8192B) was received** already points to the answer. The connection and the initial exchange succeeded; one large segment simply couldn't cross the link. Unlike the all-or-nothing failures covered in earlier posts (connection refused, name resolution failure, missing backends), this class of issue occurs **only at a size boundary**. That's why reproduction looks flaky and misdiagnosis stretches into days. ### The decisive difference from 504 Gateway Time-out 504 is easy to confuse with this. The distinction is clear. | Distinction | 504 timeout class | This post (MTU class) | |---|---|---| | Failure layer | Proxy/backend timeout mismatch | Packet physically cannot cross the link | | Raising timeouts | Effective (align them and it resolves) | **No effect** (it never arrives no matter how long you wait) | | Response-size dependency | None (a slow query 504s even on a small response) | Yes (small responses always succeed) | | Error shape | Explicit 504 response | Hang with no response → client timeout | **If raising timeouts yields zero improvement, you're in this post's territory.** ## Symptom fingerprint table: rule out non-MTU causes first There are causes you should exclude before you suspect MTU. | Symptom fingerprint | MTU suspicion | Alternate cause and first-check command | |---|---|---| | ① ping OK + small request OK + hang only on large response | **Very high** | Effectively an MTU smoking gun. `ping -M do -s 1472 ` | | ② No response after TLS Client Hello (large cert chain) | **Very high** | Server Hello exceeds MTU. `openssl s_client -connect ...` | | ③ Only between a specific node pair; Pods on the same node are fine | **High** | Only the inter-node encapsulation path is broken. `ip link show \| grep -E 'vxlan\|flannel\|cilium'` | | ④ Only when traffic goes through VPN/WireGuard | **High** | Stacked encryption overhead. `wg show`, `ip link show wg0` | | ⑤ Intermittent and independent of request size | Low | conntrack table full → `dmesg \| grep nf_conntrack` / NetworkPolicy is a size-independent full block | | ⑥ Drops after a fixed elapsed time (time-based, not size-based) | Low | keepalive/idle timeout, ingress proxy buffer. `kubectl describe ingress` | The conntrack control case for ⑤ looks like this. If you see this log, drop off the MTU path. ```bash $ dmesg -T | grep -i conntrack [Mon Aug 24 09:12:31 2026] nf_conntrack: nf_conntrack: table full, dropping packet $ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count = 262144 net.netfilter.nf_conntrack_max = 262144 ``` The topics being ruled out are covered in separate posts. If the connection itself is refused regardless of size, see [Endpoints \ · no endpoints available: 7 Causes and a 30-Second Diagnosis](/blog/endpoints-noneno-endpoints-available-7가지-원인과-30초-진단법). If health-check issues are shaking at the same time, see [K8s Liveness/Readiness probe failed·connection refused — causes and fixes](/blog/k8s-livenessreadiness-probe-failedconnection-refused-원인별-해결). ## 30-second identification sequence: binary-search the path MTU The key is to **run this both on the node and inside the Pod**. The node takes the physical NIC path; the Pod takes a completely different path through veth → cni0 → tunnel interface. Checking only on the node and declaring things "fine" is the most common misdiagnosis. ### Step 1 — DF-bit ping binary search (~10 seconds) ```bash # Pod 안에서 실행. -M do = Don't Fragment 고정 $ kubectl exec -it client-pod -- ping -M do -s 1472 -c 1 10.244.2.15 PING 10.244.2.15 (10.244.2.15) 1472(1500) bytes of data. ping: local error: message too long, mtu=1450 $ kubectl exec -it client-pod -- ping -M do -s 1422 -c 1 10.244.2.15 1430 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.51 ms ``` **Expected healthy result**: largest passing `-s` value + 28 (IP 20 + ICMP 8) = actual path MTU. In the example above, 1422 + 28 = **1450**. The interface claims 1500, but only 1450 actually gets through. When an intermediate device on the path replies, the message is different. ```bash From 10.0.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1450) ``` If you see this message, you're actually lucky. It means **PMTUD is alive**, and you are not in the black-hole scenario in section 5. ### Step 2 — Find the interface MTU mismatch (~10 seconds) ```bash # 노드에서 $ ip link show | grep -E '^[0-9]+:|mtu' | grep -E 'eth0|cni0|flannel|vxlan|cilium|tunl' 2: eth0: mtu 1500 qdisc mq state UP 4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN 5: cni0: mtu 1500 qdisc noqueue state UP 7: vethb31a4f2@if3: mtu 1500 ... ``` **This is the culprit.** `flannel.1` is 1450, but `cni0` and `veth` are 1500. The Pod builds a 1500-byte frame; at VXLAN encapsulation time that becomes 1500 + 50 = 1550, which exceeds the physical NIC's 1500. A correctly aligned cluster should look like this. ``` 2: eth0: ... mtu 1500 4: flannel.1: ... mtu 1450 5: cni0: ... mtu 1450 7: vethb31a4f2@if3: ... mtu 1450 ``` Always check inside the Pod as well. ```bash $ kubectl exec -it client-pod -- ip link show eth0 3: eth0@if7: mtu 1500 qdisc noqueue state UP ``` If the Pod interior is still 1500, then even if you already fixed the CNI config, **existing Pods have not picked it up**. ### Step 3 — Pinpoint the shrink hop on the path ```bash $ kubectl exec -it client-pod -- tracepath 10.244.2.15 1?: [LOCALHOST] pmtu 1500 1: 10.244.1.1 0.132ms 2: 10.0.1.1 0.418ms pmtu 1450 3: 10.244.2.15 0.522ms reached Resume: pmtu 1450 hops 3 back 3 ``` The hop where the `pmtu` value drops is the shrink point. ### Step 4 — Confirm the hang-start boundary If you have an endpoint that lets you control response size as a parameter, hit the boundary directly. ```bash $ for n in 500 1000 1400 1500 2000; do echo -n "size=$n -> " kubectl exec -it client-pod -- curl -s --max-time 5 \ -o /dev/null -w '%{http_code}\n' "http://api-svc/echo?bytes=$n" || echo TIMEOUT done size=500 -> 200 size=1000 -> 200 size=1400 -> 200 size=1500 -> TIMEOUT size=2000 -> TIMEOUT ``` **1400 works, 1500 and up die** — once you have that boundary, diagnosis is done. Move on to the numbers. ### Step 5 — Narrow the segment with a 3-way matrix | Path | Test command | Meaning if it fails | |---|---|---| | Same-node Pod ↔ Pod | `ping -M do -s 1472` to a same-node Pod IP | Fails without going through encapsulation → bridge/veth MTU misconfig | | Pod ↔ node IP | Same command from the Pod to the node IP | veth ↔ host NIC mismatch | | Node ↔ node | Same command from a node to another node's IP | Underlay (cloud VPC / physical switch) MTU problem | **Same-node Pods work, but Pods on different nodes don't** → confirmed as a tunnel-interface overhead calculation problem. This is the most common pattern. ## Per-CNI overhead tables and remediations The formula is simple. ``` Pod MTU = underlay link MTU − encapsulation overhead (sum overheads if they stack) ``` ### Encapsulation overhead | Encapsulation | Overhead | Notes | |---|---|---| | Flannel VXLAN | 50 B | Default | | Flannel host-gw | 0 B | No encapsulation; L2 adjacency required | | Calico IPIP | 20 B | Default mode | | Calico VXLAN | 50 B | IPv4 | | Cilium VXLAN | 50 B | | | Cilium Geneve | 50 B or more | Grows with option headers | | WireGuard encryption | IPv4 60 B / IPv6 80 B | Calico does not add it to encapsulation; subtract only this (per its docs). Check other CNIs' docs | ### Cloud underlay MTU × CNI cross-calculation table | Underlay MTU | Flannel VXLAN | Calico IPIP | Calico VXLAN | Cilium VXLAN | Cilium VXLAN + WireGuard | |---|---|---|---|---|---| | AWS 9001 (jumbo) | 8951 | 8981 | 8951 | 8951 | 8871 (assuming −80) | | AWS/on-prem 1500 | 1450 | 1480 | 1450 | 1450 | 1370 | | GCP 1460 | **1410** | 1440 | 1410 | 1410 | 1330 | | Via VPN 1400 | 1350 | 1380 | 1350 | 1350 | 1270 | This is a typical GCP incident. The default VPC MTU on GCP is 1460 ([GCP MTU docs](https://cloud.google.com/vpc/docs/mtu)), so if an MTU of 1450 carried over from a 1500 environment is hard-coded, you overrun by 10 bytes. The correct value is **1410**. Flannel and Calico compute the MTU from the node interface when you do not set one ([Flannel configuration docs](https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md)), so first check whether a fixed value was ever set. In a hybrid cluster (AWS jumbo 9001 nodes + on-prem 1500 nodes), you must **key off the smallest underlay MTU**. ### Flannel configuration ```bash $ kubectl edit configmap kube-flannel-cfg -n kube-flannel ``` ```json { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan", "MTU": 1410 } } ``` **Restart scope**: you need a DaemonSet restart, then a rollout of the workload Pods. ```bash $ kubectl rollout restart daemonset kube-flannel-ds -n kube-flannel $ kubectl rollout restart deployment -n <앱 네임스페이스> --all ``` If you restart only the CNI DaemonSet, `flannel.1` changes but **existing Pod veths stay at 1500**. A veth is created when the Pod sandbox is created, so Pods must be recreated for the change to take effect. Missing this step is why people often conclude "I changed the config and it didn't help." ### Calico configuration By default Calico auto-detects the MTU from node configuration and the encapsulation modes you enabled. Set an explicit value only when auto-detection is wrong; the new MTU applies only to new workloads ([Calico MTU docs](https://docs.tigera.io/calico/latest/networking/configuring/mtu)). For operator installs, patch the Installation resource: ```bash $ kubectl patch installation.operator.tigera.io default --type merge \ -p '{"spec":{"calicoNetwork":{"mtu":1410}}}' ``` For manifest installs, edit `calico-config` as below. ```bash $ kubectl patch configmap calico-config -n kube-system \ --type merge -p '{"data":{"veth_mtu":"1410"}}' # IPIP 모드 터널 MTU $ kubectl set env daemonset/calico-node -n kube-system FELIX_IPINIPMTU=1440 # VXLAN 모드 $ kubectl set env daemonset/calico-node -n kube-system FELIX_VXLANMTU=1410 $ kubectl rollout restart daemonset calico-node -n kube-system ``` **Restart scope**: after restarting `calico-node`, a workload Pod rollout is mandatory. ### Cilium configuration ```bash $ helm upgrade cilium cilium/cilium \ --namespace kube-system --reuse-values \ --set MTU=1410 $ kubectl rollout restart daemonset cilium -n kube-system $ kubectl rollout restart deployment -n <앱 네임스페이스> --all ``` If you also enable WireGuard node-to-node encryption (`encryption.enabled=true`, `encryption.type=wireguard`), you must account for the WireGuard overhead too. The math differs by CNI (Calico does not add it to encapsulation), so list every option that is actually on and confirm with that CNI's docs. ### Verify the change took effect ```bash # 새로 뜬 Pod에서 확인 $ kubectl exec -it <신규-pod> -- ip link show eth0 3: eth0@if11: ... mtu 1410 ... $ kubectl exec -it <신규-pod> -- ping -M do -s 1382 -c 1 <상대 Pod IP> 1390 bytes from ...: icmp_seq=1 ttl=62 time=0.44 ms ``` ### When you can't change MTU: MSS clamping If a managed CNI blocks the change, or you can't get change approval, you can force TCP MSS down to the path MTU on the node. ```bash # 모든 노드에서 실행 (SYN 패킷의 MSS를 경로 MTU에 맞춤) sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \ -j TCPMSS --clamp-mss-to-pmtu # PMTUD가 아예 안 되는 환경이면 고정값으로 sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \ -j TCPMSS --set-mss 1370 # 확인 sudo iptables -t mangle -L FORWARD -n -v | grep TCPMSS ``` **The limitations are clear.** - **Applies to TCP only.** It rewrites the MSS option on SYN packets, so UDP is out of scope. - **Ineffective for QUIC (HTTP/3).** It's UDP-based, so the MSS concept doesn't exist. If you use gRPC over HTTP/3, DoQ, etc., adjusting MTU at the source is the only fix. - Large DNS (UDP 53) responses and UDP traffic inside VXLAN are also unprotected. - It disappears on reboot, so persist it with `iptables-persistent` or a boot script. ## Failure branch: you lowered MTU and it's still broken If you matched the table and even recreated Pods, but the symptom is unchanged, suspect a PMTUD black hole. Here's the mechanism. The signal an intermediate device uses to say "this packet is too big" is **ICMP type 3 code 4 (Destination Unreachable / Fragmentation Needed)**. If a firewall drops that, the sender never learns to shrink, keeps sending large packets, and they vanish silently. No error log is left behind. ### How to confirm ```bash # 송신 노드에서 ICMP 수신 여부 관찰 $ sudo tcpdump -ni any 'icmp[icmptype] == 3 and icmp[icmpcode] == 4' -c 5 listening on any, link-type LINUX_SLL (Linux cooked v1) 0 packets captured ``` If you keep sending large packets and get **0 packets captured**, the black hole is confirmed. In a healthy case you should capture something like this. ``` 10:14:22.331 IP 10.0.1.1 > 10.0.2.31: ICMP 10.244.2.15 unreachable - need to frag (mtu 1450), length 556 ``` Also check the PMTU cache the kernel has learned. ```bash $ ip route get 10.244.2.15 10.244.2.15 via 10.0.1.1 dev eth0 src 10.0.2.31 mtu 1450 ``` If the `mtu` field is missing entirely, nothing has been learned. ### ICMP policy checkpoints for Korean environments This often happens because of security policies that block ICMP wholesale. Check the following in order. | Check target | Checkpoint | |---|---| | NCP ACG | Whether inbound/outbound rules include an allow for the ICMP protocol | | NCP Network ACL | Whether ICMP is set to Deny in the subnet-level ACL | | KT Cloud firewall | Whether a firewall policy has an ICMP allow rule; check VPC-to-VPC paths separately | | AWS Security Group | Whether `ICMP - Destination Unreachable (type 3)` is allowed | | Internal firewall / UTM | Blanket ICMP-deny policy; whether type 3 code 4 is excepted | | DPI / IPS appliances | Selective drops caused by ICMP payload inspection | If change approval is hard, or out-of-your-control paths are mixed in, **MSS clamping is effectively the only practical fix**. It locks the size in at connection setup and does not depend on PMTUD. The UDP/QUIC limitations mentioned earlier still apply. ### Preventing recurrence - At CNI install time, **pin MTU explicitly**. If you leave it to autodetect, it silently drifts on node replacement or cloud changes. - Put an `ip link show` MTU-alignment check on the node bootstrap checklist when adding nodes. - In hybrid clusters, unify on the **minimum underlay MTU**. - Add one synthetic monitor that "fetches a response of 8KB or more" so you catch size-boundary regressions early. - Whenever you newly enable WireGuard or a VPN, recalculate overhead. ### Summary checklist 1. **Symptom call** — ping OK + small request OK + hang only on large response? Did you rule out conntrack and NetworkPolicy? 2. **3-way matrix** — where does it break: same node / Pod↔node / node↔node? 3. **Plug into the table** — underlay MTU − encapsulation overhead (sum if stacked) = target Pod MTU 4. **Apply the config** — CNI config change + DaemonSet restart + **workload Pod rollout** 5. **Check PMTUD** — does ICMP type 3 code 4 actually arrive? If not, MSS clamping ## FAQ **Q. Ping works fine — how can this still be MTU?** A. Default ping uses a 56-byte payload (84 bytes total), so it passes on any MTU. MTU problems only show up on large packets that exceed the link limit. You have to set the DF bit and grow the size, as in `ping -M do -s 1472`, to confirm. **Q. Will raising the curl timeout fix it?** A. No. The packet cannot cross the link, so it will never arrive no matter how long you wait. Timeout tuning helps the 504 class, where proxy/backend timeouts are misaligned — a different failure layer from this symptom. **Q. I changed MTU and nothing changed. What did I miss?** A. Two classics. First, you restarted only the CNI DaemonSet and never rolled out workload Pods, so existing veths still have the old MTU (check with `kubectl exec -- ip link show eth0`). Second, ICMP type 3 code 4 is dropped by a firewall, so PMTUD is a black hole — in that case you need MSS clamping. ## Sources · checked 2026-10-04 - [Calico, Configure MTU](https://docs.tigera.io/calico/latest/networking/configuring/mtu) — header sizes: IPIP 20 B, VXLAN IPv4 50 B / IPv6 70 B, WireGuard IPv4 60 B / IPv6 80 B; GCE 1460 → IPIP 1440, VXLAN 1410; MTU auto-detection; operator `calicoNetwork.mtu` - [Flannel, Configuration](https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md) — MTU calculated automatically and written to subnet.env - [Flannel, Backends](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md) — `MTU` option for the vxlan backend - [Google Cloud, Maximum transmission unit](https://cloud.google.com/vpc/docs/mtu) — default VPC MTU 1460

#MSS clamping

Diagnosing Kubernetes MTU Issues — When ping Works but Large Responses Hang