Diagnosing Kubernetes MTU Issues — When ping Works but Large Responses Hang

ping replies 100% of the time, but curl hangs at 40KB

The symptoms in this installment are a different flavor from the previous ones. The connection isn't failing outright — the connection succeeds perfectly, then hangs only past a certain size.

A typical reproduction log looks like this.

# 1) ping 완전 정상
$ kubectl exec -it client-pod -- ping -c 4 10.244.2.15
PING 10.244.2.15 (10.244.2.15) 56(84) bytes of data.
64 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.412 ms
64 bytes from 10.244.2.15: icmp_seq=2 ttl=62 time=0.388 ms
--- 10.244.2.15 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss

# 2) 작은 응답 정상 (약 200B)
$ kubectl exec -it client-pod -- curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
    http://api-svc/health
200 187

# 3) 큰 응답 무한 대기 (약 80KB)
$ kubectl exec -it client-pod -- curl -v --max-time 10 http://api-svc/api/list
* Connected to api-svc (10.96.31.7) port 80 (#0)
> GET /api/list HTTP/1.1
> Host: api-svc
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
<
* Operation timed out after 10001 milliseconds with 8192 bytes received

TLS makes it even more confusing. When the certificate chain is large, Client Hello goes out but Server Hello never arrives — it just stalls.

$ kubectl exec -it client-pod -- openssl s_client -connect internal-api:443 -servername internal-api
CONNECTED(00000003)
write to 0x... [0x...] (318 bytes => 318 (0x13E))
# ... 여기서 아무것도 오지 않고 정지

At this point the app team says "the server is slow" and the infra team says "the app isn't producing a response." Both are wrong. The fact that the HTTP headers arrived and part of the body (8192B) was received already points to the answer. The connection and the initial exchange succeeded; one large segment simply couldn't cross the link.

Unlike the all-or-nothing failures covered in earlier posts (connection refused, name resolution failure, missing backends), this class of issue occurs only at a size boundary. That's why reproduction looks flaky and misdiagnosis stretches into days.

The decisive difference from 504 Gateway Time-out

504 is easy to confuse with this. The distinction is clear.

If raising timeouts yields zero improvement, you're in this post's territory.

Symptom fingerprint table: rule out non-MTU causes first

There are causes you should exclude before you suspect MTU.

The conntrack control case for ⑤ looks like this. If you see this log, drop off the MTU path.

$ dmesg -T | grep -i conntrack
[Mon Aug 24 09:12:31 2026] nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144

The topics being ruled out are covered in separate posts. If the connection itself is refused regardless of size, see kubectl get endpoints <none>·Service connection refused 5-minute diagnosis. If health-check issues are shaking at the same time, see K8s Liveness/Readiness probe failed·connection refused — causes and fixes.

30-second identification sequence: binary-search the path MTU

The key is to run this both on the node and inside the Pod. The node takes the physical NIC path; the Pod takes a completely different path through veth → cni0 → tunnel interface. Checking only on the node and declaring things "fine" is the most common misdiagnosis.

Step 1 — DF-bit ping binary search (~10 seconds)

# Pod 안에서 실행. -M do = Don't Fragment 고정
$ kubectl exec -it client-pod -- ping -M do -s 1472 -c 1 10.244.2.15
PING 10.244.2.15 (10.244.2.15) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450

$ kubectl exec -it client-pod -- ping -M do -s 1422 -c 1 10.244.2.15
1430 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.51 ms

Expected healthy result: largest passing -s value + 28 (IP 20 + ICMP 8) = actual path MTU. In the example above, 1422 + 28 = 1450. The interface claims 1500, but only 1450 actually gets through.

When an intermediate device on the path replies, the message is different.

From 10.0.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1450)

If you see this message, you're actually lucky. It means PMTUD is alive, and you are not in the black-hole scenario in section 5.

Step 2 — Find the interface MTU mismatch (~10 seconds)

# 노드에서
$ ip link show | grep -E '^[0-9]+:|mtu' | grep -E 'eth0|cni0|flannel|vxlan|cilium|tunl'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
7: vethb31a4f2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

This is the culprit. flannel.1 is 1450, but cni0 and veth are 1500. The Pod builds a 1500-byte frame; at VXLAN encapsulation time that becomes 1500 + 50 = 1550, which exceeds the physical NIC's 1500.

A correctly aligned cluster should look like this.

2: eth0: ... mtu 1500
4: flannel.1: ... mtu 1450
5: cni0: ... mtu 1450
7: vethb31a4f2@if3: ... mtu 1450

Always check inside the Pod as well.

$ kubectl exec -it client-pod -- ip link show eth0
3: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP

If the Pod interior is still 1500, then even if you already fixed the CNI config, existing Pods have not picked it up.

Step 3 — Pinpoint the shrink hop on the path

$ kubectl exec -it client-pod -- tracepath 10.244.2.15
 1?: [LOCALHOST]      pmtu 1500
 1:  10.244.1.1       0.132ms
 2:  10.0.1.1         0.418ms pmtu 1450
 3:  10.244.2.15      0.522ms reached
     Resume: pmtu 1450 hops 3 back 3

The hop where the pmtu value drops is the shrink point.

Step 4 — Confirm the hang-start boundary

If you have an endpoint that lets you control response size as a parameter, hit the boundary directly.

$ for n in 500 1000 1400 1500 2000; do
    echo -n "size=$n -> "
    kubectl exec -it client-pod -- curl -s --max-time 5 \
      -o /dev/null -w '%{http_code}\n' "http://api-svc/echo?bytes=$n" || echo TIMEOUT
  done
size=500  -> 200
size=1000 -> 200
size=1400 -> 200
size=1500 -> TIMEOUT
size=2000 -> TIMEOUT

1400 works, 1500 and up die — once you have that boundary, diagnosis is done. Move on to the numbers.

Step 5 — Narrow the segment with a 3-way matrix

Same-node Pods work, but Pods on different nodes don't → confirmed as a tunnel-interface overhead calculation problem. This is the most common pattern.

Per-CNI overhead tables and remediations

The formula is simple.

Pod MTU = underlay link MTU − encapsulation overhead (sum overheads if they stack)

Encapsulation overhead

Cloud underlay MTU × CNI cross-calculation table

This is why incidents are so common on GCP. The underlay is 1460, but if Flannel's default of 1450 is left in place, you overrun by 10 bytes. The correct value is 1410.

In a hybrid cluster (AWS jumbo 9001 nodes + on-prem 1500 nodes), you must key off the smallest underlay MTU. As jumbo frames become the default while on-prem 1500 still mixes in, this class of incident has been on the rise.

Flannel configuration

$ kubectl edit configmap kube-flannel-cfg -n kube-flannel

{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan",
    "MTU": 1410
  }
}

Restart scope: you need a DaemonSet restart, then a rollout of the workload Pods.

$ kubectl rollout restart daemonset kube-flannel-ds -n kube-flannel
$ kubectl rollout restart deployment -n <앱 네임스페이스> --all

If you restart only the CNI DaemonSet, flannel.1 changes but existing Pod veths stay at 1500. A veth is created when the Pod sandbox is created, so Pods must be recreated for the change to take effect. Missing this step is why people often conclude "I changed the config and it didn't help."

Calico configuration

$ kubectl patch configmap calico-config -n kube-system \
  --type merge -p '{"data":{"veth_mtu":"1410"}}'

# IPIP 모드 터널 MTU
$ kubectl set env daemonset/calico-node -n kube-system FELIX_IPINIPMTU=1440
# VXLAN 모드
$ kubectl set env daemonset/calico-node -n kube-system FELIX_VXLANMTU=1410
$ kubectl rollout restart daemonset calico-node -n kube-system

Restart scope: after restarting calico-node, a workload Pod rollout is mandatory.

Cilium configuration

$ helm upgrade cilium cilium/cilium \
  --namespace kube-system --reuse-values \
  --set MTU=1410

$ kubectl rollout restart daemonset cilium -n kube-system
$ kubectl rollout restart deployment -n <앱 네임스페이스> --all

If you also enable WireGuard node-to-node encryption (encryption.enabled=true, encryption.type=wireguard), you must add that overhead on top, per the table above. As eBPF-based CNIs spread and encapsulation options multiply, you need the habit of listing every option that's actually on, then summing.

Verify the change took effect

# 새로 뜬 Pod에서 확인
$ kubectl exec -it <신규-pod> -- ip link show eth0
3: eth0@if11: ... mtu 1410 ...

$ kubectl exec -it <신규-pod> -- ping -M do -s 1382 -c 1 <상대 Pod IP>
1390 bytes from ...: icmp_seq=1 ttl=62 time=0.44 ms

When you can't change MTU: MSS clamping

If a managed CNI blocks the change, or you can't get change approval, you can force TCP MSS down to the path MTU on the node.

# 모든 노드에서 실행 (SYN 패킷의 MSS를 경로 MTU에 맞춤)
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

# PMTUD가 아예 안 되는 환경이면 고정값으로
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1370

# 확인
sudo iptables -t mangle -L FORWARD -n -v | grep TCPMSS

The limitations are clear.

• Applies to TCP only. It rewrites the MSS option on SYN packets, so UDP is out of scope.
• Ineffective for QUIC (HTTP/3). It's UDP-based, so the MSS concept doesn't exist. If you use gRPC over HTTP/3, DoQ, etc., adjusting MTU at the source is the only fix.
• Large DNS (UDP 53) responses and UDP traffic inside VXLAN are also unprotected.
• It disappears on reboot, so persist it with iptables-persistent or a boot script.

Failure branch: you lowered MTU and it's still broken

If you matched the table and even recreated Pods, but the symptom is unchanged, suspect a PMTUD black hole.

Here's the mechanism. The signal an intermediate device uses to say "this packet is too big" is ICMP type 3 code 4 (Destination Unreachable / Fragmentation Needed). If a firewall drops that, the sender never learns to shrink, keeps sending large packets, and they vanish silently. No error log is left behind.

How to confirm

# 송신 노드에서 ICMP 수신 여부 관찰
$ sudo tcpdump -ni any 'icmp[icmptype] == 3 and icmp[icmpcode] == 4' -c 5
listening on any, link-type LINUX_SLL (Linux cooked v1)
0 packets captured

If you keep sending large packets and get 0 packets captured, the black hole is confirmed. In a healthy case you should capture something like this.

10:14:22.331 IP 10.0.1.1 > 10.0.2.31: ICMP 10.244.2.15 unreachable -
  need to frag (mtu 1450), length 556

Also check the PMTU cache the kernel has learned.

$ ip route get 10.244.2.15
10.244.2.15 via 10.0.1.1 dev eth0 src 10.0.2.31 mtu 1450

If the mtu field is missing entirely, nothing has been learned.

ICMP policy checkpoints for Korean environments

This often happens because of security policies that block ICMP wholesale. Check the following in order.

If change approval is hard, or out-of-your-control paths are mixed in, MSS clamping is effectively the only practical fix. It locks the size in at connection setup and does not depend on PMTUD. The UDP/QUIC limitations mentioned earlier still apply.

Preventing recurrence

• At CNI install time, pin MTU explicitly. If you leave it to autodetect, it silently drifts on node replacement or cloud changes.
• Put an ip link show MTU-alignment check on the node bootstrap checklist when adding nodes.
• In hybrid clusters, unify on the minimum underlay MTU.
• Add one synthetic monitor that "fetches a response of 8KB or more" so you catch size-boundary regressions early.
• Whenever you newly enable WireGuard or a VPN, recalculate overhead.

Summary checklist

1. Symptom call — ping OK + small request OK + hang only on large response? Did you rule out conntrack and NetworkPolicy?
2. 3-way matrix — where does it break: same node / Pod↔node / node↔node?
3. Plug into the table — underlay MTU − encapsulation overhead (sum if stacked) = target Pod MTU
4. Apply the config — CNI config change + DaemonSet restart + workload Pod rollout
5. Check PMTUD — does ICMP type 3 code 4 actually arrive? If not, MSS clamping

FAQ

Q. Ping works fine — how can this still be MTU? A. Default ping uses a 56-byte payload (84 bytes total), so it passes on any MTU. MTU problems only show up on large packets that exceed the link limit. You have to set the DF bit and grow the size, as in ping -M do -s 1472, to confirm.

Q. Will raising the curl timeout fix it? A. No. The packet cannot cross the link, so it will never arrive no matter how long you wait. Timeout tuning helps the 504 class, where proxy/backend timeouts are misaligned — a different failure layer from this symptom.

Q. I changed MTU and nothing changed. What did I miss? A. Two classics. First, you restarted only the CNI DaemonSet and never rolled out workload Pods, so existing veths still have the old MTU (check with kubectl exec -- ip link show eth0). Second, ICMP type 3 code 4 is dropped by a firewall, so PMTUD is a black hole — in that case you need MSS clamping.