Diagnosing Kubernetes MTU Issues — When ping Works but Large Responses Hang

ping replies 100% of the time, but curl hangs at 40KB

The symptoms in this installment are a different flavor from the previous ones. The connection isn't failing outright — the connection succeeds perfectly, then hangs only past a certain size.

A typical reproduction log looks like this.

# 1) ping
$ kubectl exec -it client-pod -- ping -c 4 10.244.2.15
PING 10.244.2.15 (10.244.2.15) 56(84) bytes of data.
64 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.412 ms
64 bytes from 10.244.2.15: icmp_seq=2 ttl=62 time=0.388 ms
--- 10.244.2.15 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss

# 2) reply ( 200B)
$ kubectl exec -it client-pod -- curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
    http://api-svc/health
200 187

# 3) reply ( 80KB)
$ kubectl exec -it client-pod -- curl -v --max-time 10 http://api-svc/api/list
* Connected to api-svc (10.96.31.7) port 80 (#0)
> GET /api/list HTTP/1.1
> Host: api-svc
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
<
* Operation timed out after 10001 milliseconds with 8192 bytes received

TLS makes it even more confusing. When the certificate chain is large, Client Hello goes out but Server Hello never arrives — it just stalls.

$ kubectl exec -it client-pod -- openssl s_client -connect internal-api:443 -servername internal-api
CONNECTED(00000003)
write to 0x... [0x...] (318 bytes => 318 (0x13E))
# ...

At this point the app team says "the server is slow" and the infra team says "the app isn't producing a response." Both are wrong. The fact that the HTTP headers arrived and part of the body (8192B) was received already points to the answer. The connection and the initial exchange succeeded; one large segment simply couldn't cross the link.

Unlike the all-or-nothing failures covered in earlier posts (connection refused, name resolution failure, missing backends), this class of issue occurs only at a size boundary. That's why reproduction looks flaky and misdiagnosis stretches into days.

The decisive difference from 504 Gateway Time-out

504 is easy to confuse with this. The distinction is clear.

If raising timeouts yields zero improvement, you're in this post's territory.

Symptom fingerprint table: rule out non-MTU causes first

There are causes you should exclude before you suspect MTU.

The conntrack control case for ⑤ looks like this. If you see this log, drop off the MTU path.

$ dmesg -T | grep -i conntrack
[Mon Aug 24 09:12:31 2026] nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144

The topics being ruled out are covered in separate posts. If the connection itself is refused regardless of size, see Endpoints <none> · no endpoints available: 7 Causes and a 30-Second Diagnosis. If health-check issues are shaking at the same time, see K8s Liveness/Readiness probe failed·connection refused — causes and fixes.

30-second identification sequence: binary-search the path MTU

The key is to run this both on the node and inside the Pod. The node takes the physical NIC path; the Pod takes a completely different path through veth → cni0 → tunnel interface. Checking only on the node and declaring things "fine" is the most common misdiagnosis.

Step 1 — DF-bit ping binary search (~10 seconds)

# Pod run. -M do = Don't Fragment
$ kubectl exec -it client-pod -- ping -M do -s 1472 -c 1 10.244.2.15
PING 10.244.2.15 (10.244.2.15) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450

$ kubectl exec -it client-pod -- ping -M do -s 1422 -c 1 10.244.2.15
1430 bytes from 10.244.2.15: icmp_seq=1 ttl=62 time=0.51 ms

Expected healthy result: largest passing -s value + 28 (IP 20 + ICMP 8) = actual path MTU. In the example above, 1422 + 28 = 1450. The interface claims 1500, but only 1450 actually gets through.

When an intermediate device on the path replies, the message is different.

From 10.0.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1450)

If you see this message, you're actually lucky. It means PMTUD is alive, and you are not in the black-hole scenario in section 5.

Step 2 — Find the interface MTU mismatch (~10 seconds)

#
$ ip link show | grep -E '^[0-9]+:|mtu' | grep -E 'eth0|cni0|flannel|vxlan|cilium|tunl'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
7: vethb31a4f2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

This is the culprit. flannel.1 is 1450, but cni0 and veth are 1500. The Pod builds a 1500-byte frame; at VXLAN encapsulation time that becomes 1500 + 50 = 1550, which exceeds the physical NIC's 1500.

A correctly aligned cluster should look like this.

2: eth0: ... mtu 1500
4: flannel.1: ... mtu 1450
5: cni0: ... mtu 1450
7: vethb31a4f2@if3: ... mtu 1450

Always check inside the Pod as well.

$ kubectl exec -it client-pod -- ip link show eth0
3: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP

If the Pod interior is still 1500, then even if you already fixed the CNI config, existing Pods have not picked it up.

Step 3 — Pinpoint the shrink hop on the path

$ kubectl exec -it client-pod -- tracepath 10.244.2.15
 1?: [LOCALHOST]      pmtu 1500
 1:  10.244.1.1       0.132ms
 2:  10.0.1.1         0.418ms pmtu 1450
 3:  10.244.2.15      0.522ms reached
     Resume: pmtu 1450 hops 3 back 3

The hop where the pmtu value drops is the shrink point.

Step 4 — Confirm the hang-start boundary

If you have an endpoint that lets you control response size as a parameter, hit the boundary directly.

$ for n in 500 1000 1400 1500 2000; do
    echo -n "size=$n -> "
    kubectl exec -it client-pod -- curl -s --max-time 5 \
      -o /dev/null -w '%{http_code}\n' "http://api-svc/echo?bytes=$n" || echo TIMEOUT
  done
size=500  -> 200
size=1000 -> 200
size=1400 -> 200
size=1500 -> TIMEOUT
size=2000 -> TIMEOUT

1400 works, 1500 and up die — once you have that boundary, diagnosis is done. Move on to the numbers.

Step 5 — Narrow the segment with a 3-way matrix

Same-node Pods work, but Pods on different nodes don't → confirmed as a tunnel-interface overhead calculation problem. This is the most common pattern.

Per-CNI overhead tables and remediations

The formula is simple.

Pod MTU = underlay link MTU − encapsulation overhead (sum overheads if they stack)

Encapsulation overhead

Cloud underlay MTU × CNI cross-calculation table

This is a typical GCP incident. The default VPC MTU on GCP is 1460 (GCP MTU docs), so if an MTU of 1450 carried over from a 1500 environment is hard-coded, you overrun by 10 bytes. The correct value is 1410. Flannel and Calico compute the MTU from the node interface when you do not set one (Flannel configuration docs), so first check whether a fixed value was ever set.

In a hybrid cluster (AWS jumbo 9001 nodes + on-prem 1500 nodes), you must key off the smallest underlay MTU.

Flannel configuration

$ kubectl edit configmap kube-flannel-cfg -n kube-flannel

{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan",
    "MTU": 1410
  }
}

Restart scope: you need a DaemonSet restart, then a rollout of the workload Pods.

$ kubectl rollout restart daemonset kube-flannel-ds -n kube-flannel
$ kubectl rollout restart deployment -n < > --all

If you restart only the CNI DaemonSet, flannel.1 changes but existing Pod veths stay at 1500. A veth is created when the Pod sandbox is created, so Pods must be recreated for the change to take effect. Missing this step is why people often conclude "I changed the config and it didn't help."

Calico configuration

By default Calico auto-detects the MTU from node configuration and the encapsulation modes you enabled. Set an explicit value only when auto-detection is wrong; the new MTU applies only to new workloads (Calico MTU docs). For operator installs, patch the Installation resource:

$ kubectl patch installation.operator.tigera.io default --type merge \
  -p '{"spec":{"calicoNetwork":{"mtu":1410}}}'

For manifest installs, edit calico-config as below.

$ kubectl patch configmap calico-config -n kube-system \
  --type merge -p '{"data":{"veth_mtu":"1410"}}'

# IPIP MTU
$ kubectl set env daemonset/calico-node -n kube-system FELIX_IPINIPMTU=1440
# VXLAN
$ kubectl set env daemonset/calico-node -n kube-system FELIX_VXLANMTU=1410
$ kubectl rollout restart daemonset calico-node -n kube-system

Restart scope: after restarting calico-node, a workload Pod rollout is mandatory.

Cilium configuration

$ helm upgrade cilium cilium/cilium \
  --namespace kube-system --reuse-values \
  --set MTU=1410

$ kubectl rollout restart daemonset cilium -n kube-system
$ kubectl rollout restart deployment -n < > --all

If you also enable WireGuard node-to-node encryption (encryption.enabled=true, encryption.type=wireguard), you must account for the WireGuard overhead too. The math differs by CNI (Calico does not add it to encapsulation), so list every option that is actually on and confirm with that CNI's docs.

Verify the change took effect

# Pod check
$ kubectl exec -it <-pod> -- ip link show eth0
3: eth0@if11: ... mtu 1410 ...

$ kubectl exec -it <-pod> -- ping -M do -s 1382 -c 1 < Pod IP>
1390 bytes from ...: icmp_seq=1 ttl=62 time=0.44 ms

When you can't change MTU: MSS clamping

If a managed CNI blocks the change, or you can't get change approval, you can force TCP MSS down to the path MTU on the node.

# run (SYN MSS path MTU )
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

# PMTUD
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1370

# check
sudo iptables -t mangle -L FORWARD -n -v | grep TCPMSS

The limitations are clear.

• Applies to TCP only. It rewrites the MSS option on SYN packets, so UDP is out of scope.
• Ineffective for QUIC (HTTP/3). It's UDP-based, so the MSS concept doesn't exist. If you use gRPC over HTTP/3, DoQ, etc., adjusting MTU at the source is the only fix.
• Large DNS (UDP 53) responses and UDP traffic inside VXLAN are also unprotected.
• It disappears on reboot, so persist it with iptables-persistent or a boot script.

Failure branch: you lowered MTU and it's still broken

If you matched the table and even recreated Pods, but the symptom is unchanged, suspect a PMTUD black hole.

Here's the mechanism. The signal an intermediate device uses to say "this packet is too big" is ICMP type 3 code 4 (Destination Unreachable / Fragmentation Needed). If a firewall drops that, the sender never learns to shrink, keeps sending large packets, and they vanish silently. No error log is left behind.

How to confirm

# ICMP
$ sudo tcpdump -ni any 'icmp[icmptype] == 3 and icmp[icmpcode] == 4' -c 5
listening on any, link-type LINUX_SLL (Linux cooked v1)
0 packets captured

If you keep sending large packets and get 0 packets captured, the black hole is confirmed. In a healthy case you should capture something like this.

10:14:22.331 IP 10.0.1.1 > 10.0.2.31: ICMP 10.244.2.15 unreachable -
  need to frag (mtu 1450), length 556

Also check the PMTU cache the kernel has learned.

$ ip route get 10.244.2.15
10.244.2.15 via 10.0.1.1 dev eth0 src 10.0.2.31 mtu 1450

If the mtu field is missing entirely, nothing has been learned.

ICMP policy checkpoints for Korean environments

This often happens because of security policies that block ICMP wholesale. Check the following in order.

If change approval is hard, or out-of-your-control paths are mixed in, MSS clamping is effectively the only practical fix. It locks the size in at connection setup and does not depend on PMTUD. The UDP/QUIC limitations mentioned earlier still apply.

Preventing recurrence

• At CNI install time, pin MTU explicitly. If you leave it to autodetect, it silently drifts on node replacement or cloud changes.
• Put an ip link show MTU-alignment check on the node bootstrap checklist when adding nodes.
• In hybrid clusters, unify on the minimum underlay MTU.
• Add one synthetic monitor that "fetches a response of 8KB or more" so you catch size-boundary regressions early.
• Whenever you newly enable WireGuard or a VPN, recalculate overhead.

Summary checklist

1. Symptom call — ping OK + small request OK + hang only on large response? Did you rule out conntrack and NetworkPolicy?
2. 3-way matrix — where does it break: same node / Pod↔node / node↔node?
3. Plug into the table — underlay MTU − encapsulation overhead (sum if stacked) = target Pod MTU
4. Apply the config — CNI config change + DaemonSet restart + workload Pod rollout
5. Check PMTUD — does ICMP type 3 code 4 actually arrive? If not, MSS clamping

FAQ

Q. Ping works fine — how can this still be MTU? A. Default ping uses a 56-byte payload (84 bytes total), so it passes on any MTU. MTU problems only show up on large packets that exceed the link limit. You have to set the DF bit and grow the size, as in ping -M do -s 1472, to confirm.

Q. Will raising the curl timeout fix it? A. No. The packet cannot cross the link, so it will never arrive no matter how long you wait. Timeout tuning helps the 504 class, where proxy/backend timeouts are misaligned — a different failure layer from this symptom.

Q. I changed MTU and nothing changed. What did I miss? A. Two classics. First, you restarted only the CNI DaemonSet and never rolled out workload Pods, so existing veths still have the old MTU (check with kubectl exec -- ip link show eth0). Second, ICMP type 3 code 4 is dropped by a firewall, so PMTUD is a black hole — in that case you need MSS clamping.

Sources · checked 2026-10-04

• Calico, Configure MTU — header sizes: IPIP 20 B, VXLAN IPv4 50 B / IPv6 70 B, WireGuard IPv4 60 B / IPv6 80 B; GCE 1460 → IPIP 1440, VXLAN 1410; MTU auto-detection; operator calicoNetwork.mtu
• Flannel, Configuration — MTU calculated automatically and written to subnet.env
• Flannel, Backends — MTU option for the vxlan backend
• Google Cloud, Maximum transmission unit — default VPC MTU 1460