Kubernetes Network Latency: A Segment-by-Segment Runbook from DNS and kube-proxy to MTU and eBPF

"The code hasn't changed and the Pods have headroom, but service-to-service calls are slow." In Kubernetes, latency like this usually comes from one segment: DNS lookups, Service routing (kube-proxy), overlay networking and MTU, NetworkPolicy, or sidecar proxies. This guide measures each segment in turn and tunes only the one the data points to. eBPF shows up as a tool for both measurement and a faster dataplane.

30-second triage: narrow it down by symptom

Judge by P95 and P99, not the average. A normal average with a spiking P99 means some requests are hitting a specific segment: retransmits, DNS timeouts, or one node's path.

Step 1: Baseline each segment

Before guessing, measure four paths under the same conditions. The segment where the number jumps is your suspect.

1. Node ↔ node (node IPs)
2. Pod ↔ Pod on the same node (Pod IPs)
3. Pod ↔ Pod across nodes (Pod IPs)
4. Pod → Service name (through DNS + ClusterIP)

Pick the tool for what you're measuring:

• Throughput and retransmits: iperf3. It doesn't measure latency. The Retr column is the TCP retransmit count.
  BashCopy

  iperf3 -s                          # server Pod
  iperf3 -c <server_IP> -t 10 -P 4    # client Pod: 10 seconds, 4 parallel streams

- **Request-response latency: `netperf -t TCP_RR`**, or a load tool for your real protocol (`wrk` or `hey` for HTTP). Fix the payload size and iteration count, and record both the average and P99. `ping` only shows ICMP RTT.
- **Per-hop loss: `mtr -n -r -c 100 <target_IP>`.** Intermediate routers often deprioritize ICMP replies, so only loss or latency that **carries through to the final hop** is a real problem.

## Step 2: Check DNS latency

A Pod's `/etc/resolv.conf` usually contains `options ndots:5` and several search domains. Names with fewer than five dots (e.g. `api.example.com`) are tried with each search domain appended first, so one external name can turn into several lookups.

- For frequently called external domains, use a trailing-dot FQDN (`api.example.com.`) or lower `ndots` in the Pod's `dnsConfig.options`. Short in-cluster names are affected too, so test per service before rolling it out.
- Intermittent delays of **almost exactly 5 seconds** are a well-known case: a Linux conntrack race on UDP drops a DNS packet, and the resolver retries after its default 5-second timeout. A per-node DNS cache (NodeLocal DNSCache) shortens this path.
- Also check that CoreDNS Pods aren't CPU-throttled and that there are enough replicas for the traffic.

## Step 3: Service routing (kube-proxy)

ClusterIP traffic goes through rules that kube-proxy programs. The cost depends on the mode:

| Mode | Characteristics | Notes |
| --- | --- | --- |
| `iptables` | Rules grow with Services and endpoints; rule updates slow down too | Default in many clusters |
| `nftables` | Designed for better performance; GA in Kubernetes 1.33 | Recommended by the docs as the alternative to iptables |
| `ipvs` | Hash-based lookup | Deprecated since Kubernetes 1.35 |
| kube-proxy replacement (eBPF) | Cilium and others handle Services directly in eBPF | Provided as a CNI feature |

```bash
kubectl -n kube-system get cm kube-proxy -o yaml | grep -E '^\s*mode:'   # kubeadm; empty means iptables

If direct Pod-IP calls and Service calls show little difference, this segment isn't the cause. Changing the mode affects the whole cluster, so consider it only when Step 1 points here. Reference: Kubernetes: Virtual IPs and Service Proxies.

Step 4: Overlay, MTU and MSS

Overlays such as VXLAN, IP-in-IP and WireGuard add a tunnel header to every packet, so the Pod MTU must be smaller than the node path MTU by that header size. If the Pod MTU is too large, big packets get fragmented or rely on Path MTU Discovery; if a device on the path blocks ICMP "Fragmentation Needed", only large packets silently vanish, creating an MTU black hole.

First confirm from packets whether you're on an overlay. Capture VXLAN ports on the node NIC: if traffic shows up, you're on an encapsulated path. The standard VXLAN port is UDP 4789; Flannel and Cilium default to 8472 on Linux.

sudo tcpdump -ni <node_NIC> 'udp port 4789 or udp port 8472' -c 20

Checks, in order:

1. Interface MTU on nodes and Pods
   BashCopy

   ip -d link show                                  # node: physical NIC, flannel.1, vxlan.calico, tunl0, cilium_vxlan, etc.
   kubectl exec -it <pod> -- ip link show eth0      # Pod (if the image has the ip command)

2. **Measure path MTU:** `-s` is the ICMP payload size, so on IPv4 the real packet size is that plus the 20-byte IP header and 8-byte ICMP header.
   ```bash
   ping -M do -s 1472 -c 3 <target_IP>    # 1472 + 28 = 1500 bytes
   ping -M do -s 1422 -c 3 <target_IP>    # VXLAN: 1450 - 28
   tracepath -n <target_IP>               # PMTU per hop

Test node to node with node IPs, and Pod to Pod from inside a Pod with the other Pod's IP, to see which segment loses MTU. 3. CNI MTU settings: on IPv4 the tunnel overhead is 20 bytes for IP-in-IP, 50 for VXLAN and 60 for WireGuard. The Pod MTU must be at most "node path MTU − header size" (1450 for a 1500 path with VXLAN). Calico MTU docs

• Calico: auto-detected by default. To set it, change spec.calicoNetwork.mtu in the Installation resource (operator installs) or veth_mtu in the calico-config ConfigMap (manifest installs), then restart calico-node.
• Flannel: computes host interface MTU minus backend overhead and writes it as FLANNEL_MTU in /run/flannel/subnet.env on each node. If nodes share an L2 segment, the host-gw backend removes the tunnel entirely. Flannel Backends
• Cilium: Helm value MTU (default 0 = auto-detect). Cilium Helm reference

Running Pods keep the old MTU after a change; recreate them to pick up the new value. 4. MSS clamping (only when needed): where MTU shrinks at a VPN or tunnel boundary and PMTUD is blocked, lower the MSS of SYN packets on the router or gateway at that boundary.

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

MSS isn't a single sysctl. net.ipv4.tcp_mem is the system-wide TCP memory limit and has nothing to do with MTU or MSS.

Step 5: NetworkPolicy and the CNI dataplane

iptables evaluates the rules in a chain in order, so per-packet cost grows with rule count. eBPF dataplanes look things up in structures like hash maps and are less affected by rule growth. At a few dozen policies, though, the difference often isn't measurable. Consider swapping the dataplane only when policy processing is still the suspect after Step 4. Policy design itself is covered in Fixing NetworkPolicy issues: Calico vs Cilium.

Step 6: Service mesh sidecars

A sidecar (Envoy or similar) intercepts every request for mTLS, retries and metrics, so each hop adds latency. Compare with and without the sidecar under the same load to put a number on it.

• If the increase is large, check proxy CPU limits (throttling), access log level, and retry/timeout policies first.
• Consider excluding workloads that don't really need the mesh from injection.
• Istio also offers ambient mode, which uses per-node proxies instead of sidecars (GA in 1.24). Measure it the same way before adopting it.

Looking inside the kernel with eBPF

tcpdump only records when a packet was seen on the interface you captured. To see where inside the kernel latency or retransmits occur, use eBPF-based tools:

sudo tcpretrans          # name varies by bcc-tools packaging, e.g. tcpretrans-bpfcc
sudo tcpconnlat 10       # only connections slower than 10 ms

Write a measurable hypothesis

Instead of "it's slow", write it down by segment and number: "P99 of A → B calls (100 KB responses) is 40 ms higher only on the cross-node path; no difference on the same node." That tells you where to look next (overlay/MTU), and the same measurement confirms whether the tuning worked.

FAQ

Q1. Will switching the CNI to Cilium reduce latency automatically? A1. It can if the cause is iptables rule processing or kube-proxy, but not if the cause is DNS, MTU or sidecars. Confirm the cause with per-segment measurements first.

Q2. Why is iperf3 fine while the service is still slow? A2. iperf3 measures throughput. Request-response latency includes DNS lookups, connection setup, sidecars and application time, so measure P99 separately with netperf TCP_RR or a load tool for your real protocol.

Q3. Does a service mesh always make things slower? A3. An extra proxy hop does add latency, but how much depends on configuration and load. Set a baseline by comparing with and without the sidecar under the same conditions. For mTLS setup, see Enforcing mTLS with Istio PeerAuthentication.