Resolving K8s NetworkPolicy Communication Errors: A Calico vs Cilium (iptables vs eBPF) Comparison Guide

Kubernetes NetworkPolicy, Calico vs Cilium: An In-Depth Guide to Resolving Complex Communication Failures

In Kubernetes, NetworkPolicy is the front line of security. Questions like “I applied this policy—why isn’t traffic flowing?” and “Communication between certain Pods seems blocked; where is it getting dropped?” are walls every operations engineer hits at some point. The real skill is going beyond writing YAML that parses, and understanding how the policy is enforced at the kernel and which layer actually drops the packet.

This guide goes beyond syntax. It presents a practical method for diagnosing and fixing complex network failures in production, based on how each CNI (Container Network Interface) actually implements policy.

Why “the policy isn’t taking effect” happens: beyond the basics

Most people expect NetworkPolicy to behave like a firewall. It follows a Default Deny model: any traffic that is not explicitly allowed is blocked.

Problems start when you do not know through which mechanism, at what point, and at which layer (L3/L4) that Default Deny is applied. CNIs implement policy in fundamentally different ways, so the same YAML can behave differently on Calico versus Cilium—or introduce unexpected overhead.

Comparing policy processing by CNI: iptables vs eBPF

The core difference is where packet filtering happens. That choice drives both performance and how hard debugging will be.

Key analysis: The iptables approach stacks every policy rule onto kernel firewall chains in sequence. Once you have hundreds of rules, that produces rule-chaining overhead and can hurt performance. Cilium’s eBPF path intercepts packets inside the kernel and applies only the needed filters, which is a clear win in large clusters.

Strategies for complex communication-failure scenarios

In production you rarely see a simple Pod-to-Pod miss. You get stacked policy conflicts.

1. The pitfalls of Egress (outbound) control

Many people only think about Ingress (inbound traffic). External API calls and traffic to other namespaces are Egress. Even if intra-cluster traffic is allowed, blocking a Pod’s traffic to external IP ranges requires an explicit Egress policy. If that Egress policy is missing, it is easy to misdiagnose the Pod as simply “unable to reach the outside world.”

2. Cross-namespace communication conflicts

When a Pod in namespace A talks to a Service in namespace B, policies on both A and B apply. If A’s policy blocks a port on B and B’s policy also blocks A, which policy wins becomes the question. In practice the most restrictive policy usually wins, but you should understand each CNI’s priority model and define the broadest allow policies at the top level so behavior stays predictable.

3. Interaction between Service Mesh and NetworkPolicy

With a Service Mesh such as Istio, traffic control happens at the sidecar proxy (Envoy). If NetworkPolicy drops the packet at L3/L4, Envoy never sees it. If you run a mesh, treat NetworkPolicy as the last line of defense and leave L7 (HTTP header-based) control to mesh policies (VirtualService, AuthorizationPolicy). That split is more stable architecturally.

Hands-on: network communication failure debugging workflow (Troubleshooting Flow)

When traffic fails, blindly running kubectl describe wastes time. Use this 3-step workflow.

Assumed scenario: Pod A (Namespace: dev) $\rightarrow$ Pod B (Namespace: prod) communication failure (Port 8080)

Step 1: Review policies and scope (Declarative Check) First, list every NetworkPolicy in the involved namespaces.

# dev 네임스페이스의 정책 확인
kubectl get netpol -n dev 
# prod 네임스페이스의 정책 확인
kubectl get netpol -n prod 

• What to check: Is there an Ingress allow rule from A to B? Is there an Egress allow rule from B to A? (verify both directions)

Step 2: Diagnose state with CNI-specific tools (CNI Specific Check) Log into the node where the policy should apply and use the CNI’s own tools.

• Calico users: Check IPAM with calicoctl get ippool and policy application with calicoctl get policy.
• Cilium users: Run cilium status to confirm eBPF maps are loaded and policies are applied.

Step 3: Packet-level traffic capture (Deep Packet Inspection) If the steps above look healthy and traffic still fails, find where packets are dropped.

1. Test inside Pod A: kubectl exec -it <pod-a-name> -- tcpdump -i eth0 host <pod-b-ip> and port 8080
2. Test at the node (last line of defense): SSH to the node hosting Pod B and run tcpdump -i eth0 host <pod-a-ip> and port 8080.
   • How to read it: If packets leave Pod A but never show up in the node-level tcpdump, they were most likely dropped first by the CNI’s Egress/node-level policy.

💡 Practitioner tip: When debugging policy, I always run iptables -L -v (or the CNI-specific tool) before tcpdump, to confirm the rules were actually loaded into the kernel chains. That habit alone prevented about 80% of my misdiagnoses.

Conclusion: a policy validation checklist for stable networking

Treat NetworkPolicy as a continuous validation process, not a one-time config. Use this checklist.

1. [Basic principle] Is there a policy that applies Default Deny in every namespace?
2. [Directionality] Have you explicitly defined both Ingress and Egress? (especially for external traffic)
3. [CNI characteristics] Do you understand your CNI’s enforcement path (eBPF/iptables) and have the matching debug tools ready?
4. [Layering] If you use Service Mesh with NetworkPolicy, is L7 control in the mesh and L3/L4 blocking in NetworkPolicy?

Next time, building on these principles, we will go deep on fine-grained service-to-service policy design patterns for a Zero Trust architecture.

References: official documentation

The primary sources for the behavior, configuration, and errors in this article are the official docs below. Use them for version-specific options and exact semantics.

• Kubernetes official documentation

Frequently Asked Questions (FAQ)

Q1: I applied a NetworkPolicy, and a Pod can no longer reach the public internet (e.g. Google DNS 8.8.8.8). Why? A1: If a policy that only allows intra-cluster traffic is in effect, external traffic (Egress) must be allowed explicitly. Add an Egress policy in that Pod’s namespace that permits the external IP ranges.

Q2: Is there a policy-conflict risk if I mix Calico and Cilium, or run both CNIs? A2: Yes—very high. If the two CNIs filter traffic in different ways, it is hard to predict which rules win. In production, a single CNI for the whole cluster is the most stable choice.

Q3: Does applying a NetworkPolicy block all traffic? A3: By default, yes. You must explicitly add Ingress and Egress rules to allow only the flows you need (e.g. Service A $\rightarrow$ Service B, Port 80).