Why Your K8s L7 Network Policy Failed: A Complete Guide to eBPF-Based Debugging Workflows

K8s L7 Network Policy Failures? Root-Cause Analysis and an eBPF Debugging Guide

As cloud-native environments have matured, traffic control has moved beyond simply opening ports (L4). Application-level (L7) controls—such as allowing only requests that carry a specific header—have become essential. With the adoption of service meshes, the complexity of Kubernetes networking policy has grown exponentially.

Even so, after you apply a NetworkPolicy with a sophisticated rule like “only requests that include a specific API key should pass,” it is common—even for experienced infrastructure engineers—to see traffic allowed when it should be blocked, or blocked when it should be allowed. Re-running kubectl apply will not solve this.

This article goes past the surface-level “policy failed to apply” message, diagnoses the root causes of L7 traffic filtering at the eBPF level, and provides a practical debugging workflow so you can raise your networking troubleshooting game.

iptables vs. eBPF: Understanding How Network Policies Are Actually Processed

To understand network policy, you need to know how the kernel handles packets. Traditional processing and modern eBPF-based processing are fundamentally different mechanisms.

1. iptables-based processing (kernel chains)

iptables uses the Linux kernel’s Netfilter framework to inspect packets sequentially, chain by chain. Each policy rule exists as a single rule. Every incoming packet is matched against those rules in order, and an action (Accept/Drop) is decided.

Limitations: As the number of rules grows, matching overhead accumulates. Inspecting complex application-layer data such as HTTP headers (L7) requires kernel modules or awkward traffic splitting, which easily becomes a performance bottleneck.

2. eBPF-based processing (loaded programs)

eBPF (extended Berkeley Packet Filter) loads user-defined programs into kernel space so they run when a network packet passes a specific hook point inside the kernel. Modern CNIs such as Cilium implement networking logic this way.

Advantages:

1. Performance: Instead of walking every rule for every packet, only the needed logic is efficiently attached inside the kernel, so overhead drops dramatically.
2. Capability: Inspecting and filtering L7 protocol information directly at kernel level is structurally much easier.

💡 Practitioner’s note: I once burned a huge amount of time on an L7 policy failure. I assumed “the policy just didn’t apply.” The real cause was that eBPF did not treat certain flows (for example Keep-Alive packets) as policy inspection targets and was bypassing them. The key is understanding the packet lifecycle behind the abstract “failure” message.

Three Traps That Cause L7 Filtering to Fail

L7 policy failures usually fall into one of these three buckets.

1. Order dependency

Order matters. If a catch-all Allow All sits too high, or a general Drop runs before a rule that needs an exception, traffic can be allowed or blocked before the intended logic ever runs.

2. Overhead and state

L7 filtering is inherently stateful. For example, “allow only requests whose X-API-Key maps to a valid session cookie” may require validating that key (including a DB lookup). When that check is attempted at kernel level, you can hit performance overhead—or the policy engine may treat it as excessive load and skip inspection.

3. Flow mismatch

This is the most common mistake. Policies usually focus only on requests (Ingress). L7 communication is a round trip of request and response. Even with a perfect Ingress policy, a missing Egress policy—or a service mesh mutating traffic through a sidecar—can make the policy engine treat the mutated packet as a new flow and skip inspection.

Practical debugging workflow: combining tcpdump and cilium status

To get past these traps, cross-check the policy engine’s intent against the actual packet flow.

Step 1: Check policy-engine state (What should happen?)

First, confirm the policy was actually loaded into the kernel and which rules are active. With Cilium, cilium status is the strongest tool.

# 활성화된 네트워크 정책 목록 및 상태 확인
cilium status
# 특정 네임스페이스의 정책 상세 확인 (버전 및 환경에 따라 명령어 상이)
kubectl get netpol -n <namespace>

If cilium status reports the policy as healthy but traffic is still blocked, the problem is likely the packet flow, not the policy definition.

Step 2: Capture and analyze real packets (What is happening?)

Use tcpdump on the network interface of the failing Pod and look at the packets with your own eyes.

Suppose the api-server Pod should receive requests with a specific header from the frontend Pod, but it does not.

# 문제가 발생하는 Pod의 노드에서 실행 (가장 정확함)
# eth0는 실제 네트워크 인터페이스 이름으로 변경 필요
sudo tcpdump -i eth0 -nn host <target_pod_ip> and port 8080 -w capture_fail.pcap

Open the captured .pcap in Wireshark and look for patterns like these:

• Packets never arrive: Likely an L3/L4 firewall or a basic CNI routing problem.
• Packets arrive but are dropped on an unexpected port: L7 filtering ran, but the policy engine did not treat the packet as an allowed flow and dropped it.
• Headers missing or mutated: A service mesh or proxy mutated the traffic, and the headers the policy engine inspects were already lost.

Step 3: Review a policy YAML example (best practice)

When you define L7 policy, use http matching explicitly, and use metadata when you need to keep validation logic clearly separated.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-secure-api
  namespace: api-ns
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - HTTP
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: HTTP
          port: 8080
      # L7 매칭을 명시적으로 정의
      rules:
        - operation: GET
          headers:
            # 반드시 이 헤더와 값이 포함되어야 함
            X-API-Key:
              str: "secure-key-123"

Conclusion: A final checklist for reliable L7 network control

An L7 network policy is not just a config file—it is kernel-level behavior. When a policy fails, inspect in this order:

1. [Policy definition] Did the YAML explicitly declare an L7 type such as policyTypes: [HTTP]?
2. [Policy load] Did cilium status (or equivalent) confirm the policy was loaded into the kernel?
3. [Flow verification] Did tcpdump show that the failing traffic actually reaches the Pod, and that headers were not mutated?
4. [Exceptions] Was the policy overridden by an overly broad Allow rule, and was essential Keep-Alive traffic excepted?

Internalize this workflow and you will stop at “the packet was dropped at step 3 for reason X” instead of the vague “policy apply failed.”

References: official docs

The primary sources for the behavior, configuration, and errors in this article are the official docs below. Use them for version-specific options and exact semantics.

• Kubernetes official documentation

FAQ

Q1: After applying an L7 policy, health-check traffic gets blocked. What should I do? A1: Health checks are usually periodic and predictable. The safest approach is to define that traffic’s source IP/Pod and port separately, put an Allow rule at the top, or configure an exception so the policy engine ignores it.

Q2: Can I do L7 filtering with vanilla Kubernetes NetworkPolicy, without Cilium? A2: The base Kubernetes NetworkPolicy spec does not support L7 filtering (HTTP headers) directly. You need a CNI that extends L7 via eBPF, such as Cilium.

Q3: I captured packets with tcpdump but cannot see header information. Why? A3: The packet has likely already passed a layer such as a service-mesh sidecar proxy, where L7 data was abstracted or stripped. In that case, the proxy logs or sidecar traffic are often more accurate than tcpdump.