The Kubernetes CNI MTU Fragmentation Trap: Why Encapsulated RTP
calricon / field note
Discover why CNI overlay encapsulation causes MTU
The Kubernetes CNI MTU Fragmentation Trap: Why Encapsulated RTP Drops WebRTC Audio
DevOps and real-time communications engineers deploying media servers, WebRTC SFUs, and SIP gateways (such as LiveKit, Janus, Asterisk, or FreeSWITCH) across Kubernetes clusters frequently encounter severe audio degradation, garbled voice quality, and intermittent one-way audio. While inter-pod HTTP microservices operate flawlessly, real-time media streams suffer catastrophic packet loss. In WebRTC environments, video frames freeze instantly, audio roboticizes, and DTLS-SRTP key exchanges randomly fail during call initialization, despite worker nodes exhibiting low CPU and memory utilization.
The root cause lies in the silent Maximum Transmission Unit (MTU) mismatch introduced by Kubernetes Container Network Interface (CNI) overlay encapsulation. Standard physical Ethernet interfaces operate with an MTU of 1500 bytes. When CNIs like Calico, Flannel, or Cilium encapsulate pod egress traffic inside VXLAN or Geneve UDP headers, they append 50 to 64 bytes of encapsulation overhead. When media pods dispatch large UDP packets (such as bundled Opus audio frames, video keyframes, or DTLS handshake certificates) without MTU compensation, the intermediate physical network fragments or silently drops packets due to disabled IP reassembly, muting active media channels.
Problem Diagnosis & Root Cause Architecture
To understand why Kubernetes overlay networks selectively destroy real-time audio and video while leaving web traffic unaffected, one must examine UDP encapsulation and Path MTU Discovery (PMTUD) behavior.
- Overlay Encapsulation Header Overhead: Standard CNI overlays wrap raw pod IP packets inside an outer UDP tunnel. VXLAN requires a 50-byte outer header (20B IP + 8B UDP + 8B VXLAN + 14B Ethernet), while Geneve or WireGuard encryption consumes up to 64 to 80 bytes. Consequently, the effective maximum packet size inside the pod virtual ethernet interface (
veth) shrinks from 1500 bytes to 1450 or 1420 bytes. - Failure of Path MTU Discovery for UDP: Unlike TCP connections that negotiate Maximum Segment Size (MSS) during the three-way handshake and employ Path MTU Discovery, UDP is a connectionless protocol. Media servers emit UDP datagrams without MSS negotiation. If the application constructs a 1480-byte packet and the overlay interface allows 1500 bytes, the outer VXLAN packet exceeds the physical node's 1500-byte wire limit.
- Intermediate Network Packet Dropping: When the outer encapsulated packet exceeds 1500 bytes, the worker node kernel must fragment the packet at the physical NIC. However, cloud infrastructure providers and hardware top-of-rack switches frequently drop fragmented UDP packets or rate-limit IP fragments to protect against denial-of-service floods.
- DTLS-SRTP Handshake Failure: During WebRTC session establishment, the client and media pod exchange TLS certificates over UDP. Certificate chains typically range between 1200 and 2000 bytes. When encapsulation forces fragment drops, the DTLS handshake never completes, leaving the media channel completely dormant and dropping the call before audio connects.
Default CNI overlay installations assume TCP workloads where MSS clamping conceals MTU mismatches. Real-time UDP media datagrams bypass TCP MSS clamping, colliding with outer encapsulation headers and triggering catastrophic fragment loss.
Why Standard Implementations Fail
Relying on Default TCP MSS Clamping
Administrators often enable clamp-mss: true in the CNI configuration or apply iptables rules targeting TCP SYN packets. While this prevents large HTTP responses from fragmenting, it has zero impact on real-time media. RTP, RTCP, and WebRTC media streams run exclusively over UDP, rendering TCP MSS clamping completely ineffective.
Increasing Pod CPU and Memory Requests
Scaling pod compute allocations or bumping replica counts does nothing to alleviate network layer fragmentation. The packet drop occurs at the kernel virtual bridge and physical network interface boundary, completely outside the application container's execution runtime.
Enabling Jumbo Frames on Worker Nodes Only
Configuring mtu 9000 on the physical interfaces of Kubernetes worker nodes fails in hybrid cloud or multi-tenant cloud VPC environments. If underlying cloud virtual routers or transit gateways enforce a standard 1500-byte limit, encapsulated packets that leave the physical node are silently discarded by the cloud provider fabric.
Authenticated Step-by-Step Resolution Playbook
Step 1: Determine Physical MTU and Calculate Pod Safe Boundaries
Verify the maximum wire MTU supported across physical host interfaces and calculate the required pod MTU by subtracting CNI encapsulation overhead.
# Identify physical interface MTU on Kubernetes worker nodes
ip link show eth0
# Encapsulation Overhead Table:
# - Plain routed (no overlay, BGP/Direct): 1500 MTU
# - Calico VXLAN: 1500 - 50 = 1450 MTU
# - Flannel VXLAN: 1500 - 50 = 1450 MTU
# - Cilium VXLAN + Geneve: 1500 - 64 = 1436 MTU
# - WireGuard Encryption: 1500 - 80 = 1420 MTUStep 2: Reconfigure CNI DaemonSet with Explicit MTU Settings
Update your CNI configuration to automatically provision pod virtual interfaces (veth) with the correct reduced MTU, ensuring media applications never generate oversized unfragmented frames.
# For Calico CNI: Update the FelixConfiguration custom resource
cat << 'EOF' | kubectl apply -f -
apiVersion: projectcalico.internal/v3
kind: FelixConfiguration
metadata:
name: default
spec:
# Enforce 1450 MTU across all pod veth interfaces for VXLAN
ipipMTU: 1450
vxlanMTU: 1450
wireguardMTU: 1420
EOF
# For Cilium CNI: Apply explicit MTU via Helm or ConfigMap
kubectl patch configmap cilium-config -n kube-system --type merge \
-p '{"data":{"mtu":"1436"}}'Restart CNI daemon pods across the cluster to propagate the MTU configuration:
# Restart CNI daemons to apply MTU settings
kubectl rollout restart daemonset calico-node -n calico-system
kubectl rollout status daemonset calico-node -n calico-systemStep 3: Enforce Native Direct Routing (BGP) for Real-Time Media Nodes
Whenever possible, eliminate overlay encapsulation entirely for media worker nodes by switching Calico or Cilium to native routing (Direct Routing/BGP), restoring a full 1500-byte unencapsulated MTU.
# Configure Calico IPPool for Direct Routing within local subnets
cat << 'EOF' | kubectl apply -f -
apiVersion: projectcalico.internal/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
cidr: https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://10.244.0.0/16%26source%3Dgmail%26ust%3D1794171086433000%26sa%3DE&source=gmail&ust=1794171500596000&sa=E
# Use VXLAN only when crossing cross-subnet boundaries
vxlanMode: CrossSubnet
natOutgoing: true
nodeSelector: all()
EOFStep 4: Configure Media Applications with Safe Packetization Boundaries
Configure RTP and WebRTC media server profiles (such as Asterisk, FreeSWITCH, or Janus) to respect conservative packet limits and avoid generating oversized UDP frames.
# In Janus Gateway: /etc/janus/janus.jcfg
general: {
# Restrict maximum WebRTC packet size below CNI MTU
mtu = 1400
}
# In Asterisk: /etc/asterisk/rtp.conf
[general]
# Limit maximum RTP packetization length
rtpend = 20000
rtpstart = 10000
strictrtp = yesStep 5: Verify MTU Alignment and Zero Packet Fragmentation in Production
Verify that newly provisioned pods inherit the corrected MTU and conduct packet size sweeps using ping with the Don't Fragment (DF) bit set.
# Confirm pod virtual interface MTU inside a running media pod
kubectl exec -n telephony media-pod-0 -- ip link show eth0
# Verify packet path using DF-bit test from inside the media pod
# 1450 MTU allows maximum payload of 1450 - 28 (20B IP + 8B ICMP) = 1422 bytes
kubectl exec -n telephony media-pod-0 -- ping -c 4 -M do -s 1422 10.244.2.15
# Monitor physical node network interface for UDP fragment drops
cat /proc/net/snmp | grep -E "(Ip:.*Frag|Ip:.*Reasm)"
# Inspect active WebRTC DTLS handshake success rate
kubectl logs -n telephony media-pod-0 | grep -i "DTLS handshake complete"Aligning pod interface MTUs with outer CNI encapsulation limits eliminates packet drops, ensuring clean WebRTC DTLS handshakes and crystal-clear RTP audio delivery across Kubernetes clusters.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.