The rtpengine Kernel Forwarding Fallback Trap: Why Silent Userspace
calricon / field note
Learn why rtpengine silently falls back to userspace RTP
The rtpengine Kernel Forwarding Fallback Trap: Why Silent Userspace Offload Exhausts Carrier CPU
In carrier-grade VoIP media architectures paired with Kamailio or OpenSIPS, rtpengine relies on the xt_RTPENGINE Linux kernel module to forward established RTP media streams directly inside kernel space. By bypassing userspace socket queues once media endpoints are learned, a single media proxy node can sustain thousands of concurrent G.711 or Opus calls with minimal CPU overhead. However, production telephony clusters frequently experience a sudden, catastrophic capacity collapse following routine OS security patches or firewall reloads: media node CPU spikes to 100%, ksoftirqd threads saturate, UDP receive buffers overflow, and callers report severe robotic audio and packet loss at just 20% of normal peak concurrency. The root cause is silent userspace forwarding fallback—where missing xt_RTPENGINE kernel bindings, flushed netfilter chains, or mismatched forwarding table IDs force rtpengine to process every 20ms RTP packet through userspace syscalls.
Problem Diagnosis & Root Cause Architecture
When a new SIP call is negotiated via Kamailio, rtpengine initially handles the first few RTP packets in userspace to learn NAT-mapped source addresses and verify media symmetry. Once endpoints stabilize, the daemon pushes forwarding rules into /proc/rtpengine/control for a designated forwarding table (typically table = 0), allowing xt_RTPENGINE to intercept and forward subsequent packets directly in the netfilter hook. Silent userspace degradation occurs when any link in this kernel-offload chain breaks:
- Silent Startup Fallback Without --no-fallback: By default, if the
xt_RTPENGINEkernel module fails to load (for example, after an unattended Linux kernel upgrade where DKMS headers are missing),rtpenginelogs a single warning and silently falls back to userspace-only forwarding. Because the daemon still binds UDP ports and answers Kamailiongcontrol commands, health checks pass while per-packet syscall overhead increases by an order of magnitude. - Netfilter Chain Flushing by Firewall Managers: Tools such as
ufw,firewalld, or dynamicfail2banreloads frequently flush or reorder the netfilterINPUTchain. If theRTPENGINEtarget rule (or managednftableshook) is purged or placed below an explicitACCEPTrule for the RTP port range, incoming UDP media packets bypassxt_RTPENGINEentirely and travel straight up to the userspace socket. - Forwarding Table ID Desynchronization: Each
rtpengineinstance binds to a specific kernel forwarding table ID (0through63). If/etc/rtpengine/rtpengine.confspecifiestable = 0while the netfilter rule specifies--id 1(or a stale table remains locked after an unclean restart), the kernel module never matches active streams. - Userspace Context-Switch and rmem Saturation: At 2,000 concurrent bidirectional G.711 calls (50 packets per second per direction), the media proxy must process 200,000 packets per second. In userspace mode, every packet incurs a kernel-to-user copy, a
recvfrom()wakeup, and asendto()syscall, exhausting CPU cycles and overflowingnet.core.rmem_maxbuffers.
"An rtpengine node running in silent userspace fallback behaves like a ticking time bomb: it passes every SIP signaling health check at low traffic, then collapses into packet loss and CPU starvation the moment peak carrier concurrency arrives."
Why Standard Implementations Fail
Omitting no-fallback in Daemon Configuration
Most default package installations leave no-fallback = false (the implicit default). In high-availability clusters fronted by Kamailio rtpengine_manage(), allowing a degraded node to stay in rotation rather than failing fast prevents Kamailio from failing over to healthy peer media nodes.
Relying on Unmanaged Manual iptables Rules
Hardcoding legacy iptables -I INPUT -p udp -j RTPENGINE --id 0 commands in static boot scripts breaks whenever modern Linux distributions switch to nftables or when firewall daemons reload rulesets mid-day.
Ignoring Kernel Header Synchronization During OS Upgrades
Automated patching pipelines often upgrade linux-image-* packages without installing matching linux-headers-* packages, leaving ngcp-rtpengine-kernel-dkms unbuilt on the new kernel after a reboot.
Validated Step-by-Step Resolution Playbook
Step 1: Audit Kernel Module Status, Table Bindings, and Active Netfilter Hooks
Verify whether xt_RTPENGINE is loaded, inspect active forwarding tables in /proc/rtpengine/list, and confirm that packets are actually hitting the kernel hook:
# Verify xt_RTPENGINE kernel module is loaded
lsmod | grep -i xt_RTPENGINE
# Inspect active kernel forwarding tables (must show table ID 0)
cat /proc/rtpengine/list
# Check managed nftables chain status (exits 0 if healthy)
rtpengine --nftables-status --config-file=/etc/rtpengine/rtpengine.conf
# Inspect live kernel vs userspace packet counters via rtpengine-ctl
rtpengine-ctl -ip https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.1:2223%26source%3Dgmail%26ust%3D1794632908394000%26sa%3DE&source=gmail&ust=1794633071246000&sa=E list numsessions
cat /proc/rtpengine/0/status 2>/dev/null || echo "Table 0 not active in kernel!"Step 2: Rebuild and Pin DKMS Kernel Module Across Kernel Versions
Ensure matching kernel headers are installed and force DKMS to compile and load xt_RTPENGINE across all installed kernels:
# Install matching headers for the running kernel and rebuild DKMS module
apt-get update && apt-get install -y linux-headers-$(uname -r) ngcp-rtpengine-kernel-dkms
dkms autoinstall
modprobe xt_RTPENGINE
# Persist module loading across reboots
echo "xt_RTPENGINE" > /etc/modules-load.d/rtpengine.confStep 3: Enforce no-fallback and Managed nftables Chains in rtpengine.conf
Configure rtpengine to manage its own nftables chain automatically and refuse to start if kernel offload is unavailable, forcing immediate failover in Kamailio:
# /etc/rtpengine/rtpengine.conf
# Production carrier configuration enforcing kernel-space RTP forwarding
[rtpengine]
interface = internal/https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://10.10.1.10%26source%3Dgmail%26ust%3D1794632908394000%26sa%3DE;external/https://www.google.com/url?q%3Dhttp://203.0.113.10%26source%3Dgmail%26ust%3D1794632908394000%26sa%3DE&source=gmail&ust=1794633071246000&sa=E
listen-ng = https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.1:2223%26source%3Dgmail%26ust%3D1794632908394000%26sa%3DE&source=gmail&ust=1794633071246000&sa=E
listen-cli = https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.1:9900%26source%3Dgmail%26ust%3D1794632908394000%26sa%3DE&source=gmail&ust=1794633071246000&sa=E
table = 0
no-fallback = true
nftables-chain = rtpengine
port-min = 30000
port-max = 40000
tos = 184
delete-delay = 30
num-threads = 16Step 4: Tune Linux Network Buffers and Systemd Service Dependencies
Harden the systemd unit so rtpengine waits for xt_RTPENGINE and configures adequate UDP socket buffers for initial call setup bursts before kernel offload takes over:
# /etc/sysctl.d/99-rtpengine-tuning.conf
# Optimize UDP socket buffers for high-concurrency RTP setup bursts
net.core.rmem_max = 33554432
net.core.rmem_default = 16777216
net.core.wmem_max = 33554432
net.core.netdev_max_backlog = 10000# Apply sysctl parameters and restart rtpengine
sysctl --system
systemctl restart rtpengine-daemonStep 5: Verify In-Kernel Packet Offload Under Live Media Traffic
Confirm that rtpengine bound cleanly to kernel table 0, the nftables hook is receiving packets, and no-fallback is actively enforced:
# Verify active nftables rtpengine chain and packet counters
nft list chain inet filter rtpengine
# Confirm kernel table 0 is registered and active
grep -E "table|packets" /proc/rtpengine/0/status
# Verify systemd status shows no userspace fallback warnings
journalctl -u rtpengine-daemon -n 30 --no-pager | grep -iE "kernel|table|fallback"Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.