The Kamailio HTABLE Lock Contention Trap: Why Auto-Expire Sweeps
calricon / field note
Discover how default HTABLE slot sizing and timer sweeps trigger
The Kamailio HTABLE Lock Contention Trap: Why Auto-Expire Sweeps Freeze SIP Worker Pools
In high-throughput telecommunications architectures, the Kamailio SIP proxy relies on the memory-resident htable (Hash Table) module to handle stateful real-time operations without incurring relational database latency. Telephony engineers standardly leverage hash tables for SIP flood mitigation, caller ID velocity tracking, registration rate limiting, and IP reputation caching. To prevent memory exhaustion from stale records, tables are routinely configured with automated time-to-live expiration (autoexpire). However, under peak traffic spikes exceeding 1,500 calls per second or during large-scale network scanning floods, carrier-grade proxies suffer an abrupt operational stall: inbound SIP INVITE and REGISTER transactions cease processing, UDP socket receive buffers surge to capacity, and upstream carrier SBCs trigger trunk failovers due to transaction timeouts. The failure is not caused by CPU exhaustion or network saturation, but by coarse-grained mutex lock contention inside the hash table module during periodic expiration sweeps.
Problem Diagnosis & Root Cause Architecture
When Kamailio initializes an htable, it allocates an array of hash slots (buckets) protected by shared mutex locks in shared memory. Every read, write, or increment operation executed within the routing script must acquire the mutual exclusion lock assigned to that specific hash slot. The architectural breakdown occurs through the intersection of undersized bucket arrays and the synchronous execution model of Kamailio's timer process:
- Coarse Slot Array Allocation: By default, an uncalibrated hash table definition defaults to a small power-of-two size (such as
size=9, providing only 512 hash slots). When tracking tens of thousands of active IP addresses or user records, hundreds of disparate keys hash into identical buckets, forming lengthy linked lists. - Synchronous Timer Sweep Serialization: A dedicated internal timer process awakens at regular intervals (defined by
timer_interval) to purge expired keys. The timer traverses each bucket sequentially, locking the slot's mutex before scanning its linked list of memory records. - Worker Pool Starvation: While the timer process holds the slot lock to inspect, unchain, and free memory for multiple expired entries, worker processes attempting to route incoming SIP packets that map to that same slot are forced into kernel sleep states.
- Socket Buffer Congestion and Packet Eviction: Because Kamailio worker processes handle inbound UDP datagrams synchronously from shared sockets, blocked worker processes halt packet ingestion. Unserviced packets accumulate in the Linux kernel UDP receive queue (
Recv-Q) until the buffer overflows and packets are silently discarded.
"Hash table auto-expiration in shared memory requires fine-grained bucket distribution. When thousands of dynamic records are constrained within hundreds of hash slots, timer maintenance sweeps hold bucket mutexes across extended execution slices. This converts concurrent SIP worker pools into serialized wait queues, driving kernel socket buffer overflows and carrier timeout cascades."
Why Standard Implementations Fail
The Undersized Default Slot Assumption
Most standard Kamailio deployments instantiate hash tables using basic module parameters such as modparam("htable", "htable", "ipban=>size=9;autoexpire=300"). The parameter size=9 specifies an exponent representing 2^9 (512) buckets. In modern carrier environments handling continuous scanning traffic and dynamic caller records, table populations routinely swell to 80,000 entries. This results in an average load factor exceeding 150 items per linked list. When the auto-expiration timer traverses these deep chains, memory deallocation and pointer rewrites keep the bucket lock held for milliseconds at a time, creating severe contention with incoming SIP worker traffic.
Monolithic Table Co-Location
Telephony administrators frequently consolidate heterogeneous data structures into a single shared hash table. High-velocity transient metrics (such as one-second sliding window transaction counters) are placed alongside long-lived records (such as 24-hour persistent geographic IP blacklists). Because the auto-expire daemon must inspect every populated slot to evaluate expiration timestamps, long-lived entries unnecessarily prolong the inspection loop, amplifying the frequency and duration of lock conflicts against real-time routing threads.
Aggressive Timer Sweep Intervals
When administrators observe memory consumption growing inside Kamailio shared memory pools, a common misconfiguration is to reduce timer_interval from the default 20 seconds down to 2 or 5 seconds. Rather than resolving memory pressure, this change multiplies the rate of bucket lock acquisitions by the timer process. The timer continually contends with SIP worker processes, driving high system CPU utilization in kernel futex arbitration and preventing workers from servicing inbound transactions.
Validated Step-by-Step Resolution Playbook
Step 1: Diagnose Mutex Lock Contention and Socket Buffers
Verify whether SIP packet processing delays correlate with kernel socket queue accumulation and mutex contention. Inspect the UDP receive queues for the Kamailio listening interface and verify hash table record counts:
# Check Linux kernel UDP receive queue saturation on port 5060
ss -u -a -p -n | grep :5060
# Inspect Kamailio shared memory hash table sizing and record volume
kamcmd htable.dump ipban | grep -c "name"
# Monitor lock wait states and context switches across Kamailio worker processes
pidstat -w -p $(pgrep -d, kamailio) 1 5Step 2: Calculate and Configure Fine-Grained Bucket Sizing
Reconfigure hash table slot sizing so that the total number of buckets exceeds the anticipated peak key population by at least a factor of two. Setting size=14 yields 16,384 buckets, while size=16 allocates 65,536 buckets, ensuring the average bucket depth remains below 1.5 entries even under massive traffic spikes.
# /etc/kamailio/kamailio-htable.cfg
# Optimize module parameters for high-concurrency environments
loadmodule "htable.so"
# Allocate 65,536 buckets (2^16) for high-rate IP tracking
# Allocate 16,384 buckets (2^14) for caller velocity counters
modparam("htable", "htable", "ip_tracking=>size=16;autoexpire=180;")
modparam("htable", "htable", "rate_limits=>size=14;autoexpire=60;")
# Maintain clean segregation for persistent lists without autoexpire
modparam("htable", "htable", "static_allowlist=>size=12;autoexpire=0;")
# Set timer sweep interval to balance memory reclamation and lock stability
modparam("htable", "timer_interval", 30)Step 3: Segregate Volatile Counters from Long-Lived State
Separate sub-minute traffic metrics from extended reputation records. Place ephemeral rate-limiting keys into short-lived tables with shallow expiration timers, while routing long-term policy checks to dedicated, non-expiring or asynchronously managed tables:
# Kamailio routing script logic: segregating table interactions
route[REQINIT] {
# Extract source IP for connection tracking
$var(src_ip) = $si;
# Check long-lived static allowlist first (no lock expiration contention)
if ($sht(static_allowlist=>$var(src_ip)) != $null) {
return;
}
# Evaluate short-lived sliding window counter
if ($sht(rate_limits=>$var(src_ip)) != $null) {
if ($sht(rate_limits=>$var(src_ip)) > 50) {
sl_send_reply("503", "Traffic Velocity Exceeded");
exit;
}
$sht(rate_limits=>$var(src_ip)) = $sht(rate_limits=>$var(src_ip)) + 1;
} else {
$sht(rate_limits=>$var(src_ip)) = 1;
}
}Step 4: Optimize Operating System UDP Socket Buffers
Prevent packet drops during unavoidable microsecond lock acquisitions by expanding the Linux network core memory limits and tuning the Kamailio socket receiver buffers:
# /etc/sysctl.d/99-kamailio-telephony.conf
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.core.rmem_default = 33554432
net.core.netdev_max_backlog = 10000
# Apply sysctl settings immediately
# sysctl -p /etc/sysctl.d/99-kamailio-telephony.confIn kamailio.cfg, expand the per-process socket buffer configuration to absorb transient micro-bursts:
# Kamailio global configuration settings
auto_bind_ipv4 = 1
children = 16
tcp_children = 8
# Expand UDP receive buffer allocation to 32MB per listener
open_sock_buffer_size = 33554432Step 5: Verify Lock Freedom and Real-Time Performance
Subject the reconfigured proxy to a sustained SIP registration and call setup stress test using SIPp. Monitor the operational metrics to confirm that bucket depth remains optimal and socket queues remain clear:
# Execute high-throughput registration scenario
sipp -sn uac -r 1500 -rp 1000 -m 30000 -i 192.168.10.50 https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://192.168.10.20:5060%26source%3Dgmail%26ust%3D1794487080311000%26sa%3DE&source=gmail&ust=1794487146917000&sa=E
# Inspect htable statistics and bucket occupancy in real time
kamcmd stats.get_statistics htable:
# Verify that kernel UDP receive queue remains at zero under load
watch -n 1 'ss -u -a -p -n | grep :5060'Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.