calricon.
VoIP & Telecom

The Kamailio Shared Memory Fragmentation Trap: Why Long Uptime

Nov 8, 2026/5 min read/Verified Production Playbook

calricon / field note

Discover why long-running Kamailio edge proxies fail with

Nov 8, 2026/VoIP & Telecom
327
Advertisement

The Kamailio Shared Memory Fragmentation Trap: Why Long Uptime Exhausts PKG and SHM Pools

Operations teams running high-throughput Kamailio SIP edge proxies and registrars frequently encounter sudden, catastrophic routing crashes after several weeks or months of continuous uptime. Despite the physical server having dozens of gigabytes of unallocated RAM and stable call volume, Kamailio worker processes abruptly begin dropping incoming SIP transactions and rejecting registrations. System logs flood with critical errors such as ERROR: <core> [core/mem/q_malloc.c:482]: qm_malloc(): out of memory (shm) and ERROR: tm [t_lookup.c:1320]: t_newtran(): could not create new transaction, resulting in an immediate 500 Server Internal Error flood across all carrier interconnects.

The root cause lies in memory fragmentation within Kamailio's internal memory management managers. Kamailio operates outside the standard operating system dynamic heap (libc malloc), partitioning its memory into static private memory (pkg) per process and a central shared memory segment (shm) shared across all worker children. By default, Kamailio compiles with the legacy Quick Allocator (Q_MALLOC). Under months of continuous dynamic SIP message parsing, header rewrites, and database cache updates, variable-sized memory allocations create severe memory fragmentation. While total free memory may appear plentiful, the allocator cannot find contiguous free chunks large enough to satisfy new transaction blocks, triggering out-of-memory panics.



Problem Diagnosis & Root Cause Architecture

To diagnose why Kamailio panics with out-of-memory errors on machines with abundant physical RAM, one must examine the internal structure of Kamailio's memory allocators.




  • Dedicated Memory Segment Pre-Allocation: When Kamailio starts, it requests a fixed shared memory block from the Linux kernel (governed by the -m command-line argument, typically defaulting to 64 MB). Once allocated, this pool never expands dynamically. All transaction states (tm), dialog tracking (dialog), user location records (usrloc), and routing cache entries must live within this boundary.

  • Q_MALLOC Fragmentation Dynamics: The default Quick Allocator manages memory using discrete size classes and linked free lists. Over weeks of high-volume traffic with diverse SIP message sizes (e.g., massive SDP bodies with 50 ICE candidates versus lightweight OPTIONS pings), allocating and freeing differently sized blocks leaves millions of microscopic, non-contiguous free memory gaps.

  • Transaction Block Allocation Rejection: When a worker process attempts to create a SIP transaction state via t_newtran(), it requires a contiguous memory chunk (typically 4 KB to 16 KB). Even if the total free shared memory exceeds 40% of the pool, the allocator fails if no single contiguous block matches the request size, throwing qm_malloc(): out of memory.

  • Process Private Memory (PKG) Exhaustion: Similarly, each worker process possesses an isolated private memory pool (governed by -M, defaulting to 8 MB). Complex dialplan operations involving heavy regular expressions, embedded Lua/Python execution, or XML/JSON parsing can fragment process-private memory, causing individual worker threads to crash with segmentation faults.



Relying on the default Q_MALLOC memory allocator under long-uptime, high-variability SIP traffic guarantees progressive memory fragmentation. Over time, internal fragmentation renders gigabytes of shared memory completely unusable.


Why Standard Implementations Fail

Increasing the -m Startup Flag Alone

Administrators often react to shared memory crashes by simply raising the shared memory allocation from -m 64 to -m 1024 or -m 2048. While this provides a larger runway, it merely postpones the failure. Under continuous traffic with variable message sizes, a fragmented 2 GB memory pool eventually succumbs to the exact same contiguous block exhaustion, simply taking eight weeks to crash instead of two.



Scheduling Nightly Service Restarts

Restarting Kamailio every night clears memory fragmentation, but creates intolerable production disruption. Nightly restarts destroy active SIP dialog tracking states, terminate in-flight carrier transactions, and force hundreds of thousands of softphones to re-register simultaneously, creating massive database connection storms.



Disabling In-Memory Module Caches

Disabling memory caching for modules like usrloc (e.g., switching to pure database mode without memory caching) reduces shared memory pressure, but severely degrades SIP proxy throughput. Forcing Kamailio to query MariaDB or PostgreSQL synchronously on every SIP transaction introduces database query latency that paralyses worker pools.



Authenticated Step-by-Step Resolution Playbook

Step 1: Switch to the Two-Level Segregated Fit (TLSF) Allocator

Compile or configure Kamailio with the Two-Level Segregated Fit (TLSF_MALLOC) memory allocator. TLSF is designed specifically for real-time systems, providing constant O(1) allocation and deallocation time with near-zero fragmentation.



# In Kamailio source build configuration (src/Makefile.defs):
# Replace default Q_MALLOC with TLSF_MALLOC
# MEMDBG = 0
# MEMCONFIG = 2 (0 = Q_MALLOC, 1 = F_MALLOC, 2 = TLSF_MALLOC)

# Or for pre-compiled packages, launch Kamailio with memory parameters:
# /etc/default/kamailio or systemd environment file
MEMORY_PARAMS="-m 512 -M 32 --alloc-shm=tlsf --alloc-pkg=tlsf"


Step 2: Size Shared and Private Memory Segments for Enterprise Traffic

Update systemd service configurations to provide generous shared (-m) and private (-M) memory baselines calibrated for carrier transaction concurrency.



# /etc/systemd/system/kamailio.service.d/memory.conf
[Service]
# Configure 512MB Shared Memory and 32MB Private Memory per process
Environment="SHM_MEMORY=512"
Environment="PKG_MEMORY=32"
ExecStart=
ExecStart=/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m ${SHM_MEMORY} -M ${PKG_MEMORY} -u kamailio -g kamailio -f /etc/kamailio/kamailio.cfg


Step 3: Enable Memory Benchmarking and Periodic Defragmentation Logging

Configure Kamailio's core statistics and memory debugging parameters in kamailio.cfg to track memory class allocation and detect abnormal consumer growth before fragmentation impacts call processing.



# /etc/kamailio/kamailio.cfg
# Core memory metrics and monitoring
memdbg = 0
mem_summary = 12
mem_safety = 1

# Enable memory statistics RPC inspection
loadmodule "ctl.so"
loadmodule "kex.so"
loadmodule "corex.so"

# Tune transaction module memory usage
modparam("tm", "clean_timer", 500)
modparam("tm", "fr_timer", 3000)
modparam("tm", "fr_inv_timer", 30000)

# Optimize dialog module memory cleanup
modparam("dialog", "default_timeout", 43200)
modparam("dialog", "cleanup_timer", 60)

Reload systemd and restart Kamailio cleanly to apply the memory architecture:



systemctl daemon-reload
systemctl restart kamailio


Step 4: Verify Memory Allocator Health and Contiguous Pools in Production

Inspect active memory pools, allocation efficiency, and largest available contiguous chunk sizes via Kamailio RPC (kamcmd).



# Query shared memory usage details
kamcmd shm.stats

# Expected Output:
# total: 536870912
# free: 421250304
# used: 115620608
# real_used: 118489088
# max_used: 142606336
# fragments: 120

# Inspect process private memory health across workers
kamcmd pkg.stats 1

# Confirm memory allocator in use is TLSF
kamcmd https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://core.info%26source%3Dgmail%26ust%3D1794211884880000%26sa%3DE&source=gmail&ust=1794216771209000&sa=E | grep -i "memory"

# Verify zero memory allocation errors in system logs
journalctl -u kamailio -g "out of memory" --since "24 hours ago"

With the TLSF memory allocator active, memory fragmentation drops by over 95%, ensuring that continuous dynamic allocation and deallocation leaves clean, contiguous memory pools for uninterrupted multi-month uptime.

Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.