Fix Linux systemd TasksMax and EAGAIN Fork Failures
calricon / field note
Eliminate Linux EAGAIN fork and thread creation failures under
Fix Linux systemd TasksMax and EAGAIN Fork Failures
High-concurrency Linux services such as Asterisk, FreeSWITCH, PostgreSQL, and multi-threaded Go or JVM applications suddenly fail with pthread_create failed: Resource temporarily unavailable or fork: retry: Resource temporarily unavailable (errno 11 / EAGAIN), even when host RAM, CPU, and global kernel.pid_max have ample capacity.
The verified root cause is cgroup PID exhaustion enforced by systemd's TasksMax and DefaultTasksMax limits. Because systemd maps TasksMax directly to the cgroup pids.max controller—and counts every kernel thread as a task—services running under conservative distro defaults (often 15% of pid_max, such as 4915, or 512 in user scopes) hit their cgroup ceiling long before exhausting system memory.
Problem Diagnosis & Root Cause Architecture
Unlike legacy sysvinit daemons governed solely by PAM limits.conf, every systemd unit and user slice executes inside a dedicated cgroup with an independent task ceiling. Four architectural factors trigger EAGAIN:
- Threads Count Toward pids.max: Each worker thread created via
pthread_createconsumes one slot in the unit'sTasksCurrentcounter, rapidly saturatingTasksMaxon thread-pool heavy services. - Low DefaultTasksMax Ceilings: Stock
DefaultTasksMaxpercentages scale down on smaller VMs (for example,4915on 32Kpid_maxhosts), capping multi-process databases and VoIP media engines unexpectedly. - Shared User Slice Scope Contention: Transient background jobs or interactive shells spawned in the same
user-.sliceshare a singleUserTasksMaxor slicepids.maxpool, allowing one noisy worker to starve sibling processes. - Ignored /etc/security/limits.conf Changes: Raising
nprocin PAM limits does not raise the unit's cgrouppids.maxenforced by systemd.
When fork() or pthread_create() returns EAGAIN while free RAM and kernel.pid_max remain plentiful, compare TasksCurrent against TasksMax on the failing systemd unit.
Why Standard Implementations Fail
1. Editing /etc/security/limits.conf for systemd Services
Systemd services launched by PID 1 bypass PAM session limits entirely; updating /etc/security/limits.conf leaves the cgroup pids.max ceiling unchanged.
2. Editing Unit Files Under /lib/systemd/system Directly
Modifying vendor unit files directly gets overwritten on package upgrades; persistent overrides belong in /etc/systemd/system/<unit>.service.d/ drop-ins or via systemctl set-property.
3. Updating DefaultTasksMax Without Reloading Units
Changing DefaultTasksMax in /etc/systemd/system.conf requires systemctl daemon-reload (or daemon-reexec / reboot for active slices) before existing units inherit the updated cap.
Step-by-Step Resolution Playbook
Step 1: Audit TasksCurrent, TasksMax, and System pid_max
Inspect the active task count and ceiling on the affected service alongside the host-wide DefaultTasksMax and pid_max values:
systemctl show --property=TasksCurrent,TasksMax,TasksAccounting asterisk.service
systemctl show --property=DefaultTasksMax
cat /proc/sys/kernel/pid_max
systemd-cgtop -n 1Step 2: Raise TasksMax Live and Persistently via set-property
Apply a higher TasksMax limit immediately without dropping active connections using systemctl set-property, which writes a persistent drop-in under /etc/systemd/system/<unit>.service.d/:
sudo systemctl set-property asterisk.service TasksMax=32768
systemctl cat asterisk.serviceStep 3: Configure Global DefaultTasksMax and User Slice Overrides
For hosts running many high-thread services or user-scoped workers, create clean drop-in overrides for system.conf and user-.slice:
sudo mkdir -p /etc/systemd/system.conf.d /etc/systemd/system/user-.slice.d
cat << 'EOF' | sudo tee /etc/systemd/system.conf.d/90-tasksmax.conf
[Manager]
DefaultTasksMax=32768
EOF
cat << 'EOF' | sudo tee /etc/systemd/system/user-.slice.d/40-tasksmax.conf
[Slice]
TasksMax=16384
EOF
sudo systemctl daemon-reloadStep 4: Verify Updated Cgroup pids.max and Thread Creation
Confirm that systemd updated the unit's TasksMax property and live cgroup pids.max file, and verify zero EAGAIN errors in service logs:
systemctl show --property=TasksCurrent,TasksMax asterisk.service
systemctl show --property=DefaultTasksMax
journalctl -u asterisk --since "5 minutes ago" | grep -i "Resource temporarily unavailable"Confirm that TasksMax reports 32768 and that thread creation spikes complete without EAGAIN errors.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.