Fixing Docker BuildKit Cache and Log Bloat on Linux CI Runners
calricon / field note
Docker CI/CD runners and production hosts frequently fill
Fixing Docker BuildKit Cache and Log Bloat on Linux CI Runners
On Linux CI/CD runners (GitHub Actions self-hosted runners, GitLab Runner, Jenkins) and Docker Compose production hosts, deployments and image builds frequently abort with no space left on device inside /var/lib/docker. Even when only a handful of containers are running, df -h shows the root or Docker volume pinned at 98%–100%.
The root causes are unbounded json-file container logs (which default to max-size: -1, unlimited) and BuildKit build cache accumulation (whose built-in garbage collector only triggers when capacity ratios or 48-hour/60-day age thresholds are crossed).
Problem Diagnosis & Root Cause Architecture
Pinpointing what is consuming /var/lib/docker requires separating logs, writable container layers, images, and BuildKit cache:
- Unlimited
json-fileContainer Logs: Docker's default logging driver writes all containerstdout/stderrto/var/lib/docker/containers/<id>/*-json.logwith no size cap. A single verbose or crash-looping service can silently grow a 40 GB log file. - Lenient BuildKit GC Thresholds: On Linux Docker Engine, default BuildKit garbage collection retains unused cache for up to 60 days (or 48 hours for ephemeral cache) and derives capacity limits from total disk size—allowing CI builds to consume tens of gigabytes before automatic pruning starts.
- Custom Builder Isolation: Running
docker system pruneonly cleans the defaultdockerbuilder. Customdocker buildxbuilders keep separate cache stores that require explicitdocker buildx duanddocker buildx pruneinspection. - Container Writable Layer Writes: Apps writing temp files, caches, or uploads directly to the container filesystem instead of mounted volumes bloat
overlay2upperdirs and risk inode exhaustion (df -i).
Production Rule: Never delete files manually inside/var/lib/docker. Configure globaljson-filelog rotation (max-size+max-file) and BuildKit GC limits (defaultKeepStorage) in/etc/docker/daemon.json, and schedule filtered pruning rather than blind-afwipes.
Why Standard Implementations Fail
1. Running docker system prune -af on a Cron Job
Scheduling unattended docker system prune -af wipes all stopped debug containers and every cached base image, forcing CI pipelines to re-download gigabytes of layers on every build while still leaving multi-gigabyte active *-json.log files untouched.
2. Editing daemon.json Without Restarting Docker and Recreating Containers
BuildKit GC settings and logging defaults in /etc/docker/daemon.json are not hot-reloaded by systemctl reload docker—you must run systemctl restart docker, and existing containers must be recreated (docker compose up -d --force-recreate) to adopt the new log rotation limits.
3. Using --volumes During Automated Cleanup
Adding --volumes to a scheduled prune permanently destroys unmounted or stopped-stack database volumes, turning a routine disk cleanup into data loss.
Validated Step-by-Step Resolution Playbook
Step 1: Attribute Usage Across Images, Build Cache, and Logs
Inspect Docker's space breakdown, BuildKit cache usage, and oversized container JSON logs before deleting anything:
df -h /var/lib/docker && df -i /var/lib/docker
docker system df
docker buildx du
sudo find /var/lib/docker/containers/ -name "*-json.log" -exec ls -lh {} + | sort -k5 -rh | sed -n '1,15p'Step 2: Reclaim Immediate Space Safely Without Touching Volumes
Prune stale BuildKit cache older than 24 hours and unused images, and truncate any runaway *-json.log file safely:
docker builder prune --filter "until=24h" -f
docker image prune -a --filter "until=72h" -f
sudo truncate -s 0 /var/lib/docker/containers/<container-id>/*-json.logStep 3: Enforce Log Rotation and BuildKit GC in /etc/docker/daemon.json
Configure bounded json-file rotation (30 MB max per container) and BuildKit garbage collection in /etc/docker/daemon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"builder": {
"gc": {
"enabled": true,
"defaultKeepStorage": "15GB"
}
}
}Step 4: Restart Docker Daemon and Recreate Existing Containers
Apply the daemon configuration (with live-restore enabled if supported, or during a maintenance window) and recreate running Compose services so existing containers pick up log rotation:
sudo systemctl restart docker
docker buildx inspect --bootstrap
docker compose up -d --force-recreateStep 5: Verify Active GC Policy and Log Limits
Confirm BuildKit's GC policy is loaded, verify container log options, and monitor docker system df over subsequent CI builds:
docker inspect --format='{{.HostConfig.LogConfig}}' <container-name>
docker system dfConfirm max-size: 10m and max-file: 3 appear on the container's LogConfig, and BuildKit cache remains capped near 15GB across repeated pipeline runs.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.