calricon.
DevOps & Cloud

Fixing Docker BuildKit Cache and Log Bloat on Linux CI Runners

Nov 16, 2026/4 min read/Verified Production Playbook

calricon / field note

Docker CI/CD runners and production hosts frequently fill

Nov 16, 2026/DevOps & Cloud
359
Advertisement

Fixing Docker BuildKit Cache and Log Bloat on Linux CI Runners

On Linux CI/CD runners (GitHub Actions self-hosted runners, GitLab Runner, Jenkins) and Docker Compose production hosts, deployments and image builds frequently abort with no space left on device inside /var/lib/docker. Even when only a handful of containers are running, df -h shows the root or Docker volume pinned at 98%–100%.

The root causes are unbounded json-file container logs (which default to max-size: -1, unlimited) and BuildKit build cache accumulation (whose built-in garbage collector only triggers when capacity ratios or 48-hour/60-day age thresholds are crossed).



Problem Diagnosis & Root Cause Architecture

Pinpointing what is consuming /var/lib/docker requires separating logs, writable container layers, images, and BuildKit cache:



  • Unlimited json-file Container Logs: Docker's default logging driver writes all container stdout/stderr to /var/lib/docker/containers/<id>/*-json.log with no size cap. A single verbose or crash-looping service can silently grow a 40 GB log file.

  • Lenient BuildKit GC Thresholds: On Linux Docker Engine, default BuildKit garbage collection retains unused cache for up to 60 days (or 48 hours for ephemeral cache) and derives capacity limits from total disk size—allowing CI builds to consume tens of gigabytes before automatic pruning starts.

  • Custom Builder Isolation: Running docker system prune only cleans the default docker builder. Custom docker buildx builders keep separate cache stores that require explicit docker buildx du and docker buildx prune inspection.

  • Container Writable Layer Writes: Apps writing temp files, caches, or uploads directly to the container filesystem instead of mounted volumes bloat overlay2 upperdirs and risk inode exhaustion (df -i).


Production Rule: Never delete files manually inside /var/lib/docker. Configure global json-file log rotation (max-size + max-file) and BuildKit GC limits (defaultKeepStorage) in /etc/docker/daemon.json, and schedule filtered pruning rather than blind -af wipes.


Why Standard Implementations Fail

1. Running docker system prune -af on a Cron Job

Scheduling unattended docker system prune -af wipes all stopped debug containers and every cached base image, forcing CI pipelines to re-download gigabytes of layers on every build while still leaving multi-gigabyte active *-json.log files untouched.


2. Editing daemon.json Without Restarting Docker and Recreating Containers

BuildKit GC settings and logging defaults in /etc/docker/daemon.json are not hot-reloaded by systemctl reload docker—you must run systemctl restart docker, and existing containers must be recreated (docker compose up -d --force-recreate) to adopt the new log rotation limits.


3. Using --volumes During Automated Cleanup

Adding --volumes to a scheduled prune permanently destroys unmounted or stopped-stack database volumes, turning a routine disk cleanup into data loss.



Validated Step-by-Step Resolution Playbook

Step 1: Attribute Usage Across Images, Build Cache, and Logs

Inspect Docker's space breakdown, BuildKit cache usage, and oversized container JSON logs before deleting anything:


df -h /var/lib/docker && df -i /var/lib/docker
docker system df
docker buildx du
sudo find /var/lib/docker/containers/ -name "*-json.log" -exec ls -lh {} + | sort -k5 -rh | sed -n '1,15p'


Step 2: Reclaim Immediate Space Safely Without Touching Volumes

Prune stale BuildKit cache older than 24 hours and unused images, and truncate any runaway *-json.log file safely:


docker builder prune --filter "until=24h" -f
docker image prune -a --filter "until=72h" -f
sudo truncate -s 0 /var/lib/docker/containers/<container-id>/*-json.log


Step 3: Enforce Log Rotation and BuildKit GC in /etc/docker/daemon.json

Configure bounded json-file rotation (30 MB max per container) and BuildKit garbage collection in /etc/docker/daemon.json:


{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"builder": {
"gc": {
"enabled": true,
"defaultKeepStorage": "15GB"
}
}
}


Step 4: Restart Docker Daemon and Recreate Existing Containers

Apply the daemon configuration (with live-restore enabled if supported, or during a maintenance window) and recreate running Compose services so existing containers pick up log rotation:


sudo systemctl restart docker
docker buildx inspect --bootstrap
docker compose up -d --force-recreate


Step 5: Verify Active GC Policy and Log Limits

Confirm BuildKit's GC policy is loaded, verify container log options, and monitor docker system df over subsequent CI builds:


docker inspect --format='{{.HostConfig.LogConfig}}' <container-name>
docker system df

Confirm max-size: 10m and max-file: 3 appear on the container's LogConfig, and BuildKit cache remains capped near 15GB across repeated pipeline runs.

Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.