How to free space on a Gitlab CI server
von Stefan Moises
If you run your own Gitlab CI server, there may come a day when you run out of disc space, no matter how much space you have :) At least if you are also using Gitlab Runners in combination with Docker and maybe the integrated Docker Registry, too, your hard drive will fill up over time. In this blog post, I will give you a few hints what you can do to prevent this and how to cleanup space from time to time.
1. Check the Docker storage driver
First of all, check what internal storage driver Docker uses. If you are running Docker for quite some time or are on an older linux kernel, you may still use the so called "AUFS" storage driver. If possible, you should definately switch to the "Overlay2" storage driver, see e.g. https://docs.docker.com/storage/storagedriver/overlayfs-driver/
If you really, really cannot switch to Overlay2, you can use this script to cleanup the AUFS folders etc. that Docker uses.
If you already use Overlay2 or you can easily switch, congrats. Well, sort of ... because your pain won't end here :) Overlay2 is a lot better concerning disc space than AUFS, but it will still fill up your HD over time ...
2. Utilize the Docker cleanup commands
If you use Gitlab CI and you are running quite some projects launching Docker containers etc., you will end up with lots of images, containers, volumes etc. over time. One thing that really helps is to regularly call
docker system prune -f
Be aware that the "-f" forces the command to execute immediately without further warnings :) See https://docs.docker.com/engine/reference/commandline/system_prune/ for more information. This command will "remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes". If you want to remove all images and also the volumes, add "-a --volumes" to the call:
docker system prune -f -a --volumes
This should free up quite some GB on your Gitlab server. And normally, it shouldn't hurt much, because Gitlab can always re-download the images and fire up new containers and create volumes if needed. Usually, everything related to Docker which is created in a Gitlab CI process is temporary anyways.
So my recommendation is, create a cronjob which will prune your Docker stuff at least once a week, maybe every weekend and see if that helps. We are pruning every night.
3. Take care of your Docker Registry
But even if you have switched to Overlay2 and if you call "docker system prune" once a week or even every night, your HD may become fuller and fuller and you will still run out of disc space eventually.
Then you are probably also using the Gitlab Docker Registry and you manage some larger images with it. If you do so, and if you are often editing Dockerfiles, pushing images etc., the Registry will grow and grow, adding layer after layer. And by default, there is no process which will ever clean up all those image layers. And you probably won't even care at all for the old layers in your images, all you need is the latest version, right?
I have found a clever little bash script which will cleanup all that unneeded Docker Registry stuff for you, which will save you a lot of space, if you are maintaining some large Docker images in your Registry.
We are running it once a week via cron and it really helps to keep the HD space free.
4. Further space savers
We are using cache dirs for the docker runners and also for a general PHP Composer cache to save some download and deployment time for PHP projects. The directories are mounted volumes on the host, e.g. "/cache" and "/var/cache" (defined in "/etc/gitlab-runner/config.toml"):
tls_verify = false
hostname = "docker.local"
image = "php:7.1"
disable_cache = false
volumes = ["/var/cache:/composer_cache:rw", "/cache", "/gitlab_docker_transfer:/gitlab_docker_transfer:rw"]
cache_dir = "/cache"
The runner cache can also grow between 20 and 40 GB for us in a day, so we are also cleaning those directories every once in a while to save some more space if needed.
Also, make sure you don't keep the CI artifacts too long, if you have lots of projects and deployments. You can set that in your .gitlab-ci.yml file, e.g.
expire_in: 1 days
So, hopefully, you will have less sleepless nights and lots of free space on your Gitlab server in the future!