Garnix is a generally great product. However, I recently decided to host my own cache CI for the following reasons:
- Garnix has no pay-as-you-go plan. While the 25$ individual plan is way over my budget, trying not up run out the narrow CI minutes included free plan is such a pain in the ass.
- Hard to debug as it's a SaaS product. Recently I am receiving lots of "unexpected end of file" "cannot connect to garnix builder" "foo doesn't exist in cahe.garnix.io" etc. What's worse, all these buggy hours are billed. The last time I used it, it literally stucked for more than 2 hours and drains about 1000 minutes of my credits, for compiling NOTHING.
More decisively, I got nothing better to do in my life. Or worse, I want to waste some time on these nerdy stuff to escape from the reality.
Utilizing Nix's builtin tools
Nix as package manager per se is a fairly powerful build system. Sandbox, storage management, remote build, etc.
The idea is a simple script that runs
nix build .#nixosConfigurations."$host".config.system.build.toplevel --out-link ".cache/$host"`
for every machine in flake.nix. So the top level deriviation is built and a output link is created to prevent it from garbage collected.
Here comes the tricky part, storage and distribution.
Attic
Let's review the needs:
- managing signing secrets
- storage management
- garbage collection
- skip what's already cached upstream
- preferably compression and de-duplication
- preferably external S3 storage support
- provide binary cache via http(s)
It seems that NixOS's official infra currently runs a centralized server, which runs hydra jobs, calls build machines, and uploads everything to S3. Unfortunately there isn't a way to "skip upstream cache" as they are the upstream per se.
I eventually found two candidates:
Obviously the latter one is more scalable. Let's go.
As part of NixOS ecosystem, the setup of atticd is highly procedural, unfortunately. (What's wrong with you ...)
The first thing is create a token:
atticd-atticadm make-token \ --sub "merrkry" \ --validity '1y' \ --pull '*' \ --push '*' \ --delete '*' \ --create-cache '*' \ --configure-cache '*' \ --configure-cache-retention '*' \ --destroy-cache '*'
atticd-atticadm is a wrapper that allows you to authenticate with polkit.
So far we have created a token under merrkry ("subject of the JWT token", no idea what that means), with a validity of a year, and the permission to perform any operation under any cache (matched by *).