Remote building for NixOS on Raspberry Pi

2020.04.28 Nix NixOS Embedded

contents

• Set up a Packet ARM server
• Check you can non-interactively access the Nix tools
• Modify your configuration.nix on the Pi to use the remote builder
• Build!
• Troubleshooting
• Improvements

This is a low-friction method to use a Packet.net ARM server to run remote builds for NixOS packages on Raspberry Pi, which takes an age on its own, and can be accelerated by quite a large extent by using a remote 32-core ARM server. The experience of building remotely in this way is pretty smooth.

This post loosely follows the wiki page here.

Set up a Packet ARM server¶

• Sign up and select: c2.large.arm in AMS1
• Choose NixOS as the OS

NixOS is available on a minority of the servers they provide, so you have to look around.

You don't need to do anything with this server, simply start it and copy the IP address needed to access it.

You will need to have added an SSH key to your account for it to be deployed with, and have the private key present on the Raspberry Pi, however.

Check you can non-interactively access the Nix tools¶

I have the SSH key at ~/.ssh/packet, so running the following, I get:

$ ssh -i ~/.ssh/packet root@123.210.123.210 nix-store --version
nix-store (Nix) 2.2

Create your SSH config:

mkdir /root/.ssh
touch /root/.ssh/config

And add the following, using the server's IP:

# ephemeral build server
Host builder
    HostName 123.210.123.210
    User root
    IdentityFile ~/.ssh/packet

This will have to be /root/.ssh/config for using nixos-rebuild.

Now we can:

$ ssh builder nix-store --version
nix-store (Nix) 2.2

Modify your configuration.nix on the Pi to use the remote builder¶

{

  # Add remote build machine
  nix.buildMachines = [{
    hostName = "builder";
    system = "aarch64-linux";
    maxJobs = 1;
    speedFactor = 2;
    supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
    mandatoryFeatures = [ ];
  }];
  nix.distributedBuilds = true;
  # Speeds things up by downloading dependencies remotely:
  nix.extraOptions = ''
    builders-use-substitutes = true
  '';

}

Then run:

sudo nixos-rebuild switch

to evaluate the new configuration.

Build!¶

Add the packages you want to build into environment.systemPackages and build, disabling local jobs, so everything should happen on the server.

sudo nixos-rebuild -j0 switch

Or build one thing explicity.

nix-build -j0 your-derivation.nix
nix-build -j0 thingy

It's a rather nice sight, having an SSH session running htop on the remote server, while watching the build log fly past on the build output on the Raspberry Pi.

Troubleshooting¶

You may get:

cannot build on 'ssh://builder': cannot connect to 'builder': ssh: Could not resolve hostname builder: Name or service not known

If this happens, you should also get at the end of the build:

error: unable to start any build; either increase '--max-jobs' or enable remote builds

if your local Nix can't reach the server. This is probably because it doesn't have a working SSH config, or the config is for the wrong user.

Improvements¶

It could be possible to spin up the packet server using Nixops, but this is entirely undocumented. There is a packet plugin however, but I can't for the life of me find out how to add it. This might be the topic of a future blog post.