Packaging a Rust project for Nix

2019-06-29 Nix | NixOS | Nixpkgs | Rust
[draft]

This is a walkthrough of making a simple Nix package, in this case a small Rust project, url-bot-rs.

It is also a relatively simple workflow for building packages and adding/testing these packages on a NixOS system.

This covers some useful concepts in Nix/NixOS, including:

Derivations
Overlays
The structure of Nixpkgs

Packaging in Nix involves defining build instructions to build a project, allowing Nix tools to build it, so that the resulting binaries etc. may be included in a Nix environment, or NixOS.

After that, a module may be used to generate configuration and other system-wide items to support the running of the packaged software.

Monolithic `default.nix`¶

You often see this file in the root of project repositories.

This is a simple default.nix expression for building a Rust project using the buildRustPackage function.

with import <nixpkgs> { };

rustPlatform.buildRustPackage rec {
  name = "url-bot-rs-${version}";
  version = "0.2.0";
  src = ./.;
  buildInputs = [ openssl pkgconfig sqlite ];

  checkPhase = "";
  cargoSha256 = "sha256:1bk7kr6i5xh7b45caf93i096cbblajrli0nixx9m76m3ya7vnbp5";

  meta = with stdenv.lib; {
    description = "Minimal IRC URL bot in Rust";
    homepage = https://github.com/nuxeh/url-bot-rs;
    license = licenses.isc;
    maintainers = [ maintainers.tailhook ];
    platforms = platforms.all;
  };
}

This can be placed in the project root and used in the following ways, for example:

nix-build invokes Cargo to build the package.
nix-env -if default.nix installs the package in your nix environment.
nix-shell if there is no shell.nix.

Notes:

This expression takes all files in the same directory as the source.
cargoSha256 = "sha256:1bk7kr6i5xh7b45caf93i096cbblajrli0nixx9m76m3ya7vnbp5"; rather quirkily, is obtained by attempting to build, and taking the SHA from the error given as "got".

Splitting up the simple `default.nix`¶

The monolithic default.nix can be split up into a derivation, and a couple of small expressions to:

Import Nixpkgs
Overlay the derivation onto Nixpkgs
Call the derivation (which is now a Nix function)

Overlays provide a simple method for a package to be added to a NixOS system's configuration.nix without it having to be present in upstream Nixpkgs, which is useful for local testing.

From the single Nix expression, we make three separate files, which together achieve the same result, but with more flexibility.

Preparation¶

Create the overlay.nix and default.nix content, and stubs for other files:

PNAME=url-bot-rs
mkdir "${PNAME}"

tee ${PNAME}/overlay.nix << EOF
self: super: {
  ${PNAME} = self.callPackage ./derivation.nix {};
}
EOF

tee ${PNAME}/default.nix << EOF
let
  pkgs = import <nixpkgs> {
    config = {};
    overlays = [
      (import ./overlay.nix)
    ];
  };
in pkgs.${PNAME}
EOF

touch ${PNAME}/{derivation,module}.nix

`derivation.nix`¶

This is an implementation of a Nixpkgs-like function, with inputs to take any Nix packages needed to build the project, and returning the package itself.

{ stdenv, rustPlatform, cmake, pkgconfig, sqlite, openssl }:

rustPlatform.buildRustPackage rec {
  name = "url-bot-rs-${version}";
  version = "0.2.0";
  src = ./.;
  buildInputs = [ openssl pkgconfig sqlite ];

  checkPhase = "";
  cargoSha256 = "sha256:1bk7kr6i5xh7b45caf93i096cbblajrli0nixx9m76m3ya7vnbp5";

  meta = with stdenv.lib; {
    description = "Minimal IRC URL bot in Rust";
    homepage = https://github.com/nuxeh/url-bot-rs;
    license = licenses.isc;
    maintainers = [ maintainers.tailhook ];
    platforms = platforms.all;
  };
}

`overlay.nix`¶

Use an overlay to allow the local derivation to be overlayed onto Nixpkgs.

self: super: {
  url-bot-rs = self.callPackage ./derivation.nix {};
}

`default.nix`¶

This file is normally used for:

Building a project with nix-build.
Entering the project environment with nix-shell, if there is no shell.nix present.
Installing the package with nix-env.

However, rather than having the build instructions directly in default.nix, we use the above derivation.nix instead, via the overlay in overlay.nix.

let
  pkgs = import <nixpkgs> {
    config = {};
    overlays = [
      (import ./overlay.nix)
    ];
  };
in pkgs.url-bot-rs

This reddit post was the original inspiration for all of this.

Preparing for integration into Nixpkgs¶

The only difference between the derivation.nix above and a derivation suitable for submitting to Nixpkgs, is that it needs to fetch the source from somewhere, rather than using the local source. To get the source from GitHub, we can use the fetchFromGitHub function in Nixpkgs.

`derivation.nix`¶

{ stdenv, rustPlatform, fetchFromGitHub, cmake, pkgconfig, sqlite, openssl }:

rustPlatform.buildRustPackage rec {
  name = "url-bot-rs-${version}";
  version = "0.2.0";
  src = fetchFromGitHub {
    owner = "nuxeh";
    repo = "url-bot-rs";
    rev = "c2a7bb93019d150643fb14c5160f493882506128";
    sha256 = "0xhs5shqrnq8d9i7j863xfc4zpk28k2js6yqsadmbfnncac34nac";
  };

  buildInputs = [ openssl pkgconfig sqlite ];

  checkPhase = "";
  cargoSha256 = "sha256:1bk7kr6i5xh7b45caf93i096cbblajrli0nixx9m76m3ya7vnbp5";

  meta = with stdenv.lib; {
    description = "Minimal IRC URL bot in Rust";
    homepage = https://github.com/nuxeh/url-bot-rs;
    license = licenses.isc;
    maintainers = [ maintainers.tailhook ];
    platforms = platforms.all;
  };
}

In order to get the inputs required for fetchFromGitHub, run:

nix-prefetch-github <owner> <project>

In this case:

nix-prefetch-github nuxeh url-bot-rs

This will give the computed SHA for master, used to verify the downloaded source.

Remember to do this from within a nix environment if not on NixOS, e.g. first

source ~/.nix-profile/etc/profile.d/nix.sh

Building locally¶

Change directory to wherever your default.nix lives.

Enter the Nix environment if using Nix on a host OS:

source ~/.nix-profile/etc/profile.d/nix.sh

Then run:

nix-build

On recent Debian, if you see:

error: cloning builder process: Operation not permitted

You should do the following (issue):

sudo sysctl kernel.unprivileged_userns_clone=1

Installing locally¶

nix-env -if default.nix

Adding the package to `configuration.nix`¶

For local testing we can add the package as an overlay, locally, to the system's evaluated Nixpkgs. This can also be a method of adding a local package to a system without adding it to Nixpkgs at all.

Place the required files on your system¶

The new package can be overlayed onto an existing system configuration (/etc/nixos/configuration.nix) so that the package can be build and installed onto the system.

This is quite simple to do, locally, on a test system, without having to use a modified nixpkgs repository, or change the nixpkgs currently used by the system.

Create the overlays directory, and a folder for our package's overlay:

mkdir -p /root/overlays/url-bot-rs

Copy the derivation.nix file from above to:

/root/overlays/url-bot-rs/derivation.nix

Copy the overlay.nix file from above to:

/root/overlays/url-bot-rs/overlay.nix

You should end up with the following directory structure:

/root/overlays
└── url-bot-rs
    ├── derivation.nix
    └── overlay.nix

Nixpkgs overlays are apparently found automatically in the directory ~/.config/nixpkgs/overlays if it exists, although I must have missed something, since it doesn't appear to work.

Integrate into `configuration.nix`¶

Apply the overlay by adding one of the following to your /etc/nixos/configuration.nix:

{
  # ...
  nixpkgs.config.packageOverrides = import /root/overlays/url-bot-rs/overlay.nix pkgs;
}

or:

{
  # ...
  nixpkgs.overlays = [
    (import /root/overlays/url-bot-rs/overlay.nix)
  ];
}

Add the package to the system. Both of the above allow you to add url-bot-rs to systemPackages as follows:

{
  # ...
  environment.systemPackages = with pkgs; [
    url-bot-rs
  ];
}

Then build with:

nixos-rebuild switch

Notes:

On build, you may have to correct the Cargo SHA, once more.
An automatic overlay look-up process, which doesn't seem to work for me, is described in the Nixpkgs manual.

Useful links¶

Creating a module¶

The next step in preparing the project for integration into a NixOS system and/or Nixpkgs, is to create a module, which is an extension of the configuration.nix, adding options for configuring the package, creating configuration files, and anything else needed to be added to the Nix store/system, in order for the package to run. This may also include such things as adding systemd units, system users, etc, if required.

This will be covered in a later post.

The basic syntax for a module could be something like:

{ config, lib, pkgs, ... }:

with lib;

let

in {

  options = {

  };

  config = mkIf cfg.enable {

  };

}

Then we add a variable to access the configuration options given in the configuration, and a definition of the configuration options we are about to create.

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.url-bot-rs;

in {

  options = {

    services.url-bot-rs = {


    };

  };

  config = mkIf cfg.enable {

  };

}

A reasonable place to start would be to add some configuration options. We'll start with user, group, and state directory, which are pretty universal. We'll add a package description variable, since this is used multiple times later on. Also, add the enable option.

{ config, lib, pkgs, ... }:

with lib;

let
    cfg = config.services.url-bot-rs;
    pkgDesc = "url-bot-rs irc bot";

in {

  options = {

    services.url-bot-rs = {

      enable = mkEnableOption "${pkgName}";

      user = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs user";
      };

      group = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs group";
      };

      stateDir = mkOption {
        type = types.path;
        default = "/var/lib/url-bot-rs/";
        description = "state directory for url-bot-rs";
        example = "/home/bob/.url-bot-rs/";
      };

    };

  };

  config = mkIf cfg.enable {

  };

}

And then, the associated implementations for these.

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.url-bot-rs;
  pkgDesc = "url-bot-rs irc bot";

in {

  options = {

    services.url-bot-rs = {

      enable = mkEnableOption "url-bot-rs";

      user = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs user";
      };

      group = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs group";
      };

      stateDir = mkOption {
        type = types.path;
        default = "/var/lib/url-bot-rs/";
        description = "state directory for url-bot-rs";
        example = "/home/bob/.url-bot-rs/";
      };

    };

  };

  config = mkIf cfg.enable {

    users.users.${cfg.user} = {
      name = cfg.user;
      group = cfg.group;
      home = cfg.stateDir;
      createHome = true;
      description = pkgDesc;
    };

    users.groups.${cfg.user} = {
      name = cfg.group;
    };

    systemd.services.url-bot-rs = {
      description = pkgDesc;
      wantedBy = [ "multi-user.target" ];
      after = [ "network-online.target" ];
      serviceConfig = {
        User = cfg.user;
        Group = cfg.group;
        Restart = "always";
	WorkingDirectory = cfg.stateDir;
        ExecStart = ''
          ${pkgs.url-bot-rs}/bin/url-bot-rs
        '';
      };
    };

  };

}

So now we have users implemented, and we have a systemd unit to run the service.

Setting the user's home directory to the state directory path, and enabling createHome is a fairly handy way to have the state directory created for you.

The only thing that remains for a basic module is to generate a configuration file. The purest way to do this is to write the configuration file to the nix store, which can be done using pkgs.writeText.

A simple way to generate a configuration file is to put the configuration contents in a string, which can then have configuration options inserted into it. This is placed in the let block.

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.url-bot-rs;
  pkgDesc = "url-bot-rs irc bot";

  confFile = pkgs.writeText "url-bot-rs.conf" ''
    [features]
    report_metadata = false
    report_mime = false
    mask_highlights = false
    send_notice = false
    history = false
    invite = false
    autosave = false
    send_errors_to_poster = false

    [parameters]
    url_limit = 10
    accept_lang = "en"
    status_channels = []

    [database]
    path = "${cfg.databasePath}"
    type = "sqlite"

    [connection]
    nickname = "url-bot-rs"
    nick_password = ""
    alt_nicks = ["url-bot-rs_"]
    username = "url-bot-rs"
    realname = "url-bot-rs"
    server = "chat.freenode.net"
    port = 6697
    password = ""
    use_ssl = true
    encoding = "UTF-8"
    channels = []
    user_info = "Feed me URLs."
    source = "https://github.com/nuxeh/url-bot-rs"
    ping_time = 180
    ping_timeout = 10
    burst_window_length = 8
    max_messages_in_burst = 15
    should_ghost = false
  '';

in {

  options = {

    services.url-bot-rs = {

      enable = mkEnableOption "url-bot-rs";

      user = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs user";
      };

      group = mkOption {
        type = types.str;
        default = "url-bot-rs";
        description = "url-bot-rs group";
      };

      stateDir = mkOption {
        type = types.path;
        default = "/var/lib/url-bot-rs/";
        description = "state directory for url-bot-rs";
        example = "/home/bob/.url-bot-rs/";
      };

      databasePath = mkOption {
        type = types.str;
        default = "history.db";
        description = "database path for url-bot-rs";
      };

    };

  };

  config = mkIf cfg.enable {

    users.users.${cfg.user} = {
      name = cfg.user;
      group = cfg.group;
      home = cfg.stateDir;
      createHome = true;
      description = pkgDesc;
    };

    users.groups.${cfg.user} = {
      name = cfg.group;
    };

    systemd.services.url-bot-rs = {
      description = pkgDesc;
      wantedBy = [ "multi-user.target" ];
      after = [ "network-online.target" ];
      serviceConfig = {
        User = cfg.user;
        Group = cfg.group;
        Restart = "always";
	WorkingDirectory = cfg.stateDir;
        ExecStart = ''
          ${pkgs.url-bot-rs}/bin/url-bot-rs --conf ${confFile}
        '';
      };
    };

  };

}

pkgs.writeText writes the string it's given to a file in the nix store, and the resulting path is stored in the confFile variable. This allows us to use that variable in the systemd unit, to provide the binary with the configuration path.

Variables from the configuration can be substituted into the configuration, as shown above with cfg.databasePath.

Integrate into `configuration.nix`¶

To build locally, import the module to your configuration.nix.

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      /root/overlays/url-bot-rs/module.nix
    ];
}

Then, minimally, enable the service:

{
    services.url-bot-rs.enable = true;
}

And as with any nix service, you can provide whatever configuration options you wish to that have been implemented by the module.

{
    services.url-bot-rs.enable = {
        enable = true;
        # ...
    };
}

Alternatives to `buildRustPackage`¶

This derivation invokes Cargo once for the whole project, so whole build is handled as a single call to Cargo.

There are alternatives, such as:

These parse Cargo.lock, generating a more complex derivation, expressing each Rust dependency of the crate, to be built individually by buildRustPackage.

This has a number of advantages, e.g. not having to rebuild all the project's dependencies for each build, and giving Nix the ability to cache each dependency individually.

Such a derivation for this project, built with crate2nix is available on GitHub. It should be noted that the length of these generated files is quite large, thousands of lines.

eipi.xyz

Packaging a Rust project for Nix

contents

Monolithic default.nix¶

Splitting up the simple default.nix¶

Preparation¶

derivation.nix¶

overlay.nix¶

default.nix¶

Preparing for integration into Nixpkgs¶

derivation.nix¶

Building locally¶

Installing locally¶

Adding the package to configuration.nix¶

Place the required files on your system¶

Integrate into configuration.nix¶

Useful links¶

Creating a module¶

Integrate into configuration.nix¶

Alternatives to buildRustPackage¶

Monolithic `default.nix`¶

Splitting up the simple `default.nix`¶

`derivation.nix`¶

`overlay.nix`¶

`default.nix`¶

`derivation.nix`¶

Adding the package to `configuration.nix`¶

Integrate into `configuration.nix`¶

Integrate into `configuration.nix`¶

Alternatives to `buildRustPackage`¶