Nix Environments ¶

Warning

The Nix setup for the Digital Marketplace is no longer maintained. Parts may be broken or missing. This document is purely for historic information. See the developer setup instead.

This page documents a totally optional (yet rather convenient) means of automatically setting up reproducible development environments for some Digital Marketplace components. For a user with Nix installed, spawning an ephemeral development environment for a nix-ified project can be as simple as performing a:

$ nix-shell --pure .

from the source directory.

Who’s nick?¶

Nix describes itself as a functional package manager, though it’s possibly better thought of as a build tool, as Nix has a fairly broad definition of what a “package” can be. A “package” can be anything from a single file to a meta-package of other “packages” or even a whole runtime environment built from these dependency building blocks.

Packages’ build instructions are described in .nix files using Nix’s own (quite pure) functional programming language . Nix focuses on attempting to build “pure” packages - that is, packages whose only external references are to files derived from other “pure” packages. This allows Nix to provide supposedly totally reproducible environments given a set of instructions, with applications that behave in a way totally independent of the underlying impure environment.

To do this, Nix forgoes the traditional unix filesystem heirarchy (/usr/lib, /bin and all your old friends) in favour of installing everything in a hash-based-named directory under /nix/store. Items under the Nix store are only bound together into a specific runtime environment as and when needed (either through some clever symlink tricks or by very selectively and specifically including certain items in a particular environment’s environment variables on demand). Breaking the system’s reliance on fixed filesystem paths enables Nix to have an arbitrary number of different versions of components (that would normally cause conflicts) installed at the same time without problems.

If you’ve ever needed two versions of OpenSSL installed at the same time (each used by a different piece of software), you’ll find there’s a problem because only one file can occupy /usr/lib/libssl.so at once. It’s possible to use some tricks to build depending software against specific versions of libraries installed in specific locations, or in /usr/local/lib, but doing so is build tool-specific and each project’s build system will vary in how reliably this works, and we’re not even mentioning getting the build to use the correct match of libraries to headers. In Nix this isn’t an issue because only the specific version of the library you asked for is included in the environment at build time and the resultant software is made to reference this library in its specific location.

TL;DR is with Nix, there’s nothing stopping you running a number of different entire distributions at the same time on top of the same kernel, even with entirely different libc versions. Note that from a technical perspective, this is not the same thing as containerization or virtualization - there is no os-policed enforcement of environment separation (either from a security or resource partitioning point of view) - it is simply that one application stack’s dependencies are completely separable from another’s. Such enforcement can be added if so desired, but in development environments that’s not the feature we’re generally looking for and more often than not gets in the way.

This makes it sound like a lot of compiling is involved, but because Nix is able to identify unique packages by hash (the hash used is based on the full set of input parameters given to the package’s build environment, so it’s possible to determine a package’s identifying hash before building it), Nix can fetch pre-compiled versions of popular packages from its “binary cache” - a build farm run by the Nix project. Little building is usually required, but when it is it is of course totally transparent to the Nix user.

nixpkgs is the standard set of package definitions that Nix tends to use. It contains a similar selection of software as would be found in a typical Linux distribution (or maybe Homebrew). It’s actually just a tree of .nix files and is maintained in GitHub similarly how to any other open source project would be.

Nix officially runs on x86 Linux and MacOS (technically known as “darwin”) (and unofficially on a few others) and can operate completely independently of any existing system package manager. The concept has been taken further with the NixOS project - a Linux distribution built entirely on top of Nix, using it to produce an entirely “stateless” system. That is, the only parts of the system that maintain state are the user data areas and parts of var, using Nix to even build its configuration files and populate /etc with these when system configuration is changed. I don’t advocate going this far just yet, at least not for development.

nixpkgs master is a continually-rolling collection of software which gets a stable branch forked off approximately every 6 months. At time of writing the most recent one is 19.03. These stable branches are synchronized with NixOS releases, so you may occasionally see the stable branch referred to as e.g. nixos-19.03.

Nix works on MacOS (“darwin”). That said, it doesn’t work quite as well as it does on Linux, for two reasons. Firstly the proprietary nature of the underlying platform makes purity more difficult for some of the lower level system libraries. Secondly there are just fewer users than on Linux, so packages get less attention and testing.

The Nix manual provides further information on Nix and the nixpkgs manual details various conventions used throughout nixpkgs and the many useful tools and shortcuts it provides for those writing expressions in the nix language.

default.nix ¶

One of the modes in which one can use Nix is through nix-shell. Given a Nix expression (usually residing in a .nix file), nix-shell will ensure all the dependencies specified by the expression are present on the system and then launch a shell in an ephemeral environment into which it has injected those dependencies.

If this sounds a bit like virtualenv, it is - only it works for the whole system environment, not just python dependencies. It has sometimes been described as a “whole-system virtualenv”. A major difference is that as soon as the user ends the shell session the environment is as good as gone - it holds no state itself.

A useful way to maintain this environment description is with a default.nix file in a project’s root directory. The relevance of the filename default.nix is simply that it’s the default filename nix-shell will look for when invoked in a directory and no other name is provided.

The –pure flag ¶

A neat trick that nix-shell can do, through use of the --pure flag, is clean the provided environment of everything but the nix-provided dependencies specified by the supplied nix expression. Including (and this is largely the point) your system-provided tools and libraries. This is extremely useful to reduce the liklihood an operation you’re performing isn’t inadvertantly referencing “impure” system components and therefore is as reproducible as possible. This can be quite critical if the inner operation you’re performing is building software and you want to make sure the result is only dependent on your nix-reproducible environment.

default.nix files in Digital Marketplace repositories ¶

Several Digital Marketplace repositories now include an experimental default.nix which, used with nix-shell, should be able to provide a full execution environment for development and testing.

The idea is that a user with Nix installed should be able to perform a:

$ nix-shell --pure .

and obtain a fully working development environment (it may be necessary to blow away your node_modules/ and bower_components/ directories before running make run_all to get them re-fetched again as they will have been constructed using a different node version).

The environment’s python version can be chosen by editing the definition of pythonPackages - probably by editing the line:

pythonPackages = pkgs.python36Packages;

to reference e.g. python27Packages. The line might of course not look exactly like that in all default.nix files. default.nix has been set up to use a separate virtualenv for each python version so it should be possible to drop out & back in to different python versions without any extra fuss.

Caveats ¶

You’ll notice that if you use the --pure flag with nix-shell, it gives you a really very pure environment - it won’t even put vi in your environment. In fact, nothing from your underlying system installation will be directly available. This is great for reasons outlined in the section on The –pure flag, but could indeed be quite annoying for day to day use. There are numerous approaches that could be taken to make this easier. It’s up to you which one you prefer. Some suggestions:

Don’t use the --pure flag and live with the danger of mixed up libraries. This will allow you to continue using your underlying installed tools.
Only use nix-shell for execution of the actual project commands, either dropping out of nix-shell whenever you’re not using the project commands, or leaving another shell active in the same directory for performing those commands. This is probably the approach I’ll take.
Add your tool of choice to the dependencies using a local.nix, causing Nix to provide this tool for you. See the section on local.nix.

Something to note is that this is not a full nixification - for the majority of the python dependencies, it just creates a virtualenv and calls pip at the last stage of its initialization. This means that the installed python modules aren’t pure in the same way proper Nix packages are. The worst effect this should have is maybe occasionally having to blow away your venv* dirs after you’ve updated your nix channels.

There’s also a possibility that pip will too aggressively cache compiled modules that it installs and not realize that it’s being asked to build/install modules against a totally different set of headers & libs as it was last time it was invoked. I haven’t actually see this be a problem yet, so we’ll probably cross that bridge when we come to it.

MacOS users will, for now, have to comment out the watchdog dependency in requirements_for_test.txt as it appears to need some impure dependencies to access system libraries (?). But there are possibly a few workarounds for this that can be discussed. The first that comes to mind is adding the nixpkgs package for watchdog (pythonPackages.watchdog, which, strangely enough, does work fine on MacOS even though I can’t quite figure out exactly how its build environment differs) to your local.nix.

The $PS1 (custom prompt) provided may not be to everyone’s taste. I tried to include a bunch of useful information in it, including a shortName for each project which some may find objectionable (this is so that it’s clear if you’ve absent-mindedly cd’d to a different project directory while remaining in a nix-shell). As shown in the example, this can also be personalized in local.example.nix, or over time we might agree on a better default one.

As much as this document talks about the “total reproducibility” of the environment, the truth is of course that default.nix bases itself on whatever your systems current default nixpkgs is set to track. This is embodied in the line:

pkgs = import <nixpkgs> {};

If you stick to a stable branch, your results should remain mostly… stable… across nixpkgs updates, but of course different people with nixpkgs set to track different branches/releases could get different results.

It would be perfectly possible to replace this line with e.g.:

nixpkgs = (import <nixpkgs> {}).fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs-channels";
    rev = "0d4431cfe90b2242723ccb1ccc90714f2f68a609";
    sha256 = "0iil6dx91widz66avnbs4m8lhygmadhyma1m2kbq57iwj73yql3w";
};

and achieve a much higher degree of reproducibility through something akin to “pinning” the environment to a specific snapshot of nixpkgs. This is an approach that some people take, but it could well become annoying as we’d have to continually be bumping this version to stay up to date with security releases and the like. I suggest we play this by ear for now.

local.nix ¶

The default.nix files have been designed to look aside at evaluation time for a file named local.nix. The contents are expected to be a nix expression defining a function. If the file is present, this function will be called applying first the args passed to the original default.nix and then oldAttrs, the attrset originally applied to mkDerivation. If that explanation means nothing to you, read the section of the manual describing the nix language. Alternatively, each repository with a default.nix should include a local.example.nix which should outline the basic idea and can be copied and adapted to a users needs.

local.nix should have been added to the .gitignore so we shouldn’t end up committing our personal settings.

default.nix in the functional tests repo ¶

A slightly different approach has been taken in the digitalmarketplace-functional-tests repository, mostly because I am much less familiar with bundler than I am with virtualenv and am less confident in hijacking some of its functionality. Nix comes with a tool called bundix which is able to generate a .nix file from a Gemfile and Gemfile.lock. This means that the gems get installed as actual nix packages, inheriting the purity benefits of doing so (though notably not all of the benefits of using properly maintained packages from nixpkgs). The bundix tool is included in this env to make it possible for people to regenerate gemset.nix.

Disadvantages of this approach are that the gemset.nix file needs to be kept in sync with the Gemfile.lock to remain useful and it’s not always obvious when things have slipped slightly out of sync. Also you’re no longer using the same tool to install your language-level dependencies everywhere so it may not be obvious when subtle problems creep in which don’t affect the Nix environment, but do affect a bundler-generated one.

If we really loved this approach, it is possible to do the same thing for python projects requirements.txt files using a tool called pypi2nix (I’m not super-keen on it personally, but am open to exploring it). Note that at time of writing it is being discussed whether to introduce a requirements.txt-generation step into our toolchain so it could end up being quite natural to autogenerate a .nix file at the same time.