Scrive Nix Workshop
Welcome to Scrive Nix Workshop! This book contains the workshop materials used for training Haskell developers at Scrive. (We are hiring Haskell developers!)
We are open sourcing this content so that the Nix community can also benefit from the content and make contribution to improve it.
The content in this workshop is currently organized based on the workshop sessions we have held at Scrive. With the materials open sourced, we welcome to changes to make this more catered for the general audience in the Nix community. We hope that the current materials are still useful for more people to learn about Nix.
Nix is a rather difficult subject with a steep learning curve, even for us! So this is also our chance to learn from the rest of the Nix community. If there is any mistake or misconception in this book, do let us know!
Build Instructions
A HTML version of this workshop is published at https://scrive.github.io/nix-workshop/. The source code of this workshop is available on GitHub. This workshop material is built using mdbook.
To view this book, first install Nix, and then run:
make serve
You can then browse to http://localhost:3000 to view the book webpage.
License
Copyright (c) 2020-2021 Scrive AB. This work is licensed under both the Creative Commons Attribution-ShareAlike 4.0 International License and the MIT license.
Introduction
What is Nix?
Programming Language
-
Dynamically typed - Similar semantics to JavaScript and Lisp.
-
Functional programming - Higher order functions, immutability, etc.
-
Lazy - Values are not evaluated until needed.
Package Manager
-
Packages as special Nix objects that produce derivations and build artifacts.
-
One package can serve as build input of another package.
-
Multiple versions of the "same" package can be present on the same system.
Build System
-
Packages are built from source code.
-
Build artifacts of packages are cached based on content address (SHA256 checksum).
-
Multi language / multi repository build system.
- Language agnostic.
- Construct your own build system pipeline.
Operating System
-
Nix itself is a pseudo operating system.
- Rich set of Nix packages that can typically be found in OS packages.
-
Nix packages can co-exist non-destructively with native OS packages.
-
All Nix artifacts are stored in
/nix. -
Global "installation" is merely a set of symlinks to Nix artifacts in
/nix/store.
-
-
Lightweight activation of global Nix packages.
-
Add
~/.nix-profile/bin/to$PATH. -
Call
source ~/.nix-profile/etc/profile.d/nix.shto activate Nix. -
Otherwise Nix is almost invisible to users if it is not activated.
-
-
NixOS is a full Linux operating system.
Reproducibility
-
Key differentiation of Nix as compared to other solutions.
-
Nix packages are built inside a lightweight sandbox.
-
No containerization.
-
Sanitize all environment variables.
-
Special
$HOMEdirectory at/homeless-shelter. -
Reset date to Unix time 0.
-
Very difficult to accidentally escape the sandbox.
-
-
Content-addressable storage.
-
Addresses of Nix packages are based on a checksum of the source code, plus other factors such as CPU architecture and operating system.
-
If the checksum of the source code changes, the addresses of the derivation and any build artifacts also change.
-
If the address of a dependency changes, the addresses of the derivation and build artifact also change.
-
Installation
Download available at https://nixos.org/download.html.
Simplest way is to run:
$ curl -L https://nixos.org/nix/install | sh
After installation, you might need to relogin to your shell to reload the environment. Otherwise, run the following to use Nix immediately:
source ~/.nix-profile/etc/profile.d/nix.sh
You may want to configure to load this automatically in ~/.bashrc or similar
file.
Update
If you have installed Nix before but have not updated it for a while, you should update it with:
nix-channel --update
This helps ensure we are installing the latest version of packages in global installation and global imports.
Learning Resources for Nix
-
- Nix Manual - Information about
nix-the-command and the Nix Language. - Nixpkgs Manual - Information about the Nix Package Set (
nixpkgs). How to extend and customise packages, how each language ecosystem is packaged, etc. - NixOS Manual - Information about the NixOS operating system. How to install/configure/update NixOS, the DSL for describing configuration options, etc.
- Nix Manual - Information about
-
nix.dev - Pragmatic guide on how to use Nix productively.
-
Awesome Nix - Curated list of Nix resources.
-
Nix Pills - Alternative Nix tutorial. Takes a bottom-up approach, explaining Nix and Nixpkgs design patterns along the way.
Install Global Packages
For newcomers, we can think of Nix as a supercharged package manager. Similar to traditional package managers, we can use Nix to install packages globally. Nix "installs" a global package by adding symlinkgs to ~/.nix-profile/bin, which should be automatically included in your shell's $PATH.
$ nix-env -i hello
installing 'hello-2.10'
building '/nix/store/mlfrpy1ahv3arh2n23p45vdpm0p4nl1x-user-environment.drv'...
created 39 symlinks in user environment
$ hello
Hello, world!
$ which hello
/home/user/.nix-profile/bin/hello
$ readlink $(which hello)
/nix/store/ylhzcjbchfihsrpsg0dxx9niwzp35y63-hello-2.10/bin/hello
Uninstall
$ nix-env --uninstall hello
uninstalling 'hello-2.10'
While convenient, global packages pollute the global environment of our system. Next we will look at how Nix shell can provide a local shell environment that provide the same dependencies that we need.
Nix Shell
You can use Nix packages without installing them globally on your machine. This is a good way to bring in the tools you need for individual projects.
$ nix-shell -p hello
[nix-shell:nix-workshop]$ hello
Hello, world!
Using Multiple Packages
$ nix-shell -p nodejs ghc cabal-install
[nix-shell:nix-workshop]$ which node
/nix/store/ndkzg5kpyp92mlzh5h66l4j393x6b256-nodejs-12.19.0/bin/node
[nix-shell:nix-workshop]$ which ghc
/nix/store/sbqnpfnx4w8jb7jq2yb71pifihwqy2a5-ghc-8.8.4/bin/ghc
[nix-shell:nix-workshop]$ which cabal
/nix/store/060x141b9fz2pm6yz4zn3i0ncavbdbf7-cabal-install-3.2.0.0/bin/cabal
Using shell.nix
It is common practice for Nix-using projects to provide a shell.nix file that specifies the shell environment. The nix-shell command reads this file, allowing us to create reproducible shell environments without using -p. These environments can provide access to any tool written in any language, without polluting the global environment. We will cover the use of shell.nix in a later chapter.
Nix Repl
nix repl
Welcome to Nix version 2.3.8. Type :? for help.
nix-repl> "Hello World!"
"Hello World!"
Nix Primitives
Strings
nix-repl> "hello"
"hello"
Booleans
nix-repl> true
true
nix-repl> false
false
nix-repl> true && false
false
nix-repl> true || false
true
Null
nix-repl> null
null
nix-repl> true && null
error: value is null while a Boolean was expected, at (string):1:1
Numbers
nix-repl> 1
1
nix-repl> 2
2
nix-repl> 1 + 2
3
String Interpolation
nix-repl> name = "John"
nix-repl> name
"John"
nix-repl> "Hello, ${name}!"
"Hello, John!"
Multiline Strings
nix-repl> ''
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Nullam augue ligula, pharetra quis mi porta.
- ${name}
''
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.\n Nullam augue ligula, pharetra quis mi porta.\n\n- John\n"
String Concatenation
nix-repl> "Hello " + "World"
"Hello World"
nix-repl> "Hello " + 123
error: cannot coerce an integer to a string, at (string):1:1
Set / Object
nix-repl> object = { foo = "foo val"; bar = "bar val"; }
nix-repl> object
{ bar = "bar val"; foo = "foo val"; }
nix-repl> object.foo
"foo val"
nix-repl> object.bar
"bar val"
Merge Objects
nix-repl> a = { foo = "foo val"; bar = "bar val"; }
nix-repl> b = { foo = "override"; baz = "baz val"; }
nix-repl> a // b
{ bar = "bar val"; baz = "baz val"; foo = "override"; }
Inherit
nix-repl> foo = "foo val"
nix-repl> bar = "bar val"
nix-repl> { foo = foo; bar = bar; }
{ bar = "bar val"; foo = "foo val"; }
nix-repl> { inherit foo bar; }
{ bar = "bar val"; foo = "foo val"; }
Inherit From Object
nix-repl> let
object = {
foo = "foo val";
bar = "bar val";
};
in
{
foo = object.foo;
baz = "baz val";
}
{ baz = "baz val"; foo = "foo val"; }
nix-repl> let
object = {
foo = "foo val";
bar = "bar val";
};
in
{
inherit (object) foo;
baz = "baz val";
}
{ baz = "baz val"; foo = "foo val"; }
List
nix-repl> list = [ "hello" 123 { foo = "foo"; } ]
nix-repl> list
[ "hello" 123 { ... } ]
nix-repl> builtins.elemAt list 2
{ foo = "foo"; }
List Concatenation
nix-repl> [ 1 2 ] ++ [ "foo" "bar" ]
[ 1 2 "foo" "bar" ]
Nix Expressions
If Expression
nix-repl> if true then 0 else 1
0
nix-repl> if false then 0 else "foo"
"foo"
nix-repl> if null then 1 else 2
error: value is null while a Boolean was expected
Let Expression
nix-repl> let
foo = "foo val";
bar = "bar val, ${foo}";
in
{ inherit foo bar; }
{ bar = "bar val, foo val"; foo = "foo val"; }
With Expression
nix-repl> let
object = {
foo = "foo val";
bar = "bar val";
};
in
[
object.foo
object.bar
]
[ "foo val" "bar val" ]
nix-repl> let
object = {
foo = "foo val";
bar = "bar val";
};
in
with object; [
foo
bar
]
[ "foo val" "bar val" ]
Function
nix-repl> greet = name: "Hello, ${name}!"
nix-repl> greet "Alice"
"Hello, Alice!"
nix-repl> greet "Bob"
"Hello, Bob!"
Curried Function
nix-repl> secret-greet = code: name:
if code == "secret"
then "Hello, ${name}!"
else "Nothing here"
nix-repl> secret-greet "secret" "John"
"Hello, John!"
nix-repl> nothing = secret-greet "wrong"
nix-repl> nothing "Alice"
"Nothing here"
nix-repl> nothing "Bob"
"Nothing here"
Named Arguments
nix-repl> greet = { name, title }: "Hello, ${title} ${name}"
nix-repl> greet { title = "Ms."; name = "Alice"; }
"Hello, Ms. Alice"
nix-repl> greet { name = "Alice"; }
error: anonymous function at (string):1:2 called without required argument 'title', at (string):1:1
Default Arguments
nix-repl> greet = { name ? "Anonymous", title ? "Ind." }: "Hello, ${title} ${name}"
nix-repl> greet {}
"Hello, Ind. Anonymous"
nix-repl> greet { name = "Bob"; }
"Hello, Ind. Bob"
nix-repl> greet { title = "Mr."; }
"Hello, Mr. Anonymous"
Lazy Evaluation
nix-repl> err = throw "something went wrong"
nix-repl> err
error: something went wrong
nix-repl> if true then 1 else err
1
nix-repl> if false then 1 else err
error: something went wrong
nix-repl> object = { foo = err; bar = "bar val"; }
nix-repl> object.bar
"bar val"
nix-repl> object.foo
error: something went wrong
Sequencing
nix-repl> builtins.seq err true
error: something went wrong
nix-repl> builtins.seq object true
true
nix-repl> builtins.deepSeq object true
error: something went wrong
File Management in Nix
String to File
nix-repl> builtins.toFile "hello.txt" "Hello World!"
"/nix/store/r4mvpxzh7rgrm4j831b2yi90zq64grqm-hello.txt"
$ cat /nix/store/r4mvpxzh7rgrm4j831b2yi90zq64grqm-hello.txt
Hello World!
Path
nix-repl> ./.
/path/to/nix-workshop
nix-repl> ./code/01-getting-started
/path/to/nix-workshop/code/01-getting-started
nix-repl> ./not-found
/path/to/nix-workshop/not-found
Path Concatenation
nix-repl> ./. + "code/01-getting-started"
/path/to/nix-workshop/code/01-getting-started
Read File
nix-repl> builtins.readFile ./code/03-nix-basics/03-files/hello.txt
"Hello World!"
nix-repl> builtins.readFile /nix/store/r4mvpxzh7rgrm4j831b2yi90zq64grqm-hello.txt
"Hello World!"
nix-repl> builtins.readFile (builtins.toFile "hello" "Hello World!")
"Hello World!"
Path
nix-repl> builtins.path { path = ./.; }
"/nix/store/s0c3cc8k6dy51zx9xicfprsl9r35zvf6-nix-workshop"
nix-repl> "${./.}"
"/nix/store/s0c3cc8k6dy51zx9xicfprsl9r35zvf6-nix-workshop"
$ ls /nix/store/s0c3cc8k6dy51zx9xicfprsl9r35zvf6-nix-workshop
01-getting-started 02-nix-commands ...
The exact address changes every time the directory is updated.
Named Path
nix-repl> workshop = builtins.path { path = ./.; name = "first-scrive-workshop"; }
nix-repl> workshop
"/nix/store/fp0lw035xhxqwgfqifxlb430lyw48r7m-first-scrive-workshop"
nix-repl> builtins.readFile (workshop + "/code/03-nix-basics/03-files/hello.txt")
"Hello World!"
Content-Addressable Path
The files hello.txt and hello-2.txt
both have the same content "Hello World!", but they produce different artifacts
in the Nix store, i.e. the name of a Nix artifact depends on the name of the
original file / directory.
nix-repl> builtins.path { path = ./code/03-nix-basics/03-files/hello.txt; }
"/nix/store/925f1jb1ajrypjbyq7rylwryqwizvhp0-hello.txt"
nix-repl> builtins.path { path = ./code/03-nix-basics/03-files/hello-2.txt; }
"/nix/store/bghk1lsjcylfm05j00zj5j42lv09i79z-hello-2.txt"
Solution: give a fixed name to path artifacts:
nix-repl> builtins.path {
name = "hello.txt";
path = ./code/03-nix-basics/03-files/hello-2.txt;
}
"/nix/store/925f1jb1ajrypjbyq7rylwryqwizvhp0-hello.txt"
Fetch URL
nix-repl> example = builtins.fetchurl "https://scrive.com/robots.txt"
nix-repl> example
[0.0 MiB DL] downloading 'https://scrive.com/robots.txt'"/nix/store/r98i29hkzwyykm984fpr4ldbai2r8lhj-robots.txt"
nix-repl> example
"/nix/store/r98i29hkzwyykm984fpr4ldbai2r8lhj-robots.txt"
$ cat /nix/store/r98i29hkzwyykm984fpr4ldbai2r8lhj-robots.txt
User-agent: *
Sitemap: https://scrive.com/sitemap.xml
Disallow: /amnesia/
Disallow: /api/
URLs are only fetched once locally!
Fetch Tarball
nix-repl> nodejs-src = builtins.fetchTarball
"https://nodejs.org/dist/v14.15.0/node-v14.15.0-linux-x64.tar.xz"
nix-repl> nodejs-src
"/nix/store/6wkj0blipzdqbsvwv03qy57n4l33scpw-source"
$ ls /nix/store/6wkj0blipzdqbsvwv03qy57n4l33scpw-source
bin CHANGELOG.md include lib LICENSE README.md share
SHA256 Checksum
Make sure that the content retrieved is the same for all users.
nix-repl> nodejs-src = builtins.fetchTarball {
name = "nodejs-src";
url = "https://nodejs.org/dist/v14.15.0/node-v14.15.0-linux-x64.tar.xz";
sha256 = "14jmakaxmlllyyprydc6826s7yk50ipvmwwrkzf6pdqis04g7a9v";
}
nix-repl> nodejs-src
"/nix/store/6wkj0blipzdqbsvwv03qy57n4l33scpw-source"
Importing Modules
Importing Nix Modules
We have the following files in 03-nix-basics/04-import:
"foo val"
[ "bar val 1" "bar val 2" ]
let
foo = import ./foo.nix;
bar = import ./bar.nix;
in
{ inherit foo bar; }
nix-repl> import ./code/03-nix-basics/04-import/foo.nix
"foo val"
nix-repl> import ./code/03-nix-basics/04-import/bar.nix
[ "bar val 1" "bar val 2" ]
nix-repl> import ./code/03-nix-basics/04-import
{ bar = [ ... ]; foo = "foo val"; }
Importing Global Modules
nix-repl> nixpkgs = import <nixpkgs> {}
nix-repl> nixpkgs.lib.stringLength "hello"
5
Importing a Tarball
Let's say we have the same foo.nix now available as a gist. We can import that file by asking Nix to fetch the tarball generated by GitHub:
nix-repl> gist = builtins.fetchTarball "https://gist.github.com/soareschen/d41e9b764018da4d2336644329c915e4/archive/476c45eaba13e23316cdca781bda7ec68676397b.tar.gz"
nix-repl> foo = import (gist + "/foo.nix")
nix-repl> foo
"foo val"
Pinning a Tarball Version
We have tested our remote foo.nix hosted on GitHub Gist, expecting it
to have the same content as the local foo.nix we have.
But the gist can be updated, yet we want to ensure that the content remains
the same. This can be done by pinning the SHA256 checksum of foo.nix to
what we expect using nix-prefetch-url:
$ nix-prefetch-url --type sha256 --unpack "https://gist.github.com/soareschen/d41e9b764018da4d2336644329c915e4/archive/476c45eaba13e23316cdca781bda7ec68676397b.tar.gz"
unpacking...
[0.0 MiB DL]
path is '/nix/store/vik2vk9ifbyps9pvhqa89px0c76cvaxz-476c45eaba13e23316cdca781bda7ec68676397b.tar.gz'
1ig5g6gvys26ka11z0wx08l72h8g5rr7p4fywk905sabdknf92yx
We can then copy the SHA256 checksum to make sure that the file content at the URL never unexpectedly changes:
nix-repl> gist = builtins.fetchTarball {
url = "https://gist.github.com/soareschen/d41e9b764018da4d2336644329c915e4/archive/476c45eaba13e23316cdca781bda7ec68676397b.tar.gz";
sha256 = "1ig5g6gvys26ka11z0wx08l72h8g5rr7p4fywk905sabdknf92yx";
}
Pinning Nixpkgs
We can use the same approach to pin nixpkgs itself. By pinning nixpkgs to a
specific commit ID and SHA256 checksum, we can be sure that everyone that uses
our Nix module are using the exact version of nixpkgs that we have specified.
For instance, we can pin the nixpkgs version we use to commit
c1e5f8723ceb684c8d501d4d4ae738fef704747e:
$ nix-prefetch-url --type sha256 --unpack \
> "https://github.com/NixOS/nixpkgs/archive/c1e5f8723ceb684c8d501d4d4ae738fef704747e.tar.gz"
unpacking...
[19.6 MiB DL]
path is '/nix/store/7ik3kdki828cnva46vnis87ha6axjk7n-c1e5f8723ceb684c8d501d4d4ae738fef704747e.tar.gz'
02k3l9wnwpmq68xmmfy4wb2panqa1rs04p1mzh2kiwn0449hl86j
Now we can import our pinned nixpkgs:
nix-repl> nixpkgs-src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/c1e5f8723ceb684c8d501d4d4ae738fef704747e.tar.gz";
sha256 = "02k3l9wnwpmq68xmmfy4wb2panqa1rs04p1mzh2kiwn0449hl86j";
}
nix-repl> nixpkgs = import nixpkgs-src {}
nix-repl> nixpkgs.lib.stringLength "hello"
5
Pinning nixpkgs is highly encouraged when developing Nix modules. Without pinning, your users may run your modules on a version of nixpkgs that is several months old.
We will go through in a later chapter on how to use
niv to automate the management of pinned
remote Nix packages.
Nix Derivation Basics
First import a pinned version of nixpkgs so that we all get the same result:
nix-repl> nixpkgs-src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/c1e5f8723ceb684c8d501d4d4ae738fef704747e.tar.gz";
sha256 = "02k3l9wnwpmq68xmmfy4wb2panqa1rs04p1mzh2kiwn0449hl86j";
}
nix-repl> nixpkgs = import nixpkgs-src {}
We use the pinned version of nixpkgs so that everyone following the
tutorial will get the exact same derivation.
Standard Derivation
nix-repl> hello-drv = nixpkgs.stdenv.mkDerivation {
name = "hello.txt";
unpackPhase = "true";
installPhase = ''
echo -n "Hello World!" > $out
'';
}
nix-repl> hello-drv
«derivation /nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv»
$ cat /nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
Derive([("out","/nix/store/f6qq9bwv0lxw5glzjmin1y1r1s3kangv-hello.txt","","")],...)
$ nix show-derivation /nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
{
"/nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv": {
"outputs": {
"out": {
"path": "/nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt"
}
},
"inputSrcs": [
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
...
}
}
Building Derivation
We can now build our derivation:
$ nix-build /nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
/nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt
$ cat /nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt
Hello World!
This may take some time to load on your computer, as Nix fetches the essential build tools that are commonly needed to build Nix packages.
We can also build the derivation within Nix repl using the :b command:
nix-repl> :b hello-drv
[1 built, 0.0 MiB DL]
this derivation produced the following outputs:
out -> /nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt
Tracing Derivation
Our hello-drv produce the same output as hello.txt in previous chapter,
but produce different output in the Nix store. (previously we had
/nix/store/925f1jb1ajrypjbyq7rylwryqwizvhp0-hello.txt)
We can trace the dependencies of the derivation back to its source:
$ nix-store --query --deriver /nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt
/nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
$ nix-store --query --deriver /nix/store/925f1jb1ajrypjbyq7rylwryqwizvhp0-hello.txt
unknown-deriver
Our hello.txt built from stdenv.mkDerivation is built from a derivation
artifact hello.txt.drv, but our hello.txt created from builtins.path
has no deriver.
In other words, the Nix artifacts are different because they are produced from
different derivations.
We can further trace the dependencies of hello.txt.drv:
$ nix-store -qR /nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv
/nix/store/03f77phmfdmsbfpcc6mspjfff3yc9fdj-setup-hook.sh
...
That's a lot of dependencies! Where are they being used? We will learn about it in the next chapter.
Derivation in a Nix File
We save the same earlier derivation we defined inside a Nix file named
hello.nix. Now we can build our derivation directly:
$ nix-build 04-derivations/01-derivation-basics/hello.nix
/nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt
We can also get the derivation without building it using nix-instantiate:
$ nix-instantiate 04-derivations/01-derivation-basics/hello.nix
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/ad6c51ia15p9arjmvvqkn9fys9sf1kdw-hello.txt.drv
Ignore the warning from nix-instantiate, as we don't care whether the derivation
is deleted during Nix garbage collection.
Notice that both the derivation and the build output have the same hash
as the earlier result we had in nix repl.
Caching Nix Build Artifacts
We create hello-sleep.nix as a variant of
hello.nix which sleeps for 10 seconds in its buildPhase.
(We will go through how each phase works in the next chapter)
The 10 seconds sleep simulates the time taken to compile a program.
We can see what happens when we try to build the same Nix derivation
multiple times.
First, instantiating a derivation is not affected by the build time:
$ time nix-instantiate 04-derivations/01-derivation-basics/hello-sleep.nix
/nix/store/58ngrpwgv6hl633a1iyjbmjqlbdqjw92-hello.txt.drv
real 0m0,217s
user 0m0,179s
sys 0m0,032s
The first time we build hello-sleep.nix, it is going to take about 10 seconds.
We can also see the logs we printed during the build phase is shown:
$ time nix-build 04-derivations/01-derivation-basics/hello-sleep.nix
these derivations will be built:
/nix/store/58ngrpwgv6hl633a1iyjbmjqlbdqjw92-hello.txt.drv
building '/nix/store/58ngrpwgv6hl633a1iyjbmjqlbdqjw92-hello.txt.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
Building hello world...
Finished building hello world!
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/lm801yriwjj4298ry74hdv5j0rpkpacq-hello.txt
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
patching script interpreter paths in /nix/store/lm801yriwjj4298ry74hdv5j0rpkpacq-hello.txt
checking for references to /build/ in /nix/store/lm801yriwjj4298ry74hdv5j0rpkpacq-hello.txt...
/nix/store/lm801yriwjj4298ry74hdv5j0rpkpacq-hello.txt
real 0m12,202s
user 0m0,371s
sys 0m0,084s
But the next time we build hello-sleep.nix, it will take no time to build,
and there is no build output:
$ time nix-build 03-nix-basics/05-derivation/hello-sleep.nix
/nix/store/lm801yriwjj4298ry74hdv5j0rpkpacq-hello.txt
real 0m0,310s
user 0m0,256s
sys 0m0,047s
Nix determines whether a derivation needs to be rebuilt based on the input
derivation. For our case, in both calls to hello-sleep.nix,
nix-build instantiates the derivation behind the scene: i.e.
/nix/store/58ngrpwgv6hl633a1iyjbmjqlbdqjw92-hello.txt.drv.
So it determines that the result has previously already
been built, and reuses the same Nix artifact.
Derivation as File
With the duck-typing nature of Nix, derivations act just like files in Nix.
We can actually treat the hello-drv we defined earlier as a file and
read from it:
nix-repl> builtins.readFile hello-drv
querying info about missing paths"Hello World!"
How does that work? Internally Nix lazily builds a
derivation when it is evaluated, and turns it into
a file path. We can verify that by using builtins.toPath:
nix-repl> builtins.toPath hello-drv
"/nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt"
With this property, we can also import derivations from a Nix file, and then use it as if the derivation has been built:
nix-repl> hello = import ./code/04-derivations/01-derivation-basics/hello.nix
nix-repl> builtins.readFile hello
querying info about missing paths"Hello World!"
We can even use a derivation as a string. Nix automatically builds the derivation when it is evaluated as a string:
nix-repl> "path of hello: ${hello}"
"path of hello: /nix/store/z449wrqvwncs8clk7bsliabv1g1ci3n3-hello.txt"
Dependencies
Previously we have built toy derivations with dummy output. In practice, Nix derivations are used for building programs, with build artifacts such as compiled binaries being the derivation output.
We can demonstrate this property by "building" a greet program. First we have to import a pinned version of nixpkgs as before. To simplify the process we abstract it out into a nixpkgs.nix at the root directory.
Now we build a greet program in greet.nix:
let
nixpkgs = import ../../nixpkgs.nix;
greet = nixpkgs.stdenv.mkDerivation {
name = "greet";
unpackPhase = "true";
buildPhase = ''
echo "building greet..."
sleep 3
'';
installPhase = ''
mkdir -p $out/bin
cat <<'EOF' > $out/bin/greet
#!/usr/bin/env bash
echo "Hello, $1!"
EOF
chmod +x $out/bin/greet
'';
};
in
greet
$ nix-build code/04-derivations/02-dependencies/greet.nix
these derivations will be built:
/nix/store/97lmyym0isl0ism7pfnv1b0ls4cahpi8-greet.drv
building '/nix/store/97lmyym0isl0ism7pfnv1b0ls4cahpi8-greet.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
building greet...
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin
patching script interpreter paths in /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin/greet: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash"
checking for references to /build/ in /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet...
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
Now we can run greet:
$ result/bin/greet John
Hello, John!
Let's try to see what's inside the produced greet script:
$ cat result/bin/greet
#!/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
echo "Hello, $1!"
The shebang to the bash shell has been modified to pin to the Nix version of Bash.
Upper Greet
Our greet program can now be used as a dependency to other derivations.
Let's create an upper-greet derivation that convert any greet result to
upper case.
let
nixpkgs = import ../../nixpkgs.nix;
inherit (nixpkgs) coreutils;
greet = import ./greet.nix;
in
nixpkgs.stdenv.mkDerivation {
name = "upper-greet";
unpackPhase = "true";
buildPhase = ''
echo "building upper-greet..."
sleep 3
'';
installPhase = ''
mkdir -p $out/bin
cat <<'EOF' > $out/bin/upper-greet
#!/usr/bin/env bash
${greet}/bin/greet "$@" | ${coreutils}/bin/tr [a-z] [A-Z]
EOF
chmod +x $out/bin/upper-greet
'';
}
Show Derivation
First we instantiate upper-greet.drv without building it yet:
drv=$(nix-instantiate 04-derivations/02-dependencies/upper-greet.nix)
We can use nix show-derivation to find out the dependency graph of the
derivation of upper-greet:
$ nix show-derivation $drv
{
"/nix/store/n61g8616l7g7zv32q52yrzmzr850mjp0-upper-greet.drv": {
"outputs": {
"out": {
"path": "/nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet"
}
},
"inputSrcs": [
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"inputDrvs": {
"/nix/store/7gby8zic1p851ap63q1vdpwy7z1db85c-coreutils-8.32.drv": [
"out"
],
"/nix/store/97lmyym0isl0ism7pfnv1b0ls4cahpi8-greet.drv": [
"out"
],
"/nix/store/l54djrh1n7d8zdfn26w7v6zjh5wp7faa-bash-4.4-p23.drv": [
"out"
],
"/nix/store/x9why09hwx2pcnmw0fw7hhh1511hyskl-stdenv-linux.drv": [
"out"
]
},
...
"env": {
"buildInputs": "",
"buildPhase": "echo \"building upper-greet...\"\nsleep 3\n",
"builder": "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash",
...
"installPhase": "mkdir -p $out/bin\n\ncat <<'EOF' > $out/bin/upper-greet\n#!/usr/bin/env bash\n/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin/greet \"$@\" | /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/tr [a-z] [A-Z]\nEOF\n\nchmod +x $out/bin/upper-greet\n",
"name": "upper-greet",
...
We can see that greet.drv is included as one of inputDrvs. This means that when
upper-greet.drv is being built, greet.drv will have to be built first.
The output path of upper-greet.drv is listed in outputs. This shows that
the output hash of a derivation is fixed, regardless of the content of the
build result.
This is also why the output path of greet.drv is used directly in env.installPhase
of upper-greet.drv, even for the case when greet.drv has not been built.
Build Derivation
$ nix-build 04-derivations/02-dependencies/upper-greet.nix
these derivations will be built:
/nix/store/n61g8616l7g7zv32q52yrzmzr850mjp0-upper-greet.drv
building '/nix/store/n61g8616l7g7zv32q52yrzmzr850mjp0-upper-greet.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
building upper-greet...
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet/bin
patching script interpreter paths in /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet
/nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet/bin/upper-greet: interpreter directive changed from "/usr/bin/env bash" to "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash"
checking for references to /build/ in /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet...
/nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet
As expected, the greet results are turned into upper case.
$ result/bin/upper-greet John
HELLO, JOHN!
The absolute paths to greet and coreutils are extended:
$ cat result/bin/upper-greet
#!/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin/greet "$@" | /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/tr [a-z] [A-Z]
Runtime Dependency
If we query the references of the upper-greet output (not the derivation),
we can see that greet is still a runtime dependency of upper-greet.
$ nix-store --query --references /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet
/nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32
/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
We can use nix why-depends to find out why Nix thinks greet is a runtime
dependency to upper-greet:
$ nix why-depends /nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
/nix/store/dj2vp64gbja0bp65lngrw9q4lrm1a8r3-upper-greet
╚═══bin/upper-greet: …ash-4.4-p23/bin/bash./nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin/greet "$@" | /nix/sto…
=> /nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet
Fibonacci
Nix dependencies can be nested arbitrarily deep. We can demonstrate that by building a fibonacci Nix derivation with the following behavior:
- Answers are produced in
$out/answer. - Each build takes 3 seconds to produce the answer.
fib(0)is 0, andfib(1)is 1.fib(n)depends on the answers fromfib(n-1)andfib(n-2).- The builds are prefixed with a name, so that we can force Nix to re-evaluate the whole sequence by changing the name.
let
nixpkgs = import ../../nixpkgs.nix;
inherit (nixpkgs) stdenv;
prefixed-fib = prefix:
let fib = n:
assert builtins.isInt n;
assert n >= 0;
let
n-str = builtins.toString n;
in
if n == 0 || n == 1
then
stdenv.mkDerivation {
name = "${prefix}-fib-${n-str}";
unpackPhase = "true";
buildPhase = ''
echo "Producing base case fib(${n-str})..."
sleep 3
echo "The answer to fib(${n-str}) is ${n-str}"
'';
installPhase = ''
mkdir -p $out
echo "${n-str}" > $out/answer
'';
}
else
let
fib-1 = fib (n - 1);
fib-2 = fib (n - 2);
n-1-str = builtins.toString (n - 1);
n-2-str = builtins.toString (n - 2);
in
stdenv.mkDerivation {
name = "${prefix}-fib-${n-str}";
unpackPhase = "true";
buildPhase = ''
fib_1=$(cat ${fib-1}/answer)
fib_2=$(cat ${fib-2}/answer)
echo "Calculating the answer of fib(${n-str}).."
echo "Given fib(${n-1-str}) = $fib_1,"
echo "and given fib(${n-2-str}) = $fib_2.."
sleep 3
answer=$(( $fib_1 + $fib_2 ))
echo "The answer to fib(${n-str}) is $answer"
'';
installPhase = ''
mkdir -p $out
echo "$answer" > $out/answer
'';
}
;
in fib;
in
prefixed-fib
To make sure we build the fibonacci sequence from scratch each time, we can use
$(date +%s) with the Unix timestamp as the prefix to our builds.
Let's try fib(0) and fib(1):
$ time nix-build -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$(date +%s)\" 0"
these derivations will be built:
/nix/store/yyy5fz5rsws6a812c9xc5ps1hwh9lm98-1605561354-fib-0.drv
building '/nix/store/yyy5fz5rsws6a812c9xc5ps1hwh9lm98-1605561354-fib-0.drv'...
...
no configure script, doing nothing
building
Producing base case fib(0)...
The answer to fib(0) is 0
...
checking for references to /build/ in /nix/store/9kvw6x88l9nx42mvrgzif60a72h66fqz-1605561354-fib-0...
/nix/store/9kvw6x88l9nx42mvrgzif60a72h66fqz-1605561354-fib-0
real 0m4,763s
user 0m0,476s
sys 0m0,130s
$ time nix-build -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$(date +%s)\" 1"
these derivations will be built:
/nix/store/qs2pc54dmd21xlhlqgzwmgfj98y1kr8n-1605561412-fib-1.drv
building '/nix/store/qs2pc54dmd21xlhlqgzwmgfj98y1kr8n-1605561412-fib-1.drv'...
...
Producing base case fib(1)...
The answer to fib(1) is 1
...
checking for references to /build/ in /nix/store/426cpqvvr7lbg92hywmbw7552vggpway-1605561412-fib-1...
/nix/store/426cpqvvr7lbg92hywmbw7552vggpway-1605561412-fib-1
real 0m5,279s
user 0m0,415s
sys 0m0,102s
So both fib(0) and fib(1) takes roughly 4~5 seconds to build.
Let's try fib(2):
$ time nix-build -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$(date +%s)\" 2"
these derivations will be built:
/nix/store/r8n1v9ifk0q6mf75j35scn3s2dwa03j7-1605561535-fib-1.drv
/nix/store/xxrzkbsnq1jpp4s5fdkpklxr3fbc0aq6-1605561535-fib-0.drv
/nix/store/1bgy17jbakr9yn1yz0bnwhnnz1y1xsdc-1605561535-fib-2.drv
building '/nix/store/xxrzkbsnq1jpp4s5fdkpklxr3fbc0aq6-1605561535-fib-0.drv'...
...
Producing base case fib(0)...
The answer to fib(0) is 0
...
checking for references to /build/ in /nix/store/8k74irn8w8rzpccd30wdn8589nq0wdv5-1605561535-fib-0...
building '/nix/store/r8n1v9ifk0q6mf75j35scn3s2dwa03j7-1605561535-fib-1.drv'...
...
Producing base case fib(1)...
The answer to fib(1) is 1
...
checking for references to /build/ in /nix/store/p716qrpmyi4649k2njfy7gb97ksyi5y3-1605561535-fib-1...
building '/nix/store/1bgy17jbakr9yn1yz0bnwhnnz1y1xsdc-1605561535-fib-2.drv'...
...
Calculating the answer of fib(2)..
Given fib(1) = 1,
and given fib(0) = 0..
The answer to fib(2) is 1
...
checking for references to /build/ in /nix/store/qnaad56wgknlgki9r3kpmr4fhc7x8vxv-1605561535-fib-2...
/nix/store/qnaad56wgknlgki9r3kpmr4fhc7x8vxv-1605561535-fib-2
real 0m12,599s
user 0m0,658s
sys 0m0,198s
So building fib(2) causes fib(1) and fib(0) to also be built.
With this going on, if we are going to build fib(5), then in total it is going to take a lot of time!
but if we have built fib(4) already, then building fib(5) will be very fast.
Let's fix our prefix to see Nix cache in effect:
$ prefix=$(date +%s)
$ time nix-build -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$prefix\" 4"
these derivations will be built:
/nix/store/gz9bgzmna8v7pw5giclfhrk81dp1z0rw-1605561962-fib-1.drv
/nix/store/y2waqw60jqawallz4q3r64iwrrihnd1p-1605561962-fib-0.drv
/nix/store/nwj4bmrqgfv1fkvhh00bl2v03c3zqpy1-1605561962-fib-2.drv
/nix/store/17d2r2dd52qvmfa5k3dm9gkl1k09wdb8-1605561962-fib-3.drv
/nix/store/w4w4la01p9a2i8mlg6fm15il6vmgcqzl-1605561962-fib-4.drv
building '/nix/store/y2waqw60jqawallz4q3r64iwrrihnd1p-1605561962-fib-0.drv'...
...
Calculating the answer of fib(4)..
Given fib(3) = 2,
and given fib(2) = 1..
The answer to fib(4) is 3
...
checking for references to /build/ in /nix/store/c26q5vs8vdfr268nqgamjk8bcypf8b7r-1605561962-fib-4...
/nix/store/c26q5vs8vdfr268nqgamjk8bcypf8b7r-1605561962-fib-4
real 0m19,486s
user 0m0,910s
sys 0m0,253s
Now run fib(5):
$ time nix-build -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$prefix\" 5"
these derivations will be built:
/nix/store/nyb25403l4m5n69y3djlffsyzvpwyv6g-1605561962-fib-5.drv
building '/nix/store/nyb25403l4m5n69y3djlffsyzvpwyv6g-1605561962-fib-5.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
Calculating the answer of fib(5)..
Given fib(4) = 3,
and given fib(3) = 2..
The answer to fib(5) is 5
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/p4iq9jaiid469ycgjm5v3ks3w0v35spi-1605561962-fib-5
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
patching script interpreter paths in /nix/store/p4iq9jaiid469ycgjm5v3ks3w0v35spi-1605561962-fib-5
checking for references to /build/ in /nix/store/p4iq9jaiid469ycgjm5v3ks3w0v35spi-1605561962-fib-5...
/nix/store/p4iq9jaiid469ycgjm5v3ks3w0v35spi-1605561962-fib-5
real 0m7,916s
user 0m0,461s
sys 0m0,151s
Now only fib-5.drv needs to be built.
Lazy Evaluation
A derivation like fib(10) is going to take a long time to build. So we don't really want to build
it unless we actually need it. In fact, we also wouldn't want to build fib(0) through fib(10)
unless they are actually needed.
With Nix's lazy evaluation strategy, we in fact get the laziness property that none of the fibonacci derivations are going to be built unless they are needed:
$ time nix-instantiate -E "import ./code/04-derivations/03-fibonacci/fib.nix \"$(date +%s)\" 10"
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv
real 0m0,235s
user 0m0,182s
sys 0m0,038s
We can see that while fib(10) has fib(0) through fib(9) as its dependencies,
but they are not being built just yet.
$ nix-store -qR /nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv | grep fib
/nix/store/31xdxhxynqndaiym043bjjky0l229vlg-1605562192-fib-1.drv
/nix/store/r4wkf6774ahva1zchk77kpvjf14xigrl-1605562192-fib-0.drv
/nix/store/8agah8vidgxi6yp9ki066yimfr16kigg-1605562192-fib-2.drv
/nix/store/ka3glx20pgzkvgan88xl87xki51y5pmi-1605562192-fib-3.drv
/nix/store/v5zra6kayssdg04n7xpjljqa1q5jjyqn-1605562192-fib-4.drv
/nix/store/1cqwdhnk8f55lxlajmjw6rzq2lq12x5l-1605562192-fib-5.drv
/nix/store/qrbrgdv16f7mc2xfalrgmypfz6c7yljq-1605562192-fib-6.drv
/nix/store/p5chc4l9mjqw5871lkd6har4hyjp55fj-1605562192-fib-7.drv
/nix/store/gn9hp9jcsfclrsdx6qlvjd051w4rsx8b-1605562192-fib-8.drv
/nix/store/6av3vlibm4knm4m3djcrfrhhb6jck3zx-1605562192-fib-9.drv
/nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv
Input Derivations
If we inspect the derivation, the derivations fib-9.drv and fib-8.drv
are listed as one of the input derivations:
$ nix show-derivation /nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv
{
"/nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv": {
"outputs": {
"out": {
"path": "/nix/store/g7415lrzl6b43vnw58dgkxg5nzbjplp0-1605562192-fib-10"
}
},
"inputSrcs": [
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"inputDrvs": {
"/nix/store/6av3vlibm4knm4m3djcrfrhhb6jck3zx-1605562192-fib-9.drv": [
"out"
],
"/nix/store/gn9hp9jcsfclrsdx6qlvjd051w4rsx8b-1605562192-fib-8.drv": [
"out"
],
"/nix/store/l54djrh1n7d8zdfn26w7v6zjh5wp7faa-bash-4.4-p23.drv": [
"out"
],
"/nix/store/x9why09hwx2pcnmw0fw7hhh1511hyskl-stdenv-linux.drv": [
"out"
]
},
...
"env": {
"buildInputs": "",
"buildPhase": "fib_1=$(cat /nix/store/zr1ziy94ig8isdg0gdliz0sm59abh2l1-1605562192-fib-9/answer)\nfib_2=$(cat /nix/store/zvivby98nbjlb6pszp6qla4v1r6zwj82
-1605562192-fib-8/answer)\n\necho \"Calculating the answer of fib(10)..\"\necho \"Given fib(9) = $fib_1,\"\necho \"and given fib(8) = $fib_2..\"\n\nsleep 3\n\
nanswer=$(( $fib_1 + $fib_2 ))\necho \"The answer to fib(10) is $answer\"\n",
...
Building Actual Derivation
We can build them later on when nix-build is actually called, or we can get the cached result
from else where before building them.
$ time nix-build /nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv
these derivations will be built:
/nix/store/31xdxhxynqndaiym043bjjky0l229vlg-1605562192-fib-1.drv
/nix/store/r4wkf6774ahva1zchk77kpvjf14xigrl-1605562192-fib-0.drv
/nix/store/8agah8vidgxi6yp9ki066yimfr16kigg-1605562192-fib-2.drv
/nix/store/ka3glx20pgzkvgan88xl87xki51y5pmi-1605562192-fib-3.drv
/nix/store/v5zra6kayssdg04n7xpjljqa1q5jjyqn-1605562192-fib-4.drv
/nix/store/1cqwdhnk8f55lxlajmjw6rzq2lq12x5l-1605562192-fib-5.drv
/nix/store/qrbrgdv16f7mc2xfalrgmypfz6c7yljq-1605562192-fib-6.drv
/nix/store/p5chc4l9mjqw5871lkd6har4hyjp55fj-1605562192-fib-7.drv
/nix/store/gn9hp9jcsfclrsdx6qlvjd051w4rsx8b-1605562192-fib-8.drv
/nix/store/6av3vlibm4knm4m3djcrfrhhb6jck3zx-1605562192-fib-9.drv
/nix/store/dwjxm9rqxfbhf4m8nbg5wzddx1j4rcpl-1605562192-fib-10.drv
building '/nix/store/r4wkf6774ahva1zchk77kpvjf14xigrl-1605562192-fib-0.drv'...
...
Calculating the answer of fib(10)..
Given fib(9) = 34,
and given fib(8) = 21..
The answer to fib(10) is 55
...
checking for references to /build/ in /nix/store/g7415lrzl6b43vnw58dgkxg5nzbjplp0-1605562192-fib-10...
/nix/store/g7415lrzl6b43vnw58dgkxg5nzbjplp0-1605562192-fib-10
real 0m39,818s
user 0m1,260s
sys 0m0,413s
Evaluation-Time Dependencies
In our original fib.nix, the build output of earlier fibonacci
numbers are used during the build phase of the current derivation.
But if we somehow uses the earlier fibonacci numbers to build
the derviation itself, Nix would behave quite differently.
let
fib-1 = fib (n - 1);
fib-2 = fib (n - 2);
n-1-str = builtins.toString (n - 1);
n-2-str = builtins.toString (n - 2);
fib-1-answer = nixpkgs.lib.removeSuffix "\n"
(builtins.readFile "${fib-1}/answer");
fib-2-answer = nixpkgs.lib.removeSuffix "\n"
(builtins.readFile "${fib-2}/answer");
in
stdenv.mkDerivation {
name = "${prefix}-fib-${n-str}";
unpackPhase = "true";
buildPhase = ''
echo "Calculating the answer of fib(${n-str}).."
echo "Given fib(${n-1-str}) = ${fib-1-answer},"
echo "and given fib(${n-2-str}) = ${fib-2-answer}.."
sleep 3
answer=$(( ${fib-1-answer} + ${fib-2-answer} ))
echo "The answer to fib(${n-str}) is $answer"
'';
...
}
Let's try to instantiate the serialized version of fib(4):
$ time nix-instantiate -E "import ./code/04-derivations/03-fibonacci/fib-serialized.nix \"$(date +%s)\" 4"
building '/nix/store/97h18adc1358s8ri9mjzmnbvbbsj7p0a-1606145205-fib-1.drv'...
...
Producing base case fib(1)...
The answer to fib(1) is 1
...
building '/nix/store/fyizpk51qr2k6pm6v2pbqynfqf8ws68p-1606145205-fib-0.drv'...
...
Producing base case fib(0)...
The answer to fib(0) is 0
...
building '/nix/store/8qglzk0vq984mx65f3993rqjpipc3q9j-1606145205-fib-2.drv'...
...
Calculating the answer of fib(2)..
Given fib(1) = 1,
and given fib(0) = 0..
The answer to fib(2) is 1
...
building '/nix/store/4p8xy858gwc348576h6rz39a3l7wk64l-1606145205-fib-3.drv'...
...
Calculating the answer of fib(3)..
Given fib(2) = 1,
and given fib(1) = 1..
The answer to fib(3) is 2
...
/nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv
real 0m20,016s
user 0m1,221s
sys 0m0,261s
What happened here? fib(0) to fib(3) are built even though we are just
instantiating fib(4).
Inspecting Input Derivation
Showing the derivation of fib-4.drv gives us a better idea:
$ nix show-derivation /nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv
{
"/nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv": {
"outputs": {
"out": {
"path": "/nix/store/fgq0j5mlqpy99mfdfc3v4bvbd6wr2slg-1606145205-fib-4"
}
},
...
"inputDrvs": {
"/nix/store/l54djrh1n7d8zdfn26w7v6zjh5wp7faa-bash-4.4-p23.drv": [
"out"
],
"/nix/store/x9why09hwx2pcnmw0fw7hhh1511hyskl-stdenv-linux.drv": [
"out"
]
},
"env": {
"buildInputs": "",
"buildPhase": "echo \"Calculating the answer of fib(4)..\"\necho \"Given fib(3) = 2,\"\necho \"and given fib(2) = 1..\"\n\nsleep 3\n\nanswer=$(( 2 + 1 ))\necho \"The answer to fib(4) is $answer\"\n",
...
Thanks to builtins.readFile, the results for fib(3) and fib(2) have in fact
been calculated and inlined inside the the derivation itself. They are no longer
listed in the input derivation.
Caching Problem of Evaluation-Time Dependencies
In fact, fib(3) and fib(2) are not even shown as dependencies in fib(4) anymore:
$ nix-store -qR --include-outputs /nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv | grep fib
/nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv
/nix/store/fgq0j5mlqpy99mfdfc3v4bvbd6wr2slg-1606145205-fib-4
This has a consequence in caching Nix dependencies. Not knowing fib(0) to
fib(3) are actually input to fib(4), it would be difficult to properly
cache these dependencies to Cachix.
Import From Derivation
Evaluation-time dependencies can also occur if we import a .nix file from a derivation.
Since Nix has to read the file content to be able to import the .nix file, the
derivation has to be built at evaluation time. This is more commonly known as
Import From Derivation, or
IFD in short.
IFD should be avoided if possible. However as we will see in later chapters, there are valid use cases that can only be solved using IFD.
Non-Lazy Build
If we build fib(4) now, indeed only fib(4) itself is being built.
$ nix-build /nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv
these derivations will be built:
/nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv
building '/nix/store/c29ap9ljazs7k0jx687hnm3s0rgsz2vm-1606145205-fib-4.drv'...
...
Calculating the answer of fib(4)..
Given fib(3) = 2,
and given fib(2) = 1..
The answer to fib(4) is 3
...
/nix/store/fgq0j5mlqpy99mfdfc3v4bvbd6wr2slg-1606145205-fib-4
The same effect can also happen if we import .nix files from derivation outputs.
Lesson learnt: parallelization in Nix can still be tricky. Try your best to include dependencies as input derivations, and lazily refer to the built output of dependencies only during build time.
Raw Derivation
We have previously used stdenv.mkDerivation to define toy derivations without
looking into how derivations work. Here we will go deeper into Nix derivations,
starting with the most basic derivation construct, builtins.derivation.
From the repl, we can see that builtins.derivation is a function:
nix-repl> builtins.derivation
«lambda @ /nix/store/qxayqjmlpqnmwg5yfsjjayw220ls8i2r-nix-2.3.8/share/nix/corepkgs/derivation.nix:4:1»
Since builtins.derviation is more primitive as compared to stdenv.mkDerivation,
the way we can build a derivation is also more involved:
nix-repl> builtins.derivation {
name = "hello";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
''
echo "Hello World!" > $out
''
];
}
«derivation /nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv»
System
First we have to supply a system attribute, which we set it to
the current OS we are running on. It is most common to have the
system values as "x86_64-linux" or "x86_64-darwin".
nix-repl> builtins.currentSystem
"x86_64-linux"
The system attribute is required because Nix supports cross compilation.
So we can also define derivations that are built on different platforms
than the one we are on.
Builder
The builder attribute expects a file path to an executable script that is
called when the derivation is built. To keep things simple, we use the bash
shell from nixpkgs.bash as the builder program.
The args attribute is used to specify the command line line arguments
passed to the builder program. Since bash itself do not know how to
build the program we want, we pass the command string using -c
to execute the bash script echo "Hello World!" > $out
Now we can try to build the derivation and see that it works:
$ nix-build /nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv
these derivations will be built:
/nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv
building '/nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv'...
/nix/store/dsgf85gxzw167v320sy08as72c0hk8wd-hello
$ cat /nix/store/dsgf85gxzw167v320sy08as72c0hk8wd-hello
Hello World!
Explicit Dependencies
Inside builtins.derivation, almost all dependencies have to be provided
explicitly, even the bash shell that we are running on. Since we specify
bash as the builder program, it is also shown in the list of inputDrvs
of our derivation.
$ nix show-derivation /nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv
{
"/nix/store/hbsv13kn5imfri16f6g2l5c2jy6dfmxl-hello.drv": {
"outputs": {
"out": {
"path": "/nix/store/dsgf85gxzw167v320sy08as72c0hk8wd-hello"
}
},
"inputSrcs": [],
"inputDrvs": {
"/nix/store/l54djrh1n7d8zdfn26w7v6zjh5wp7faa-bash-4.4-p23.drv": [
"out"
]
},
"platform": "x86_64-linux",
"builder": "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash",
"args": [
"-c",
"echo \"Hello World!\" > $out\n"
],
"env": {
"builder": "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash",
"name": "hello",
"out": "/nix/store/dsgf85gxzw167v320sy08as72c0hk8wd-hello",
"system": "x86_64-linux"
}
}
}
Inspecting the Build Environment
We can use the env command to inspect the environment variables inside our build script.
Let's try and build a derivation that prints the environment to the terminal.
nix-repl> builtins.derivation {
name = "env";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
''
set -x
ls -la .
ls -la /
env
touch $out
''
];
}
«derivation /nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv»
$ nix-build /nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv
+ nix-build /nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv
these derivations will be built:
/nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv
building '/nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv'...
+ ls -la .
bash: line 1: ls: command not found
+ ls -la /
bash: line 2: ls: command not found
+ env
bash: line 3: env: command not found
+ touch /nix/store/blcl4m2vgga6i86kh13nqlvx1l2ha7v5-env
bash: line 4: touch: command not found
builder for '/nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv' failed with exit code 127
error: build of '/nix/store/4nq2kgcmryhwjh5sg05jgwsd4ixh81ia-env.drv' failed
Not good, with builtins.derivation, not even basic commands like ls, env, and touch
are provided. (As seen previously, echo is provided though)
Instead, we also have to specify our build dependencies explicitly with nixpkgs.coreutils
providing the basic shell commands:
nix-repl> builtins.derivation {
name = "env";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
''
set -x
${nixpkgs.coreutils}/bin/ls -la .
${nixpkgs.coreutils}/bin/ls -la /
${nixpkgs.coreutils}/bin/env
${nixpkgs.coreutils}/bin/touch $out
''
];
}
«derivation /nix/store/c4bp5bvx73fz9jf1si64i00as30k9fga-env.drv»
$ nix-build /nix/store/c4bp5bvx73fz9jf1si64i00as30k9fga-env.drv
these derivations will be built:
/nix/store/c4bp5bvx73fz9jf1si64i00as30k9fga-env.drv
building '/nix/store/c4bp5bvx73fz9jf1si64i00as30k9fga-env.drv'...
+ /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/ls -la .
total 8
drwx------ 2 nixbld nixbld 4096 Nov 30 19:09 .
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:09 ..
+ /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/ls -la /
total 32
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:09 .
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:09 ..
drwxr-xr-x 2 nixbld nixbld 4096 Nov 30 19:09 bin
drwx------ 2 nixbld nixbld 4096 Nov 30 19:09 build
drwxr-xr-x 4 nixbld nixbld 4096 Nov 30 19:09 dev
drwxr-xr-x 2 nixbld nixbld 4096 Nov 30 19:09 etc
drwxr-xr-x 3 nixbld nixbld 4096 Nov 30 19:09 nix
dr-xr-xr-x 410 nobody nogroup 0 Nov 30 19:09 proc
drwxrwxrwt 2 nixbld nixbld 4096 Nov 30 19:09 tmp
+ /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/env
out=/nix/store/3rv14i75j4wyp6n9fila5rll4f99yksi-env
builder=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
NIX_LOG_FD=2
system=x86_64-linux
PWD=/build
HOME=/homeless-shelter
TMP=/build
NIX_STORE=/nix/store
TMPDIR=/build
name=env
TERM=xterm-256color
TEMPDIR=/build
SHLVL=1
NIX_BUILD_CORES=8
TEMP=/build
PATH=/path-not-set
NIX_BUILD_TOP=/build
_=/nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/env
+ /nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/touch /nix/store/3rv14i75j4wyp6n9fila5rll4f99yksi-env
/nix/store/3rv14i75j4wyp6n9fila5rll4f99yksi-env
Nix Sandbox
From above we can see that the build environment inside a Nix build script is sandboxed.
According to Nix manual:
If set to true, builds will be performed in a sandboxed environment, i.e., they’re isolated from the normal file system hierarchy and will only see their dependencies in the Nix store, the temporary build directory, private versions of /proc, /dev, /dev/shm and /dev/pts (on Linux), and the paths configured with the sandbox-paths option. This is useful to prevent undeclared dependencies on files in directories such as /usr/bin. In addition, on Linux, builds run in private PID, mount, network, IPC and UTS namespaces to isolate them from other processes in the system (except that fixed-output derivations do not run in private network namespace to ensure they can access the network).
Nix sandbox should be enabled by default. You can check your sandbox configuration with:
$ nix show-config | grep sandbox
extra-sandbox-paths =
sandbox = true
sandbox-build-dir = /build
sandbox-dev-shm-size = 50%
sandbox-fallback = true
sandbox-paths = /bin/sh=/nix/store/w0xp1k96c1dvmx6m4wl1569cdzy47w5r-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox
Capturing Build Environment
We can capture the build environment as a file by saving the output of env to $out.
nix-repl> builtins.derivation {
name = "env";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
"${nixpkgs.coreutils}/bin/env > $out"
];
}
«derivation /nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv»
$ nix-build /nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv
these derivations will be built:
/nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv
building '/nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv'...
/nix/store/6kjgg8j3y44g1ja95swqdd1v8xp6mwi1-env
$ cat /nix/store/6kjgg8j3y44g1ja95swqdd1v8xp6mwi1-env
out=/nix/store/6kjgg8j3y44g1ja95swqdd1v8xp6mwi1-env
builder=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
NIX_LOG_FD=2
system=x86_64-linux
PWD=/build
HOME=/homeless-shelter
TMP=/build
NIX_STORE=/nix/store
TMPDIR=/build
name=env
TERM=xterm-256color
TEMPDIR=/build
SHLVL=1
NIX_BUILD_CORES=8
TEMP=/build
PATH=/path-not-set
NIX_BUILD_TOP=/build
_=/nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/env
Nix Shell
Nix achieves reproducible build by carefully setting/unsetting the appropriate environment variables, so that our derivations are always built with the same environment regardless of where it is being built.
However since the derivation is built in a sandboxed environment, it may be difficult to debug when there are build errors, or rapid prototyping with the source code changed frequently.
We can get almost the same environment as inside nix build by entering a Nix shell.
$ nix-shell --pure --run env /nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv
__ETC_PROFILE_SOURCED=1
DISPLAY=:1
out=/nix/store/6kjgg8j3y44g1ja95swqdd1v8xp6mwi1-env
builder=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
USER=user
system=x86_64-linux
PWD=/path/to/nix-workshop
HOME=/home/user
TMP=/run/user/1000
NIX_STORE=/nix/store
TMPDIR=/run/user/1000
name=env
IN_NIX_SHELL=pure
TERM=xterm-256color
TEMPDIR=/run/user/1000
SHLVL=3
NIX_BUILD_CORES=8
TEMP=/run/user/1000
LOGNAME=user
PATH=/nix/store/lf467z8nr5y50q1vqnlbhpv2jachx3cs-bash-interactive-4.4-p23/bin:/home/user/.nix-profile/bin:...
NIX_BUILD_TOP=/run/user/1000
_=/usr/bin/env
Our pure Nix environment look pretty similar to the environment we captured in nix-build.
There are however a few differences, in particular with $PATH.
According the manual
for the --pure option in nix-shell:
If this flag is specified, the environment is almost entirely cleared before the interactive shell is started, so you get an environment that more closely corresponds to the “real” Nix build. A few variables, in particular
HOME,USERandDISPLAY, are retained. Note that (depending on your Bash installation)/etc/bashrcis still sourced, so any variables set there will affect the interactive shell.
We can compare the differences by diffing the output of both environments:
$ drv=/nix/store/39ah25v6iwlka3jl2angxrlx00mk2ijd-env.drv
$ diff --color <(cat $(nix-build $drv)) <(nix-shell $drv --pure --run env)
In contrast, the default impure Nix shell keeps all existing environment variables, and only add or override variables that are introduced by the derivation.
$ nix-shell $drv --run env
Environment Variables
If we observe the captured build environment, almost all attributes we passed to
builtins.derivation are converted into environment variables.
In fact, we can define any number of attributes to be used as environment variables inside our build script.
nix-repl> builtins.derivation {
name = "foo";
foo = "foo val";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
"echo $foo > $out"
];
}
«derivation /nix/store/v1i0khcvxy5bkyv2iq0kqzhcbfcfml8m-foo.drv»
We can see from the build output that the value of $foo is in fact
captured.
$ nix-build /nix/store/v1i0khcvxy5bkyv2iq0kqzhcbfcfml8m-foo.drv
these derivations will be built:
/nix/store/v1i0khcvxy5bkyv2iq0kqzhcbfcfml8m-foo.drv
building '/nix/store/v1i0khcvxy5bkyv2iq0kqzhcbfcfml8m-foo.drv'...
/nix/store/zmgp33rl2sh3l32syhq4h8gph3f4s1k9-foo
$ cat /nix/store/zmgp33rl2sh3l32syhq4h8gph3f4s1k9-foo
foo val
We can also get the same $foo variable set when entering Nix shell:
$ nix-shell --pure --run 'echo $foo' /nix/store/v1i0khcvxy5bkyv2iq0kqzhcbfcfml8m-foo.drv
foo val
Setting Dependencies as Variables
We can set out dependencies as custom attributes in a derivation and then refer to them as environment variables during the build.
For example, we can add the greet package we defined earlier
and set it as $greet in the shell.
nix-repl> greet = import ./04-derivations/02-dependencies/greet.nix
nix-repl> builtins.derivation {
inherit greet;
name = "greet-alice";
system = builtins.currentSystem;
builder = "${nixpkgs.bash}/bin/bash";
args = [
"-c"
"$greet/bin/greet Alice > $out"
];
}
«derivation /nix/store/68gdf6z0rjcyl8xcwix3gfafndsa50jj-greet-alice.drv»
$ nix-build /nix/store/68gdf6z0rjcyl8xcwix3gfafndsa50jj-greet-alice.drv
these derivations will be built:
/nix/store/68gdf6z0rjcyl8xcwix3gfafndsa50jj-greet-alice.drv
building '/nix/store/68gdf6z0rjcyl8xcwix3gfafndsa50jj-greet-alice.drv'...
/nix/store/dd290zmn983fs1w33nnq9gyh3cnj2jif-greet-alice
$ cat /nix/store/dd290zmn983fs1w33nnq9gyh3cnj2jif-greet-alice
Hello, Alice!
Standard Derivation
builtins.derivation provides the minimal functionality to define a Nix
derivation. However all dependencies have to be manually managed, which
can be pretty cumbersome. In practice, most Nix derivations are built
on top of stdenv.mkDerivation, which provide many battery-included
functionalities that helps make defining derivations easy.
The tradeoff is that stdenv.mkDerivation is much more complex than
builtins.derivation. With the detour to understand
builtins.derivation first, we can hopefully have an easier time
understanding stdenv.mkDerivation
Inspecting Build Environment
We can inspect the standard environment in similar way.
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "inspect";
unpackPhase = "true";
buildPhase = ''
set -x
ls -la .
ls -la /
env
set +x
'';
installPhase = "touch $out";
}
«derivation /nix/store/vdyp9cxs0li87app03vm8zbxmq0lhw5l-inspect.drv»
$ nix-build /nix/store/vdyp9cxs0li87app03vm8zbxmq0lhw5l-inspect.drv
these derivations will be built:
/nix/store/vdyp9cxs0li87app03vm8zbxmq0lhw5l-inspect.drv
building '/nix/store/vdyp9cxs0li87app03vm8zbxmq0lhw5l-inspect.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
++ ls -la .
total 16
drwx------ 2 nixbld nixbld 4096 Nov 30 19:42 .
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:42 ..
-rw-r--r-- 1 nixbld nixbld 5013 Nov 30 19:42 env-vars
++ ls -la /
total 32
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:42 .
drwxr-x--- 9 nixbld nixbld 4096 Nov 30 19:42 ..
drwxr-xr-x 2 nixbld nixbld 4096 Nov 30 19:42 bin
drwx------ 2 nixbld nixbld 4096 Nov 30 19:42 build
drwxr-xr-x 4 nixbld nixbld 4096 Nov 30 19:42 dev
drwxr-xr-x 2 nixbld nixbld 4096 Nov 30 19:42 etc
drwxr-xr-x 3 nixbld nixbld 4096 Nov 30 19:42 nix
dr-xr-xr-x 405 nobody nogroup 0 Nov 30 19:42 proc
drwxrwxrwt 2 nixbld nixbld 4096 Nov 30 19:42 tmp
++ env
...
unpackPhase=true
propagatedBuildInputs=
stdenv=/nix/store/ajq5dfwn4hzlx1qf2xxwb6rj8a7s65nm-stdenv-linux
TZ=UTC
OLDPWD=/build
out=/nix/store/a226brzfy71vr6vkfy4m188qs9f7k7g7-inspect
CONFIG_SHELL=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
buildInputs=
builder=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
...
buildPhase=set -x
ls -la .
ls -la /
env
set +x
PATH=/nix/store/cr86kfhzfwa558mzav4rnfkbz00hw27w-patchelf-0.12/bin:/nix/store/ppfvi0cfcpdr83klw5kx6si2l260n1gh-gcc-wrapper-9.3.0/bin:...
NIX_BUILD_TOP=/build
depsBuildTargetPropagated=
NIX_ENFORCE_PURITY=1
SIZE=size
nativeBuildInputs=
LD=ld
patches=
depsTargetTargetPropagated=
_=/nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin/env
++ set +x
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/a226brzfy71vr6vkfy4m188qs9f7k7g7-inspect
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
patching script interpreter paths in /nix/store/a226brzfy71vr6vkfy4m188qs9f7k7g7-inspect
checking for references to /build/ in /nix/store/a226brzfy71vr6vkfy4m188qs9f7k7g7-inspect...
/nix/store/a226brzfy71vr6vkfy4m188qs9f7k7g7-inspect
As we can see, our standard environment is quite more complicated than the minimal environment
provided by builtins.derivation. We also have a number of executables added to $PATH,
which we can use without specifying them as dependencies.
Capturing the Build Environment
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "env";
unpackPhase = "true";
installPhase = "env > $out";
}
«derivation /nix/store/5rgcvwndbc4525ypbb0r1vgqpbxgcy2g-env.drv»
$ cat $(nix-build /nix/store/5rgcvwndbc4525ypbb0r1vgqpbxgcy2g-env.drv)
...
unpackPhase=true
propagatedBuildInputs=
stdenv=/nix/store/ajq5dfwn4hzlx1qf2xxwb6rj8a7s65nm-stdenv-linux
TZ=UTC
OLDPWD=/build
out=/nix/store/rkjhcjhdj6ba7r7n7fasq8gmzxi5hk72-env
CONFIG_SHELL=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
buildInputs=
builder=/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash
...
Build Inputs
stdenv.mkDerivation also provides a convenient way of adding dependencies
to appropriate environment variables with the buildInputs attribute.
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "greet-alice";
buildInputs = [ greet ];
unpackPhase = "true";
installPhase = "greet Alice > $out";
}
«derivation /nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv»
$ nix-build /nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv
these derivations will be built:
/nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv
building '/nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv'...
...
/nix/store/kp32rzq63barqa55q3mf761gsggi2bq6-greet-alice
$ cat /nix/store/kp32rzq63barqa55q3mf761gsggi2bq6-greet-alice
Hello, Alice!
We can check that greet is added to $PATH using Nix shell:
$ drv=/nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv
$ nix-shell $drv --pure --run "command -v greet"
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin/greet
$ nix-shell $drv --pure --run 'echo $PATH' | tr ':' '\n'
...
/nix/store/2shqhfsyzz4rnfyysbzgyp5kbfk29750-coreutils-8.32/bin
/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet/bin
...
stdenv also adds the build inputs to other variables.
$ nix-shell $drv --pure --run 'echo $NIX_LDFLAGS'
-rpath /nix/store/kp32rzq63barqa55q3mf761gsggi2bq6-greet-alice/lib64 -rpath /nix/store/kp32rzq63barqa55q3mf761gsggi2bq6-greet-alice/lib
Note that the paths /nix/store/kp32rzq63barqa55q3mf761gsggi2bq6-greet-alice/lib
does not exist, but stdenv still sets the variables anyway.
Stdenv Script
How do stdenv.mkDerivation do the magic compared to builtins.derivation?
We can find out by first inspecting the derivation:
$ nix show-derivation $drv
{
"/nix/store/in40c5fl13ziqzds3wfg2ag7ax2xmq5l-greet-alice.drv": {
...
"builder": "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash",
"args": [
"-e",
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"env": {
"buildInputs": "/nix/store/l6xy4qjr8x3ni16skfilw0fvnda13szq-greet",
"builder": "/nix/store/qdp56fi357fgxxnkjrwx1g67hrk775im-bash-4.4-p23/bin/bash",
...
"installPhase": "greet Alice > $out",
"name": "greet-alice",
...
"stdenv": "/nix/store/ajq5dfwn4hzlx1qf2xxwb6rj8a7s65nm-stdenv-linux",
...
}
}
}
stdenv is also using bash as the builder, and have it evaluate the
script at /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh.
Let's see what's inside there:
$ cat /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
source $stdenv/setup
genericBuild
So the magic is hidden inside $stdenv/setup, with the $stdenv
variable set to /nix/store/ajq5dfwn4hzlx1qf2xxwb6rj8a7s65nm-stdenv-linux.
We can open it and see what's inside.
nix-shell $drv --run 'cat $stdenv/setup'
There are quite a lot of shell scripts happening. If we search through the script,
we can see that environment variables such as buildInputs, buildPhase, and
installPhase are being referred inside $stdenv/setup.
In other words, instead of having to figure how to setup various environment
variables to work with various dependencies, $stdenv/setup provides a higher
level abstraction of doing the setup for us. We just have to define
the build inputs and steps that we need, and $stdenv/setup will fill
in the missing pieces from us.
In fact, $stdenv/setup is also being sourced when we enter a Nix shell of a
stdenv derivation.
From the nix-shell manual:
The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself. It will then start an interactive shell in which all environment variables defined by the derivation path have been set to their corresponding values, and the script $stdenv/setup has been sourced. This is useful for reproducing the environment of a derivation for development.
Deriving Environment at Build Time
One question we might ask is, why is stdenv doing the heavy lifting steps only
at build time inside a shell script. We could as well parse the dependencies
inside Nix at evaluation time, and produce a derivation with everything
setup already.
However recall from the previous example of
fib-serialized.nix. If we try to peek into the content of a dependency derivation,
that would instead become an evaluation time dependency. If stdenv is
looking into the content of all dependencies inside Nix, then we can
only know how to build the derivation after all dependencies have been built.
Instead, stdenv avoids this to allow the derivation dependencies to
be built in parallel by Nix. With that, we can only read the content
of our dependencies at build time, which happens inside the build script.
Build Phases
stdenv provides many different phases, with default behavior of
what to run if no script for that phase is provided.
Many of the phases follow the build steps introduced by Autotools. When we are building non-C/C++ projects, only a few phases are essential. Still, it is useful to take a quick look at what phases are there, and what they offers.
The nixpkgs manual has the full list of phases.
The phases Attribute
We can force stdenv to run only specific phases by specifying them
in the phases attribute.
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "hello";
phases = [ "installPhase" ];
installPhase = "echo 'Hello World!' > $out";
}
«derivation /nix/store/m09hj2xs3yc45y3d4rdm8wks7cay00ak-hello.drv»
$ nix-build /nix/store/m09hj2xs3yc45y3d4rdm8wks7cay00ak-hello.drv
these derivations will be built:
/nix/store/m09hj2xs3yc45y3d4rdm8wks7cay00ak-hello.drv
building '/nix/store/m09hj2xs3yc45y3d4rdm8wks7cay00ak-hello.drv'...
installing
/nix/store/hpj3y6as9s07444qi6nap0f5dp5k84b6-hello
$ cat /nix/store/hpj3y6as9s07444qi6nap0f5dp5k84b6-hello
Hello World!
You may notice that the build log for this version of hello is much shorter
than the usual output of standard derivations. Messages such as
post-installation fixup are not shown here.
This is because those are implicit steps performed in phases such as fixupPhase.
We will go through later why those phases are there. But as you can see, these
phases can be disabled by explicitly specifying the phases to run.
There is also no need to explicitly skip required phases like unpackPhase,
which we previously set to true.
Unpack Phase
The unpack phase is used to unpack source code into temporary directories
to be used for compilation. By default, unpackPhase unpacks the source
code specified in $src, and if none is provided, it will abort with an error.
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "hello";
installPhase = "echo 'Hello World!' > $out";
}
«derivation /nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv»
$ nix-build /nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv
these derivations will be built:
/nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv
building '/nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv'...
unpacking sources
variable $src or $srcs should point to the source
builder for '/nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv' failed with exit code 1
error: build of '/nix/store/cx3p30jp0y0l8ixl426drsp81vcqagpr-hello.drv' failed
This is why when we don't have any source code, we have to explicitly skip
the unpackPhase by telling it to run true instead.
Let's try to see what is done in unpackPhase when we give it some source directories.
nix-repl> nixpkgs.stdenv.mkDerivation {
name = "fibonacci-src";
src = ./04-derivations/03-fibonacci;
installPhase = ''
set -x
pwd
ls -la .
cp -r ./ $out/
set +x
'';
}
«derivation /nix/store/8l0s01nk0fc1zicb9qkdmpwsw01qr5p8-fibonacci.drv»
$ nix-build /nix/store/8l0s01nk0fc1zicb9qkdmpwsw01qr5p8-fibonacci.drv
these derivations will be built:
/nix/store/8l0s01nk0fc1zicb9qkdmpwsw01qr5p8-fibonacci.drv
building '/nix/store/8l0s01nk0fc1zicb9qkdmpwsw01qr5p8-fibonacci.drv'...
unpacking sources
unpacking source archive /nix/store/a5f73yy0a8dn0p12pfriqbqyag0ksfkq-03-fibonacci
source root is 03-fibonacci
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
++ pwd
/build/03-fibonacci
++ ls -la .
total 16
drwxr-xr-x 2 nixbld nixbld 4096 Jan 1 1970 .
drwx------ 3 nixbld nixbld 4096 Dec 1 09:19 ..
-rw-r--r-- 1 nixbld nixbld 1772 Jan 1 1970 fib-serialized.nix
-rw-r--r-- 1 nixbld nixbld 1602 Jan 1 1970 fib.nix
++ cp -r ./ /nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci/
++ set +x
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci
strip is /nix/store/bnjps68g8ax6abzvys2xpx12imrx8949-binutils-2.31.1/bin/strip
patching script interpreter paths in /nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci
checking for references to /build/ in /nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci...
/nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci
$ ls -la /nix/store/qqd4msyqya0xhqxcyra0lf7v09z2q522-fibonacci
total 5728
dr-xr-xr-x 2 user user 4096 Jan 1 1970 .
drwxr-xr-x 5403 user user 5849088 Dez 1 10:19 ..
-r--r--r-- 1 user user 1602 Jan 1 1970 fib.nix
-r--r--r-- 1 user user 1772 Jan 1 1970 fib-serialized.nix
As we can see, unpackPhase copies the content of the source code specified in $src
into the temporary build directory. It also modifies the chmod permissions to
allow write permission to the files and directories.
Patch Phase
Configure Phase
Build Phase
Check Phase
Install Phase
Fixup Phase
Dependency Management
We have seen in previous chapters that Nix makes it easy to construct complex build dependencies with several benefits:
- Non-related dependencies can be built in parallel.
- Reproducible builds make it easy to cache and reuse dependencies.
However Nix does not provide any mechanism for dependency resolution, e.g. choose from multiple versions of dependencies and determining the most suitable versions.
As an example, we will build hypothetical Nix packages resembling
Minecraft crafting recipes
with versioning schemes following semver.
Let's first try to build our first version of pickaxe, which is
made of wood:
pickaxe
- 1.0.0
- stick ^1.0.1
- planks ~2.1.0
stick
- 1.0.3
- 1.1.2
- 2.0.0
planks
- 2.1.0
- 2.1.1
- 2.2.1
The first step in deciding the appropriate versions to be used to build
pickaxe-1.0.0 is to rule out invalid versions. With that, stick-2.0.0
is ruled out because it is outside of the ^1.0.1 range. Similarly
planks-2.2.1 is outside the bound for ~2.1.0.
After filtering out the invalid versions, there are still multiple versions
of stick and planks available. As a result there can be multiple
version candidates for building pickaxe-1.0.0. For example, we can use
stick-1.0.3 and planks-2.1.1. But are those the best versions to be used?
Depending on the dependency resolution algorithm used, we may get different
answers. Though in general, we can usually expect the algorithm to choose
the latest versions that are compatible with the required range. So we should
expect to get stick-1.1.2 and planks-2.1.1 as the answers.
Nested Dependencies
In reality, dependency resolution can be more complicated because of
nested dependencies. Let's say both stick and planks both depend
on wood:
stick
- 1.0.3
- wood ^1.5.0
- 1.1.2
- wood ~2.0.0
planks
- 2.1.0
- wood ^2.0.0
- 2.1.1
- wood ~2.3.0
wood
- 1.5.0
- 2.0.1
- 2.3.2
In such case, the only solution is to use stick-1.1.2 and planks-2.1.0,
because the other version combinations do not have a common wood version
usable by both stick and planks.
Package Managers
Dependency resolution is a complex topic on its own. Different languages have their own package managers that deal with dependency resolution differently. e.g. cabal-install, npm, mvn, etc. There are also OS-level package managers that have to deal with dependencies resolution. e.g. apt (for Debian and Ubuntu), rpm (Fedora), pacman (Arch Linux, Manjaro), etc.
To support package management across multiple languages and multiple platforms, Nix has its own unique challenge of managing dependencies. At this point, Nix itself do not provide any mechanism for resolving dependencies. Instead Nix users have to come out with their own higher level design patterns to resolve dependencies, such as in nixpkgs.
Package Registry
For a dependency resolution algorithm to determine what versions of dependency to use, it must first refer to a registry that contains all versions available to all packages. Each package manager have their own registry, e.g. Hackage, npm registry, Debian registry, etc.
Package registries are usually mutable databases that are constantly updated. This creates an issue with reproducibility: the result given back from a dependency resolution algorithm depends on the mutable state of the registry at the time the algorithm is executed.
In other words, say if we try to resolve the dependencies of
pickaxe-1.0.0 today, we may get stick-1.1.2 and planks-2.1.0.
But if we resolve the same dependencies tomorrow, we might get
stick-1.1.3 because new version of stick is published.
To make it worse, stick-1.1.3 may contain unexpected changes
that causes pickaxe-1.0.0 to break.
Version Pinning
Even without Nix, there is a strong use case to pin the versions to a particular snapshot of the registry. This is to make sure that, no matter when we try to resolve the dependencies, we will always get back the same dependencies.
Package Lock
One common approach is to create a lock file containing the result of
running the dependency resolution algorithm, and include the lock file into the version control system (e.g. GIT). For instance the lock file could be package-lock.json (for npm) or
cabal.project.freeze (for haskell projects). With the lock files available, we can even
skip dependency resolution in the future, and just use the result in the
lock file.
Registry Snapshot
An alternative approach would be to specify the snapshot of the
package registry itself. For example, cabal accepts an
index-state option for us to specify a timestamp of the
Hackage snapshot that it should resolve the dependencies from.
With that we can specify the timestamp of the time we first
build our dependencies, and not worry about new versions of
packages being added in the future.
However there can still be other variables that can affect the outcome. For example, the package manager itself may update the dependency resolution algorithm, so we may still get different results depending on the version of package manager used.
Upgrading Dependencies
The strategies for pinning dependencies does not eliminate the need to resolving the plans in the first place, or the need to upgrade or install new dependencies.
In the ideal world, we would like to be able to just specify the dependencies we need, and have the package manager give us the best versions that just work. But reality is messy, and dependencies can have breaking changes all the time.
Versioning Schemes
There are many attempts at coming up with versioning schemes that carry breakage information with them, such as semver and PVP. However they require package authors to manually follow the rules, and rules can be broken, intentionally or not.
Exponential Versions
In reality, each combination of dependency versions produce
a unique version of the full package that needs to be tested.
There is never just one version of pickaxe-1.0.0, but
exponential number of versions of pickaxe-1.0.0 depending
on the versions of stick, planks, and their transitive
dependencies.
To make matters worse, real world software also tend to have implicit dependencies on the runtime environment, such as executables, shared libraries, and operating system APIs.
So for each of the versions of pickaxe-1.0.0 with pinned
dependencies, we would also have multiple versions of that
software for different platforms, e.g. Linux, MacOS, Windows,
Android, iOS, etc. Even among these platforms, there are
also multiple releases of the platform, e.g. Debian 10,
Ubuntu 20.04, MacOS Big Sur, etc.
Mono Versioning
Despite all these complexities, we still like to pretend
that there is only one or few versions of pickaxe-1.0.0
ever existed. One way to tame down this complexity is
through mono versioning.
Monorepo
The simplest kind of mono versioning is by having a single repository that contains all its components and dependencies. For each commit in this repository, there is exactly one version each component and dependecy. We simply ignore the possibility of other valid combinations of component versions, and not support them.
Lockfiles in Monorepo
Package managers such as godep check the source code of dependencies
into a monorepo.
As an alternative, we can check just the lockfiles into the repository,
and have the package managers fetch them separately.
Checking the lock file is still effectively mono-versioning the dependencies. For each commit in the repository, we support only the exact dependencies specified in the lockfile of that commit. We simply pretend that no other versions of the dependencies are available.
Mono Registry
Taking the idea to extreme, we can also freeze all dependencies in a package registry and provide only one version of each dependency at any point in time.
This is the approach for registries such as Stackage, which guarantees that all Haskell dependencies only have one version that always work.
Mono registry tend to work more cleanly together with monorepo. In a project, we can specify just the snapshot version of the mono registry that we are using, and there is no need for messy details such as generating the lockfiles in the first place.
Mono Environment
Nixpkgs is also a mono registry for the standard packages in Nix. For each
version of nixpkgs, there is exactly one version of packages such as bash,
gcc, etc. But since these packages used to be provided by operating
systems, we can say that nixpkgs is also providing a mono environment
to our software.
When we create a monorepo with pinned nixpkgs, we are not only providing exactly one version of each dependencies, but also exactly one version of the environment to run on.
Mono environment restricts the specification of our software so that it does not just run on platforms such as any version of Linux or any version of Debian 10. We just pretend that there is exact one version of OS as specified in nixpkgs.
Pros and Cons of Mono Versioning
There is a fundamental difference in philosophy between multi-versioning and mono-versioning that makes it difficult for the two camps to reconcile. At its core, there are a few factors involved.
Stability
Mono-versioning places much higher value in stability. People in this camp want to make sure each version of the software always work. They achieve that by significantly limiting the number of versions of the software, and thoroughly testing softwares before upgrading any version.
Mono-versioning tend to put emphasis in LTS (long term support) releases, where its components are guaranteed to not have any breaking changes and be given years of support.
Rapid Releases
Mono versioning tend to suffer in providing slower releases. When a new version of component is available, it has to be tested to not break any other component before the new version can be released.
In contrast, multi-versioning allows a new component to be released immediately. This allows software to independently upgrade the dependencies, at the risk of it may break on some of the software.
Blurring the Line
There is no clear cut off whether the mono-versioning or multi-versioning approaches are better. In practice, we tend to take a hybrid approach in large software projects.
For example, in a company each team may have different monorepos for their projects to manage their own dependencies. The full release of the software suite is then a multi-versioning combination of each team's projects, which can break during development. Finally in production, the exact versions of each subprojects are pinned to specific versions before deployment.
Although Nix is more suitable for mono-versioning development, some of its features also make it easier to manage multi-versioned projects, by building a mono "Nixified" version of the projects.
Basic Haskell Project
We will take a quick look on the Nix structure for a trivial Haskell project, in haskell-project-v1.
Main.hs:
module Main where
main :: IO ()
main = putStrLn "Hello, Haskell!"
haskell-project.cabal:
cabal-version: 2.4
name: haskell-project
version: 0.1.0.0
license: ISC
build-type: Simple
executable hello
main-is: Main.hs
build-depends: base >=4.13
default-language: Haskell2010
Nix Dependency Management with Niv
We will use multiple Nix sources including nixpkgs and Haskell.nix
in our Haskell projects. However specifying the Nix dependencies
explicitly like in nixpkgs.nix can be a bit
cumbersome.
let
nixpkgs-src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/fcc81bc974fabd86991b8962bd30a47eb43e7d34.tar.gz";
sha256 = "1ysjmn79pl7srlzgfr35nsxq43rm1va8dqp60h09nlmw2fsq9zrc";
};
nixpkgs = import nixpkgs-src {};
in
nixpkgs
Instead we can use niv to
manage the dependencies for us. Niv allows us to easily add
any remote sources as a Nix dependency, and provide them
in a single sources object.
We can initialize niv in the project directory as follows:
$ nix-shell -j4 -E \
'let nixpkgs = import ./nixpkgs.nix;
in nixpkgs.mkShell { buildInputs = [ nixpkgs.niv ]; }'
[nix-shell]$ 05-package-management/haskell-project-v1
[nix-shell]$ niv init
By default, niv will initialize with the latest nixpkgs version available. We can explicitly override the commit version of nixpkgs to the one in this tutorial.
[nix-shell]$ niv update nixpkgs --branch nixpkgs-unstable \
--rev c1e5f8723ceb684c8d501d4d4a
e738fef704747e
Update nixpkgs
Done: Update nixpkgs
We can also add new dependencies such as Haskell.nix using
niv add:
[nix-shell]$ niv add input-output-hk/haskell.nix \
--rev 180779b7f530dcd2a45c7d00541f0f3e3d8471b5
Adding package haskell.nix
Writing new sources file
Done: Adding package haskell.nix
Two new files, nix/sources.json and sources.nix will be created
by niv. To load the source dependencies, we can simply do
sources = import ./nix/sources.nix {} to import the source object.
The source files are then available in the corresponding
attributes of the sources object, such as sources.nixpkgs.
Naive Attempt
Let's try to create a naive default.nix that tries to build with cabal directly:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsPkgs = nixpkgs.haskell.packages.ghc8102;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
in
nixpkgs.stdenv.mkDerivation {
inherit src;
name = "haskell-project";
buildInputs = [
hsPkgs.ghc
hsPkgs.cabal-install
];
builPhase = ''
cabal build all
'';
installPhase = ''
cabal install --installdir=$out --install-method=copy
'';
}
-
We use
builtins.pathto include our Haskell source code, with a filter function to filter out the localdist-newstyledirectory. -
We use GHC 8.10.2 provided from
nixpkgs.haskell.packages.ghc8102. -
We add
ghcandcabal-installintobuildInputs.
Try building it:
$ nix-build 05-package-management/haskell-project-v1/nix/01-naive/
these derivations will be built:
/nix/store/w1yscims73lrypddqcnri2vphqfnbim6-haskell-project.drv
building '/nix/store/w1yscims73lrypddqcnri2vphqfnbim6-haskell-project.drv'...
unpacking sources
unpacking source archive /nix/store/rc0pr7b71fm84az7d3gk4pdk62v8s0j0-haskell-project-src
source root is haskell-project-src
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
Config file path source is default config file.
Config file /homeless-shelter/.cabal/config not found.
Writing default configuration to /homeless-shelter/.cabal/config
dieVerbatim: user error (cabal: Couldn't establish HTTP connection. Possible cause: HTTP proxy server
is down.
)
builder for '/nix/store/w1yscims73lrypddqcnri2vphqfnbim6-haskell-project.drv' failed with exit code 1
error: build of '/nix/store/w1yscims73lrypddqcnri2vphqfnbim6-haskell-project.drv' failed
Not good. Cabal tries to access the network to get the current Hackage registry state and fails. There is good reason for this - there is no way for Nix to know that cabal's access to network is reproducible.
We can still use it as a Nix shell to build our project manually, because there is network access in Nix shell.
$ nix-shell 05-package-management/haskell-project-v1/nix/01-naive/
[nix-shell]$ cd 05-package-management/haskell-project-v1/haskell
[nix-shell]$ cabal build all
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following will be built (use -v for more details):
- haskell-project-0.1.0.0 (exe:hello) (first run)
Configuring executable 'hello' for haskell-project-0.1.0.0..
Preprocessing executable 'hello' for haskell-project-0.1.0.0..
Building executable 'hello' for haskell-project-0.1.0.0..
[1 of 1] Compiling Main ( Main.hs, /mnt/gamma/scrive/nix-workshop/code/05-package-management/haskell-project-v1/haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.2/haskell-project-0.1.0.0/x/hello/build/hello/hello-tmp/Main.o )
Linking /mnt/gamma/scrive/nix-workshop/code/05-package-management/haskell-project-v1/haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.2/haskell-project-0.1.0.0/x/hello/build/hello/hello ...
Default Attempt
We can instead try the default way of building Haskell packages in Nix. There is a full tutorial by Gabriel. Here we will just take a quick tour.
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsPkgs = nixpkgs.haskell.packages.ghc8102;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
-
We use the
hsPkgs.callCabal2nixfunction to create a nixpkgs-style package. -
We then call
hsPkgs.callPackageto "instantiate" our project with the dependencies taken fromhsPkgs.
Now try to build it:
$ nix-build 05-package-management/haskell-project-v1/nix/02-nixpkgs/
building '/nix/store/8rgnd9620lf287i0nw4j3z4wb01pd36a-cabal2nix-haskell-project.drv'...
installing
these derivations will be built:
/nix/store/as92yri0vvfi5yck5gajckfx064fy0qy-haskell-project-0.1.0.0.drv
building '/nix/store/as92yri0vvfi5yck5gajckfx064fy0qy-haskell-project-0.1.0.0.drv'...
...
Preprocessing executable 'hello' for haskell-project-0.1.0.0..
Building executable 'hello' for haskell-project-0.1.0.0..
[1 of 1] Compiling Main ( Main.hs, dist/build/hello/hello-tmp/Main.o )
Linking dist/build/hello/hello ...
...
/nix/store/3aq34n1ba3pvl6cs6f63xd737fz6604r-haskell-project-0.1.0.0
$ /nix/store/3aq34n1ba3pvl6cs6f63xd737fz6604r-haskell-project-0.1.0.0/bin/hello
Hello, Haskell!
That works. We can also create a separate shell.nix to derive a Nix shell
environment based on our Haskell environment.
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsPkgs = nixpkgs.haskell.packages.ghc8102;
project = import ./default.nix;
in
nixpkgs.mkShell {
name = "cabal-shell";
inputsFrom = [ project.env ];
buildInputs = [
hsPkgs.cabal-install
];
}
We use nixpkgs.mkShell to create a Nix derivation that is explicitly used
for Nix shell. inputsFrom propagates all build inputs of a derivation to
the new derivation. We use project.env which is a sub-derivation given
by callCabal2nix which contains the GHC shell environment for building
our project.
Notice that we have to explicitly provide cabal-install as buildInput
to our shell derivation. This shows that internally, the Haskell packages
in nixpkgs are built by directly calling GHC, skipping cabal entirely.
How Haskell in Nixpkgs Work
The Haskell packages in nixpkgs are mono-versioned. This means for each
Haskell package such as base, aeson, etc, there is exactly one version
provided by a Haskell packages set. There are however multiple versions
of Haskell packages in nixpkgs, determined by the GHC versions.
For instance, nixpkgs.haskell.packages.ghc8102 contains mono-versioned
Haskell packages that works in GHC 8.10.2, while
nixpkgs.haskell.packages.ghc884 contains mono-versioned Haskell packages
that works in GHC 8.8.4.
$ nix-shell 05-package-management/haskell-project-v1/nix/02-nixpkgs/shell.nix
[nix-shell]$ cd 05-package-management/haskell-project-v1/haskell/
[nix-shell]$ cabal build all
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following will be built (use -v for more details):
- haskell-project-0.1.0.0 (exe:hello) (first run)
Configuring executable 'hello' for haskell-project-0.1.0.0..
Preprocessing executable 'hello' for haskell-project-0.1.0.0..
Building executable 'hello' for haskell-project-0.1.0.0..
[1 of 1] Compiling Main ( Main.hs, /mnt/gamma/scrive/nix-workshop/code/05-package-management/haskell-project-v1/haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.2/haskell-project-0.1.0.0/x/hello/build/hello/hello-tmp/Main.o )
Linking /mnt/gamma/scrive/nix-workshop/code/05-package-management/haskell-project-v1/haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.2/haskell-project-0.1.0.0/x/hello/build/hello/hello ...
Stackage Upstream
The mono versions of Haskell packages used to follow Stackage LTS, which is also mono-versioned. However recently the team have switched to Stackage nightly to reduce the maintenance burden.
Callpackage Pattern
As we discussed in previous chapter, Nix itself does not provide any mechanism for dependency resolution. So nixpkgs come out with the Callpackage design pattern to manage dependencies in nixpkgs.
In short, we define new packages in function form which accept dependencies as function inputs. Let's call these functions partial packages, since they are packages with dependencies yet to be supplied.
For example, the pickaxe package we defined previously would
have a partial package definition that looks something like:
let pickaxe = { stick, planks }: ...
The partial package is then instantiated into a Nix derivation
by calling nixpkgs.callPackage with the package set containing
all its dependencies as partial packages.
let minePackages = {
wood = { ... }: ...;
stick = { wood, ... }: ...;
planks = { wood, ... }: ...;
...
}
in
nixpkgs.callPackage minePackages pickaxe {}
The nixpkgs.callPackage automagically inspects the function arguments
as dependencies in the package set, and construct a dependency graph
that connects all packages with their dependencies. If this succeeds
we get a Nix derivation with the dependency derivations provided
to our partial package.
Dependency Injection
callPackage is essentially a dependency injection pattern, where
components can specify what they need without hardcoding the
reference. This allows dependencies to be overridden before
the callPackage is called.
Using functional programming techniques, it is relatively trivial to compose partial package functions so that dependencies can be overridden either locally or globally. For example, nixpkgs use the override pattern to allow dependencies of a package be overridden.
While functional programming makes it easy to override dependencies, it does not make it easy to inspect the dependency graph to find out what is overridden. This is the downside of composing using closures as blackboxes, as compared to composing ADTs (algebraic data types) as whiteboxes.
Because of this, heavy usage of callPackage and override may impact
readability and maintainability. Readers of a Nix code base may no longer
be able to statically infer which final versions of dependencies are
linked to a package.
Haskell.nix
There is an alternative approach to managing Haskell dependencies using Haskell.nix. Unlike the mono-versioned Haskell packages in nixpkgs, Haskell.nix gives more flexibilities and allows interoperability with the multi-versioned approach of package management with cabal.
As we will see in the next chapter, the multi-versioned approach of Haskell.nix makes it much easier to add bleeding-edge dependencies from Hackage. Haskell.nix also offers many other features, such as cross compiling Haskell packages.
The biggest hurdle of adopting Haskell.nix is unfortunately to properly add Haskell.nix as a dependency in your Nix project. From the first section of the project readme:
Help! Something isn't working
The #1 problem that people have when using haskell.nix is that they find themselves building GHC. This should not happen, but you must follow the haskell.nix setup instructions properly to avoid it. If you find this happening to you, please check that you have followed the getting started instructions and consult the corresponding troubleshooting section.
As mentioned, the most important step to start using Haskell.nix is to
add the hydra.iohk.io Nix cache to your ~/.config/nix/nix.conf:
trusted-public-keys = [...] hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= [...]
substituters = [...] https://hydra.iohk.io [...]
Haskell.nix-based derivation
Aside from that first hurdle, defining a Haskell.nix-based Nix derivation
is relatively straightforward. First we define a project.nix:
let
sources = import ../sources.nix {};
haskell-nix = import sources."haskell.nix" {};
nixpkgs = haskell-nix.pkgs;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = nixpkgs.haskell-nix.cabalProject {
inherit src;
compiler-nix-name = "ghc8102";
};
in
project
We use the version of Haskell.nix managed by niv and import it. Here we
also use the version of nixpkgs provided by Haskell.nix, which adds
additional functionalities in nixpkgs.haskell-nix. We then call the
function haskell-nix.pkgs.haskell-nix.cabalProject to define a
cabal-based Haskell.nix project.
We provide the Haskell source code through the src attribute, and
also a compiler-nix-name field to specify the GHC version we want
to use, GHC 8.10.2.
Project Outputs
To actually build our Haskell project, we define a default.nix to
build the hello executable we have in our project:
let
project = import ./project.nix;
in
project.haskell-project.components.exes.hello
A Haskell.nix project can have multiple derivation outputs for
each cabal component. For our case, we do not have any library
but have one executable named hello. To load that, the executable
is unfortunately located in a long and obscure path
project.haskell-project.components.exes.hello.
$ nix-build 05-package-management/haskell-project-v1/nix/03-haskell.nix/
trace: No index state specified, using the latest index state that we know about (2020-12-04T00:00:00Z)!
these derivations will be built:
/nix/store/xdpklq1y86h6jw6d8fyw6xwhr93l8g73-haskell-project-exe-hello-0.1.0.0-config.drv
/nix/store/j0azqvy5iccbfqp6s0gbfwdgjjp8x2ji-haskell-project-exe-hello-0.1.0.0-ghc-8.10.2-env.drv
/nix/store/mp20hw7kjpqfwqsspjff0h8w8qng8n9d-haskell-project-exe-hello-0.1.0.0.drv
...
/nix/store/yr533l33zrpri7n47lsfm2cih5i0800a-haskell-project-exe-hello-0.1.0.0
$ /nix/store/yr533l33zrpri7n47lsfm2cih5i0800a-haskell-project-exe-hello-0.1.0.0/bin/hello
Hello, Haskell!
Version Conflicts in Haskell Dependencies
In the previous chapter we have created a trivial Haskell project with no
dependency other than base. Let's look at what happen when our Haskell
project have some dependencies, which happen to conflict with the
dependencies available in nixpkgs.
We first define a new Haskell project, haskell-project-v2:
cabal-version: 2.4
name: haskell-project
version: 0.1.0.0
license: ISC
build-type: Simple
executable hello
main-is: Main.hs
build-depends: base >=4.13
, yaml == 0.11.3.0
default-language: Haskell2010
We add a new dependency yaml and explicitly requiring version 0.11.3.0.
At the time of writing, the latest version of yaml available on
Hackage is 0.11.5.0.
However let's pretend there are
breaking changes
in 0.11.5.0, and we only support 0.11.3.0.
Nixpkgs
Let's try to follow our previous approach to define our Haskell derivation
using callCabal2nix. If we do that and try to build it, we would
run into an error:
$ nix-build 05-package-management/haskell-project-v2/nix/01-nixpkgs-conflict/
building '/nix/store/3gab06pwqjc16wdqhj5akxk21g1z0qnx-cabal2nix-haskell-project.drv'...
these derivations will be built:
/nix/store/242x69pl2la3lb201qd57rghisrwclpy-haskell-project-0.1.0.0.drv
building '/nix/store/242x69pl2la3lb201qd57rghisrwclpy-haskell-project-0.1.0.0.drv'...
...
Setup: Encountered missing or private dependencies:
yaml ==0.11.3.0
builder for '/nix/store/242x69pl2la3lb201qd57rghisrwclpy-haskell-project-0.1.0.0.drv' failed with exit code 1
error: build of '/nix/store/242x69pl2la3lb201qd57rghisrwclpy-haskell-project-0.1.0.0.drv' failed
Why is that so?
If we try to enter Nix shell, it will still succeed. But if we try to build
our Haskell project in Nix shell, we will find out that the package yaml
has to be explicitly built by cabal:
$ nix-shell 05-package-management/haskell-project-v2/nix/01-nixpkgs-conflict/shell.nix
[nix-shell]$ cd 05-package-management/haskell-project-v2/haskell/
[nix-shell]$ cabal --dry-run build all
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following would be built (use -v for more details):
- yaml-0.11.3.0 (lib) (requires build)
- haskell-project-0.1.0.0 (exe:hello) (first run)
Problem with Mono-versioning
If we look into
nixpkgs source code,
we can in fact see that the version of yaml available in nixpkgs is the latest,
0.11.5.0.
As discussed earlier, with the mono-versioning approach by nixpkgs, there is exactly one version of each package available. Mono-versioning conflicts can happen when we need packages that are either older or newer than the version provided by nixpkgs.
In theory we could switch to a version of nixpkgs that has yaml-0.11.3.0,
however we would then have to buy into the versions of other Haskell packages
available at that time.
Overriding Versions
Nixpkgs provides a workaround for mono-versioning conflicts, by using the
override pattern.
We can override the version of yaml to the one we want as follows:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsLib = nixpkgs.haskell.lib;
hsPkgs-original = nixpkgs.haskell.packages.ghc8102;
hsPkgs = hsPkgs-original.override {
overrides = hsPkgs-old: hsPkgs-new: {
yaml = hsPkgs-new.callHackage
"yaml" "0.11.3.0" {};
};
};
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
Essentially, we refer to the original haskell package set provided as
hsPkgs-original, and we call hsPkgs-original.override to produce
a new package set hsPkgs with yaml overridden to 0.11.3.0.
Using callHackage, we can fetch the version of yaml from the
Hackage snapshot in nixpkgs. With that we can just provide the
string "0.11.3.0" to specify the version that we want. Note
however that this only works if the version can be found in the
given Hackage snapshot, which may be outdated over time.
An issue with overriding dependencies this way is that the override
affects the entire Haskell package set. This means that all other
Haskell packages that depend on yaml will also get 0.11.3.0
instead of 0.11.5.0. As a result, this may have the unintended
ripple effect of breaking other Haskell packages that we depends
on.
Building Overridden Package
If we try to build our Haskell derivation with overridden yaml,
it would work this time:
$ nix-build 05-package-management/haskell-project-v2/nix/02-nixpkgs-override/
these derivations will be built:
/nix/store/l0v04zz36b5s5r3qc2jisvggyc0gkj5w-remove-references-to.drv
/nix/store/bg5z6b7m24fxqn8qq2l2w8c0w30wkbp3-yaml-0.11.3.0.drv
/nix/store/p5sr404mzr8bnqqprv72lxczdr9cnnim-haskell-project-0.1.0.0.drv
...
/nix/store/0m0mr11ncii3z4zkn9z0xkwk4nswprqm-haskell-project-0.1.0.0
We can also enter the Nix shell to verify that this time, cabal will
not try to build yaml for us:
$ nix-shell 05-package-management/haskell-project-v2/nix/02-nixpkgs-override/shell.nix
[nix-shell]$ cd 05-package-management/haskell-project-v2/haskell/
[nix-shell]$ cabal --dry-run build all
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following would be built (use -v for more details):
- haskell-project-0.1.0.0 (exe:hello) (first run)
Haskell.nix
In comparison with the mono-versioned nixpkgs, Haskell.nix is much more flexible in allowing any version of Haskell packages that are supported by cabal. So we can leave the Nix project unchanged and still build it successfully:
$ nix-build 05-package-management/haskell-project-v2/nix/03-haskell.nix/
trace: No index state specified, using the latest index state that we know about (2020-12-04T00:00:00Z)!
building '/nix/store/v4pf9jffq0dh6xang25qviwb77947s7s-plan-to-nix-pkgs.drv'...
Using index-state 2020-12-04T00:00:00Z
Warning: The package list for 'hackage.haskell.org-at-2020-12-04T000000Z' is
18603 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: Requested index-state2020-12-04T00:00:00Z is newer than
'hackage.haskell.org-at-2020-12-04T000000Z'! Falling back to older state
(2020-12-03T20:14:57Z).
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following would be built (use -v for more details):
- base-compat-0.11.2 (lib) (requires download & build)
- base-orphans-0.8.3 (lib) (requires download & build)
...
- aeson-1.5.4.1 (lib) (requires download & build)
- yaml-0.11.3.0 (lib) (requires download & build)
- haskell-project-0.1.0.0 (exe:hello) (first run)
these derivations will be built:
these derivations will be built:
...
/nix/store/y8vbr0b6y8bzgmadj0rfjp3d2rzx5wgs-yaml-lib-yaml-0.11.3.0-config.drv
/nix/store/fhmib5kqsxl82r1z23mm59njw2dn0c8v-yaml-lib-yaml-0.11.3.0-ghc-8.10.2-env.drv
/nix/store/j837v0cxk9dxqpxfjfngii007hq8wn3w-yaml-lib-yaml-0.11.3.0.drv
/nix/store/nxwvfjaj40adyq002khld7ngnq3wggn7-haskell-project-exe-hello-0.1.0.0-config.drv
/nix/store/y27wbd58f5d1k3lzbzpr5qcc4pgqrxg2-haskell-project-exe-hello-0.1.0.0-ghc-8.10.2-env.drv
/nix/store/b4i7xhnha8007zqxd4gidsf7xyy338an-haskell-project-exe-hello-0.1.0.0.drv
...
/nix/store/phm2jk6xnvxsgp640r66cwgipc62kbc5-haskell-project-exe-hello-0.1.0.0
$ /nix/store/phm2jk6xnvxsgp640r66cwgipc62kbc5-haskell-project-exe-hello-0.1.0.0/bin/hello
Hello, Haskell!
Transitive Haskell Version Conflicts
Let's look at another example of nixpkgs-based Haskell project, where version conflicts are propagated to transitive dependencies.
We have another Haskell project,
haskell-project-v3
with a dependency on the latest QuickCheck-2.14.2:
cabal-version: 2.4
name: haskell-project
version: 0.1.0.0
license: ISC
build-type: Simple
executable hello
main-is: Main.hs
build-depends: base >=4.13
, QuickCheck == 2.14.2
default-language: Haskell2010
Now try building the project with the default callCabal2nix:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsPkgs = nixpkgs.haskell.packages.ghc8102;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
We should see that the building failed, because the
version of nixpkgs
we are using only has QuickCheck-2.13.2 in it.
$ nix-build 05-package-management/haskell-project-v3/nix/01-nixpkgs-conflict
building '/nix/store/12qzjbk3514sj9v4j99brbxr4b83bzy5-cabal2nix-haskell-project.drv'...
installing
these derivations will be built:
/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv
...
Setup: Encountered missing or private dependencies:
QuickCheck ==2.14.2
builder for '/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv' failed with exit code 1
error: build of '/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv' failed
Using Latest Hackage Package
Now we try using the override pattern in the previous chapter to override the
version of QuickCheck:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsLib = nixpkgs.haskell.lib;
hsPkgs-original = nixpkgs.haskell.packages.ghc8102;
hsPkgs = hsPkgs-original.override {
overrides = hsPkgs-old: hsPkgs-new: {
QuickCheck = hsPkgs-new.callHackage "QuickCheck" "2.14.2" {};
};
};
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
This time the build actually failed with another error:
$ nix-build 05-package-management/haskell-project-v3/nix/01-nixpkgs-conflict
building '/nix/store/12qzjbk3514sj9v4j99brbxr4b83bzy5-cabal2nix-haskell-project.drv'...
installing
these derivations will be built:
/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv
...
Configuring haskell-project-0.1.0.0...
CallStack (from HasCallStack):
$, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:1024:20 in Cabal-3.2.0.0:Distribution.Simple.Configure
configureFinalizedPackage, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:477:12 in Cabal-3.2.0.0:Distribution.Simple.Configure
configure, called at libraries/Cabal/Cabal/Distribution/Simple.hs:625:20 in Cabal-3.2.0.0:Distribution.Simple
confHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:65:5 in Cabal-3.2.0.0:Distribution.Simple.UserHooks
configureAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:180:19 in Cabal-3.2.0.0:Distribution.Simple
defaultMainHelper, called at libraries/Cabal/Cabal/Distribution/Simple.hs:116:27 in Cabal-3.2.0.0:Distribution.Simple
defaultMain, called at Setup.hs:2:8 in main:Main
Setup: Encountered missing or private dependencies:
QuickCheck ==2.14.2
builder for '/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv' failed with exit code 1
error: build of '/nix/store/d5jyli1v9y12il9wyzs5x6gyx6q68gig-haskell-project-0.1.0.0.drv' failed
soares@soares-workstation:~/scrive/nix-workshop/code$ nix-build 05-package-management/haskell-project-v3/nix/02-nixpkgs-not-found
building '/nix/store/ybvimf0jbsbx997588kbipkpbq97iv3d-all-cabal-hashes-component-QuickCheck-2.14.2.drv'...
tar: */QuickCheck/2.14.2/QuickCheck.json: Not found in archive
tar: */QuickCheck/2.14.2/QuickCheck.cabal: Not found in archive
tar: Exiting with failure status due to previous errors
builder for '/nix/store/ybvimf0jbsbx997588kbipkpbq97iv3d-all-cabal-hashes-component-QuickCheck-2.14.2.drv' failed with exit code 2
cannot build derivation '/nix/store/73pfc98xfaj7l818nznr8r4gbls5xmls-cabal2nix-QuickCheck-2.14.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/73pfc98xfaj7l818nznr8r4gbls5xmls-cabal2nix-QuickCheck-2.14.2.drv' failed
(use '--show-trace' to show detailed location information)
What happened this time? If we check the
commit log
of our nixpkgs version, we will find out that the nixpkgs we have is commited
on 9 November 2020, but on
Hackage
QuickCheck-2.14.2 is only released on 14 November 2020.
Hackage Index in Nixpkgs
Inside the override call, when we call callHackage to get a Haskell package
from Hackage, we are really just downloading the Haskell source from Hackage
based on the snapshot cached in nixpkgs.
Recall from the principal of reproducibility, with just a version number, there
is no way Nix can tell if we will always get the exact same source code from
Hackage every time a source code is requested. In theory the
QuickCheck-2.14.2 we fetched today may be totally different from the
QuickCheck-2.14.2 we fetch tomorrow, or when it is fetched by someone else.
Nixpkgs solves this by computing the content hash of every Hackage package at the time of snapshot. So we can know for sure that the same Hackage pacakge we fetch with the same nixpkgs will always give back the same result. Nixpkgs also does implicit patching on some Hackage package, if their default configuration breaks.
One option for us may be to simply update to the latest version of nixpkgs
so that it contains QuickCheck-2.14.2. However nixpkgs do not immediately
update the Hackage snapshot every time a new Haskell pacakge is published.
Rather there is usually 1~2 weeks lag as the Haskell packages are updated
in bulk. So we can't rely on that if we want to use a Haskell package
just published an hour ago.
Furthermore, updating nixpkgs also means all other packages in nixpkgs also being updated. That may result in breaking some of our own Nix packages.
In theory we could use a stable Nix channel like nixos-20.09
instead of nixpkgs-unstable so that it is safer to update nixpkgs.
But stability always needs tradeoff with rapid releases,
so it will take even longer before the Hackage snapshot update
is propagated there.
CallHackageDirect
If we want to get the latest Hackage package beyond what is available
in nixpkgs, we can instead use callHackageDirect to directly
download the package from Hackage, skipping nixpkgs entirely:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsLib = nixpkgs.haskell.lib;
hsPkgs-original = nixpkgs.haskell.packages.ghc8102;
hsPkgs = hsPkgs-original.override {
overrides = hsPkgs-old: hsPkgs-new: {
QuickCheck = hsPkgs-new.callHackageDirect {
pkg = "QuickCheck";
ver = "2.14.2";
sha256 = "0rx4lz5rj0s1v451cq6qdxhilq4rv9b9lnq6frm18h64civ2pwbq";
} {};
};
};
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
callHackageDirect works similarly to other ways of fetching source code,
such as builtins.fetchTarball and builtins.fetchgit. In fact,
we can also override a Haskell dependency with a GitHub commit or
source tarball. Here we just need to provide an additional information,
which is the SHA256 checksum of the package.
There is currently straightforward way to compute the hash, but we can
first supply a dummy hash such as
0000000000000000000000000000000000000000000000000000,
then copy the correct hash from the hash mismatch error when
the derivation is built.
Transitive Conflicts
If we try to build it this time however, we are greeted with another error:
$ nix-build 05-package-management/haskell-project-v3/nix/03-nixpkgs-transitive-deps
these derivations will be built:
/nix/store/6my008rqdjb9kbmx0pr80c0zc0fyqqyh-QuickCheck-2.14.2.drv
/nix/store/lvclw4q2jmk89v5ppkfw3mr72qb8ch2d-haskell-project-0.1.0.0.drv
building '/nix/store/6my008rqdjb9kbmx0pr80c0zc0fyqqyh-QuickCheck-2.14.2.drv'...
...
Configuring QuickCheck-2.14.2...
CallStack (from HasCallStack):
$, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:1024:20 in Cabal-3.2.0.
...
defaultMain, called at Setup.lhs:8:10 in main:Main
Setup: Encountered missing or private dependencies:
splitmix ==0.1.*
builder for '/nix/store/6my008rqdjb9kbmx0pr80c0zc0fyqqyh-QuickCheck-2.14.2.drv' failed with exit code 1
cannot build derivation '/nix/store/lvclw4q2jmk89v5ppkfw3mr72qb8ch2d-haskell-project-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/lvclw4q2jmk89v5ppkfw3mr72qb8ch2d-haskell-project-0.1.0.0.drv' failed
So it turns out that QuickCheck-2.14.2 depends on splitmix ==0.1.*, but nixpkgs only
have splitmix-0.0.5, despite splitmix-0.1 has been released since May 2020.
We can see this as the effect of transitive dependency update. If splitmix is upgraded
to version 0.1, it will break many packages that directly depend on it, which
in turns breaks other packages that indirectly depend on it. With nixpkgs's
mono-versioning approach, there is no easy way around this other than upgrading
all affecting packages at once, or upgrading none of them. Mono-versioning
is hard!
Still, we can workaround this by overriding splitmix as well:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsLib = nixpkgs.haskell.lib;
hsPkgs-original = nixpkgs.haskell.packages.ghc8102;
hsPkgs = hsPkgs-original.override {
overrides = hsPkgs-old: hsPkgs-new: {
QuickCheck = hsPkgs-new.callHackageDirect {
pkg = "QuickCheck";
ver = "2.14.2";
sha256 = "0rx4lz5rj0s1v451cq6qdxhilq4rv9b9lnq6frm18h64civ2pwbq";
} {};
splitmix = hsPkgs-new.callHackage
"splitmix" "0.1.0.3" {};
};
};
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
Infinite Recursion
If we build this, we once again get another error: the infamous infinite recursion error:
$ nix-build --show-trace 05-package-management/haskell-project-v3/nix/04-nixpkgs-infinite-recursion/
error: while evaluating the attribute 'buildInputs' of the derivation 'haskell-project-0.1.0.0' at /nix/store/kk346951sg2anjjh8cgfbmrijg983z5q-nixpkgs-src/pkgs/development/haskell-modules/generic-builder.nix:291:3:
while evaluating the attribute 'propagatedBuildInputs' of the derivation 'QuickCheck-2.14.2' at /nix/store/kk346951sg2anjjh8cgfbmrijg983z5q-nixpkgs-src/pkgs/development/haskell-modules/generic-builder.nix:291:3:
while evaluating the attribute 'buildInputs' of the derivation 'splitmix-0.1.0.3' at /nix/store/kk346951sg2anjjh8cgfbmrijg983z5q-nixpkgs-src/pkgs/development/haskell-modules/generic-builder.nix:291:3:
while evaluating the attribute 'propagatedBuildInputs' of the derivation 'async-2.2.2' at /nix/store/kk346951sg2anjjh8cgfbmrijg983z5q-nixpkgs-src/pkgs/development/haskell-modules/generic-builder.nix:291:3:
while evaluating the attribute 'buildInputs' of the derivation 'hashable-1.3.0.0' at /nix/store/kk346951sg2anjjh8cgfbmrijg983z5q-nixpkgs-src/pkgs/development/haskell-modules/generic-builder.nix:291:3:
infinite recursion encountered, at undefined position
So QuickCheck depends on splitmix, but splitmix tests also indirectly depend on
QuickCheck, causing a cyclic dependency. Unfortunately the callPackage
pattern do not have a way to deal with cyclic dependencies, so we have to
manually find ways around it.
Fortunately in this case, it is simple to avoid it by no running unit tests on
splitmix:
let
sources = import ../sources.nix {};
nixpkgs = import sources.nixpkgs {};
hsLib = nixpkgs.haskell.lib;
hsPkgs-original = nixpkgs.haskell.packages.ghc8102;
hsPkgs = hsPkgs-original.override {
overrides = hsPkgs-old: hsPkgs-new: {
QuickCheck = hsPkgs-new.callHackageDirect {
pkg = "QuickCheck";
ver = "2.14.2";
sha256 = "0rx4lz5rj0s1v451cq6qdxhilq4rv9b9lnq6frm18h64civ2pwbq";
} {};
splitmix = hsLib.dontCheck
(hsPkgs-new.callHackage
"splitmix" "0.1.0.3" {});
};
};
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = hsPkgs.callCabal2nix "haskell-project" src;
in
hsPkgs.callPackage project {}
Now our package finally builds.
Haskell.nix
In comparison, there is no manual intervention needed for Haskell.nix-based derivation:
let
sources = import ../sources.nix {};
haskell-nix = import sources."haskell.nix" {};
nixpkgs = haskell-nix.pkgs;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = nixpkgs.haskell-nix.cabalProject {
inherit src;
compiler-nix-name = "ghc8102";
};
in
project
Hoepfully this shows why we prefer to use Haskell.nix, especially if we want it to work the same way as cabal and get the latest Haskell packages from Hackage.
Multi-versioned Haskell Packages
As we seens in the previous chapters, the mono-versioned Haskell packages provided by nixpkgs do not always provide the exact versions of Haskell packages that we need. When version conflicts happen, it often require manual intervention, because there is no mechanism to automatically resolve the new set of dependency versions based on custom requirements.
In comparison, package managers like cabal-install are built with multi-versioned Haskell packages in mind. We can give various requirements of our dependencies to cabal by specifying version bounds. If a solution exists, cabal can automatically generate a dependency graph for us.
Haskell.nix
Haskell.nix takes multi-versioned approach toward managing Haskell packages in Nix. It does so by making use of cabal-install to resolve the dependency graph, and converting it into Nix expressions.
Since Haskell.nix uses cabal-install for the actual package management, it can usually work out of the box for existing cabal-based Haskell projects.
Nevertheless, Cabal offers a wide range of features, some of which are not currently well supported by Haskell.nix. Because of this, it is important to understand how Haskell.nix works internally. This would help us understand why certain limitations are present in Haskell.nix.
Haskell.nix also provides other mode of Haskell development in Nix, such as Stack-based Haskell projects. However in this chapter we will focus only on using Haskell.nix for cabal projects.
Hackage Index
To understand how Haskell.nix resolves dependencies, we have to first understand how Cabal resolves dependencies. Recall from the chapter Dependency Management that for a package manager to resolve dependencies, it has to run the dependency resolution algorithm against the state of a package registry.
For the case of Cabal, the dependency resolution is done against the Hackage registry. More specifically, Cabal resolves the dependencies by downloading the entire state of the Hackage index to your local machine.
Every time new packages are added to Hackage, a new Hackage index snapshot is generated, and the entire index has to be re-downloaded. (Technically, the update is incremental with append-only update to the snapshot. However this relies on explicit HTTP caching mechanism to work.)
Currently the Hackage index snapshot takes about 100 MiB after compression. Furthermore the official Hackage bandwidth is pretty low and do not always stay available. Because of this, it can often take a long time to update cabal, especially when running it for the first time.
When we build Haskell projects with cabal, the local Hackage
index is not updated automatically. Instead we have to manually
run cabal update to update the local Hackage index. As a result,
it is a common problem that a new Haskell package is published
to Hackage, but it cannot be found in local builds because
the local index is not updated.
Accessing Hackage Index Inside Nix
Since Haskell.nix uses Cabal to resolve the dependencies inside Nix, it needs to download and provide the Hackage snapshot inside of a Nix build.
By default, Cabal downloads the Hackage snapshot into
~/.cabal/packages. However since there is no home directory
inside a Nix build, we need to put it somewhere local.
This can be done by specifying a --remote-repo-cache
option when running Cabal.
Even so, the next challenge is that we need to download the Hackage snapshot, and save it to the Nix store. The SHA256 checksum of the Hackage snapshot is also needed, so that Nix can be sure that all Nix builds are using the exact version of Hackage snapshot.
Fortunately Haskell.nix takes care of all these details for us. However this comes at a cost: Haskell.nix needs to regularly update and convert the Hackage snapshot into Nix expressions.
In fact, Haskell.nix manages this through the Hackage.nix project. It triggers regular CI builds to update to the latest Hackage snapshot, and save the result as Nix expressions. Haskell.nix itself is also regularly updated, to update the Hackage snapshot through updating the version of Hackage.nix used in Haskell.nix.
Regardless, the state update do not run continuously, but rather usually once every day. There can be up to a few days delay for the Hackage snapshot to be included in Haskell.nix. This also means that when a new Haskell package is published to Hackage, we cannot always use it immediately in Haskell.nix.
If we really insist on getting the bleeding edge Haskell packages, a workaround would be that we can maintain our own Hackage snapshot. Since Hackage.nix is open source, there is no problem doing that. However this would also adds a lot of overhead, as we would have to regularly keep up to date our own version of Hackage.nix. Because of this, we recommend readers to be patient and wait for few days for a new Haskell packge to become available in Haskell.nix.
Freezing Hackage Index
The Hackage index at hackage.haskell.org is constantly updated, and there is no straightforward way to go back in time and get the old state of Hackage at a time in the past.
This can cause issue when we try to download the Hackage snapshot from Nix.
To ensure reproducibility, the downloaded 01-index.tar.gz is supposed
to have the exact same content for everyone by checking the SHA256
checksum. But since the Hackage index keeps changing, we cannot ask
Nix to just download the index from the URL.
As a workaround, Hackage instead guarantees that the index state is updated in append-only mode. With this Haskell.nix uses a workaround to truncate the downloaded index with a known length of the snapshot at a particular time during fetching. This helps make sure that when Nix finally sees the index archive, the content is exactly the same regardless of when the code is evaluated.
Nevertheless, this workaround is one part of the complexity in Haskell.nix.
In the daily update to Hackage.nix, it has to regularly check the new
length of the latest Hackage index, while keeping the old lengths for
each day in the past. As a .tar.gz file, the format is opaque, and
it needs to rely on the obscure interaction of Hackage with the file
compression to create a reproducible index snapshot.
Source Repository Package
Haskell.nix also allows Haskell packages to be provided outside of Hackage.
This can be done by adding
source-repository-package
fields in your cabal.project file. Haskell.nix automatically takes into
consideration these fields when resolving the depdendencies.
One additional detail that is needed when using source-repository-package
with Haskell.nix is that you may optionally needs to add the SHA256
checksum of the Haskell package with a --sha256 comment line. More details
here.
Resolving Dependencies
We have learned earlier that evaluation-time dependencies can cause various issues in Nix. However Haskell.nix uses evaluation-time dependency, or more specifically Import From Derivation (IFD), to resolve the dependency graph of our Haskell projects.
To understand why IFD is needed, we need to first look at how Cabal resolves the dependencies, and where it stores the dependency graph.
We can ask Cabal to resolve the dependencies of our Haskell project
by running cabal configure. Inside this command, Cabal will read the
cabal.project and .cabal file, read the local Hackage snapshot,
and try to come out with a dependency graph as the solution. When successful,
cabal stores the result locally in the dist-newstyle/cache directory.
Inside the dist-newstyle/cache directory, there is a plan.json file
which contains the dependency graph that third party tools such as
Haskell.nix can use to extract the result. There are libraries such as
cabal-plan that are available,
which Haskell.nix uses to parse the dependency graph.
Now remember that to build a Haskell project with Nix, we need to construct one or more Nix derivations that encapsulate our project and all its dependencies. Since a Haskell project contains many dependencies, we want to define each Haskell package as their own Nix derivation. This way Nix can parallellize the build for each of our dependencies, and also reuse the build result if only some of the dependencies changes.
However to construct our derivation, we need to first call cabal configure
to resolve the dependency graph, and then extract the result from plan.json.
In other words, to construct the actual Nix derivation of our project, we need
to first construct another derivation that produces a build plan that tells us
what dependencies are needed.
Derivation Plan
In Haskell.nix, the build plan that is produced from calling Cabal is called a plan. The name might be a bit confusing, but essentially it represents the evaluation-time dependency of a Haskell project, of which the Nix derivation that needs to be built first before the actual Nix derivation for the Haskell project can be constructed.
Recall that Nix cannot identify which derivation is an evaluation-time dependency of another package. This means that if we try to inspect the derivation for our Haskell project, the derivation for the plan would never show up there. This also means that the plan cannot be cached easily.
Haskell.nix exports the plan for a Haskell project in
the plan-nix attribute. We can build this to see how the plan looks like:
$ nix-build -A plan-nix 05-package-management/haskell-project-v3/nix/06-haskell.nix/project.nix
trace: No index state specified, using the latest index state that we know about (2020-12-04T00:00:00Z)!
checking outputs of '/nix/store/5v1835nh26s8dldssz03cysm3aay6d7q-plan-to-nix-pkgs.drv'...
Using index-state 2020-12-04T00:00:00Z
Warning: The package list for 'hackage.haskell.org-at-2020-12-04T000000Z' is
18610 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: Requested index-state2020-12-04T00:00:00Z is newer than
'hackage.haskell.org-at-2020-12-04T000000Z'! Falling back to older state
(2020-12-03T20:14:57Z).
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following would be built (use -v for more details):
- splitmix-0.1.0.3 (lib) (requires download & build)
- random-1.2.0 (lib) (requires download & build)
- QuickCheck-2.14.1 (lib) (requires download & build)
- haskell-project-0.1.0.0 (exe:hello) (first run)
/nix/store/pgdqan44d8ky1yz0d687d2nhqsqflc48-plan-to-nix-pkgs
$ cat /nix/store/pgdqan44d8ky1yz0d687d2nhqsqflc48-plan-to-nix-pkgs/default.nix
{
pkgs = hackage:
{
packages = {
"ghc-prim".revision = (((hackage."ghc-prim")."0.6.1").revisions).default;
"mtl".revision = (((hackage."mtl")."2.2.2").revisions).default;
"rts".revision = (((hackage."rts")."1.0").revisions).default;
"QuickCheck".revision = (((hackage."QuickCheck")."2.14.1").revisions).default;
...
}
compiler = {
version = "8.10.2";
nix-name = "ghc8102";
packages = {
"ghc-prim" = "0.6.1";
"mtl" = "2.2.2";
...
}
}
}
...
}
We can see that the plan contains quite detailed information on the exact versions of dependencies that are needed to build our Haskell project.
Caching Plans with Materialization
The dependency resolution algorithm is not a cheap operation to run, especially
in large Haskell projects. This is why Cabal caches the result inside of
dist-newstyle/cache instead of recomputing it every time commands such as
cabal build are called.
In the case of Nix, we know that a Nix build can be cached and reused if there are no changes in the build inputs. However for the case of a materialized plan, the result of the plan depends on the input source code of the Haskell project. This means that every time the source code changes, the plan needs to be recomputed from scratch.
Furthermore, computing the materialized plan requires a number of dependencies that are not needed when building the Haskell project itself. On the other hand, We know that the materialized plan only changes when either the dependencies list is updated, or when the Hackage index state is updated. So we should be able to cache the materialized plan so that there is no need for Haskell.nix to recompute the dependencies all the time.
In Haskell.nix, the act of manually caching the derivation plan is called
materialization.
To put it simply, we can materialize the derivation plan by copying the
build output of the plan-nix derivation and check that into the source
control of the project.
If we already have the plan files cached in version control, we can then
pass the path to the .nix file as the materialized attribute when defining
our Haskell.nix project.
Example Materialized Haskell Project
The Nix project 07-haskell.nix-materialized contains an example Haskell.nix project that provides basic support for materialization.
As compared to the simplified version, the project.nix has a few more additional fields passed to Haskell.nix:
{ useMaterialization ? true }:
let
sources = import ../sources.nix {};
haskell-nix = import sources."haskell.nix" {};
nixpkgs = haskell-nix.pkgs;
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
project = nixpkgs.haskell-nix.cabalProject {
inherit src;
compiler-nix-name = "ghc8102";
index-state = "2020-12-04T00:00:00Z";
materialized = if useMaterialization
then ./plan else null;
plan-sha256 = if useMaterialization
then nixpkgs.lib.removeSuffix "\n"
(builtins.readFile ./plan-hash.txt)
else null;
exactDeps = true;
};
in
project
The project.nix file is now a function accepting a useMaterialization argument.
If it is set to true, then the materialized plan is passed to Haskell.nix
through the materialized attribute.
We also need to explicitly provide Haskell.nix the version of Hackage snapshot
we want to use, which we hard code to "2020-12-04T00:00:00Z".
We set the exactDeps option here to true, so that when the materialized
plan is out of sync, we get error when running cabal build inside the Nix shell.
There are also a new plan directory, a plan-hash.txt file, and a sync-materialized.sh script:
#!/usr/bin/env bash
plan=$(nix-build -j4 --no-out-link --arg useMaterialization false -A plan-nix project.nix)
rm -rf plan
cp -r $plan plan
find plan -type d -exec chmod 755 {} \;
nix-hash --base32 --type sha256 plan > plan-hash.txt
The script sync-materialized.sh creates a Haskell.nix project with
useMaterialization set to false. It then copies the build output of plan-nix
to the plan/ directory, and compute the SHA256 hash of the directory and save it to
plan-hash.txt.
As we can see, a Haskell.nix project using materialized plan is a bit more involved. However the performance trade off of using materialized plan in a large Haskell project can be significant. The example project template can hopefully serve as a reference on how to setup a Haskell.nix project with materialization.
Syncing Materialized Plan
When we explicitly cache the materialized plans, there is a risk of the plan file diverging from the actual Haskell dependencies when the .cabal file is updated. It can also introduce additional noise in git commits when dependencies are updated. This can introduce friction in Haskell projects, as all team members become responsible to update the materialized plans when the Haskell dependencies are updated.
Checking in dependencies metadata into version control is a common problem
in software projects, particularly Nix projects. As discussed in
Dependency Management, this is a
necessary evil to make sure that our dependencies are reproducible.
Syncing materialized plans can be seen as the same methods as
syncing other dependencies lock files such as cabal.project.freeze
and package-lock.json.
Nevertheless, the current workflow for managing materialization plans in Haskell.nix is not very convenient, and there are a lot of improvements that could have been made. We can only hope that these steps can be improved in future versions of Haskell.nix.
Or perhaps there are better ways of managing Haskell dependencies, as described in the next chapter.
Caching Haskell.nix Dependencies
One of the downside of multi-versioning approach taken by Haskell.nix is that it is much more difficult to provide a general Nix cache for Haskell.nix projects.
When Haskell projects are built using unmodified nixpkgs, there is usually no need to build any Haskell dependencies at all. Instead the Haskell packages are downloaded directly from cache.nixos.org.
This is possible thanks to the mono-versioning approach of nixpkgs. Since there is exactly one version of each package and their dependencies, the build result of the same dependency can always be reused for multiple packages. As such, the number of Nix packages that need to be cached are relatively small, and the default NixOS cache takes care of caching all of them.
In contrast, in Haskell.nix the dependencies for a package can change depending on the requirements of its dependents. So instead of caching one copy of each version of Haskell package, there is a combinatory explosion of packages to be cached - one for each possible combination of transitive dependencies for that package alone.
Because of this, it is common that building a Haskell.nix project also requires building all the Haskell dependencies. We will discuss about how to use Cachix to cache such Nix dependencies in the future.
Other Package Management Strategies
So far we have covered two approaches to managing dependencies in Nix. Nixpkgs takes a mono-versioning approach by providing exactly one version of each package in a particular version of nixpkgs. In contrast, Haskell.nix captures the entire Hackage snapshot and uses Cabal to resolve the dependencies with the plan derivation being an evaluation-time Nix dependency.
However these two are not the only approaches to managing dependencies in Nix. In fact, other languages like Node.js and Rust have come out with an alternative approach, which is much closer to generating traditional package lock files.
Size of Package Registry
Compared to mainstream languages like Node.js, the Haskell ecosystem is much smaller and has much less packages available. Despite that, the state of the Hackage registry is pretty large, taking over 100 MiB after compression.
Capturing the entire state of the package registry is obviously not scaleable. If the Haskell ecosystem were to grow to the size of Node.js, not only the nixpkgs and Haskell packages in nixpkgs have to be re-architected, but also Cabal itself has to come up with better ways of managing dependencies.
Compared to Cabal, package managers like npm do not simply download the entire state of the package registries. Instead, the package registries provide APIs for the package managers to query the available versions of specific packages. This way, the local package managers only have to query the relevant packages they need, and ignore the rest of the packages exist in the registry.
This approach has the advantage of significantly reducing the bandwidth required to compute the dependency graph. However since it requires network communication, this cannot be used inside a Nix build. After all, there is no way Nix can know if a package registry always give back the same answer given the same set of queries.
Generating Nix Plans
Languages like Node.js and Rust come out with tools like node2nix and cargo2nix to convert the dependency resolution results directly into Nix expressions.
For these tools to work, they have to run outside of a Nix build to query
the package registry, and then generate the Nix expressions. Typically,
the Nix expressions are also derived directly from the lock files such as
package-lock.json and Cargo.lock.
The generated Nix expressions directly construct the dependency graph
required to build a Node.js or Rust project. When nix-build is called,
Nix can then easily use the dependency graph to download and build the
dependencies concurrently.
Similar Approach In Haskell
While there is a similarly named cabal2nix command for Haskell, that is actually tied to the mono-versioning approach of nixpkgs. As a result, cabal2nix never computes any dependency graph, but instead simply generates a Nix expression that ask for the dependencies from nixpkgs.
There is no reason why Haskell cannot adopt similar approaches as node2nix or cargo2nix. It might be possible that such approaches are more difficult to achieve, due to how Cabal and Hackage index works. But that should not be an excuse to stop us from trying.
Whether to use Nix or not, if we want Haskell to gain wider adoption, then the package manager Cabal needs significant improvement to be made. We need to improve Cabal to not just have better integrations with tools like Nix and IDEs, but also make it much easier for users to use.
Caching Nix Packages
Nix can be used to build all kinds of packages from the ground up. A Nix packages can be as simple as a hello world output, or as large as projects such as GHC and GCC. If every time we try to build our Nix package from scratch together with all its dependencies, it is going to take unreasonably long time.
Fortunately, NixOS provides
a cache for most packages provided by nixpkgs.
The cache is usually enabled by default in your local nix.conf file.
Thanks to this, we can simply download the packages in nixpkgs instead of
building them from scratch.
The default cache from cache.nixos.org is usually sufficient if we are
just using Nix to build small projects. But what if we are using Nix to
build large projects that take hours to build? What if we need to
use Nix to build something like
fib(50)?
Cachix
Cachix is a cloud service for us to easily host our own Nix cache. The service is free to use for public cache, and paid plan is available for private Nix cache.
Other than Cachix, there are other options available for hosting Nix cache, such as nix-serve and hydra. However they require rolling up your own infrastructure to host the servers, which can take quite a bit of effort.
Cachix is by far the easiest way to setup a Nix cache. In this section we will go through the details of caching Nix packages using Cachix. There are also official documentation available on using Cachix, which you should check out.
To get started using Cachix, first sign up for an account at https://app.cachix.org/signup. Or if you have already signed up, login at https://app.cachix.org/login. After logging in, create a new binary cache at https://app.cachix.org/cache.
Choose a unique name for your cache. For write access, leave with the default option "API tokens". We will also use the default public read access, since our test cache do not contain sensitive data.
After creating the cache, we will also need to create an auth token to push to our cache. Go to https://app.cachix.org/personal-auth-tokens to generate one.
Enter a suitable description for your Cachix token, then click the "Generate" button.
The web page will then show us the generated token, which we can later
enter in our shell. For the purpose of this workshop, also save the
auth token as cachix-token in the nix-workshop/config
directory. (Remember to exclude the cachix authtoken prefix).
You should keep the generated token secret, as it can be used to push to the Cachix store you own, as well as accessing private cache. Having a proper description can help us identify the tokens that we are no longer using, so that we can revoke them at a later time.
Now that we have setup our Cachix account, we can install Cachix on our local machine using Nix:
$ nix-channel --update
...
$ nix-env -i cachix
...
$ cachix --version
cachix 0.6.0
At the point of writing the version of our cachix command is 0.6.0.
We can then configure Nix to use our Cachix store by running:
cachix use my-awesome-nix-cache
(Replace the store name my-awesome-nix-cache with the Cachix store name
that you have created)
Docker
It is difficult to test whether a Nix cache works when using a single machine. Since Nix also locally caches a Nix package after building it, we can't really verify locally that Nix will download from our cache the next time we build it on a fresh machine.
Although we could force rebuild everything using nix-collect-garbage, that
would also destroy all local builds we have. So we might not want to use that
nuclear option.
For the purpose of this workshop, we provide a Dockerfile that you can use to enter an Ubuntu Docker container with fresh Nix environment. You can enter the container by simply running:
make docker
Inside the docker container, you can run as the nix user with the workshop
directory mounted at ~/nix-workshop.
The Docker container is started with the --rm option, so storage for the
local Nix store is reclaimed when you exit the container. You can run
multiple copies of the container in separate shell, to test whether the
Nix packages are cached properly.
We still have to configure our Docker container to use the Cachix store
that we have just created. To simplify the configuration, save your
Cachix credentials in nix-workshop/config, with
the file cachix-store containing the name of your Cachix store,
and cachix-token containing the auth token we have just created
earlier.
When entering the Docker container, it will automatically source
scripts/setup.sh to read the config
files and run the following:
CACHIX_STORE=$(cat ~/nix-workshop/config/cachix-store)
cachix use $CACHIX_STORE
cachix authtoken $(cat ~/nix-workshop/config/cachix-token)
Caching Fibonacci
Now that our Cachix store is setup properly, we can try to actually
cache some Nix builds.
We will reuse the fib.nix
derivation that we defined in earlier
chapter.
First of all, we use nix-instantiate to instantiate our Nix derivation,
and then save the result into $drv. We will create the fib(4) derivation
with the prefix "foo". (You can choose your own prefix here)
$ drv=$(nix-instantiate -E '
import ./code/04-derivations/03-fibonacci/fib.nix
"foo" 4
')
...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
When running nix-instantiate, it will produce the warning line
warning: you did not specify '--add-root' ... which we can usually ignore.
Now we can build $drv, which can take a while to finish.
$ echo $drv
/nix/store/a4qb7vq7ws2q01jd5a07zpml5hw381nl-foo-fib-4.drv
$ nix-build --no-out-link $drv
...
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
Single Build Output
Now that we have built our derivation, how do we actually cache it using
Cachix? At the most basic level, we can use cachix push to push
a particular Nix store path to Cachix:
$ nix-build --no-out-link $drv | cachix push $CACHIX_STORE
compressing and pushing /nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4 (288.00 B)
All done.
In our naive attempt, we simply pipe the build output path of fib(4) to
cachix push. From the output we can see that Cachix has pushed a single path to
our store.
Now if we try to build fib(4) again in another container, we can see that
Nix would fetch the result directly from Cachix:
$ nix-build --no-out-link $drv
these paths will be fetched (0.00 MiB download, 0.00 MiB unpacked):
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
copying path '/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4' from 'https://scrive-nix-workshop.cachix.org'...
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
Everything looks fine so far. But in fact fib(0) to fib(3) have
not yet been cached in our store. So Nix will still have to rebuild
the dependencies when they are needed, such as when entering
Nix shell:
$ nix-shell $drv
these derivations will be built:
/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv
/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv
/nix/store/k65i01s85dix9xcgxyaggc8l13lx1rrz-foo-fib-2.drv
/nix/store/wgcp26v3g23x9i9iqiirn20pgmv4mgki-foo-fib-3.drv
The dependencies also need to be rebuilt when we try to build something like
fib(6), which depends on not only fib(4) but also fib(3):
$ drv=$(nix-instantiate -E '
import ./code/04-derivations/03-fibonacci/fib.nix
"foo" 6
')
$ nix-build --no-out-link $drv
these derivations will be built:
/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv
/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv
/nix/store/k65i01s85dix9xcgxyaggc8l13lx1rrz-foo-fib-2.drv
/nix/store/wgcp26v3g23x9i9iqiirn20pgmv4mgki-foo-fib-3.drv
/nix/store/m3sspba1wz9ffp5qyjplg8fjbnhy7d73-foo-fib-5.drv
/nix/store/zyhq28hxak4jk7xak6lixa4lbfxdjwvz-foo-fib-6.drv
...
How cachix push works
The cachix push command works by reading the Nix store paths from STDIN,
and push each of the path to the specified Cachix store. More specifically,
cachix push pushes the closure of the Nix path.
For the case above, the build result of fib(4) does not have any runtime
dependency, so only the build itself is pushed to Cachix.
We can also try pushing fib-4.drv itself to Cachix, and we can see that
it pushes the .drv derivation of all its dependencies as well.
$ echo $drv | cachix push $CACHIX_STORE
compressing and pushing /nix/store/0hfyfy1wxlri4gdcmikg7v0ybvpkl3yl-Python-3.8.6.tar.xz.drv (856.00 B)
compressing and pushing /nix/store/0vjq3889mc2z9v02hcw072ay0fivbshx-nuke-references.drv (1.41 KiB)
compressing and pushing /nix/store/0rgf63snfi078knpghs1jf2q3913gd17-bootstrap-stage4-gcc-wrapper-10.2.0.drv (7.05 KiB)
...
compressing and pushing /nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv (1.41 KiB)
...
Similarly if we instead try to push the build result of
upper-greet.nix,
we can see that the build result of
greet.nix is pushed as well.
$ drv=$(nix-instantiate ./code/04-derivations/02-dependencies/upper-greet.nix)
$ nix-build --no-out-link $drv | cachix push $CACHIX_STORE
these derivations will be built:
/nix/store/wirssa651gwxv6z8ik78ac05c7f9ml3b-greet.drv
/nix/store/aqznnbrbplwm2mvybzhp6wxw5inrq2aj-upper-greet.drv
...
compressing and pushing /nix/store/391ab6p11dh6gk4crc9a8pxym7c2v7lc-upper-greet (704.00 B)
compressing and pushing /nix/store/ws8lpdazs14zjcsgifkpdy060qsiakk6-greet (568.00 B)
All done.
However for neither approach can push the build output of fib(0) to fib(3),
which are build dependencies of fib(4).
Caching Build Dependencies
In practice, when we are are caching a derivation such as fib(4), we would also
want to cache it together with all its build dependencies, including fib(0) to
fib(3) which we have just built. To do that we have to use cachix push together
with other Nix commands.
Recall that the nix-store -qR command gives us all the dependencies of a
derivation:
$ nix-store -qR $drv
/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv
/nix/store/03f77phmfdmsbfpcc6mspjfff3yc9fdj-setup-hook.sh
...
We can grep specifically for the fib dependencies that we are interested in:
$ nix-store -qR $drv | grep fib-
/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv
/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv
/nix/store/k65i01s85dix9xcgxyaggc8l13lx1rrz-foo-fib-2.drv
/nix/store/wgcp26v3g23x9i9iqiirn20pgmv4mgki-foo-fib-3.drv
/nix/store/a4qb7vq7ws2q01jd5a07zpml5hw381nl-foo-fib-4.drv
To push the build result of all dependencies, we can add the
--include-outputs option:
$ nix-store -qR --include-outputs $drv | grep fib-
/nix/store/20flzbyx97kly3n34krlmjg9awjn6a5z-foo-fib-3
/nix/store/52j5p1a03vi8dxn7rh4s8y6n5ml318rq-foo-fib-0
/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv
/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv
/nix/store/k65i01s85dix9xcgxyaggc8l13lx1rrz-foo-fib-2.drv
/nix/store/wgcp26v3g23x9i9iqiirn20pgmv4mgki-foo-fib-3.drv
/nix/store/a4qb7vq7ws2q01jd5a07zpml5hw381nl-foo-fib-4.drv
/nix/store/c7lwn4mfn3pk0hhvc98lg1r6z6c8pb6c-foo-fib-1
/nix/store/qih0iazs5yl3dg694a2fz0jzzlxzy7k8-foo-fib-2
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
Finally, we don't really need to push the .drv files themselves
to Cachix, as Nix always regenerate them during evaluation of the
.nix files. We can use grep -v to exclude them:
$ nix-store -qR --include-outputs $drv | grep -v .drv | grep fib-
/nix/store/20flzbyx97kly3n34krlmjg9awjn6a5z-foo-fib-3
/nix/store/52j5p1a03vi8dxn7rh4s8y6n5ml318rq-foo-fib-0
/nix/store/c7lwn4mfn3pk0hhvc98lg1r6z6c8pb6c-foo-fib-1
/nix/store/qih0iazs5yl3dg694a2fz0jzzlxzy7k8-foo-fib-2
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
Now that we got the list of build outputs to push, we can then pipe them
to cachix push:
$ nix-build $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
compressing and pushing /nix/store/20flzbyx97kly3n34krlmjg9awjn6a5z-foo-fib-3 (288.00 B)
compressing and pushing /nix/store/52j5p1a03vi8dxn7rh4s8y6n5ml318rq-foo-fib-0 (288.00 B)
compressing and pushing /nix/store/c7lwn4mfn3pk0hhvc98lg1r6z6c8pb6c-foo-fib-1 (288.00 B)
compressing and pushing /nix/store/qih0iazs5yl3dg694a2fz0jzzlxzy7k8-foo-fib-2 (288.00 B)
compressing and pushing /nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4 (288.00 B)
All done.
This time if we try to build fib(6) on a fresh machine, it will download the cache
result of both fib(3) and fib(4) from Cachix:
$ nix-build -E '
import ./code/04-derivations/03-fibonacci/fib.nix
"foo" 6
'
these derivations will be built:
/nix/store/m3sspba1wz9ffp5qyjplg8fjbnhy7d73-foo-fib-5.drv
/nix/store/zyhq28hxak4jk7xak6lixa4lbfxdjwvz-foo-fib-6.drv
these paths will be fetched (61.76 MiB download, 260.53 MiB unpacked):
/nix/store/20flzbyx97kly3n34krlmjg9awjn6a5z-foo-fib-3
/nix/store/zdq2p21pq836n3k1xkh4yb8wkvl9fy0l-foo-fib-4
...
building '/nix/store/m3sspba1wz9ffp5qyjplg8fjbnhy7d73-foo-fib-5.drv'...
...
building '/nix/store/zyhq28hxak4jk7xak6lixa4lbfxdjwvz-foo-fib-6.drv'...
...
/nix/store/vrzxqqj6q11lgpizsd78r2cx2c7zfban-foo-fib-6
Evaluation-time Dependencies
As mentioned in chapter 14, It is much more tricky to find the evaluation time dependencies and push them to Cachix.
If we try to Cache the build dependencies of
fib-serialized.nix,
it wouldn't really workd.
$ drv=$(nix-instantiate -E '
import ./code/04-derivations/03-fibonacci/fib-serialized.nix
"foo" 4
')
building '/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv'...
...
building '/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv'...
...
building '/nix/store/bg0kqrl14p99y1k0g47gcx7a4ik4qk1m-foo-fib-3.drv'...
...
$ nix-build $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
compressing and pushing /nix/store/rlllddnljlv1qzlizdr97q5wbzlqpq5k-foo-fib-4 (288.00 B)
All done.
At this point, there is no simple way to find out all evaluation time dependencies to push. But in case we really want to push all evaluation time dependencies, there is still one nuclear option.
Push All Nix Derivations
cachix push provides a watch-exec command to watch the global Nix
store, and push all new paths that are added to the Nix store during the
execution of our command.
$ cachix watch-exec $CACHIX_STORE nix-build -- \
--no-out-link \
-E 'import ./code/04-derivations/03-fibonacci/fib-serialized.nix
"foo" 4'
...
Watching /nix/store for new store paths ...
building '/nix/store/hj0g7hn703axx44x29l27xb1nrdg83rh-foo-fib-0.drv'...
compressing and pushing /nix/store/52j5p1a03vi8dxn7rh4s8y6n5ml318rq-foo-fib-0 (288.00 B)
building '/nix/store/6mc3ccymdyfmqacrq5vyc43zb2gl81ml-foo-fib-1.drv'...
...
compressing and pushing /nix/store/c7lwn4mfn3pk0hhvc98lg1r6z6c8pb6c-foo-fib-1 (288.00 B)
building '/nix/store/74x1nl7paqin5zcrkkj94bbkm25shpx9-foo-fib-2.drv'...
...
compressing and pushing /nix/store/ia5arkikhbyd9drjzxm2lqgr5a1b6n9m-foo-fib-2 (288.00 B)
building '/nix/store/bg0kqrl14p99y1k0g47gcx7a4ik4qk1m-foo-fib-3.drv'...
...
compressing and pushing /nix/store/ykvgv0hvpm93glrjzpyb7hkashq5rr1q-foo-fib-3 (288.00 B)
these derivations will be built:
/nix/store/m25lspbpyv09vl3pz3sf3q20ndcrilq9-foo-fib-4.drv
building '/nix/store/m25lspbpyv09vl3pz3sf3q20ndcrilq9-foo-fib-4.drv'...
...
/nix/store/rlllddnljlv1qzlizdr97q5wbzlqpq5k-foo-fib-4
Stopped watching /nix/store and waiting for queue to empty ...
compressing and pushing /nix/store/rlllddnljlv1qzlizdr97q5wbzlqpq5k-foo-fib-4 (288.00 B)
Waiting to finish: 1 pushing, 0 in queue
Done.
We can see during the build that Nix pushes a lot of things to Cachix, including the downloaded tarballs from sources like nixpkgs, which are in fact also evaluation time dependencies.
Now if we go to a fresh machine and try to build fib-serialized, we can see
that this time it is correctly downloading the build results of fib(0) to
fib(4) from Cachix:
$ nix-build -E '
import ./code/04-derivations/03-fibonacci/fib-serialized.nix
"foo" 6'
...
copying path '/nix/store/c7lwn4mfn3pk0hhvc98lg1r6z6c8pb6c-foo-fib-1' from 'https://scrive-nix-workshop.cachix.org'...
copying path '/nix/store/52j5p1a03vi8dxn7rh4s8y6n5ml318rq-foo-fib-0' from 'https://scrive-nix-workshop.cachix.org'...
copying path '/nix/store/ia5arkikhbyd9drjzxm2lqgr5a1b6n9m-foo-fib-2' from 'https://scrive-nix-workshop.cachix.org'...
copying path '/nix/store/ykvgv0hvpm93glrjzpyb7hkashq5rr1q-foo-fib-3' from 'https://scrive-nix-workshop.cachix.org'...
copying path '/nix/store/rlllddnljlv1qzlizdr97q5wbzlqpq5k-foo-fib-4' from 'https://scrive-nix-workshop.cachix.org'...
...
building '/nix/store/9ykk0mg8b72nr73hfb82vs49pfyqinzg-foo-fib-5.drv'...
...
these derivations will be built:
/nix/store/wfqhsi5rccz5wb620axn1w3sbjhalw1s-foo-fib-6.drv
building '/nix/store/wfqhsi5rccz5wb620axn1w3sbjhalw1s-foo-fib-6.drv'...
/nix/store/7rx619rpisnh9sw2g3sk6yq07jb563yh-foo-fib-6
There is also cachix watch-store command that provides very coarse-grained
control for caching everything to Cachix.
This can be useful if we do not care about what is being cached, and
instead just want to cache everything.
However as we will see in the next chapter, there might be things that we do not want to cache to Cachix, such as proprietary source code or secret credentials.
Caching Haskell Nix Packages
Similar to the previous chapter, we can cache our Haskell.nix project in similar way.
$ drv=$(nix-instantiate ./code/05-package-management/haskell-project-v3/nix/07-haskell.nix-materialized)
$ nix-build $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
...
/nix/store/8vrdfinxxnwczn4jzknm44bsn3k5nghl-haskell-project-exe-hello-0.1.0.0
compressing and pushing /nix/store/8vrdfinxxnwczn4jzknm44bsn3k5nghl-haskell-project-exe-hello-0.1.0.0 (3.60 MiB)
compressing and pushing /nix/store/6apx83l6ss3hkn0kd4z4rkjbkgs0w4w2-default-Setup-setup (18.07 MiB)
compressing and pushing /nix/store/3pfy3dd8ch77km1wkwd6cdgqn57d4347-haskell-project-exe-hello-0.1.0.0-config (304.02 KiB)
compressing and pushing /nix/store/b2j1nrsjr8cpzmk58d476fc2snz17w75-ghc-8.10.2 (1.71 GiB)
...
All done.
As simple as it might look, the naive approach however has some flaws, especially when dealing with private projects.
Leaking Source Code
The first issue with pushing everything is source code contamination, i.e. the source code of the project leaking to the cache. For instance, suppose we modify the main function to print "Hello, World!" instead of "Hello, Haskell!":
$ sed -i 's/Hello, Haskell!/Hello, World!/g' ./code/05-package-management/haskell-project-v3/haskell/Main.hs
$ cat ./code/05-package-management/haskell-project-v3/haskell/Main.hs
module Main where
main :: IO ()
main = putStrLn "Hello, World!"
If we try to rebuild our Haskell project and push it to Cachix, we can notice
that the modified source code is also pushed as well.
(Notice the drv=$(nix-instantiate ...) assignment has to be re-run to get the
new derivation with the modified source)
$ drv=$(nix-instantiate ./code/05-package-management/haskell-project-v3/nix/07-haskell.nix-materialized)
$ nix-build $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
...
/nix/store/hkqkig7y1dx96qbdwkhk0anb0xdmx6hm-haskell-project-exe-hello-0.1.0.0
compressing and pushing /nix/store/hkqkig7y1dx96qbdwkhk0anb0xdmx6hm-haskell-project-exe-hello-0.1.0.0 (3.60 MiB)
compressing and pushing /nix/store/wq6ry5x7b5x3ld0d7wd2wx3vkxp4wi66-haskell-project-src (1.49 KiB)
All done.
We can list the files in /nix/store/6a049f3fv8x2rdxv34k14cxrwi9an43f-haskell-project-src
and verify that it indeed contains our modified source code. Yikes!
$ ls -la /nix/store/6a049f3fv8x2rdxv34k14cxrwi9an43f-haskell-project-src
total 180
dr-xr-xr-x 2 user user 4096 Jan 1 1970 .
drwxr-xr-x 1 user user 151552 Jan 7 15:41 ..
-r--r--r-- 1 user user 15 Jan 1 1970 .gitignore
-r--r--r-- 1 user user 65 Jan 1 1970 Main.hs
-r--r--r-- 1 user user 46 Jan 1 1970 Setup.hs
-r--r--r-- 1 user user 12 Jan 1 1970 cabal.project
-r--r--r-- 1 user user 307 Jan 1 1970 haskell-project.cabal
$ cat /nix/store/6a049f3fv8x2rdxv34k14cxrwi9an43f-haskell-project-src/Main.hs
module Main where
main :: IO ()
main = putStrLn "Hello, World!"
Pushing source code to Cachix might not be a big deal for open source projects. However this may be an issue for propritary projects with strict IP policies. This could be partially mitigated by paying for a private Cachix store. But we just have to be aware of it and be careful.
Leaking Secrets
Even for the case of open source projects, indiscriminately pushing everything to Cachix still carries another risk, which is accidentally leaking secrets such as authentication credentials.
Suppose that we have some security credentials stored locally in the secret.key
file in the project directory. Since the file is included in .gitignore, it is
not pushed to the git repository.
$ echo secret > ./code/05-package-management/haskell-project-v3/haskell/secret.key
$ ls -la ./code/05-package-management/haskell-project-v3/haskell/
total 32
drwxrwxr-x 2 user user 4096 Jan 7 15:58 .
drwxrwxr-x 4 user user 4096 Dec 8 08:23 ..
-rw-rw-r-- 1 user user 26 Jan 7 15:58 .gitignore
-rw-r--r-- 1 user user 67 Jan 7 15:45 Main.hs
-rw-r--r-- 1 user user 46 Dec 7 08:37 Setup.hs
-rw-rw-r-- 1 user user 12 Dec 7 08:37 cabal.project
-rw-rw-r-- 1 user user 307 Jan 7 09:35 haskell-project.cabal
-rw-rw-r-- 1 user user 7 Jan 7 15:58 secret.key
But is secret.key being included when pushing to Cachix? Let's find out:
$ drv=$(nix-instantiate ./code/05-package-management/haskell-project-v3/nix/07-haskell.nix-materialized)
$ nix-build $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
...
compressing and pushing /nix/store/nrmyzkww87ndyp44jkn56hrra8m9d9vy-haskell-project-exe-hello-0.1.0.0 (3.60 MiB)
compressing and pushing /nix/store/ryz8an9z9bw7j1357k9b5w99fxvnhb74-haskell-project-src (1.69 KiB)
All done.
$ ls -la /nix/store/ryz8an9z9bw7j1357k9b5w99fxvnhb74-haskell-project-src
total 188
dr-xr-xr-x 2 user user 4096 Jan 1 1970 .
drwxr-xr-x 1 user user 155648 Jan 7 16:00 ..
-r--r--r-- 1 user user 26 Jan 1 1970 .gitignore
-r--r--r-- 1 user user 67 Jan 1 1970 Main.hs
-r--r--r-- 1 user user 46 Jan 1 1970 Setup.hs
-r--r--r-- 1 user user 12 Jan 1 1970 cabal.project
-r--r--r-- 1 user user 307 Jan 1 1970 haskell-project.cabal
-r--r--r-- 1 user user 7 Jan 1 1970 secret.key
$ cat /nix/store/ryz8an9z9bw7j1357k9b5w99fxvnhb74-haskell-project-src/secret.key
secret
That's not good! Our local security credentials have been leaked to Cachix! If we also have a public Cachix store, the credentials can potentially be obtained by anyone!
The real culprit is in how we create our source derivation in
project.nix:
src = builtins.path {
name = "haskell-project-src";
path = ../../haskell;
filter = path: type:
let
basePath = builtins.baseNameOf path;
in
basePath != "dist-newstyle"
;
};
Previously, we made a naive attempt of filtering our source directory and
excluding only the dist-newstyle directory to avoid rebuilding the Nix
build when the directory is modified by local cabal runs. However if
we want to push our source code to Cachix, we better be much more careful.
Gitignore.nix
One way we can protect local secrets is by filtering out all gitignored files so that our source code is close to a fresh git checkout when copied into the Nix store. This can be done using Nix helper libraries such as gitignore.nix.
Using gitignore.nix, we can now create a new haskell-project-v4 project with the source filtered with gitignore.nix:
gitignore = (import sources."gitignore.nix" {
inherit (nixpkgs) lib;
}).gitignoreSource;
src = nixpkgs.lib.cleanSourceWith {
name = "haskell-project-src";
src = gitignore ../../haskell;
};
We first add gitignore.nix into sources using niv, and then import
it as above. Following that, we use gitignore ../../haskell to
filter the gitignored files in the haskell directory. We then
use nixpkgs.lib.cleanSourceWith as a hack to give the filtered source a
name haskell-project-src, so that we can grep for it during inspection.
Now if we try to build our derivation, we should get the project source with the local secret filtered out:
$ drv=$(nix-instantiate ./code/06-infrastructure/haskell-project-v4/nix/01-gitignore-src)
$ nix-store -qR --include-outputs $drv | grep haskell-project-src
/nix/store/mhlj5xql8g6ib1wna4g9pc6cpraiz1q8-haskell-project-src-root
$ ls -la /nix/store/mhlj5xql8g6ib1wna4g9pc6cpraiz1q8-haskell-project-src-root
total 140
dr-xr-xr-x 2 nix nix 4096 Jan 1 1970 .
drwxr-xr-x 1 nix nix 114688 Jan 11 11:21 ..
-r--r--r-- 1 nix nix 26 Jan 1 1970 .gitignore
-r--r--r-- 1 nix nix 67 Jan 1 1970 Main.hs
-r--r--r-- 1 nix nix 46 Jan 1 1970 Setup.hs
-r--r--r-- 1 nix nix 12 Jan 1 1970 cabal.project
-r--r--r-- 1 nix nix 307 Jan 1 1970 haskell-project.cabal
Caveats
Gitignore.nix can help us filter out files specified in .gitignore.
However it might still be possible that developers would add new secrets
locally without adding them to .gitignore. In such case, the secret
can still potentially leak to Cachix.
The best way to prevent secrets from leaking is to build from a published git or tarball URL. That way it will be less likely for us to accidentally mix up and leak the secrets in our local file systems. This will however require more complex project organization, as we have to place the Nix code separately from the source code.
Otherwise, it is still recommended to avoid pushing source code to Cachix in the first place, both for proprietary and open source projects. After all, users will almost always build a Nix project with their own local source code, or source that are fetched directly from git or remote URLs. There is rarely a need to use Cachix to distribute source code to our users.
Filtering Out Source
One simple way to filter out the source code is to filter out the name
of the source derivation using grep before pushing to Cachix:
$ nix-store -qR --include-outputs $drv \
| grep -v .drv | grep -v haskell-project-src \
| cachix push $CACHIX_STORE
Note however this may only work if no other paths pushed to Cachix depends
on the source code. This is because Cachix automatically pushes the whole
closure of a Nix path. For instance this would not work if we try to push
the .drv file of the build derivation to Cachix, because that would
also capture the source derivation as part of the closure.
This approach also would not work if there are some intermediate derivations that make copy of the original source code and modify them to produce new source derivation. The intermediate derivation may have a different name, or even a generic one, which it would be difficult for us to filter out without inspecting the derivation source.
As a result, it is best to make use of the patchPhase in
stdenv.mkDerivation to modify the source code if necessary.
Caching Nix Shell
Another way to exclude source code from derivation is by creating a Nix shell
derivation and cache that instead. Haskell.nix provides a shellFor
function that creates a Nix shell derivation from the original
Haskell.nix project we defined.
{ useMaterialization ? true }:
let
project = import ./project.nix {
inherit useMaterialization;
};
in
project.shellFor {
withHoogle = false;
}
If we inspect the derivation tree from shell.nix, we can confirm that
indeed the source code not present in the list. And so we can
safely push only the Haskell.nix dependencies to Cachix.
drv=$(nix-instantiate ./code/06-infrastructure/haskell-project-v4/nix/01-gitignore-src/shell.nix)
$ nix-store -qR --include-outputs $drv | grep haskell-project-src
We first use nix-shell --run true $drv to build only the dependencies of our shell derivation and
push them to Cachix.
$ nix-shell --run true $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
...
All done.
If we want to cache the final build artifact as well, we can still run nix-build $drv and
then push only the build output to Cachix.
$ nix-build ./code/06-infrastructure/haskell-project-v4/nix/01-gitignore-src | cachix push $CACHIX_STORE
...
compressing and pushing /nix/store/9in65nlw9s255x8zh5g7hlvbnl23rqbz-haskell-project-exe-hello-0.1.0.0 (3.60 MiB)
All done.
Double Check Leaking with Code Changes
Our attempt to cache only the Nix shell derivation seems to exclude the source code,
but is it really excluded? If we are not careful, we could easily let Nix give a
generic name like source to our source derivation. In that case it would not
be possible to detect it through grep if our source code has leaked through.
As a result, it is best to double check what is being cached by slightly modifying our source code, and then try pushing to Cachix again.
$ sed -i 's/Hello, Haskell!/Hello, World!/g' ./code/06-infrastructure/haskell-project-v4/haskell/Main.hs
$ cat ./code/06-infrastructure/haskell-project-v4/haskell/Main.hs
module Main where
main :: IO ()
main = putStrLn "Hello, World!"
$ drv=$(nix-instantiate ./code/06-infrastructure/haskell-project-v4/nix/01-gitignore-src/shell.nix)
$ nix-shell --run true $drv && nix-store -qR --include-outputs $drv | grep -v .drv | cachix push $CACHIX_STORE
All done.
$ nix-build ./code/06-infrastructure/haskell-project-v4/nix/01-gitignore-src | cachix push $CACHIX_STORE
these derivations will be built:
/nix/store/52qqdj4pq564ivyawpvfzsz2s3kv9wmp-haskell-project-exe-hello-0.1.0.0.drv
...
compressing and pushing /nix/store/fdb6b3dj79gqff0lz0xf34lrs4gpb5a0-haskell-project-exe-hello-0.1.0.0 (3.60 MiB)
All done.
As we expect, even though Main.hs has been modified, there is no new source
artifact being pushed to Cachix. Only nix-build produced a new binary, which
is then pushed to Cachix.
You can apply the same method on your own project to double check if your source code is leaking to Cachix. Even if you do not care about the source code leaking, this can still serve as a good way to check if any secret is leaking.
Caching Multiple Projects
The technique for caching Nix shell can only work if we have projects made of a single Nix derivation. If we instead have a large project with multiple source repositories, it is much harder to filter out the source code if the derivations depend on each others.
In such cases, the simple way is to use grep -v and hope that it can filter
out all the source derivations. Otherwise you may need to use project-specific
techniques to make sure that only intended Nix artifacts are being cached.
Conclusion
As we seen in this chapter, caching build results is not as straighforward if there are things that we want to prevent from being cached, such as proprietary source code or local secrets. This is probably not a big issue right now, because many people may not even be aware that their source code and secrets are leaking!
Even without considering leaking secrets, there are still too many different ways of caching build results in Nix. While this provides more flexibility for us to control what to cache, the learning curve is way too high for new users who just want to get their Nix builds cached.
Nix and Cachix may need to implement additional features to help make caching easier, and to protect sensitive data. For example, Cachix may add a command line option to exclude paths matching specific pattern to never be pushed.