picnoir/Nix
1
0
Fork 0
Code Issues Pull requests 1 Releases Wiki Activity
Nix/src/libstore/build/derivation-goal.cc

3907 lines
147 KiB
C++
Raw Normal View History

Change .cc files to use split build headers
2020-10-12 19:08:52 +02:00
#include "derivation-goal.hh"
#include "hook-instance.hh"
#include "worker.hh"
Make <nix/fetchurl.nix> a builtin builder This ensures that 1) the derivation doesn't change when Nix changes; 2) the derivation closure doesn't contain Nix and its dependencies; 3) we don't have to rely on ugly chroot hacks.
2015-07-20 04:30:16 +02:00
#include "builtins.hh"
buildenv: Eliminate global variables, other cleanup (cherry picked from commit b82f75464d1e5ae9a00d8004e5dd7b1ca05059e4)
2019-10-21 17:17:15 +02:00
#include "builtins/buildenv.hh"
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include "references.hh"
Eliminate the substituter mechanism Substitution is now simply a Store -> Store copy operation, most typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
#include "finally.hh"
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include "util.hh"
#include "archive.hh"
exportReferencesGraph: Export more complete info in JSON format This writes info about every path in the closure in the same format as ‘nix path-info --json’. Thus it also includes NAR hashes and sizes. Example: [ { "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl", "narSize": 197648, "references": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20939776 }, { "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24", "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3", "narSize": 20742128, "references": [ "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20742128 } ] Fixes #1134.
2017-01-26 20:36:20 +01:00
#include "json.hh"
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include "compression.hh"
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
#include "daemon.hh"
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 19:11:09 +01:00
#include "worker-protocol.hh"
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
#include "topo-sort.hh"
Move Callback into its own header This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.
2020-09-21 18:40:11 +02:00
#include "callback.hh"
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 23:06:23 +02:00
Change .cc files to use split build headers
2020-10-12 19:08:52 +02:00
#include <regex>
#include <queue>
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
#include <sys/types.h>
Fix #2162: use getaddrinfo instead of curl to preload NSS
2018-06-12 13:05:14 +02:00
#include <sys/socket.h>
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
#include <sys/un.h>
Fix #2162: use getaddrinfo instead of curl to preload NSS
2018-06-12 13:05:14 +02:00
#include <netdb.h>
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include <fcntl.h>
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
#include <termios.h>
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include <unistd.h>
#include <sys/mman.h>
#include <sys/utsname.h>
#include <sys/resource.h>
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if HAVE_STATVFS
#include <sys/statvfs.h>
#endif
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
Build sandbox support etc. unconditionally on Linux Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent ambiguity.
2015-12-03 16:30:19 +01:00
/* Includes required for chroot support. */
#if __linux__
In chroot builds, use a private network namespace On Linux it's possible to run a process in its own network namespace, meaning that it gets its own set of network interfaces, disjunct from the rest of the system. We use this to completely remove network access to chroot builds, except that they get a private loopback interface. This means that: - Builders cannot connect to the outside network or to other processes on the same machine, except processes within the same build. - Vice versa, other processes cannot connect to processes in a chroot build, and open ports/connections do not show up in "netstat". - If two concurrent builders try to listen on the same port (e.g. as part of a test), they no longer conflict with each other. This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 06:28:35 +02:00
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/ip.h>
* Support i686-linux builds directly on x86_64-linux Nix installations. This is implemented using the personality() syscall, which causes uname to return "i686" in child processes.
2009-01-12 17:30:32 +01:00
#include <sys/personality.h>
Fix segfault in builtin fetchurl The stack allocated for the builder was way too small (32 KB). This is sufficient for normal derivations, because they just do some setup and then exec() the actual builder. But for the fetchurl builtin derivation it's not enough. Also, allocating the stack on the caller's stack was fishy business.
2015-10-21 14:45:56 +02:00
#include <sys/mman.h>
Build sandbox support etc. unconditionally on Linux Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent ambiguity.
2015-12-03 16:30:19 +01:00
#include <sched.h>
#include <sys/param.h>
#include <sys/mount.h>
#include <sys/syscall.h>
configure: Add a flag to disable seccomp. This is needed for new arches where libseccomp support doesn't exist yet. Fixes #1878.
2018-02-18 08:35:01 +01:00
#if HAVE_SECCOMP
Add a seccomp filter to prevent creating setuid/setgid binaries This prevents builders from setting the S_ISUID or S_ISGID bits, preventing users from using a nixbld* user to create a setuid/setgid binary to interfere with subsequent builds under the same nixbld* uid. This is based on aszlig's seccomp code (47f587700d646f5b03a42f2fa57c28875a31efbe). Reported by Linus Heckemann.
2017-05-29 11:34:24 +02:00
#include <seccomp.h>
configure: Add a flag to disable seccomp. This is needed for new arches where libseccomp support doesn't exist yet. Fixes #1878.
2018-02-18 08:35:01 +01:00
#endif
Build sandbox support etc. unconditionally on Linux Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent ambiguity.
2015-12-03 16:30:19 +01:00
#define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old))
* Support i686-linux builds directly on x86_64-linux Nix installations. This is implemented using the personality() syscall, which causes uname to return "i686" in child processes.
2009-01-12 17:30:32 +01:00
#endif
Use posix_spawn_setbinpref_np to advise which architecture to run When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use. Unfortunately, posix_spawnattr_setbinpref_np does not appear to be inherited by child processes in x86_64 Rosetta 2 translations, meaning that this will not always work as expected. For example: { arm = derivation { name = "test"; system = "aarch64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; rosetta = derivation { name = "test"; system = "x86_64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = i386 ] echo It works! /usr/bin/touch $out '') ]; }; } `arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on aarch64-compiled Nix. I suspect there is a way to fix this since: $ /usr/bin/arch -arch x86_64 /bin/bash \ -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch' arm64 seems to work correctly. We may need to wait for Apple to update system_cmds in opensource.apple.com to find out how though.
2020-12-03 22:41:59 +01:00
#if __APPLE__
#include <spawn.h>
Set kern.curproc_arch_affinity=0 to escape Rosetta By default, once you enter x86_64 Rosetta 2, macOS will try to run everything in x86_64. So an x86_64 Nix will still try to use x86_64 even when system = aarch64-darwin. To avoid this we can set kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0, we ignore this preference. This is based on how https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html works. Completely undocumented, but seems to work! Note, you can verify this works with this impure Nix expression: ``` { a = derivation { name = "a"; system = "aarch64-darwin"; builder = "/bin/sh"; args = [ "-e" (builtins.toFile "builder" '' [ "$(/usr/bin/arch)" = arm64 ] [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; b = derivation { name = "b"; system = "x86_64-darwin"; builder = "/bin/sh"; args = [ "-e" (builtins.toFile "builder" '' [ "$(/usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; } ```
2021-01-12 05:05:32 +01:00
#include <sys/sysctl.h>
Use posix_spawn_setbinpref_np to advise which architecture to run When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use. Unfortunately, posix_spawnattr_setbinpref_np does not appear to be inherited by child processes in x86_64 Rosetta 2 translations, meaning that this will not always work as expected. For example: { arm = derivation { name = "test"; system = "aarch64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; rosetta = derivation { name = "test"; system = "x86_64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = i386 ] echo It works! /usr/bin/touch $out '') ]; }; } `arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on aarch64-compiled Nix. I suspect there is a way to fix this since: $ /usr/bin/arch -arch x86_64 /bin/bash \ -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch' arm64 seems to work correctly. We may need to wait for Apple to update system_cmds in opensource.apple.com to find out how though.
2020-12-03 22:41:59 +01:00
#endif
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#include <pwd.h>
#include <grp.h>
Heuristically detect if a build may have failed due to a full disk This will allow Hydra to detect that a build should not be marked as "permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
#include <nlohmann/json.hpp>
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 23:06:23 +02:00
namespace nix {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void handleDiffHook(
uid_t uid, uid_t gid,
const Path & tryA, const Path & tryB,
const Path & drvPath, const Path & tmpDir)
{
auto diffHook = settings.diffHook;
if (diffHook != "" && settings.runDiffHook) {
try {
RunOptions diffHookOptions(diffHook,{tryA, tryB, drvPath, tmpDir});
diffHookOptions.searchPath = true;
diffHookOptions.uid = uid;
diffHookOptions.gid = gid;
diffHookOptions.chdir = "/";
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto diffRes = runProgram(diffHookOptions);
if (!statusOk(diffRes.first))
throw ExecError(diffRes.first,
"diff-hook program '%1%' %2%",
diffHook,
statusToString(diffRes.first));
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (diffRes.second != "")
printError(chomp(diffRes.second));
} catch (Error & error) {
ErrorInfo ei = error.info();
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
// FIXME: wrap errors.
ei.msg = hintfmt("diff hook execution failed: %s", ei.msg.str());
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
logError(ei);
}
}
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
const Path DerivationGoal::homeDir = "/homeless-shelter";
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
DerivationGoal::DerivationGoal(const StorePath & drvPath,
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
: Goal(worker)
, useDerivation(true)
, drvPath(drvPath)
, wantedOutputs(wantedOutputs)
, buildMode(buildMode)
{
state = &DerivationGoal::getDerivation;
name = fmt(
"building of '%s' from .drv file",
StorePathWithOutputs { drvPath, wantedOutputs }.to_string(worker.store));
trace("created");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
mcExpectedBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.expectedBuilds);
worker.updateProgress();
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Build derivations in a more predictable order Derivations are now built in order of derivation name, so a package named "aardvark" is built before "baboon". Fixes #399.
2014-11-24 16:48:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv,
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
: Goal(worker)
, useDerivation(false)
, drvPath(drvPath)
, wantedOutputs(wantedOutputs)
, buildMode(buildMode)
{
this->drv = std::make_unique<BasicDerivation>(BasicDerivation(drv));
state = &DerivationGoal::haveDerivation;
name = fmt(
"building of '%s' from in-memory derivation",
StorePathWithOutputs { drvPath, drv.outputNames() }.to_string(worker.store));
trace("created");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
mcExpectedBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.expectedBuilds);
worker.updateProgress();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Prevent the .chroot directory from being
garbage-collected. (See isActiveTempFile() in gc.cc.) */
worker.store.addTempRoot(this->drvPath);
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
DerivationGoal::~DerivationGoal()
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Careful: we should never ever throw an exception from a
destructor. */
try { killChild(); } catch (...) { ignoreException(); }
try { stopDaemon(); } catch (...) { ignoreException(); }
try { deleteTmpDir(false); } catch (...) { ignoreException(); }
try { closeLogFile(); } catch (...) { ignoreException(); }
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 17:36:09 +02:00
Change .cc files to use split build headers
2020-10-12 19:08:52 +02:00
string DerivationGoal::key()
{
/* Ensure that derivations get built in order of their name,
i.e. a derivation named "aardvark" always comes before
"baboon". And substitution goals always happen before
derivation goals (due to "b$"). */
return "b$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
inline bool DerivationGoal::needsHashRewrite()
{
#if __linux__
return !useChroot;
#else
/* Darwin requires hash rewriting even when sandboxing is enabled. */
return true;
#endif
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::killChild()
{
if (pid != -1) {
worker.childTerminated(this);
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-09 00:39:24 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildUser) {
/* If we're using a build user, then there is a tricky
race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then
it won't be killed, and we'll potentially lock up in
pid.wait(). So also send a conventional kill to the
child. */
::kill(-pid, SIGKILL); /* ignore the result */
buildUser->kill();
pid.wait();
} else
pid.kill();
If a substitute closure is incomplete, build dependencies, then retry the substituter Issue #77.
2013-01-02 12:38:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
assert(pid == -1);
}
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 10:50:20 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
hook.reset();
}
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
Print only one error message if a build fails E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:25:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::timedOut(Error && ex)
{
killChild();
done(BuildResult::TimedOut, ex);
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::work()
{
(this->*state)();
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::addWantedOutputs(const StringSet & outputs)
{
/* If we already want all outputs, there is nothing to do. */
if (wantedOutputs.empty()) return;
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (outputs.empty()) {
wantedOutputs.clear();
needRestart = true;
} else
for (auto & i : outputs)
if (wantedOutputs.insert(i).second)
needRestart = true;
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
* Write build logs to disk again.
2004-06-29 11:41:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::getDerivation()
{
trace("init");
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 10:50:20 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* The first thing to do is to make sure that the derivation
exists. If it doesn't, it may be created through a
substitute. */
if (buildMode == bmNormal && worker.store.isValidPath(drvPath)) {
loadDerivation();
return;
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 10:50:20 +01:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 22:47:22 +02:00
addWaitee(upcast_goal(worker.makeSubstitutionGoal(drvPath)));
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 18:26:21 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
state = &DerivationGoal::loadDerivation;
}
Build derivations in a more predictable order Derivations are now built in order of derivation name, so a package named "aardvark" is built before "baboon". Fixes #399.
2014-11-24 16:48:04 +01:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::loadDerivation()
{
trace("loading derivation");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nrFailed != 0) {
done(BuildResult::MiscFailure, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath)));
return;
}
Build derivations in a more predictable order Derivations are now built in order of derivation name, so a package named "aardvark" is built before "baboon". Fixes #399.
2014-11-24 16:48:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* `drvPath' should already be a root, but let's be on the safe
side: if the user forgot to make it a root, we wouldn't want
things being garbage collected while we're busy. */
worker.store.addTempRoot(drvPath);
Build derivations in a more predictable order Derivations are now built in order of derivation name, so a package named "aardvark" is built before "baboon". Fixes #399.
2014-11-24 16:48:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
assert(worker.store.isValidPath(drvPath));
Use a steady clock for timeouts Fixes #1146.
2016-12-06 21:58:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Get the derivation. */
drv = std::unique_ptr<BasicDerivation>(new Derivation(worker.store.derivationFromPath(drvPath)));
Use a steady clock for timeouts Fixes #1146.
2016-12-06 21:58:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
haveDerivation();
}
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::haveDerivation()
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
trace("have derivation");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (drv->type() == DerivationType::CAFloating)
settings.requireExperimentalFeature("ca-derivations");
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 17:36:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
retrySubstitution = false;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : drv->outputsAndOptPaths(worker.store))
if (i.second.second)
worker.store.addTempRoot(*i.second.second);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check what outputs paths are not already valid. */
checkPathValidity();
bool allValid = true;
for (auto & [_, status] : initialOutputs) {
if (!status.wanted) continue;
if (!status.known || !status.known->isValid()) {
allValid = false;
break;
}
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If they are all valid, then we're done. */
if (allValid && buildMode == bmNormal) {
done(BuildResult::AlreadyValid);
return;
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
parsedDrv = std::make_unique<ParsedDerivation>(drvPath, *drv);
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* We are first going to try to create the invalid output paths
through substitutes. If that doesn't work, we'll build
them. */
if (settings.useSubstitutes && parsedDrv->substitutesAllowed())
for (auto & [_, status] : initialOutputs) {
if (!status.wanted) continue;
if (!status.known) {
warn("do not know how to query for unknown floating content-addressed derivation output yet");
/* Nothing to wait for; tail call */
return DerivationGoal::gaveUpOnSubstitution();
}
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 22:47:22 +02:00
addWaitee(upcast_goal(worker.makeSubstitutionGoal(
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
status.known->path,
buildMode == bmRepair ? Repair : NoRepair,
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 22:47:22 +02:00
getDerivationCA(*drv))));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (waitees.empty()) /* to prevent hang (no wake-up event) */
outputsSubstitutionTried();
else
state = &DerivationGoal::outputsSubstitutionTried;
}
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 02:05:54 +01:00
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 17:19:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::outputsSubstitutionTried()
{
trace("all outputs substituted (maybe)");
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback) {
done(BuildResult::TransientFailure,
fmt("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
worker.store.printStorePath(drvPath)));
return;
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If the substitutes form an incomplete closure, then we should
build the dependencies of this derivation, but after that, we
DerivationGoal: only retry if output closure incomplete is only problem
2020-10-18 14:21:53 +02:00
can still use the substitutes for this derivation itself.
If the nrIncompleteClosure != nrFailed, we have another issue as well.
In particular, it may be the case that the hole in the closure is
an output of the current derivation, which causes a loop if retried.
*/
if (nrIncompleteClosure > 0 && nrIncompleteClosure == nrFailed) retrySubstitution = true;
Improve substitution progress indicator E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-14 20:14:55 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
nrFailed = nrNoSubstituters = nrIncompleteClosure = 0;
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 17:53:23 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (needRestart) {
needRestart = false;
haveDerivation();
return;
Improve substitution progress indicator E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-14 20:14:55 +02:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
checkPathValidity();
size_t nrInvalid = 0;
for (auto & [_, status] : initialOutputs) {
if (!status.wanted) continue;
if (!status.known || !status.known->isValid())
nrInvalid++;
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildMode == bmNormal && nrInvalid == 0) {
done(BuildResult::Substituted);
return;
}
if (buildMode == bmRepair && nrInvalid == 0) {
repairClosure();
return;
}
if (buildMode == bmCheck && nrInvalid > 0)
throw Error("some outputs of '%s' are not valid, so checking is not possible",
worker.store.printStorePath(drvPath));
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Nothing to wait for; tail call */
gaveUpOnSubstitution();
boost::shared_ptr -> std::shared_ptr
2014-03-30 00:49:23 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* At least one of the output paths could not be
produced using a substitute. So we have to build instead. */
void DerivationGoal::gaveUpOnSubstitution()
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Make sure checkPathValidity() from now on checks all
outputs. */
wantedOutputs.clear();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* The inputs must be built before we can build this goal. */
if (useDerivation)
for (auto & i : dynamic_cast<Derivation *>(drv.get())->inputDrvs)
addWaitee(worker.makeDerivationGoal(i.first, i.second, buildMode == bmRepair ? bmRepair : bmNormal));
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : drv->inputSrcs) {
if (worker.store.isValidPath(i)) continue;
if (!settings.useSubstitutes)
throw Error("dependency '%s' of '%s' does not exist, and substitution is disabled",
worker.store.printStorePath(i), worker.store.printStorePath(drvPath));
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 22:47:22 +02:00
addWaitee(upcast_goal(worker.makeSubstitutionGoal(i)));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 10:50:20 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (waitees.empty()) /* to prevent hang (no wake-up event) */
inputsRealised();
else
state = &DerivationGoal::inputsRealised;
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-09 00:39:24 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::repairClosure()
{
/* If we're repairing, we now know that our own outputs are valid.
Now check whether the other paths in the outputs closure are
good. If not, then start derivation goals for the derivations
that produced those outputs. */
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Get the output closure. */
auto outputs = queryDerivationOutputMap();
StorePathSet outputClosure;
for (auto & i : outputs) {
if (!wantOutput(i.first, wantedOutputs)) continue;
worker.store.computeFSClosure(i.second, outputClosure);
}
If a substitute closure is incomplete, build dependencies, then retry the substituter Issue #77.
2013-01-02 12:38:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Filter out our own outputs (which we have already checked). */
for (auto & i : outputs)
outputClosure.erase(i.second);
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 12:42:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Get all dependencies of this derivation so that we know which
derivation is responsible for which path in the output
closure. */
StorePathSet inputClosure;
if (useDerivation) worker.store.computeFSClosure(drvPath, inputClosure);
std::map<StorePath, StorePath> outputsToDrv;
for (auto & i : inputClosure)
if (i.isDerivation()) {
auto depOutputs = worker.store.queryPartialDerivationOutputMap(i);
for (auto & j : depOutputs)
if (j.second)
outputsToDrv.insert_or_assign(*j.second, i);
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 12:42:57 +02:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check each path (slow!). */
for (auto & i : outputClosure) {
if (worker.pathContentsGood(i)) continue;
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
printError(
"found corrupted or missing path '%s' in the output closure of '%s'",
worker.store.printStorePath(i), worker.store.printStorePath(drvPath));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto drvPath2 = outputsToDrv.find(i);
if (drvPath2 == outputsToDrv.end())
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 22:47:22 +02:00
addWaitee(upcast_goal(worker.makeSubstitutionGoal(i, Repair)));
Print only one error message if a build fails E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:25:35 +02:00
else
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
addWaitee(worker.makeDerivationGoal(drvPath2->second, StringSet(), bmRepair));
Print only one error message if a build fails E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:25:35 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (waitees.empty()) {
done(BuildResult::AlreadyValid);
return;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 17:36:09 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
state = &DerivationGoal::closureRepaired;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::closureRepaired()
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 17:36:09 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
trace("closure repaired");
if (nrFailed > 0)
throw Error("some paths in the output closure of derivation '%s' could not be repaired",
worker.store.printStorePath(drvPath));
done(BuildResult::AlreadyValid);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 17:36:09 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::inputsRealised()
build: run diff-hook under --check and document diff-hook
2019-05-10 22:39:31 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
trace("all inputs realised");
runProgram: support gid, uid, chdir
2019-05-11 22:35:53 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nrFailed != 0) {
if (!useDerivation)
throw Error("some dependencies of '%s' are missing", worker.store.printStorePath(drvPath));
done(BuildResult::DependencyFailed, Error(
"%s dependencies of derivation '%s' failed to build",
nrFailed, worker.store.printStorePath(drvPath)));
return;
build: run diff-hook under --check and document diff-hook
2019-05-10 22:39:31 +02:00
}
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (retrySubstitution) {
haveDerivation();
return;
}
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Gather information necessary for computing the closure and/or
running the build hook. */
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Determine the full set of input paths. */
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 17:33:31 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* First, the input derivations. */
if (useDerivation) {
auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
* Move killUser() to libutil so that the setuid helper can use it.
2006-12-07 01:16:07 +01:00
Rework the db schema for derivation outputs Add a new table for tracking the derivation output mappings. We used to hijack the `DerivationOutputs` table for that, but (despite its name), it isn't a really good fit: - Its entries depend on the drv being a valid path, making it play badly with garbage collection and preventing us to copy a drv output without copying the whole drv closure too; - It dosen't guaranty that the output path exists; By using a different table, we can experiment with a different schema better suited for tracking the output mappings of CA derivations. (incidentally, this also fixes #4138)
2020-10-20 15:14:02 +02:00
if (settings.isExperimentalFeatureEnabled("ca-derivations") &&
((!fullDrv.inputDrvs.empty() && derivationIsCA(fullDrv.type()))
|| fullDrv.type() == DerivationType::DeferredInputAddressed)) {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* We are be able to resolve this derivation based on the
now-known results of dependencies. If so, we become a stub goal
aliasing that resolved derivation goal */
std::optional attempt = fullDrv.tryResolve(worker.store);
assert(attempt);
Derivation drvResolved { *std::move(attempt) };
Don't block while waiting for build users
2020-05-08 11:22:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto pathResolved = writeDerivation(worker.store, drvResolved);
Register the realisations for unresolved drvs Once a build is done, get back to the original derivation, and register all the newly built outputs for this derivation. This allows Nix to work properly with derivations that don't have all their build inputs available − thus allowing garbage collection and (once it's implemented) binary substitution
2021-01-27 10:03:05 +01:00
resolvedDrv = drvResolved;
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto msg = fmt("Resolved derivation: '%s' -> '%s'",
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved));
act = std::make_unique<Activity>(*logger, lvlInfo, actBuildWaiting, msg,
Logger::Fields {
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved),
});
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto resolvedGoal = worker.makeDerivationGoal(
pathResolved, wantedOutputs, buildMode);
addWaitee(resolvedGoal);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
state = &DerivationGoal::resolvedFinished;
return;
}
Wait for build users when none are available
2020-05-05 12:04:36 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & [depDrvPath, wantedDepOutputs] : fullDrv.inputDrvs) {
/* Add the relevant output closures of the input derivation
`i' as input paths. Only add the closures of output paths
that are specified as inputs. */
assert(worker.store.isValidPath(drvPath));
auto outputs = worker.store.queryPartialDerivationOutputMap(depDrvPath);
for (auto & j : wantedDepOutputs) {
if (outputs.count(j) > 0) {
auto optRealizedInput = outputs.at(j);
if (!optRealizedInput)
throw Error(
"derivation '%s' requires output '%s' from input derivation '%s', which is supposedly realized already, yet we still don't know what path corresponds to that output",
Fix the error message when a dep is missing Fix a mismatch in the errors thrown when a needed output was missing from an input derivation that was leading to a wrong and quite misleading error message
2021-01-07 11:21:43 +01:00
worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.store.computeFSClosure(*optRealizedInput, inputPaths);
} else
throw Error(
"derivation '%s' requires non-existent output '%s' from input derivation '%s'",
Fix the error message when a dep is missing Fix a mismatch in the errors thrown when a needed output was missing from an input derivation that was leading to a wrong and quite misleading error message
2021-01-07 11:21:43 +01:00
worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
}
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 21:00:15 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Second, the input sources. */
worker.store.computeFSClosure(drv->inputSrcs, inputPaths);
* Create some state directories automatically as a convenience.
2009-09-23 19:05:51 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("added input paths %s", worker.store.showPaths(inputPaths));
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* What type of derivation are we building? */
derivationType = drv->type();
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Don't repeat fixed-output derivations since they're already
verified by their output hash.*/
nrRounds = derivationIsFixed(derivationType) ? 1 : settings.buildRepeat + 1;
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 18:58:34 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Okay, try to build. Note that here we don't wait for a build
slot to become available, since we don't need one if there is a
build hook. */
state = &DerivationGoal::tryToBuild;
worker.wakeUp(shared_from_this());
* Sanity check.
2006-12-07 01:19:27 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
result = BuildResult();
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::started() {
auto msg = fmt(
buildMode == bmRepair ? "repairing outputs of '%s'" :
buildMode == bmCheck ? "checking outputs of '%s'" :
nrRounds > 1 ? "building '%s' (round %d/%d)" :
"building '%s'", worker.store.printStorePath(drvPath), curRound, nrRounds);
fmt("building '%s'", worker.store.printStorePath(drvPath));
if (hook) msg += fmt(" on '%s'", machineName);
act = std::make_unique<Activity>(*logger, lvlInfo, actBuild, msg,
Logger::Fields{worker.store.printStorePath(drvPath), hook ? machineName : "", curRound, nrRounds});
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 15:14:35 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::tryToBuild()
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
trace("trying to build");
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Obtain locks on all output paths, if the paths are known a priori.
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
The locks are automatically released when we exit this function or Nix
crashes. If we can't acquire the lock, then continue; hopefully some
other goal can start a build, and if not, the main loop will sleep a few
seconds and then retry this goal. */
PathSet lockFiles;
/* FIXME: Should lock something like the drv itself so we don't build same
CA drv concurrently */
Test nix-build with non-local-store --store Just a few small things needed fixing!
2020-12-20 20:55:21 +01:00
if (dynamic_cast<LocalStore *>(&worker.store))
/* If we aren't a local store, we might need to use the local store as
a build remote, but that would cause a deadlock. */
/* FIXME: Make it so we can use ourselves as a build remote even if we
are the local store (separate locking for building vs scheduling? */
/* FIXME: find some way to lock for scheduling for the other stores so
a forking daemon with --store still won't farm out redundant builds.
*/
for (auto & i : drv->outputsAndOptPaths(worker.store))
if (i.second.second)
lockFiles.insert(worker.store.Store::toRealPath(*i.second.second));
Pass all settings to build-remote This ensures that command line flags such as --builders get passed correctly.
2017-10-23 20:43:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!outputLocks.lockPaths(lockFiles, "", false)) {
if (!actLock)
actLock = std::make_unique<Activity>(*logger, lvlWarn, actBuildWaiting,
fmt("waiting for lock on %s", yellowtxt(showPaths(lockFiles))));
worker.waitForAWhile(shared_from_this());
return;
}
Handle log messages from build-remote This makes the progress indicator show statuses like "connecting to 'root@machine'".
2017-10-24 13:41:52 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
actLock.reset();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Now check again whether the outputs are valid. This is because
another process may have started building in parallel. After
it has finished and released the locks, we can (and should)
reuse its results. (Strictly speaking the first check can be
omitted, but that would be less efficient.) Note that since we
now hold the locks on the output paths, no other process can
build this derivation, so no further checks are necessary. */
checkPathValidity();
bool allValid = true;
for (auto & [_, status] : initialOutputs) {
if (!status.wanted) continue;
if (!status.known || !status.known->isValid()) {
allValid = false;
break;
}
}
if (buildMode != bmCheck && allValid) {
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
outputLocks.setDeletion(true);
done(BuildResult::AlreadyValid);
return;
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If any of the outputs already exist but are not valid, delete
them. */
for (auto & [_, status] : initialOutputs) {
if (!status.known || status.known->isValid()) continue;
auto storePath = status.known->path;
debug("removing invalid path '%s'", worker.store.printStorePath(status.known->path));
deletePath(worker.store.Store::toRealPath(storePath));
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally = buildMode != bmNormal || parsedDrv->willBuildLocally(worker.store);
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!buildLocally) {
switch (tryBuildHook()) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
actLock.reset();
result.startTime = time(0); // inexact
state = &DerivationGoal::buildDone;
started();
return;
case rpPostpone:
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
if (!actLock)
actLock = std::make_unique<Activity>(*logger, lvlWarn, actBuildWaiting,
fmt("waiting for a machine to build '%s'", yellowtxt(worker.store.printStorePath(drvPath))));
worker.waitForAWhile(shared_from_this());
outputLocks.unlock();
return;
case rpDecline:
/* We should do it ourselves. */
break;
}
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
actLock.reset();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
state = &DerivationGoal::tryLocalBuild;
worker.wakeUp(shared_from_this());
}
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 16:53:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::tryLocalBuild() {
Remove a redundant condition in DerivationGoal::tryLocalBuild()
2021-01-13 10:13:51 +01:00
/* Make sure that we are allowed to start a build. */
Test nix-build with non-local-store --store Just a few small things needed fixing!
2020-12-20 20:55:21 +01:00
if (!dynamic_cast<LocalStore *>(&worker.store)) {
throw Error(
"unable to build with a primary store that isn't a local store; "
"either pass a different '--store' or enable remote builds."
"\nhttps://nixos.org/nix/manual/#chap-distributed-builds");
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
unsigned int curBuilds = worker.getNrLocalBuilds();
Remove a redundant condition in DerivationGoal::tryLocalBuild()
2021-01-13 10:13:51 +01:00
if (curBuilds >= settings.maxBuildJobs) {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.waitForBuildSlot(shared_from_this());
outputLocks.unlock();
return;
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If `build-users-group' is not empty, then we have to build as
one of the members of that group. */
if (settings.buildUsersGroup != "" && getuid() == 0) {
#if defined(__linux__) || defined(__APPLE__)
if (!buildUser) buildUser = std::make_unique<UserLock>();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildUser->findFreeUser()) {
/* Make sure that no other processes are executing under this
uid. */
buildUser->kill();
} else {
if (!actLock)
actLock = std::make_unique<Activity>(*logger, lvlWarn, actBuildWaiting,
fmt("waiting for UID to build '%s'", yellowtxt(worker.store.printStorePath(drvPath))));
worker.waitForAWhile(shared_from_this());
return;
}
#else
/* Don't know how to block the creation of setuid/setgid
binaries on this platform. */
throw Error("build users are not supported on this platform for security reasons");
#endif
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
actLock.reset();
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try {
Hack to get SSH error messages from build-remote E.g. cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed. cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures Authentication failed.
2018-03-20 15:17:59 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Okay, we have to build. */
startBuilder();
Fix "Bad address" executing build hook This was observed in the deb_debian7x86_64 build: http://hydra.nixos.org/build/29973215 Calling c_str() on a temporary should be fine because the temporary shouldn't be destroyed until after the execl() call, but who knows...
2016-01-07 15:10:14 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
} catch (BuildError & e) {
outputLocks.unlock();
buildUser.reset();
worker.permanentFailure = true;
done(BuildResult::InputRejected, e);
return;
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* This state will be reached when we get EOF on the child's
log pipe. */
state = &DerivationGoal::buildDone;
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 17:19:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
started();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
static void chmod_(const Path & path, mode_t mode)
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (chmod(path.c_str(), mode) == -1)
throw SysError("setting permissions on '%s'", path);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Move/rename path 'src' to 'dst'. Temporarily make 'src' writable if
it's a directory and we're not root (to be able to update the
directory's parent link ".."). */
static void movePath(const Path & src, const Path & dst)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto st = lstat(src);
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
bool changePerm = (geteuid() && S_ISDIR(st.st_mode) && !(st.st_mode & S_IWUSR));
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (changePerm)
chmod_(src, st.st_mode | S_IWUSR);
If a substitute closure is incomplete, build dependencies, then retry the substituter Issue #77.
2013-01-02 12:38:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (rename(src.c_str(), dst.c_str()))
throw SysError("renaming '%1%' to '%2%'", src, dst);
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (changePerm)
chmod_(dst, st.st_mode);
}
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void replaceValidPath(const Path & storePath, const Path & tmpPath)
{
/* We can't atomically replace storePath (the original) with
tmpPath (the replacement), so we have to move it out of the
way first. We'd better not be interrupted here, because if
we're repairing (say) Glibc, we end up with a broken system. */
Path oldPath = (format("%1%.old-%2%-%3%") % storePath % getpid() % random()).str();
if (pathExists(storePath))
movePath(storePath, oldPath);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try {
movePath(tmpPath, storePath);
} catch (...) {
try {
// attempt to recover
movePath(oldPath, storePath);
} catch (...) {
ignoreException();
}
throw;
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
deletePath(oldPath);
}
computeFSClosure: Only process the missing/corrupt paths Issue #122.
2013-06-13 16:43:20 +02:00
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
MakeError(NotDeterministic, BuildError);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::buildDone()
{
trace("build done");
Use deterministic $TMPDIR in sandbox Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the temporary directory is now always /tmp/nix-build-<drvname>-0. This improves bitwise-exact reproducibility for builds that store $TMPDIR in their build output. (Of course, those should still be fixed...)
2015-12-02 14:59:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Release the build user at the end of this function. We don't do
it right away because we don't want another build grabbing this
uid and then messing around with our output. */
Finally releaseBuildUser([&]() { buildUser.reset(); });
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
sandboxMountNamespace = -1;
Add an option to limit the log output of builders This is mostly useful for Hydra to deal with builders that get stuck in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe, so just to be sure,
kill it. */
int status = hook ? hook->pid.kill() : pid.kill();
Show the log tail when a build fails If --no-build-output is given (which will become the default for the "nix" command at least), show the last 10 lines of the build output if the build fails.
2016-04-25 16:47:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("builder process for '%s' finished", worker.store.printStorePath(drvPath));
Show the log tail when a build fails If --no-build-output is given (which will become the default for the "nix" command at least), show the last 10 lines of the build output if the build fails.
2016-04-25 16:47:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
result.timesBuilt++;
result.stopTime = time(0);
Handle log messages from build-remote This makes the progress indicator show statuses like "connecting to 'root@machine'".
2017-10-24 13:41:52 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* So the child is gone now. */
worker.childTerminated(this);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Close the read side of the logger pipe. */
if (hook) {
hook->builderOut.readSide = -1;
hook->fromHook.readSide = -1;
} else
builderOut.readSide = -1;
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Close the log file. */
closeLogFile();
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* When running under a build user, make sure that all processes
running under that uid are gone. This is to prevent a
malicious user from leaving behind a process that keeps files
open and modifies them after they have been chown'ed to
root. */
if (buildUser) buildUser->kill();
Support user namespaces being disabled If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.
2020-10-07 22:02:36 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Terminate the recursive Nix daemon. */
stopDaemon();
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
bool diskFull = false;
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try {
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 02:46:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check the exit status. */
if (!statusOk(status)) {
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 22:05:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Heuristically check whether the build failure may have
been caused by a disk full condition. We have no way
of knowing whether the build actually got an ENOSPC.
So instead, check if the disk is (nearly) full now. If
so, we don't mark this build as a permanent failure. */
#if HAVE_STATVFS
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
if (auto localStore = dynamic_cast<LocalStore *>(&worker.store)) {
uint64_t required = 8ULL * 1024 * 1024; // FIXME: make configurable
struct statvfs st;
if (statvfs(localStore->realStoreDir.c_str(), &st) == 0 &&
(uint64_t) st.f_bavail * st.f_bsize < required)
diskFull = true;
if (statvfs(tmpDir.c_str(), &st) == 0 &&
(uint64_t) st.f_bavail * st.f_bsize < required)
diskFull = true;
}
Use AutoDelete for sandbox profile file
2015-11-15 12:08:50 +01:00
#endif
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 21:45:16 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
deleteTmpDir(false);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Move paths out of the chroot for easier debugging of
build failures. */
if (useChroot && buildMode == bmNormal)
for (auto & [_, status] : initialOutputs) {
if (!status.known) continue;
if (buildMode != bmCheck && status.known->isValid()) continue;
auto p = worker.store.printStorePath(status.known->path);
if (pathExists(chrootRootDir + p))
rename((chrootRootDir + p).c_str(), p.c_str());
}
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto msg = fmt("builder for '%s' %s",
yellowtxt(worker.store.printStorePath(drvPath)),
statusToString(status));
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!logger->isVerbose() && !logTail.empty()) {
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
msg += fmt(";\nlast %d log lines:\n", logTail.size());
for (auto & line : logTail) {
msg += "> ";
msg += line;
msg += "\n";
}
Tell user to run 'nix log' to get full build logs
2021-01-25 17:15:38 +01:00
msg += fmt("For full logs, run '" ANSI_BOLD "nix log %s" ANSI_NORMAL "'.",
worker.store.printStorePath(drvPath));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (diskFull)
msg += "\nnote: build failure may have been caused by lack of free disk space";
Refactoring
2014-02-17 23:04:52 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
throw BuildError(msg);
}
Add a --repair flag to ‘nix-store -r’ to repair derivation outputs With this flag, if any valid derivation output is missing or corrupt, it will be recreated by using a substitute if available, or by rebuilding the derivation. The latter may use hash rewriting if chroots are not available.
2012-10-02 23:13:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Compute the FS closure of the outputs and register them as
being valid. */
registerOutputs();
Provide more detailed info about build status to hydra-queue-runner In particular, hydra-queue-runner can now distinguish between remote build / substitution / already-valid. For instance, if a path already existed on the remote side, we don't want to store a log file.
2015-07-20 03:15:45 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (settings.postBuildHook != "") {
Activity act(*logger, lvlInfo, actPostBuildHook,
fmt("running post-build-hook '%s'", settings.postBuildHook),
Logger::Fields{worker.store.printStorePath(drvPath)});
PushActivity pact(act.id);
StorePathSet outputPaths;
Revert "Re-query for the derivation outputs in the post-build-hook" This reverts commit 1b1e0760335832c87516b9103b670b34662d5daf. Using `queryPartialDerivationOutputMap` assumes that the derivation exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
for (auto i : drv->outputs) {
outputPaths.insert(finalOutputs.at(i.first));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
std::map<std::string, std::string> hookEnvironment = getEnv();
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
hookEnvironment.emplace("DRV_PATH", worker.store.printStorePath(drvPath));
hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", worker.store.printStorePathSet(outputPaths))));
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
RunOptions opts(settings.postBuildHook, {});
opts.environment = hookEnvironment;
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
struct LogSink : Sink {
Activity & act;
std::string currentLine;
Revert "Merge branch 'seccomp' of https://github.com/aszlig/nix" This reverts commit 9f3f2e21edb17dbcd674539dff96efb6cceca10c, reversing changes made to 47f587700d646f5b03a42f2fa57c28875a31efbe.
2016-12-19 11:52:57 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
LogSink(Activity & act) : act(act) { }
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Sink: Use std::string_view
2020-12-02 14:00:43 +01:00
void operator() (std::string_view data) override {
for (auto c : data) {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (c == '\n') {
flushLine();
} else {
currentLine += c;
}
}
}
Progress indicator: More improvements
2017-08-15 15:31:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void flushLine() {
act.result(resPostBuildLogLine, currentLine);
currentLine.clear();
}
Improve "waiting for locks" messages These are now shown in the progress bar. Closes #3577.
2020-06-15 16:03:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
~LogSink() {
if (currentLine != "") {
currentLine += '\n';
flushLine();
}
}
};
LogSink sink(act);
Allow builders to create activities Actually, currently they can only create download activities. Thus, downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:01:21 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
opts.standardOut = &sink;
opts.mergeStderrToStdout = true;
runProgram2(opts);
}
Progress indicator: Show on what machine we're building E.g. $ nix build nixpkgs.hello --builders 'root@wendy' [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildMode == bmCheck) {
deleteTmpDir(true);
done(BuildResult::Built);
return;
}
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Delete unused redirected outputs (when doing hash rewriting). */
for (auto & i : redirectedOutputs)
deletePath(worker.store.Store::toRealPath(i.second));
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Delete the chroot (if we were using one). */
autoDelChroot.reset(); /* this runs the destructor */
Recursive Nix: Handle concurrent client connections
2019-11-04 14:27:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
deleteTmpDir(true);
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Repeat the build if necessary. */
if (curRound++ < nrRounds) {
outputLocks.unlock();
state = &DerivationGoal::tryToBuild;
worker.wakeUp(shared_from_this());
return;
}
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
} catch (BuildError & e) {
outputLocks.unlock();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
BuildResult::Status st = BuildResult::MiscFailure;
build: make needsHashRewrite a method
2019-05-12 22:47:41 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (hook && WIFEXITED(status) && WEXITSTATUS(status) == 101)
st = BuildResult::TimedOut;
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
else if (hook && (!WIFEXITED(status) || WEXITSTATUS(status) != 100)) {
}
Build derivations in a more predictable order Derivations are now built in order of derivation name, so a package named "aardvark" is built before "baboon". Fixes #399.
2014-11-24 16:48:04 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
else {
st =
dynamic_cast<NotDeterministic*>(&e) ? BuildResult::NotDeterministic :
statusOk(status) ? BuildResult::OutputRejected :
derivationIsImpure(derivationType) || diskFull ? BuildResult::TransientFailure :
BuildResult::PermanentFailure;
}
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
done(st, e);
return;
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
}
Fixed exportBuildReferenceGraph
2008-01-15 05:32:08 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
done(BuildResult::Built);
}
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::resolvedFinished() {
Register the realisations for unresolved drvs Once a build is done, get back to the original derivation, and register all the newly built outputs for this derivation. This allows Nix to work properly with derivations that don't have all their build inputs available − thus allowing garbage collection and (once it's implemented) binary substitution
2021-01-27 10:03:05 +01:00
assert(resolvedDrv);
// If the derivation was originally a full `Derivation` (and not just
// a `BasicDerivation`, we must retrieve it because the `staticOutputHashes`
// will be wrong otherwise
Derivation fullDrv = *drv;
if (auto upcasted = dynamic_cast<Derivation *>(drv.get()))
fullDrv = *upcasted;
auto originalHashes = staticOutputHashes(worker.store, fullDrv);
auto resolvedHashes = staticOutputHashes(worker.store, *resolvedDrv);
// `wantedOutputs` might be empty, which means “all the outputs”
auto realWantedOutputs = wantedOutputs;
if (realWantedOutputs.empty())
realWantedOutputs = resolvedDrv->outputNames();
for (auto & wantedOutput : realWantedOutputs) {
assert(originalHashes.count(wantedOutput) != 0);
assert(resolvedHashes.count(wantedOutput) != 0);
auto realisation = worker.store.queryRealisation(
DrvOutput{resolvedHashes.at(wantedOutput), wantedOutput}
);
// We've just built it, but maybe the build failed, in which case the
// realisation won't be there
if (realisation) {
auto newRealisation = *realisation;
newRealisation.id = DrvOutput{originalHashes.at(wantedOutput), wantedOutput};
worker.store.registerDrvOutput(newRealisation);
} else {
// If we don't have a realisation, then it must mean that something
// failed when building the resolved drv
assert(!result.success());
}
}
// This is potentially a bit fishy in terms of error reporting. Not sure
// how to do it in a cleaner way
amDone(nrFailed == 0 ? ecSuccess : ecFailed, ex);
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
HookReply DerivationGoal::tryBuildHook()
{
if (!worker.tryBuildHook || !useDerivation) return rpDecline;
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!worker.hook)
worker.hook = std::make_unique<HookInstance>();
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try {
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Send the request to the hook. */
worker.hook->sink
<< "try"
<< (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0)
<< drv->platform
<< worker.store.printStorePath(drvPath)
<< parsedDrv->getRequiredSystemFeatures();
worker.hook->sink.flush();
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Read the first line of input, which should be a word indicating
whether the hook wishes to perform the build. */
string reply;
while (true) {
Add a trace to readLine() failures Hopefully this helps to diagnose 'error: unexpected EOF reading a line' on macOS.
2021-02-05 12:11:50 +01:00
auto s = [&]() {
try {
return readLine(worker.hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the response from the build hook");
throw e;
}
}();
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (handleJSONLogMessage(s, worker.act, worker.hook->activities, true))
;
else if (string(s, 0, 2) == "# ") {
reply = string(s, 2);
break;
}
else {
s += "\n";
writeToStderr(s);
}
}
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 21:45:16 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("hook reply is '%1%'", reply);
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 21:45:16 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (reply == "decline")
return rpDecline;
else if (reply == "decline-permanently") {
worker.tryBuildHook = false;
worker.hook = 0;
return rpDecline;
}
else if (reply == "postpone")
return rpPostpone;
else if (reply != "accept")
throw Error("bad hook reply '%s'", reply);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
} catch (SysError & e) {
if (e.errNo == EPIPE) {
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
printError(
"build hook died unexpectedly: %s",
chomp(drainFD(worker.hook->fromHook.readSide.get())));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.hook = 0;
return rpDecline;
} else
throw;
}
Per-output reference and closure size checks In structured-attributes derivations, you can now specify per-output checks such as: outputChecks."out" = { # The closure of 'out' must not be larger than 256 MiB. maxClosureSize = 256 * 1024 * 1024; # It must not refer to C compiler or to the 'dev' output. disallowedRequisites = [ stdenv.cc "dev" ]; }; outputChecks."dev" = { # The 'dev' output must not be larger than 128 KiB. maxSize = 128 * 1024; }; Also fixed a bug in allowedRequisites that caused it to ignore self-references.
2018-10-22 21:49:56 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
hook = std::move(worker.hook);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Add a trace to readLine() failures Hopefully this helps to diagnose 'error: unexpected EOF reading a line' on macOS.
2021-02-05 12:11:50 +01:00
try {
machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw e;
}
Compress build logs on the fly using bzip2
2012-05-30 16:12:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Tell the hook all the inputs that have to be copied to the
remote system. */
worker_proto::write(worker.store, hook->sink, inputPaths);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
{
StorePathSet missingPaths;
for (auto & [_, status] : initialOutputs) {
if (!status.known) continue;
if (buildMode != bmCheck && status.known->isValid()) continue;
missingPaths.insert(status.known->path);
}
worker_proto::write(worker.store, hook->sink, missingPaths);
}
* Write build logs to disk again.
2004-06-29 11:41:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
hook->sink = FdSink();
hook->toHook.writeSide = -1;
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create the log file and pipe. */
Path logFile = openLogFile();
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 22:05:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
set<int> fds;
fds.insert(hook->fromHook.readSide.get());
fds.insert(hook->builderOut.readSide.get());
worker.childStarted(shared_from_this(), fds, false, false);
Add a --repair flag to ‘nix-store -r’ to repair derivation outputs With this flag, if any valid derivation output is missing or corrupt, it will be recreated by using a substitute if available, or by rebuilding the derivation. The latter may use hash rewriting if chroots are not available.
2012-10-02 23:13:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return rpAccept;
}
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Provide more detailed info about build status to hydra-queue-runner In particular, hydra-queue-runner can now distinguish between remote build / substitution / already-valid. For instance, if a path already existed on the remote side, we don't want to store a log file.
2015-07-20 03:15:45 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
int childEntry(void * arg)
{
((DerivationGoal *) arg)->runChild();
return 1;
}
Don't lock a user while doing remote builds
2020-05-14 16:00:54 +02:00
exportReferences: Check whether the path is in the input closure Fixes #2082.
2018-04-17 12:03:27 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePathSet DerivationGoal::exportReferences(const StorePathSet & storePaths)
{
StorePathSet paths;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & storePath : storePaths) {
if (!inputPaths.count(storePath))
throw BuildError("cannot export references of path '%s' because it is not in the input closure of the derivation", worker.store.printStorePath(storePath));
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.store.computeFSClosure({storePath}, paths);
}
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If there are derivations in the graph, then include their
outputs as well. This is useful if you want to do things
like passing all build-time dependencies of some path to a
derivation that builds a NixOS DVD image. */
auto paths2 = paths;
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & j : paths2) {
if (j.isDerivation()) {
Derivation drv = worker.store.derivationFromPath(j);
for (auto & k : drv.outputsAndOptPaths(worker.store)) {
if (!k.second.second)
/* FIXME: I am confused why we are calling
`computeFSClosure` on the output path, rather than
derivation itself. That doesn't seem right to me, so I
won't try to implemented this for CA derivations. */
throw UnimplementedError("exportReferences on CA derivations is not yet implemented");
worker.store.computeFSClosure(*k.second.second, paths);
}
}
}
Improve progress indicator
2017-05-16 16:09:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return paths;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
static std::once_flag dns_resolve_flag;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
static void preloadNSS() {
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
one of the glibc NSS libraries in a sandboxed child, which will fail unless the library's already
been loaded in the parent. So we force a lookup of an invalid domain to force the NSS machinery to
load its lookup libraries in the parent before any child gets a chance to. */
std::call_once(dns_resolve_flag, []() {
struct addrinfo *res = NULL;
Improve progress indicator
2017-05-16 16:09:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (getaddrinfo("this.pre-initializes.the.dns.resolvers.invalid.", "http", NULL, &res) != 0) {
if (res) freeaddrinfo(res);
}
});
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void linkOrCopy(const Path & from, const Path & to)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (link(from.c_str(), to.c_str()) == -1) {
/* Hard-linking fails if we exceed the maximum link count on a
file (e.g. 32000 of ext3), which is quite possible after a
'nix-store --optimise'. FIXME: actually, why don't we just
bind-mount in this case?
It can also fail with EPERM in BeegFS v7 and earlier versions
which don't allow hard-links to other directories */
if (errno != EMLINK && errno != EPERM)
throw SysError("linking '%s' to '%s'", to, from);
copyPath(from, to);
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::startBuilder()
build: make needsHashRewrite a method
2019-05-12 22:47:41 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Right platform? */
if (!parsedDrv->canBuildLocally(worker.store))
throw Error("a '%s' with features {%s} is required to build '%s', but I am a '%s' with features {%s}",
drv->platform,
concatStringsSep(", ", parsedDrv->getRequiredSystemFeatures()),
worker.store.printStorePath(drvPath),
settings.thisSystem,
concatStringsSep<StringSet>(", ", worker.store.systemFeatures));
build: make needsHashRewrite a method
2019-05-12 22:47:41 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (drv->isBuiltin())
preloadNSS();
build: make needsHashRewrite a method
2019-05-12 22:47:41 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __APPLE__
additionalSandboxProfile = parsedDrv->getStringAttr("__sandboxProfile").value_or("");
#endif
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Are we doing a chroot build? */
{
auto noChroot = parsedDrv->getBoolAttr("__noChroot");
if (settings.sandboxMode == smEnabled) {
if (noChroot)
throw Error("derivation '%s' has '__noChroot' set, "
"but that's not allowed when 'sandbox' is 'true'", worker.store.printStorePath(drvPath));
#if __APPLE__
if (additionalSandboxProfile != "")
throw Error("derivation '%s' specifies a sandbox profile, "
"but this is only allowed when 'sandbox' is 'relaxed'", worker.store.printStorePath(drvPath));
#endif
useChroot = true;
}
else if (settings.sandboxMode == smDisabled)
useChroot = false;
else if (settings.sandboxMode == smRelaxed)
useChroot = !(derivationIsImpure(derivationType)) && !noChroot;
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
if (auto localStoreP = dynamic_cast<LocalStore *>(&worker.store)) {
auto & localStore = *localStoreP;
if (localStore.storeDir != localStore.realStoreDir) {
#if __linux__
useChroot = true;
#else
throw Error("building using a diverted store is not supported on this platform");
#endif
}
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create a temporary directory where the build will take
place. */
tmpDir = createTempDir("", "nix-build-" + std::string(drvPath.name()), false, false, 0700);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
chownToBuilder(tmpDir);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & [outputName, status] : initialOutputs) {
/* Set scratch path we'll actually use during the build.
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
If we're not doing a chroot build, but we have some valid
output paths. Since we can't just overwrite or delete
them, we have to do hash rewriting: i.e. in the
environment/arguments passed to the build, we replace the
hashes of the valid outputs with unique dummy strings;
after the build, we discard the redirected outputs
corresponding to the valid outputs, and rewrite the
contents of the new outputs to replace the dummy strings
with the actual hashes. */
auto scratchPath =
!status.known
? makeFallbackPath(outputName)
: !needsHashRewrite()
/* Can always use original path in sandbox */
? status.known->path
: !status.known->isPresent()
/* If path doesn't yet exist can just use it */
? status.known->path
: buildMode != bmRepair && !status.known->isValid()
/* If we aren't repairing we'll delete a corrupted path, so we
can use original path */
? status.known->path
: /* If we are repairing or the path is totally valid, we'll need
to use a temporary path */
makeFallbackPath(status.known->path);
scratchOutputs.insert_or_assign(outputName, scratchPath);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 19:41:48 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* A non-removed corrupted path needs to be stored here, too */
if (buildMode == bmRepair && !status.known->isValid())
redirectedBadOutputs.insert(status.known->path);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Substitute output placeholders with the scratch output paths.
We'll use during the build. */
inputRewrites[hashPlaceholder(outputName)] = worker.store.printStorePath(scratchPath);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Additional tasks if we know the final path a priori. */
if (!status.known) continue;
auto fixedFinalPath = status.known->path;
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Additional tasks if the final and scratch are both known and
differ. */
if (fixedFinalPath == scratchPath) continue;
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Ensure scratch path is ours to use. */
deletePath(worker.store.printStorePath(scratchPath));
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Rewrite and unrewrite paths */
{
std::string h1 { fixedFinalPath.hashPart() };
std::string h2 { scratchPath.hashPart() };
inputRewrites[h1] = h2;
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
redirectedOutputs.insert_or_assign(std::move(fixedFinalPath), std::move(scratchPath));
Don't create unnecessary substitution goals for derivations
2014-11-24 16:44:35 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Construct the environment passed to the builder. */
initEnv();
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 16:11:19 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeStructuredAttrs();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Handle exportReferencesGraph(), if set. */
if (!parsedDrv->getStructuredAttrs()) {
/* The `exportReferencesGraph' feature allows the references graph
to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of
each `pathN' will be stored in a text file `nameN' in the
temporary build directory. The text files have the format used
by `nix-store --register-validity'. However, the deriver
fields are left empty. */
string s = get(drv->env, "exportReferencesGraph").value_or("");
Strings ss = tokenizeString<Strings>(s);
if (ss.size() % 2 != 0)
throw BuildError("odd number of tokens in 'exportReferencesGraph': '%1%'", s);
for (Strings::iterator i = ss.begin(); i != ss.end(); ) {
string fileName = *i++;
static std::regex regex("[A-Za-z_][A-Za-z0-9_.-]*");
if (!std::regex_match(fileName, regex))
throw Error("invalid file name '%s' in 'exportReferencesGraph'", fileName);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto storePathS = *i++;
if (!worker.store.isInStore(storePathS))
throw BuildError("'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
auto storePath = worker.store.toStorePath(storePathS).first;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Write closure info to <fileName>. */
writeFile(tmpDir + "/" + fileName,
worker.store.makeValidityRegistration(
exportReferences({storePath}), false, false));
}
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (useChroot) {
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Allow a user-configurable set of directories from the
host file system. */
dirsInChroot.clear();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Generalize extra-* settings This removes the extra-substituters and extra-sandbox-paths settings and instead makes every array setting extensible by setting "extra-<name> = <value>" in the configuration file or passing "--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
for (auto i : settings.sandboxPaths.get()) {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (i.empty()) continue;
bool optional = false;
if (i[i.size() - 1] == '?') {
optional = true;
i.pop_back();
}
size_t p = i.find('=');
if (p == string::npos)
dirsInChroot[i] = {i, optional};
else
dirsInChroot[string(i, 0, p)] = {string(i, p + 1), optional};
}
dirsInChroot[tmpDirInSandbox] = tmpDir;
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Add the closure of store paths to the chroot. */
StorePathSet closure;
for (auto & i : dirsInChroot)
try {
if (worker.store.isInStore(i.second.source))
worker.store.computeFSClosure(worker.store.toStorePath(i.second.source).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing 'sandbox-paths'");
throw;
}
for (auto & i : closure) {
auto p = worker.store.printStorePath(i);
dirsInChroot.insert_or_assign(p, p);
}
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
PathSet allowedPaths = settings.allowedImpureHostPrefixes;
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* This works like the above, except on a per-derivation level */
auto impurePaths = parsedDrv->getStringsAttr("__impureHostDeps").value_or(Strings());
Don't require --fallback to recover from disappeared binary cache NARs
2018-06-05 16:04:41 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : impurePaths) {
bool found = false;
/* Note: we're not resolving symlinks here to prevent
giving a non-root user info about inaccessible
files. */
Path canonI = canonPath(i);
/* If only we had a trie to do this more efficiently :) luckily, these are generally going to be pretty small */
for (auto & a : allowedPaths) {
Path canonA = canonPath(a);
if (canonI == canonA || isInDir(canonI, canonA)) {
found = true;
break;
}
}
if (!found)
throw Error("derivation '%s' requested impure path '%s', but it was not in allowed-impure-host-deps",
worker.store.printStorePath(drvPath), i);
* Start of concurrent garbage collection. Processes write temporary roots to a per-process temporary file in /nix/var/nix/temproots while holding a write lock on that file. The garbage collector acquires read locks on all those files, thus blocking further progress in other Nix processes, and reads the sets of temporary roots.
2005-01-31 11:27:25 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
dirsInChroot[i] = i;
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 11:55:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __linux__
/* Create a temporary directory in which we set up the chroot
environment using bind-mounts. We put it in the Nix store
to ensure that we can create hard-links to non-directory
inputs in the fake Nix store in the chroot (see below). */
chrootRootDir = worker.store.Store::toRealPath(drvPath) + ".chroot";
deletePath(chrootRootDir);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Clean up the chroot directory automatically. */
autoDelChroot = std::make_shared<AutoDelete>(chrootRootDir);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
printMsg(lvlChatty, format("setting up chroot environment in '%1%'") % chrootRootDir);
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (mkdir(chrootRootDir.c_str(), 0750) == -1)
throw SysError("cannot create '%1%'", chrootRootDir);
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 11:55:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildUser && chown(chrootRootDir.c_str(), 0, buildUser->getGID()) == -1)
throw SysError("cannot change ownership of '%1%'", chrootRootDir);
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 11:55:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create a writable /tmp in the chroot. Many builders need
this. (Of course they should really respect $TMPDIR
instead.) */
Path chrootTmpDir = chrootRootDir + "/tmp";
createDirs(chrootTmpDir);
chmod_(chrootTmpDir, 01777);
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 11:55:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create a /etc/passwd with entries for the build user and the
nobody account. The latter is kind of a hack to support
Samba-in-QEMU. */
createDirs(chrootRootDir + "/etc");
* Fix the `--fallback' switch. * Fix the substitutes tests.
2005-01-25 14:00:12 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Declare the build user's group so that programs get a consistent
view of the system (e.g., "id -gn"). */
writeFile(chrootRootDir + "/etc/group",
fmt("root:x:0:\n"
"nixbld:!:%1%:\n"
"nogroup:x:65534:\n", sandboxGid()));
* Fix the `--fallback' switch. * Fix the substitutes tests.
2005-01-25 14:00:12 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create /etc/hosts with localhost entry. */
if (!(derivationIsImpure(derivationType)))
writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n");
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Make the closure of the inputs available in the chroot,
rather than the whole Nix store. This prevents any access
to undeclared dependencies. Directories are bind-mounted,
while other inputs are hard-linked (since only directories
can be bind-mounted). !!! As an extra security
precaution, make the fake Nix store only writable by the
build user. */
Path chrootStoreDir = chrootRootDir + worker.store.storeDir;
createDirs(chrootStoreDir);
chmod_(chrootStoreDir, 01775);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildUser && chown(chrootStoreDir.c_str(), 0, buildUser->getGID()) == -1)
throw SysError("cannot change ownership of '%1%'", chrootStoreDir);
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 16:11:19 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : inputPaths) {
auto p = worker.store.printStorePath(i);
Path r = worker.store.toRealPath(p);
if (S_ISDIR(lstat(r).st_mode))
dirsInChroot.insert_or_assign(p, r);
else
linkOrCopy(r, chrootRootDir + p);
}
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 11:55:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If we're repairing, checking or rebuilding part of a
multiple-outputs derivation, it's possible that we're
rebuilding a path that is in settings.dirsInChroot
(typically the dependencies of /bin/sh). Throw them
out. */
for (auto & i : drv->outputsAndOptPaths(worker.store)) {
/* If the name isn't known a priori (i.e. floating
content-addressed derivation), the temporary location we use
should be fresh. Freshness means it is impossible that the path
is already in the sandbox, so we don't need to worry about
removing it. */
if (i.second.second)
dirsInChroot.erase(worker.store.printStorePath(*i.second.second));
}
Only substitute wanted outputs of a derivation If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#elif __APPLE__
/* We don't really have any parent prep work to do (yet?)
All work happens in the child, instead. */
#else
throw Error("sandboxing builds is not supported on this platform");
#endif
}
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 16:11:19 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (needsHashRewrite() && pathExists(homeDir))
throw Error("home directory '%1%' exists; please remove it to assure purity of builds without sandboxing", homeDir);
if (useChroot && settings.preBuildHook != "" && dynamic_cast<Derivation *>(drv.get())) {
printMsg(lvlChatty, format("executing pre-build hook '%1%'")
% settings.preBuildHook);
auto args = useChroot ? Strings({worker.store.printStorePath(drvPath), chrootRootDir}) :
Strings({ worker.store.printStorePath(drvPath) });
enum BuildHookState {
stBegin,
stExtraChrootDirs
};
auto state = stBegin;
auto lines = runProgram(settings.preBuildHook, false, args);
auto lastPos = std::string::size_type{0};
for (auto nlPos = lines.find('\n'); nlPos != string::npos;
nlPos = lines.find('\n', lastPos)) {
auto line = std::string{lines, lastPos, nlPos - lastPos};
lastPos = nlPos + 1;
if (state == stBegin) {
if (line == "extra-sandbox-paths" || line == "extra-chroot-dirs") {
state = stExtraChrootDirs;
} else {
throw Error("unknown pre-build hook command '%1%'", line);
}
} else if (state == stExtraChrootDirs) {
if (line == "") {
state = stBegin;
} else {
auto p = line.find('=');
if (p == string::npos)
dirsInChroot[line] = line;
else
dirsInChroot[string(line, 0, p)] = string(line, p + 1);
}
}
}
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 12:16:11 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Fire up a Nix daemon to process recursive Nix calls from the
builder. */
if (parsedDrv->getRequiredSystemFeatures().count("recursive-nix"))
startDaemon();
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Run the builder. */
printMsg(lvlChatty, "executing builder '%1%'", drv->builder);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create the log file. */
Path logFile = openLogFile();
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create a pipe to get the output of the builder. */
//builderOut.create();
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
builderOut.readSide = posix_openpt(O_RDWR | O_NOCTTY);
if (!builderOut.readSide)
throw SysError("opening pseudoterminal master");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::string slaveName(ptsname(builderOut.readSide.get()));
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildUser) {
if (chmod(slaveName.c_str(), 0600))
throw SysError("changing mode of pseudoterminal slave");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (chown(slaveName.c_str(), buildUser->getUID(), 0))
throw SysError("changing owner of pseudoterminal slave");
}
#if __APPLE__
else {
if (grantpt(builderOut.readSide.get()))
throw SysError("granting access to pseudoterminal slave");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#endif
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if 0
// Mount the pt in the sandbox so that the "tty" command works.
// FIXME: this doesn't work with the new devpts in the sandbox.
if (useChroot)
dirsInChroot[slaveName] = {slaveName, false};
#endif
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (unlockpt(builderOut.readSide.get()))
throw SysError("unlocking pseudoterminal");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
builderOut.writeSide = open(slaveName.c_str(), O_RDWR | O_NOCTTY);
if (!builderOut.writeSide)
throw SysError("opening pseudoterminal slave");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
// Put the pt into raw mode to prevent \n -> \r\n translation.
struct termios term;
if (tcgetattr(builderOut.writeSide.get(), &term))
throw SysError("getting pseudoterminal attributes");
When repairing a derivation, check and repair the entire output closure If we find a corrupted path in the output closure, we rebuild the derivation that produced that particular path.
2012-10-03 16:38:09 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
cfmakeraw(&term);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
throw SysError("putting pseudoterminal into raw mode");
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
result.startTime = time(0);
If a substitute closure is incomplete, build dependencies, then retry the substituter Issue #77.
2013-01-02 12:38:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Fork a child to build the package. */
ProcessOptions options;
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __linux__
if (useChroot) {
/* Set up private namespaces for the build:
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 15:48:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
- The PID namespace causes the build to start as PID 1.
Processes outside of the chroot are not visible to those
on the inside, but processes inside the chroot are
visible from the outside (though with different PIDs).
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
- The private mount namespace ensures that all the bind
mounts we do will only show up in this process and its
children, and will disappear automatically when we're
done.
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
- The private network namespace ensures that the builder
cannot talk to the outside world (or vice versa). It
only has a private loopback interface. (Fixed-output
derivations are not run in a private network namespace
to allow functions like fetchurl to work.)
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
- The IPC namespace prevents the builder from communicating
with outside processes using SysV IPC mechanisms (shared
memory, message queues, semaphores). It also ensures
that all IPC objects are destroyed when the builder
exits.
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
- The UTS namespace ensures that builders see a hostname of
localhost rather than the actual hostname.
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
We use a helper process to do the clone() to work around
clone() being broken in multi-threaded programs due to
at-fork handlers not being run. Note that we use
CLONE_PARENT to ensure that the real builder is parented to
us.
*/
CA derivations that depend on other CA derivations Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!(derivationIsImpure(derivationType)))
privateNetwork = true;
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 15:48:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
userNamespaceSync.create();
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 15:48:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
options.allowVfork = false;
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 15:48:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path maxUserNamespaces = "/proc/sys/user/max_user_namespaces";
static bool userNamespacesEnabled =
pathExists(maxUserNamespaces)
&& trim(readFile(maxUserNamespaces)) != "0";
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
usingUserNamespace = userNamespacesEnabled;
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Pid helper = startProcess([&]() {
Add an option to make non-determinism non-fatal That is, when build-repeat > 0, and the output of two rounds differ, then print a warning rather than fail the build. This is primarily to let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Drop additional groups here because we can't do it
after we've created the new user namespace. FIXME:
this means that if we're not root in the parent
namespace, we can't drop additional groups; they will
be mapped to nogroup in the child namespace. There does
not seem to be a workaround for this. (But who can tell
from reading user_namespaces(7)?)
See also https://lwn.net/Articles/621612/. */
if (getuid() == 0 && setgroups(0, 0) == -1)
throw SysError("setgroups failed");
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
size_t stackSize = 1 * 1024 * 1024;
char * stack = (char *) mmap(0, stackSize,
PROT_WRITE | PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);
if (stack == MAP_FAILED) throw SysError("allocating stack");
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 21:14:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
int flags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_PARENT | SIGCHLD;
if (privateNetwork)
flags |= CLONE_NEWNET;
if (usingUserNamespace)
flags |= CLONE_NEWUSER;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
pid_t child = clone(childEntry, stack + stackSize, flags, this);
if (child == -1 && errno == EINVAL) {
/* Fallback for Linux < 2.13 where CLONE_NEWPID and
CLONE_PARENT are not allowed together. */
flags &= ~CLONE_NEWPID;
child = clone(childEntry, stack + stackSize, flags, this);
}
if (usingUserNamespace && child == -1 && (errno == EPERM || errno == EINVAL)) {
/* Some distros patch Linux to not allow unprivileged
* user namespaces. If we get EPERM or EINVAL, try
* without CLONE_NEWUSER and see if that works.
*/
usingUserNamespace = false;
flags &= ~CLONE_NEWUSER;
child = clone(childEntry, stack + stackSize, flags, this);
}
/* Otherwise exit with EPERM so we can handle this in the
parent. This is only done when sandbox-fallback is set
to true (the default). */
if (child == -1 && (errno == EPERM || errno == EINVAL) && settings.sandboxFallback)
_exit(1);
if (child == -1) throw SysError("cloning builder process");
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFull(builderOut.writeSide.get(),
fmt("%d %d\n", usingUserNamespace, child));
_exit(0);
}, options);
Support sandbox builds by non-root users This allows an unprivileged user to perform builds on a diverted store (i.e. where the physical store location differs from the logical location). Example: $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \ 'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"' will do a build in the Nix store physically in /tmp/store but logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:45:11 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
int res = helper.wait();
if (res != 0 && settings.sandboxFallback) {
useChroot = false;
initTmpDir();
goto fallback;
} else if (res != 0)
throw Error("unable to start build process");
* Refactoring.
2009-03-23 00:53:05 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
userNamespaceSync.readSide = -1;
Improve "waiting for locks" messages These are now shown in the progress bar. Closes #3577.
2020-06-15 16:03:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Close the write side to prevent runChild() from hanging
reading from this. */
Finally cleanup([&]() {
userNamespaceSync.writeSide = -1;
});
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto ss = tokenizeString<std::vector<std::string>>(readLine(builderOut.readSide.get()));
assert(ss.size() == 2);
usingUserNamespace = ss[0] == "1";
string2Int(): Return std::optional
2021-01-08 12:22:21 +01:00
pid = string2Int<pid_t>(ss[1]).value();
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 15:48:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (usingUserNamespace) {
/* Set the UID/GID mapping of the builder's user namespace
such that the sandbox user maps to the build user, or to
the calling user (if build users are disabled). */
uid_t hostUid = buildUser ? buildUser->getUID() : getuid();
uid_t hostGid = buildUser ? buildUser->getGID() : getgid();
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 14:13:58 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile("/proc/" + std::to_string(pid) + "/uid_map",
fmt("%d %d 1", sandboxUid(), hostUid));
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile("/proc/" + std::to_string(pid) + "/setgroups", "deny");
Improve "waiting for locks" messages These are now shown in the progress bar. Closes #3577.
2020-06-15 16:03:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile("/proc/" + std::to_string(pid) + "/gid_map",
fmt("%d %d 1", sandboxGid(), hostGid));
} else {
debug("note: not using a user namespace");
if (!buildUser)
throw Error("cannot perform a sandboxed build because user namespaces are not enabled; check /proc/sys/user/max_user_namespaces");
}
Fix max-jobs option After 0ed946aa616bbf7ffe7f90d3309abdd27d875b10, max-jobs setting (-j/--max-jobs) stopped working. The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure out whether the limit is reached or not) is not incremented yet when tryBuild is started; So, the solution is to move the check to tryLocalBuild. Closes https://github.com/nixos/nix/issues/3763
2020-09-24 23:07:42 +02:00
Create /etc/passwd *after* figuring out the sandbox uid/gid Fixes build failures like # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv No user exists for uid 30012
2020-10-14 12:20:58 +02:00
/* Now that we now the sandbox uid, we can write
/etc/passwd. */
writeFile(chrootRootDir + "/etc/passwd", fmt(
"root:x:0:0:Nix build user:%3%:/noshell\n"
"nixbld:x:%1%:%2%:Nix build user:%3%:/noshell\n"
"nobody:x:65534:65534:Nobody:/:/noshell\n",
sandboxUid(), sandboxGid(), settings.sandboxBuildDir));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Save the mount namespace of the child. We have to do this
*before* the child does a chroot. */
sandboxMountNamespace = open(fmt("/proc/%d/ns/mnt", (pid_t) pid).c_str(), O_RDONLY);
if (sandboxMountNamespace.get() == -1)
throw SysError("getting sandbox mount namespace");
Fix max-jobs option After 0ed946aa616bbf7ffe7f90d3309abdd27d875b10, max-jobs setting (-j/--max-jobs) stopped working. The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure out whether the limit is reached or not) is not incremented yet when tryBuild is started; So, the solution is to move the check to tryLocalBuild. Closes https://github.com/nixos/nix/issues/3763
2020-09-24 23:07:42 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Signal the builder that we've updated its user namespace. */
writeFull(userNamespaceSync.writeSide.get(), "1");
} else
#endif
{
fallback:
options.allowVfork = !buildUser && !drv->isBuiltin();
pid = startProcess([&]() {
runChild();
}, options);
* Refactoring.
2009-03-23 00:53:05 +01:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* parent */
pid.setSeparatePG(true);
builderOut.writeSide = -1;
worker.childStarted(shared_from_this(), {builderOut.readSide.get()}, true, true);
Don't lock a user while doing remote builds
2020-05-14 16:00:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check if setting up the build environment failed. */
Add more instrumentation for #4270
2021-02-05 15:57:33 +01:00
std::vector<std::string> msgs;
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
while (true) {
Add a trace to readLine() failures Hopefully this helps to diagnose 'error: unexpected EOF reading a line' on macOS.
2021-02-05 12:11:50 +01:00
string msg = [&]() {
try {
return readLine(builderOut.readSide.get());
} catch (Error & e) {
Add more instrumentation for #4270
2021-02-05 15:57:33 +01:00
e.addTrace({}, "while waiting for the build environment to initialize (previous messages: %s)",
concatStringsSep("|", msgs));
Add a trace to readLine() failures Hopefully this helps to diagnose 'error: unexpected EOF reading a line' on macOS.
2021-02-05 12:11:50 +01:00
throw e;
}
}();
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (string(msg, 0, 1) == "\2") break;
if (string(msg, 0, 1) == "\1") {
FdSource source(builderOut.readSide.get());
auto ex = readError(source);
ex.addTrace({}, "while setting up the build environment");
throw ex;
Don't lock a user while doing remote builds
2020-05-14 16:00:54 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("sandbox setup: " + msg);
Add more instrumentation for #4270
2021-02-05 15:57:33 +01:00
msgs.push_back(std::move(msg));
Don't lock a user while doing remote builds
2020-05-14 16:00:54 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Don't lock a user while doing remote builds
2020-05-14 16:00:54 +02:00
Improve "waiting for locks" messages These are now shown in the progress bar. Closes #3577.
2020-06-15 16:03:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::initTmpDir() {
/* In a sandbox, for determinism, always use the same temporary
directory. */
#if __linux__
tmpDirInSandbox = useChroot ? settings.sandboxBuildDir : tmpDir;
#else
tmpDirInSandbox = tmpDir;
#endif
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* In non-structured mode, add all bindings specified in the
derivation via the environment, except those listed in the
passAsFile attribute. Those are passed as file names pointing
to temporary files containing the contents. Note that
passAsFile is ignored in structure mode because it's not
needed (attributes are not passed through the environment, so
there is no size constraint). */
if (!parsedDrv->getStructuredAttrs()) {
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 16:11:19 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StringSet passAsFile = tokenizeString<StringSet>(get(drv->env, "passAsFile").value_or(""));
for (auto & i : drv->env) {
if (passAsFile.find(i.first) == passAsFile.end()) {
env[i.first] = i.second;
} else {
auto hash = hashString(htSHA256, i.first);
string fn = ".attr-" + hash.to_string(Base32, false);
Path p = tmpDir + "/" + fn;
writeFile(p, rewriteStrings(i.second, inputRewrites));
chownToBuilder(p);
env[i.first + "Path"] = tmpDirInSandbox + "/" + fn;
}
}
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Progress indicator: More improvements
2017-08-15 15:31:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* For convenience, set an environment pointing to the top build
directory. */
env["NIX_BUILD_TOP"] = tmpDirInSandbox;
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Also set TMPDIR and variants to point to this directory. */
env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] = tmpDirInSandbox;
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Explicitly set PWD to prevent problems with chroot builds. In
particular, dietlibc cannot figure out the cwd because the
inode of the current directory doesn't appear in .. (because
getdents returns the inode of the mount point). */
env["PWD"] = tmpDirInSandbox;
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::initEnv()
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
env.clear();
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Most shells initialise PATH to some default (/bin:/usr/bin:...) when
PATH is not set. We don't want this, so we fill it in with some dummy
value. */
env["PATH"] = "/path-not-set";
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Set HOME to a non-existing path to prevent certain programs from using
/etc/passwd (or NIS, or whatever) to locate the home directory (for
example, wget looks for ~/.wgetrc). I.e., these tools use /etc/passwd
if HOME is not set, but they will just assume that the settings file
they are looking for does not exist if HOME is set but points to some
non-existing path. */
env["HOME"] = homeDir;
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Tell the builder where the Nix store is. Usually they
shouldn't care, but this is useful for purity checking (e.g.,
the compiler or linker might only want to accept paths to files
in the store or in the build directory). */
env["NIX_STORE"] = worker.store.storeDir;
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* The maximum number of cores to utilize for parallel building. */
env["NIX_BUILD_CORES"] = (format("%d") % settings.buildCores).str();
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
initTmpDir();
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Compatibility hack with Nix <= 0.7: if this is a fixed-output
derivation, tell the builder, so that for instance `fetchurl'
can skip checking the output. On older Nixes, this environment
variable won't be set, so `fetchurl' will do the check. */
if (derivationIsFixed(derivationType)) env["NIX_OUTPUT_CHECKED"] = "1";
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* *Only* if this is a fixed-output derivation, propagate the
values of the environment variables specified in the
`impureEnvVars' attribute to the builder. This allows for
instance environment variables for proxy configuration such as
`http_proxy' to be easily passed to downloaders like
`fetchurl'. Passing such environment variables from the caller
to the builder is generally impure, but the output of
fixed-output derivations is by definition pure (since we
already know the cryptographic hash of the output). */
if (derivationIsImpure(derivationType)) {
for (auto & i : parsedDrv->getStringsAttr("impureEnvVars").value_or(Strings()))
env[i] = getEnv(i).value_or("");
build: recover store path when replacing fails This shouldn't happen in normal circumstances, but just in case attempt to move the temporary path back if possible.
2020-01-04 17:47:52 +01:00
}
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Currently structured log messages piggyback on stderr, but we
may change that in the future. So tell the builder which file
descriptor to use for that. */
env["NIX_LOG_FD"] = "2";
/* Trigger colored output in various tools. */
env["TERM"] = "xterm-256color";
Add a --repair flag to ‘nix-store -r’ to repair derivation outputs With this flag, if any valid derivation output is missing or corrupt, it will be recreated by using a substitute if available, or by rebuilding the derivation. The latter may use hash rewriting if chroots are not available.
2012-10-02 23:13:46 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
static std::regex shVarName("[A-Za-z_][A-Za-z0-9_]*");
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::writeStructuredAttrs()
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto structuredAttrs = parsedDrv->getStructuredAttrs();
if (!structuredAttrs) return;
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto json = *structuredAttrs;
UserLock: Make more RAII-ish
2017-01-25 12:45:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Add an "outputs" object containing the output paths. */
nlohmann::json outputs;
for (auto & i : drv->outputs) {
/* The placeholder must have a rewrite, so we use it to cover both the
cases where we know or don't know the output path ahead of time. */
outputs[i.first] = rewriteStrings(hashPlaceholder(i.first), inputRewrites);
}
json["outputs"] = outputs;
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Handle exportReferencesGraph. */
auto e = json.find("exportReferencesGraph");
if (e != json.end() && e->is_object()) {
for (auto i = e->begin(); i != e->end(); ++i) {
std::ostringstream str;
{
JSONPlaceholder jsonRoot(str, true);
StorePathSet storePaths;
for (auto & p : *i)
storePaths.insert(worker.store.parseStorePath(p.get<std::string>()));
worker.store.pathInfoToJSON(jsonRoot,
exportReferences(storePaths), false, true);
}
json[i.key()] = nlohmann::json::parse(str.str()); // urgh
}
}
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile(tmpDir + "/.attrs.json", rewriteStrings(json.dump(), inputRewrites));
chownToBuilder(tmpDir + "/.attrs.json");
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 01:19:50 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* As a convenience to bash scripts, write a shell file that
maps all attributes that are representable in bash -
namely, strings, integers, nulls, Booleans, and arrays and
objects consisting entirely of those values. (So nested
arrays or objects are not supported.) */
Add an option to make non-determinism non-fatal That is, when build-repeat > 0, and the output of two rounds differ, then print a warning rather than fail the build. This is primarily to let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto handleSimpleType = [](const nlohmann::json & value) -> std::optional<std::string> {
if (value.is_string())
return shellEscape(value);
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (value.is_number()) {
auto f = value.get<float>();
if (std::ceil(f) == f)
return std::to_string(value.get<int>());
}
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 01:19:50 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (value.is_null())
return std::string("''");
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 01:19:50 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (value.is_boolean())
return value.get<bool>() ? std::string("1") : std::string("");
* Also kill all processes of the build user after the build. This is critical to prevent certain kinds of 0wnage.
2005-10-17 19:43:21 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return {};
};
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::string jsonSh;
Heuristically detect if a build may have failed due to a full disk This will allow Hydra to detect that a build should not be marked as "permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto i = json.begin(); i != json.end(); ++i) {
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!std::regex_match(i.key(), shVarName)) continue;
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto & value = i.value();
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto s = handleSimpleType(value);
if (s)
jsonSh += fmt("declare %s=%s\n", i.key(), *s);
Fix assertion failure in canonicalisePathMetaData() after hash rewriting The assertion in canonicalisePathMetaData() failed because the ownership of the path already changed due to the hash rewriting. The solution is not to check the ownership of rewritten paths. Issue #122.
2013-06-13 17:12:06 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
else if (value.is_array()) {
std::string s2;
bool good = true;
Show the log tail when a build fails If --no-build-output is given (which will become the default for the "nix" command at least), show the last 10 lines of the build output if the build fails.
2016-04-25 16:47:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto i = value.begin(); i != value.end(); ++i) {
auto s3 = handleSimpleType(i.value());
if (!s3) { good = false; break; }
s2 += *s3; s2 += ' ';
Show the log tail when a build fails If --no-build-output is given (which will become the default for the "nix" command at least), show the last 10 lines of the build output if the build fails.
2016-04-25 16:47:46 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (good)
jsonSh += fmt("declare -a %s=(%s)\n", i.key(), s2);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 18:26:21 +01:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
else if (value.is_object()) {
std::string s2;
bool good = true;
Move some code around In particular, do replacing of valid paths during repair later. This prevents us from replacing a valid path after the build fails.
2014-02-17 22:25:15 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto i = value.begin(); i != value.end(); ++i) {
auto s3 = handleSimpleType(i.value());
if (!s3) { good = false; break; }
s2 += fmt("[%s]=%s ", shellEscape(i.key()), *s3);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (good)
jsonSh += fmt("declare -A %s=(%s)\n", i.key(), s2);
}
}
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile(tmpDir + "/.attrs.sh", rewriteStrings(jsonSh, inputRewrites));
chownToBuilder(tmpDir + "/.attrs.sh");
}
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 16:33:12 +01:00
struct RestrictedStoreConfig : virtual LocalFSStoreConfig
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
{
using LocalFSStoreConfig::LocalFSStoreConfig;
const std::string name() { return "Restricted Store"; }
};
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* A wrapper around LocalStore that only allows building/querying of
paths that are in the input closures of the build or were added via
recursive Nix calls. */
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 16:33:12 +01:00
struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual LocalFSStore
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
{
ref<LocalStore> next;
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
DerivationGoal & goal;
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
RestrictedStore(const Params & params, ref<LocalStore> next, DerivationGoal & goal)
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 16:33:12 +01:00
: StoreConfig(params)
, LocalFSStoreConfig(params)
, RestrictedStoreConfig(params)
, Store(params)
, LocalFSStore(params)
, next(next), goal(goal)
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
{ }
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path getRealStoreDir() override
{ return next->realStoreDir; }
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::string getUri() override
{ return next->getUri(); }
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePathSet queryAllValidPaths() override
{
StorePathSet paths;
for (auto & p : goal.inputPaths) paths.insert(p);
for (auto & p : goal.addedPaths) paths.insert(p);
return paths;
}
Add a post-build-hook Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-07-11 20:23:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void queryPathInfoUncached(const StorePath & path,
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept override
{
if (goal.isAllowed(path)) {
try {
/* Censor impure information. */
auto info = std::make_shared<ValidPathInfo>(*next->queryPathInfo(path));
info->deriver.reset();
info->registrationTime = 0;
info->ultimate = false;
info->sigs.clear();
callback(info);
} catch (InvalidPath &) {
callback(nullptr);
}
} else
callback(nullptr);
};
Add a flag ‘--check’ to verify build determinism The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to redo the build of a derivation whose output paths are already valid. If the new output differs from the original output, an error is printed. This makes it easier to test if a build is deterministic. (Obviously this cannot catch all sources of non-determinism, but it catches the most common one, namely the current time.) For example: $ nix-build '<nixpkgs>' -A patchelf ... $ nix-build '<nixpkgs>' -A patchelf --check error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv' The --check build fails if not all outputs are valid. Thus the first call to nix-build is necessary to ensure that all outputs are valid. The current outputs are left untouched: the new outputs are either put in a chroot or diverted to a different location in the store using hash rewriting.
2014-02-18 01:01:14 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void queryReferrers(const StorePath & path, StorePathSet & referrers) override
{ }
Add a flag ‘--check’ to verify build determinism The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to redo the build of a derivation whose output paths are already valid. If the new output differs from the original output, an error is printed. This makes it easier to test if a build is deterministic. (Obviously this cannot catch all sources of non-determinism, but it catches the most common one, namely the current time.) For example: $ nix-build '<nixpkgs>' -A patchelf ... $ nix-build '<nixpkgs>' -A patchelf --check error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv' The --check build fails if not all outputs are valid. Thus the first call to nix-build is necessary to ensure that all outputs are valid. The current outputs are left untouched: the new outputs are either put in a chroot or diverted to a different location in the store using hash rewriting.
2014-02-18 01:01:14 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::map<std::string, std::optional<StorePath>> queryPartialDerivationOutputMap(const StorePath & path) override
{
if (!goal.isAllowed(path))
throw InvalidPath("cannot query output map for unknown path '%s' in recursive Nix", printStorePath(path));
return next->queryPartialDerivationOutputMap(path);
}
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 12:08:27 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override
{ throw Error("queryPathFromHashPart"); }
Keep build directory if not all expected outputs were produced Fixes issue #123 in Nixpkgs.
2012-09-18 16:11:42 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePath addToStore(const string & name, const Path & srcPath,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256,
PathFilter & filter = defaultPathFilter, RepairFlag repair = NoRepair) override
{ throw Error("addToStore"); }
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void addToStore(const ValidPathInfo & info, Source & narSource,
RepairFlag repair = NoRepair, CheckSigsFlag checkSigs = CheckSigs) override
{
next->addToStore(info, narSource, repair, checkSigs);
goal.addDependency(info.path);
}
* Make sure that lock files are cleaned up properly when building through the build hook.
2011-12-25 17:38:37 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePath addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair = NoRepair) override
{
auto path = next->addTextToStore(name, s, references, repair);
goal.addDependency(path);
return path;
}
* Don't cache transient build hook problems.
2009-04-15 08:25:02 +02:00
Restore RestrictedStore.addToStoreFromDump implementation It was accidentally removed in commit ca30abb3fb36440e5a13161c39647189036fc18f
2020-10-31 23:52:09 +01:00
StorePath addToStoreFromDump(Source & dump, const string & name,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override
{
auto path = next->addToStoreFromDump(dump, name, method, hashAlgo, repair);
goal.addDependency(path);
return path;
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void narFromPath(const StorePath & path, Sink & sink) override
{
if (!goal.isAllowed(path))
throw InvalidPath("cannot dump unknown path '%s' in recursive Nix", printStorePath(path));
LocalFSStore::narFromPath(path, sink);
}
Provide more detailed info about build status to hydra-queue-runner In particular, hydra-queue-runner can now distinguish between remote build / substitution / already-valid. For instance, if a path already existed on the remote side, we don't want to store a log file.
2015-07-20 03:15:45 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void ensurePath(const StorePath & path) override
{
if (!goal.isAllowed(path))
throw InvalidPath("cannot substitute unknown path '%s' in recursive Nix", printStorePath(path));
/* Nothing to be done; 'path' must already be valid. */
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 17:36:51 +02:00
void registerDrvOutput(const Realisation & info) override
Restrict the operations on drv outputs in recursive Nix There's currently no way to properly filter them, so disallow them altogether instead.
2020-12-11 21:12:53 +01:00
// XXX: This should probably be allowed as a no-op if the realisation
// corresponds to an allowed derivation
{ throw Error("registerDrvOutput"); }
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 17:36:51 +02:00
std::optional<const Realisation> queryRealisation(const DrvOutput & id) override
Restrict the operations on drv outputs in recursive Nix There's currently no way to properly filter them, so disallow them altogether instead.
2020-12-11 21:12:53 +01:00
// XXX: This should probably be allowed if the realisation corresponds to
// an allowed derivation
{ throw Error("queryRealisation"); }
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 17:36:51 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void buildPaths(const std::vector<StorePathWithOutputs> & paths, BuildMode buildMode) override
{
if (buildMode != bmNormal) throw Error("unsupported build mode");
StorePathSet newPaths;
for (auto & path : paths) {
if (!goal.isAllowed(path.path))
throw InvalidPath("cannot build unknown path '%s' in recursive Nix", printStorePath(path.path));
Propagate remote timeouts properly
2014-08-17 19:09:03 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
next->buildPaths(paths, buildMode);
for (auto & path : paths) {
if (!path.path.isDerivation()) continue;
auto outputs = next->queryDerivationOutputMap(path.path);
for (auto & output : outputs)
if (wantOutput(output.first, path.outputs))
newPaths.insert(output.second);
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 12:08:27 +01:00
}
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 22:05:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePathSet closure;
next->computeFSClosure(newPaths, closure);
for (auto & path : closure)
goal.addDependency(path);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 12:21:44 +02:00
}
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
BuildMode buildMode = bmNormal) override
{ unsupported("buildDerivation"); }
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void addTempRoot(const StorePath & path) override
{ }
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void addIndirectRoot(const Path & path) override
{ }
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Roots findRoots(bool censor) override
{ return Roots(); }
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void collectGarbage(const GCOptions & options, GCResults & results) override
{ }
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing
2017-03-03 15:40:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void addSignatures(const StorePath & storePath, const StringSet & sigs) override
{ unsupported("addSignatures"); }
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing
2017-03-03 15:40:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void queryMissing(const std::vector<StorePathWithOutputs> & targets,
StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown,
uint64_t & downloadSize, uint64_t & narSize) override
{
/* This is slightly impure since it leaks information to the
client about what paths will be built/substituted or are
already present. Probably not a big deal. */
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing
2017-03-03 15:40:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::vector<StorePathWithOutputs> allowed;
for (auto & path : targets) {
if (goal.isAllowed(path.path))
allowed.emplace_back(path);
else
unknown.insert(path.path);
build-hook: If there are no machines defined, quit permanently
2017-05-02 12:16:29 +02:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
next->queryMissing(allowed, willBuild, willSubstitute,
unknown, downloadSize, narSize);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
};
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 16:53:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::startDaemon()
{
settings.requireExperimentalFeature("recursive-nix");
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Store::Params params;
params["path-info-cache-size"] = "0";
params["store"] = worker.store.storeDir;
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
if (auto localStore = dynamic_cast<LocalStore *>(&worker.store))
params["root"] = localStore->rootDir;
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
params["state"] = "/no-such-path";
params["log"] = "/no-such-path";
auto store = make_ref<RestrictedStore>(params,
ref<LocalStore>(std::dynamic_pointer_cast<LocalStore>(worker.store.shared_from_this())),
*this);
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
addedPaths.clear();
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto socketName = ".nix-socket";
Path socketPath = tmpDir + "/" + socketName;
env["NIX_REMOTE"] = "unix://" + tmpDirInSandbox + "/" + socketName;
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
daemonSocket = createUnixDomainSocket(socketPath, 0600);
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
chownToBuilder(socketPath);
exportReferences: Check whether the path is in the input closure Fixes #2082.
2018-04-17 12:03:27 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
daemonThread = std::thread([this, store]() {
exportReferences: Check whether the path is in the input closure Fixes #2082.
2018-04-17 12:03:27 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
while (true) {
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Accept a connection. */
struct sockaddr_un remoteAddr;
socklen_t remoteAddrLen = sizeof(remoteAddr);
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
AutoCloseFD remote = accept(daemonSocket.get(),
(struct sockaddr *) &remoteAddr, &remoteAddrLen);
if (!remote) {
if (errno == EINTR) continue;
if (errno == EINVAL) break;
throw SysError("accepting connection");
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
closeOnExec(remote.get());
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("received daemon connection");
Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments
2018-01-23 20:13:30 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto workerThread = std::thread([store, remote{std::move(remote)}]() {
FdSource from(remote.get());
FdSink to(remote.get());
try {
daemon::processConnection(store, from, to,
daemon::NotTrusted, daemon::Recursive,
[&](Store & store) { store.createUser("nobody", 65535); });
debug("terminated daemon connection");
} catch (SysError &) {
ignoreException();
}
});
Fix #2162: use getaddrinfo instead of curl to preload NSS
2018-06-12 13:05:14 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
daemonWorkerThreads.push_back(std::move(workerThread));
Fix #2162: use getaddrinfo instead of curl to preload NSS
2018-06-12 13:05:14 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("daemon shutting down");
Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments
2018-01-23 20:13:30 +01:00
});
}
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Factor out linkOrCopy()
2018-10-02 11:22:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::stopDaemon()
Factor out linkOrCopy()
2018-10-02 11:22:13 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (daemonSocket && shutdown(daemonSocket.get(), SHUT_RDWR) == -1)
throw SysError("shutting down daemon socket");
Don't chmod symlink before moving outputs around Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 16:38:45 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (daemonThread.joinable())
daemonThread.join();
// FIXME: should prune worker threads more quickly.
// FIXME: shutdown the client socket to speed up worker termination.
for (auto & thread : daemonWorkerThreads)
thread.join();
daemonWorkerThreads.clear();
daemonSocket = -1;
Factor out linkOrCopy()
2018-10-02 11:22:13 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::addDependency(const StorePath & path)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (isAllowed(path)) return;
* A switch `-j NUMBER' to set the maximum number of parallel jobs (0 = no limit). * Add missing file to distribution.
2004-05-12 16:20:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
addedPaths.insert(path);
Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments
2018-01-23 20:13:30 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If we're doing a sandbox build, then we have to make the path
appear in the sandbox. */
if (useChroot) {
Don't allow sandbox profile except in relaxed mode This makes Darwin consistent with Linux: Nix expressions can't break out of the sandbox unless relaxed sandbox mode is enabled. For the normal sandbox mode this will require fixing #759 however.
2016-01-04 16:32:16 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("materialising '%s' in the sandbox", worker.store.printStorePath(path));
Use deterministic $TMPDIR in sandbox Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the temporary directory is now always /tmp/nix-build-<drvname>-0. This improves bitwise-exact reproducibility for builds that store $TMPDIR in their build output. (Of course, those should still be fixed...)
2015-12-02 14:59:07 +01:00
Barf when using a diverted store on macOS Fixes #1792.
2018-01-15 12:14:43 +01:00
#if __linux__
Support sandbox builds by non-root users This allows an unprivileged user to perform builds on a diverted store (i.e. where the physical store location differs from the logical location). Example: $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \ 'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"' will do a build in the Nix store physically in /tmp/store but logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:45:11 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path source = worker.store.Store::toRealPath(path);
Path target = chrootRootDir + worker.store.printStorePath(path);
debug("bind-mounting %s -> %s", target, source);
Use deterministic $TMPDIR in sandbox Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the temporary directory is now always /tmp/nix-build-<drvname>-0. This improves bitwise-exact reproducibility for builds that store $TMPDIR in their build output. (Of course, those should still be fixed...)
2015-12-02 14:59:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (pathExists(target))
throw Error("store path '%s' already exists in the sandbox", worker.store.printStorePath(path));
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto st = lstat(source);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (S_ISDIR(st.st_mode)) {
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Bind-mount the path into the sandbox. This requires
entering its mount namespace, which is not possible
in multithreaded programs. So we do this in a
child process.*/
Pid child(startProcess([&]() {
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (setns(sandboxMountNamespace.get(), 0) == -1)
throw SysError("entering sandbox mount namespace");
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
createDirs(target);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (mount(source.c_str(), target.c_str(), "", MS_BIND, 0) == -1)
throw SysError("bind mount from '%s' to '%s' failed", source, target);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
_exit(0);
}));
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
int status = child.wait();
if (status != 0)
throw Error("could not add path '%s' to sandbox", worker.store.printStorePath(path));
} else
linkOrCopy(source, target);
#else
throw Error("don't know how to make path '%s' (produced by a recursive Nix call) appear in the sandbox",
worker.store.printStorePath(path));
#endif
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Add a mechanism for derivation attributes to reference the derivation's outputs For example, you can now say: configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}"; The strings returned by the ‘placeholder’ builtin are replaced at build time by the actual store paths corresponding to the specified outputs. Previously, you had to work around the inability to self-reference by doing stuff like: preConfigure = '' configureFlags+=" --prefix $out --includedir=$dev" ''; or rely on ad-hoc variable interpolation semantics in Autoconf or Make (e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 15:12:54 +02:00
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::chownToBuilder(const Path & path)
{
if (!buildUser) return;
if (chown(path.c_str(), buildUser->getUID(), buildUser->getGID()) == -1)
throw SysError("cannot change ownership of '%1%'", path);
}
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 19:11:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void setupSeccomp()
{
#if __linux__
if (!settings.filterSyscalls) return;
#if HAVE_SECCOMP
scmp_filter_ctx ctx;
Fix exportReferencesGraph in the structured attrs case
2017-10-25 14:08:29 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!(ctx = seccomp_init(SCMP_ACT_ALLOW)))
throw SysError("unable to initialize seccomp mode 2");
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 18:36:42 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Finally cleanup([&]() {
seccomp_release(ctx);
});
Improve setting the default chroot dirs
2015-03-24 11:57:46 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nativeSystem == "x86_64-linux" &&
seccomp_arch_add(ctx, SCMP_ARCH_X86) != 0)
throw SysError("unable to add 32-bit seccomp architecture");
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nativeSystem == "x86_64-linux" &&
seccomp_arch_add(ctx, SCMP_ARCH_X32) != 0)
throw SysError("unable to add X32 seccomp architecture");
Add option to verify build determinism Passing "--option build-repeat <N>" will cause every build to be repeated N times. If the build output differs between any round, the build is rejected, and the output paths are not registered as valid. This is primarily useful to verify build determinism. (We already had a --check option to repeat a previously succeeded build. However, with --check, non-deterministic builds are registered in the DB. Preventing that is useful for Hydra to ensure that non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (nativeSystem == "aarch64-linux" &&
seccomp_arch_add(ctx, SCMP_ARCH_ARM) != 0)
printError("unable to add ARM seccomp architecture; this may result in spurious build failures if running 32-bit ARM processes");
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Prevent builders from creating setuid/setgid binaries. */
for (int perm : { S_ISUID, S_ISGID }) {
if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(chmod), 1,
SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0)
throw SysError("unable to add seccomp rule");
Add the closure of store paths to the chroot Thus, for example, to get /bin/sh in a chroot, you only need to specify /bin/sh=${pkgs.bash}/bin/sh in build-chroot-dirs. The dependencies of sh will be added automatically.
2015-03-24 11:50:22 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(fchmod), 1,
SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0)
throw SysError("unable to add seccomp rule");
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(fchmodat), 1,
SCMP_A2(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0)
throw SysError("unable to add seccomp rule");
}
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Prevent builders from creating EAs or ACLs. Not all filesystems
support these, and they're not allowed in the Nix store because
they're not representable in the NAR serialisation. */
if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(setxattr), 0) != 0 ||
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(lsetxattr), 0) != 0 ||
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(fsetxattr), 0) != 0)
throw SysError("unable to add seccomp rule");
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (seccomp_attr_set(ctx, SCMP_FLTATR_CTL_NNP, settings.allowNewPrivileges ? 0 : 1) != 0)
throw SysError("unable to set 'no new privileges' seccomp attribute");
Add basic Apple sandbox support
2015-01-06 07:27:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (seccomp_load(ctx) != 0)
throw SysError("unable to load seccomp BPF program");
#else
throw Error(
"seccomp is not supported on this platform; "
"you can bypass this error by setting the option 'filter-syscalls' to false, but note that untrusted builds can then create setuid binaries!");
#endif
#endif
}
* Delete the chroot directory automatically. * Removed some debug messages.
2007-10-27 18:06:38 +02:00
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::runChild()
{
/* Warning: in the child we should absolutely not make any SQLite
calls! */
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 02:46:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try { /* child */
Tighten permissions on chroot directories
2015-03-24 11:35:53 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
commonChildInit(builderOut);
Tighten permissions on chroot directories
2015-03-24 11:35:53 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
try {
setupSeccomp();
} catch (...) {
if (buildUser) throw;
}
* Create a /tmp with 1777 permission in the chroot. Some builders need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 18:03:03 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
bool setUser = true;
* Provide a minimal /etc/passwd in the chroot to keep some builders happy.
2008-12-11 18:44:02 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Make the contents of netrc available to builtin:fetchurl
(which may run under a different uid and/or in a sandbox). */
std::string netrcData;
try {
if (drv->isBuiltin() && drv->builder == "builtin:fetchurl")
netrcData = readFile(settings.netrcFile);
} catch (SysError &) { }
nix: add /etc/hosts with localhost entry to chroot builds.
2011-11-21 16:19:51 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __linux__
if (useChroot) {
Tighten permissions on chroot directories
2015-03-24 11:35:53 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
userNamespaceSync.writeSide = -1;
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 19:57:10 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (drainFD(userNamespaceSync.readSide.get()) != "1")
throw Error("user namespace initialisation failed");
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
userNamespaceSync.readSide = -1;
Handle repairing paths that are in build-chroot-dirs
2012-10-03 17:20:16 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (privateNetwork) {
Support building a derivation if some outputs are already valid (non-chroot case) This uses scary hash rewriting. Fixes #21.
2012-09-12 00:39:22 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Initialise the loopback interface. */
AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP));
if (!fd) throw SysError("cannot open IP socket");
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
struct ifreq ifr;
strcpy(ifr.ifr_name, "lo");
ifr.ifr_flags = IFF_UP | IFF_LOOPBACK | IFF_RUNNING;
if (ioctl(fd.get(), SIOCSIFFLAGS, &ifr) == -1)
throw SysError("cannot set loopback interface flags");
Add the pre-build hook. This hook can be used to set system-specific per-derivation build settings that don't fit into the derivation model and are too complex or volatile to be hard-coded into nix. Currently, the pre-build hook can only add chroot dirs/files through the interface, but it also has full access to the chroot root. The specific use case for this is systems where the operating system ABI is more complex than just the kernel-support system calls. For example, on OS X there is a set of system-provided frameworks that can reliably be accessed by any program linked to them, no matter the version the program is running on. Unfortunately, those frameworks do not necessarily live in the same locations on each version of OS X, nor do their dependencies, and thus nix needs to know the specific version of OS X currently running in order to make those frameworks available. The pre-build hook is a perfect mechanism for doing just that.
2015-04-18 22:56:02 +02:00
}
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Set the hostname etc. to fixed values. */
char hostname[] = "localhost";
if (sethostname(hostname, sizeof(hostname)) == -1)
throw SysError("cannot set host name");
char domainname[] = "(none)"; // kernel default
if (setdomainname(domainname, sizeof(domainname)) == -1)
throw SysError("cannot set domain name");
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Make all filesystems private. This is necessary
because subtrees may have been mounted as "shared"
(MS_SHARED). (Systemd does this, for instance.) Even
though we have a private mount namespace, mounting
filesystems on top of a shared subtree still propagates
outside of the namespace. Making a subtree private is
local to the namespace, though, so setting MS_PRIVATE
does not affect the outside world. */
if (mount(0, "/", 0, MS_PRIVATE | MS_REC, 0) == -1)
throw SysError("unable to make '/' private");
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Bind-mount chroot directory to itself, to treat it as a
different filesystem from /, as needed for pivot_root. */
if (mount(chrootRootDir.c_str(), chrootRootDir.c_str(), 0, MS_BIND, 0) == -1)
throw SysError("unable to bind mount '%1%'", chrootRootDir);
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Bind-mount the sandbox's Nix store onto itself so that
we can mark it as a "shared" subtree, allowing bind
mounts made in *this* mount namespace to be propagated
into the child namespace created by the
unshare(CLONE_NEWNS) call below.
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Marking chrootRootDir as MS_SHARED causes pivot_root()
to fail with EINVAL. Don't know why. */
Path chrootStoreDir = chrootRootDir + worker.store.storeDir;
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (mount(chrootStoreDir.c_str(), chrootStoreDir.c_str(), 0, MS_BIND, 0) == -1)
throw SysError("unable to bind mount the Nix store", chrootStoreDir);
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (mount(0, chrootStoreDir.c_str(), 0, MS_SHARED, 0) == -1)
throw SysError("unable to make '%s' shared", chrootStoreDir);
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Set up a nearly empty /dev, unless the user asked to
bind-mount the host /dev. */
Strings ss;
if (dirsInChroot.find("/dev") == dirsInChroot.end()) {
createDirs(chrootRootDir + "/dev/shm");
createDirs(chrootRootDir + "/dev/pts");
ss.push_back("/dev/full");
if (worker.store.systemFeatures.get().count("kvm") && pathExists("/dev/kvm"))
ss.push_back("/dev/kvm");
ss.push_back("/dev/null");
ss.push_back("/dev/random");
ss.push_back("/dev/tty");
ss.push_back("/dev/urandom");
ss.push_back("/dev/zero");
createSymlink("/proc/self/fd", chrootRootDir + "/dev/fd");
createSymlink("/proc/self/fd/0", chrootRootDir + "/dev/stdin");
createSymlink("/proc/self/fd/1", chrootRootDir + "/dev/stdout");
createSymlink("/proc/self/fd/2", chrootRootDir + "/dev/stderr");
}
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Fixed-output derivations typically need to access the
network, so give them access to /etc/resolv.conf and so
on. */
if (derivationIsImpure(derivationType)) {
ss.push_back("/etc/resolv.conf");
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
// Only use nss functions to resolve hosts and
// services. Don’t use it for anything else that may
// be configured for this system. This limits the
// potential impurities introduced in fixed-outputs.
writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n");
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
ss.push_back("/etc/services");
ss.push_back("/etc/hosts");
if (pathExists("/var/run/nscd/socket"))
ss.push_back("/var/run/nscd/socket");
}
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : ss) dirsInChroot.emplace(i, i);
Run builds in a pseudo-terminal This allows many programs (e.g. gcc, clang, cmake) to print colorized log output (assuming $TERM is set to a value like "xterm"). There are other ways to get colors, in particular setting CLICOLOR_FORCE, but they're less widely supported and can break programs that parse tool output.
2019-05-17 22:29:15 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Bind-mount all the directories from the "host"
filesystem that we want in the chroot
environment. */
auto doBind = [&](const Path & source, const Path & target, bool optional = false) {
debug("bind mounting '%1%' to '%2%'", source, target);
struct stat st;
if (stat(source.c_str(), &st) == -1) {
if (optional && errno == ENOENT)
return;
else
throw SysError("getting attributes of path '%1%'", source);
}
if (S_ISDIR(st.st_mode))
createDirs(target);
else {
createDirs(dirOf(target));
writeFile(target, "");
}
if (mount(source.c_str(), target.c_str(), "", MS_BIND | MS_REC, 0) == -1)
throw SysError("bind mount from '%1%' to '%2%' failed", source, target);
};
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 22:44:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : dirsInChroot) {
if (i.second.source == "/proc") continue; // backwards compatibility
doBind(i.second.source, chrootRootDir + i.first, i.second.optional);
}
Keep track of the exact build start/stop times
2016-12-07 16:09:38 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Bind a new instance of procfs on /proc. */
createDirs(chrootRootDir + "/proc");
if (mount("none", (chrootRootDir + "/proc").c_str(), "proc", 0, 0) == -1)
throw SysError("mounting /proc");
Kill builds when we get EOF on the log FD This closes a long-time bug that allowed builds to hang Nix indefinitely (regardless of timeouts) simply by doing exec > /dev/null 2>&1; while true; do true; done Now, on EOF, we just send SIGKILL to the child to make sure it's really gone.
2017-01-19 16:58:39 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Mount a new tmpfs on /dev/shm to ensure that whatever
the builder puts in /dev/shm is cleaned up automatically. */
if (pathExists("/dev/shm") && mount("none", (chrootRootDir + "/dev/shm").c_str(), "tmpfs", 0,
fmt("size=%s", settings.sandboxShmSize).c_str()) == -1)
throw SysError("mounting /dev/shm");
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Mount a new devpts on /dev/pts. Note that this
requires the kernel to be compiled with
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y (which is the case
if /dev/ptx/ptmx exists). */
if (pathExists("/dev/pts/ptmx") &&
!pathExists(chrootRootDir + "/dev/ptmx")
&& !dirsInChroot.count("/dev/pts"))
{
if (mount("none", (chrootRootDir + "/dev/pts").c_str(), "devpts", 0, "newinstance,mode=0620") == 0)
{
createSymlink("/dev/pts/ptmx", chrootRootDir + "/dev/ptmx");
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Make sure /dev/pts/ptmx is world-writable. With some
Linux versions, it is created with permissions 0. */
chmod_(chrootRootDir + "/dev/pts/ptmx", 0666);
} else {
if (errno != EINVAL)
throw SysError("mounting /dev/pts");
doBind("/dev/pts", chrootRootDir + "/dev/pts");
doBind("/dev/ptmx", chrootRootDir + "/dev/ptmx");
}
}
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Unshare this mount namespace. This is necessary because
pivot_root() below changes the root of the mount
namespace. This means that the call to setns() in
addDependency() would hide the host's filesystem,
making it impossible to bind-mount paths from the host
Nix store into the sandbox. Therefore, we save the
pre-pivot_root namespace in
sandboxMountNamespace. Since we made /nix/store a
shared subtree above, this allows addDependency() to
make paths appear in the sandbox. */
if (unshare(CLONE_NEWNS) == -1)
throw SysError("unsharing mount namespace");
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Do the chroot(). */
if (chdir(chrootRootDir.c_str()) == -1)
throw SysError("cannot change directory to '%1%'", chrootRootDir);
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (mkdir("real-root", 0) == -1)
throw SysError("cannot create real-root directory");
Revive running builds in a PID namespace
2014-12-23 17:25:06 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (pivot_root(".", "real-root") == -1)
throw SysError("cannot pivot old root directory onto '%1%'", (chrootRootDir + "/real-root"));
Support sandbox builds by non-root users This allows an unprivileged user to perform builds on a diverted store (i.e. where the physical store location differs from the logical location). Example: $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \ 'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"' will do a build in the Nix store physically in /tmp/store but logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:45:11 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (chroot(".") == -1)
throw SysError("cannot change root directory to '%1%'", chrootRootDir);
Support sandbox builds by non-root users This allows an unprivileged user to perform builds on a diverted store (i.e. where the physical store location differs from the logical location). Example: $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \ 'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"' will do a build in the Nix store physically in /tmp/store but logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:45:11 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (umount2("real-root", MNT_DETACH) == -1)
throw SysError("cannot unmount real root filesystem");
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (rmdir("real-root") == -1)
throw SysError("cannot remove real-root directory");
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Switch to the sandbox uid/gid in the user namespace,
which corresponds to the build user or calling user in
the parent namespace. */
if (setgid(sandboxGid()) == -1)
throw SysError("setgid failed");
if (setuid(sandboxUid()) == -1)
throw SysError("setuid failed");
Support user namespaces being disabled If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.
2020-10-07 22:02:36 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
setUser = false;
}
#endif
Support user namespaces being disabled If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.
2020-10-07 22:02:36 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (chdir(tmpDirInSandbox.c_str()) == -1)
throw SysError("changing into '%1%'", tmpDir);
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Close all other file descriptors. */
closeMostFDs({STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO});
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __linux__
/* Change the personality to 32-bit if we're doing an
i686-linux build on an x86_64-linux machine. */
struct utsname utsbuf;
uname(&utsbuf);
if (drv->platform == "i686-linux" &&
(settings.thisSystem == "x86_64-linux" ||
(!strcmp(utsbuf.sysname, "Linux") && !strcmp(utsbuf.machine, "x86_64")))) {
if (personality(PER_LINUX32) == -1)
throw SysError("cannot set i686-linux personality");
}
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Impersonate a Linux 2.6 machine to get some determinism in
builds that depend on the kernel version. */
if ((drv->platform == "i686-linux" || drv->platform == "x86_64-linux") && settings.impersonateLinux26) {
int cur = personality(0xffffffff);
if (cur != -1) personality(cur | 0x0020000 /* == UNAME26 */);
}
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Disable address space randomization for improved
determinism. */
int cur = personality(0xffffffff);
if (cur != -1) personality(cur | ADDR_NO_RANDOMIZE);
#endif
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Disable core dumps by default. */
struct rlimit limit = { 0, RLIM_INFINITY };
setrlimit(RLIMIT_CORE, &limit);
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
// FIXME: set other limits to deterministic values?
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Fill in the environment. */
Strings envStrs;
for (auto & i : env)
envStrs.push_back(rewriteStrings(i.first + "=" + i.second, inputRewrites));
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If we are running in `build-users' mode, then switch to the
user we allocated above. Make sure that we drop all root
privileges. Note that above we have closed all file
descriptors except std*, so that's safe. Also note that
setuid() when run as root sets the real, effective and
saved UIDs. */
if (setUser && buildUser) {
/* Preserve supplementary groups of the build user, to allow
admins to specify groups such as "kvm". */
if (!buildUser->getSupplementaryGIDs().empty() &&
setgroups(buildUser->getSupplementaryGIDs().size(),
buildUser->getSupplementaryGIDs().data()) == -1)
throw SysError("cannot set supplementary groups of build user");
Prevent a deadlock when user namespace setup fails Observed on Centos 7 when user namespaces are disabled: DerivationGoal::startBuilder() throws an exception, ~DerivationGoal() waits for the child process to exit, but the child process hangs forever in drainFD(userNamespaceSync.readSide.get()) in DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get through. Issue #4092.
2020-10-06 18:52:46 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (setgid(buildUser->getGID()) == -1 ||
getgid() != buildUser->getGID() ||
getegid() != buildUser->getGID())
throw SysError("setgid failed");
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (setuid(buildUser->getUID()) == -1 ||
getuid() != buildUser->getUID() ||
geteuid() != buildUser->getUID())
throw SysError("setuid failed");
}
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Fill in the arguments. */
Strings args;
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
const char *builder = "invalid";
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (drv->isBuiltin()) {
;
Support user namespaces being disabled If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.
2020-10-07 22:02:36 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#if __APPLE__
else {
/* This has to appear before import statements. */
std::string sandboxProfile = "(version 1)\n";
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (useChroot) {
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Lots and lots and lots of file functions freak out if they can't stat their full ancestry */
PathSet ancestry;
Run builds in a user namespace This way, all builds appear to have a uid/gid of 0 inside the chroot. In the future, this may allow using programs like systemd-nspawn inside builds, but that will require assigning a larger UID/GID map to the build. Issue #625.
2016-06-09 18:27:39 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* We build the ancestry before adding all inputPaths to the store because we know they'll
all have the same parents (the store), and there might be lots of inputs. This isn't
particularly efficient... I doubt it'll be a bottleneck in practice */
for (auto & i : dirsInChroot) {
Path cur = i.first;
while (cur.compare("/") != 0) {
cur = dirOf(cur);
ancestry.insert(cur);
}
}
Check the return code of the clone() call
2012-06-27 15:52:06 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* And we want the store in there regardless of how empty dirsInChroot. We include the innermost
path component this time, since it's typically /nix/store and we care about that. */
Path cur = worker.store.storeDir;
while (cur.compare("/") != 0) {
ancestry.insert(cur);
cur = dirOf(cur);
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Add all our input paths to the chroot */
for (auto & i : inputPaths) {
auto p = worker.store.printStorePath(i);
dirsInChroot[p] = p;
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be configurable */
if (settings.darwinLogSandboxViolations) {
sandboxProfile += "(deny default)\n";
} else {
sandboxProfile += "(deny default (with no-log))\n";
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
sandboxProfile += "(import \"sandbox-defaults.sb\")\n";
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (derivationIsImpure(derivationType))
sandboxProfile += "(import \"sandbox-network.sb\")\n";
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Add the output paths we'll use at build-time to the chroot */
sandboxProfile += "(allow file-read* file-write* process-exec\n";
for (auto & [_, path] : scratchOutputs)
sandboxProfile += fmt("\t(subpath \"%s\")\n", worker.store.printStorePath(path));
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
sandboxProfile += ")\n";
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Our inputs (transitive dependencies and any impurities computed above)
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
without file-write* allowed, access() incorrectly returns EPERM
*/
sandboxProfile += "(allow file-read* file-write* process-exec\n";
for (auto & i : dirsInChroot) {
if (i.first != i.second.source)
throw Error(
"can't map '%1%' to '%2%': mismatched impure paths not supported on Darwin",
i.first, i.second.source);
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
string path = i.first;
struct stat st;
if (lstat(path.c_str(), &st)) {
if (i.second.optional && errno == ENOENT)
continue;
throw SysError("getting attributes of path '%s", path);
}
if (S_ISDIR(st.st_mode))
sandboxProfile += fmt("\t(subpath \"%s\")\n", path);
else
sandboxProfile += fmt("\t(literal \"%s\")\n", path);
}
sandboxProfile += ")\n";
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Allow file-read* on full directory hierarchy to self. Allows realpath() */
sandboxProfile += "(allow file-read*\n";
for (auto & i : ancestry) {
sandboxProfile += fmt("\t(literal \"%s\")\n", i);
}
sandboxProfile += ")\n";
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 19:11:09 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
sandboxProfile += additionalSandboxProfile;
} else
sandboxProfile += "(import \"sandbox-minimal.sb\")\n";
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("Generated sandbox profile:");
debug(sandboxProfile);
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path sandboxFile = tmpDir + "/.sandbox.sb";
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
writeFile(sandboxFile, sandboxProfile);
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
bool allowLocalNetworking = parsedDrv->getBoolAttr("__darwinAllowLocalNetworking");
Fix sandbox fallback settings The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed. Since we don’t know ahead of time here whether sandboxing is enabled, we need to reset all of the env vars we’ve set previously. This fixes the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-13 01:02:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* The tmpDir in scope points at the temporary build directory for our derivation. Some packages try different mechanisms
to find temporary directories, so we want to open up a broader place for them to dump their files, if needed. */
Path globalTmpDir = canonPath(getEnv("TMPDIR").value_or("/tmp"), true);
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* They don't like trailing slashes on subpath directives */
if (globalTmpDir.back() == '/') globalTmpDir.pop_back();
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (getEnv("_NIX_TEST_NO_SANDBOX") != "1") {
builder = "/usr/bin/sandbox-exec";
args.push_back("sandbox-exec");
args.push_back("-f");
args.push_back(sandboxFile);
args.push_back("-D");
args.push_back("_GLOBAL_TMP_DIR=" + globalTmpDir);
args.push_back("-D");
args.push_back("IMPORT_DIR=" + settings.nixDataDir + "/nix/sandbox/");
if (allowLocalNetworking) {
args.push_back("-D");
args.push_back(string("_ALLOW_LOCAL_NETWORKING=1"));
}
args.push_back(drv->builder);
} else {
builder = drv->builder.c_str();
args.push_back(std::string(baseNameOf(drv->builder)));
}
}
#else
else {
builder = drv->builder.c_str();
args.push_back(std::string(baseNameOf(drv->builder)));
}
#endif
Communicate the structured log FD to builders Since we may use a dedicated file descriptor in the future, this allows us to change it. So builders can do if [[ -n $NIX_LOG_FD ]]; then echo "@nix { message... }" >&$NIX_LOG_FD fi
2017-09-08 11:27:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : drv->args)
args.push_back(rewriteStrings(i, inputRewrites));
Set $TERM
2019-05-19 16:56:08 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Indicate that we managed to set up the build environment. */
writeFull(STDERR_FILENO, string("\2\n"));
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Execute the program. This should not return. */
if (drv->isBuiltin()) {
try {
logger = makeJSONLogger(*logger);
Moving more code out of DerivationGoal::startBuilder()
2017-01-25 12:00:28 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
BasicDerivation & drv2(*drv);
for (auto & e : drv2.env)
e.second = rewriteStrings(e.second, inputRewrites);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (drv->builder == "builtin:fetchurl")
builtinFetchurl(drv2, netrcData);
else if (drv->builder == "builtin:buildenv")
builtinBuildenv(drv2);
else if (drv->builder == "builtin:unpack-channel")
builtinUnpackChannel(drv2);
else
throw Error("unsupported builtin function '%1%'", string(drv->builder, 8));
_exit(0);
} catch (std::exception & e) {
writeFull(STDERR_FILENO, e.what() + std::string("\n"));
_exit(1);
}
}
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Use posix_spawn_setbinpref_np to advise which architecture to run When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use. Unfortunately, posix_spawnattr_setbinpref_np does not appear to be inherited by child processes in x86_64 Rosetta 2 translations, meaning that this will not always work as expected. For example: { arm = derivation { name = "test"; system = "aarch64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; rosetta = derivation { name = "test"; system = "x86_64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = i386 ] echo It works! /usr/bin/touch $out '') ]; }; } `arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on aarch64-compiled Nix. I suspect there is a way to fix this since: $ /usr/bin/arch -arch x86_64 /bin/bash \ -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch' arm64 seems to work correctly. We may need to wait for Apple to update system_cmds in opensource.apple.com to find out how though.
2020-12-03 22:41:59 +01:00
#if __APPLE__
posix_spawnattr_t attrp;
if (posix_spawnattr_init(&attrp))
throw SysError("failed to initialize builder");
if (posix_spawnattr_setflags(&attrp, POSIX_SPAWN_SETEXEC))
throw SysError("failed to initialize builder");
if (drv->platform == "aarch64-darwin") {
Set kern.curproc_arch_affinity=0 to escape Rosetta By default, once you enter x86_64 Rosetta 2, macOS will try to run everything in x86_64. So an x86_64 Nix will still try to use x86_64 even when system = aarch64-darwin. To avoid this we can set kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0, we ignore this preference. This is based on how https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html works. Completely undocumented, but seems to work! Note, you can verify this works with this impure Nix expression: ``` { a = derivation { name = "a"; system = "aarch64-darwin"; builder = "/bin/sh"; args = [ "-e" (builtins.toFile "builder" '' [ "$(/usr/bin/arch)" = arm64 ] [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; b = derivation { name = "b"; system = "x86_64-darwin"; builder = "/bin/sh"; args = [ "-e" (builtins.toFile "builder" '' [ "$(/usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ] [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; } ```
2021-01-12 05:05:32 +01:00
// Unset kern.curproc_arch_affinity so we can escape Rosetta
int affinity = 0;
sysctlbyname("kern.curproc_arch_affinity", NULL, NULL, &affinity, sizeof(affinity));
Use posix_spawn_setbinpref_np to advise which architecture to run When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use. Unfortunately, posix_spawnattr_setbinpref_np does not appear to be inherited by child processes in x86_64 Rosetta 2 translations, meaning that this will not always work as expected. For example: { arm = derivation { name = "test"; system = "aarch64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; rosetta = derivation { name = "test"; system = "x86_64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = i386 ] echo It works! /usr/bin/touch $out '') ]; }; } `arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on aarch64-compiled Nix. I suspect there is a way to fix this since: $ /usr/bin/arch -arch x86_64 /bin/bash \ -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch' arm64 seems to work correctly. We may need to wait for Apple to update system_cmds in opensource.apple.com to find out how though.
2020-12-03 22:41:59 +01:00
cpu_type_t cpu = CPU_TYPE_ARM64;
posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL);
} else if (drv->platform == "x86_64-darwin") {
cpu_type_t cpu = CPU_TYPE_X86_64;
posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL);
}
posix_spawn(NULL, builder, NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
#else
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
execve(builder, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data());
Use posix_spawn_setbinpref_np to advise which architecture to run When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use. Unfortunately, posix_spawnattr_setbinpref_np does not appear to be inherited by child processes in x86_64 Rosetta 2 translations, meaning that this will not always work as expected. For example: { arm = derivation { name = "test"; system = "aarch64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = arm64 ] /usr/bin/touch $out '') ]; }; rosetta = derivation { name = "test"; system = "x86_64-darwin"; builder = "/bin/bash"; args = [ "-e" (builtins.toFile "test" '' set -x /usr/sbin/sysctl sysctl.proc_translated /usr/sbin/sysctl sysctl.proc_native [ "$(/usr/bin/arch)" = i386 ] echo It works! /usr/bin/touch $out '') ]; }; } `arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on aarch64-compiled Nix. I suspect there is a way to fix this since: $ /usr/bin/arch -arch x86_64 /bin/bash \ -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch' arm64 seems to work correctly. We may need to wait for Apple to update system_cmds in opensource.apple.com to find out how though.
2020-12-03 22:41:59 +01:00
#endif
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
throw SysError("executing '%1%'", drv->builder);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
} catch (Error & e) {
writeFull(STDERR_FILENO, "\1\n");
FdSink sink(STDERR_FILENO);
sink << e;
sink.flush();
_exit(1);
Squashed get CA derivations building
2020-08-07 21:09:26 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::registerOutputs()
{
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressed derivations this isn't the case.
*/
if (hook) {
bool allValid = true;
for (auto & i : drv->outputsAndOptPaths(worker.store)) {
if (!i.second.second || !worker.store.isValidPath(*i.second.second))
allValid = false;
Store the final drv outputs in memory when building remotely The `DerivationGoal` has a variable storing the “final” derivation output paths that is used (amongst other things) to fill the environment for the post build hook. However this variable wasn't set when the build-hook is used, causing a crash when both hooks are used together. Fix this by setting this variable (from the informations in the db) after a run of the post build hook.
2020-12-09 10:38:04 +01:00
else
finalOutputs.insert_or_assign(i.first, *i.second.second);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (allValid) return;
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
}
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::map<std::string, ValidPathInfo> infos;
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Set of inodes seen during calls to canonicalisePathMetaData()
for this build's outputs. This needs to be shared between
outputs to allow hard links between outputs. */
InodesSeen inodesSeen;
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path checkSuffix = ".check";
bool keepPreviousRound = settings.keepFailed || settings.runDiffHook;
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::exception_ptr delayedException;
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* The paths that can be referenced are the input closures, the
output paths, and any paths that have been built via recursive
Nix calls. */
StorePathSet referenceablePaths;
for (auto & p : inputPaths) referenceablePaths.insert(p);
for (auto & i : scratchOutputs) referenceablePaths.insert(i.second);
for (auto & p : addedPaths) referenceablePaths.insert(p);
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* FIXME `needsHashRewrite` should probably be removed and we get to the
real reason why we aren't using the chroot dir */
auto toRealPathChroot = [&](const Path & p) -> Path {
return useChroot && !needsHashRewrite()
? chrootRootDir + p
: worker.store.toRealPath(p);
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
};
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check whether the output paths were created, and make all
output paths read-only. Then get the references of each output (that we
might need to register), so we can topologically sort them. For the ones
that are most definitely already installed, we just store their final
name so we can also use it in rewrites. */
StringSet outputsToSort;
struct AlreadyRegistered { StorePath path; };
struct PerhapsNeedToRegister { StorePathSet refs; };
std::map<std::string, std::variant<AlreadyRegistered, PerhapsNeedToRegister>> outputReferencesIfUnregistered;
std::map<std::string, struct stat> outputStats;
for (auto & [outputName, _] : drv->outputs) {
auto actualPath = toRealPathChroot(worker.store.printStorePath(scratchOutputs.at(outputName)));
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
outputsToSort.insert(outputName);
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Updated wanted info to remove the outputs we definitely don't need to register */
auto & initialInfo = initialOutputs.at(outputName);
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Don't register if already valid, and not checking */
initialInfo.wanted = buildMode == bmCheck
|| !(initialInfo.known && initialInfo.known->isValid());
if (!initialInfo.wanted) {
outputReferencesIfUnregistered.insert_or_assign(
outputName,
AlreadyRegistered { .path = initialInfo.known->path });
continue;
}
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
struct stat st;
if (lstat(actualPath.c_str(), &st) == -1) {
if (errno == ENOENT)
throw BuildError(
"builder for '%s' failed to produce output path for output '%s' at '%s'",
worker.store.printStorePath(drvPath), outputName, actualPath);
throw SysError("getting attributes of path '%s'", actualPath);
}
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
#ifndef __CYGWIN__
/* Check that the output is not group or world writable, as
that means that someone else can have interfered with the
build. Also, the output should be owned by the build
user. */
if ((!S_ISLNK(st.st_mode) && (st.st_mode & (S_IWGRP | S_IWOTH))) ||
(buildUser && st.st_uid != buildUser->getUID()))
throw BuildError(
"suspicious ownership or permission on '%s' for output '%s'; rejecting this build output",
actualPath, outputName);
#endif
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Canonicalise first. This ensures that the path we're
rewriting doesn't contain a hard link to /etc/shadow or
something like that. */
canonicalisePathMetaData(actualPath, buildUser ? buildUser->getUID() : -1, inodesSeen);
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
debug("scanning for references for output '%s' in temp location '%s'", outputName, actualPath);
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Pass blank Sink as we are not ready to hash data at this stage. */
NullSink blank;
auto references = worker.store.parseStorePathSet(
scanForReferences(blank, actualPath, worker.store.printStorePathSet(referenceablePaths)));
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
outputReferencesIfUnregistered.insert_or_assign(
outputName,
PerhapsNeedToRegister { .refs = references });
outputStats.insert_or_assign(outputName, std::move(st));
Pass lists/attrsets to bash as (associative) arrays
2017-10-25 13:01:50 +02:00
}
Support special attributes in structured attributes derivations E.g. __noChroot and allowedReferences now work correctly. We also now check that the attribute type is correct. For instance, instead of allowedReferences = "out"; you have to write allowedReferences = [ "out" ]; Fixes #2453.
2018-09-28 12:43:01 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto sortedOutputNames = topoSort(outputsToSort,
{[&](const std::string & name) {
return std::visit(overloaded {
/* Since we'll use the already installed versions of these, we
can treat them as leaves and ignore any references they
have. */
[&](AlreadyRegistered _) { return StringSet {}; },
[&](PerhapsNeedToRegister refs) {
StringSet referencedOutputs;
/* FIXME build inverted map up front so no quadratic waste here */
for (auto & r : refs.refs)
for (auto & [o, p] : scratchOutputs)
if (r == p)
referencedOutputs.insert(o);
return referencedOutputs;
},
}, outputReferencesIfUnregistered.at(name));
}},
{[&](const std::string & path, const std::string & parent) {
// TODO with more -vvvv also show the temporary paths for manual inspection.
return BuildError(
"cycle detected in build of '%s' in the references of output '%s' from output '%s'",
worker.store.printStorePath(drvPath), path, parent);
}});
Add support for passing structured data to builders Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-25 16:42:07 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::reverse(sortedOutputNames.begin(), sortedOutputNames.end());
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & outputName : sortedOutputNames) {
auto output = drv->outputs.at(outputName);
auto & scratchPath = scratchOutputs.at(outputName);
auto actualPath = toRealPathChroot(worker.store.printStorePath(scratchPath));
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto finish = [&](StorePath finalStorePath) {
/* Store the final path */
finalOutputs.insert_or_assign(outputName, finalStorePath);
/* The rewrite rule will be used in downstream outputs that refer to
use. This is why the topological sort is essential to do first
before this for loop. */
if (scratchPath != finalStorePath)
outputRewrites[std::string { scratchPath.hashPart() }] = std::string { finalStorePath.hashPart() };
};
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::optional<StorePathSet> referencesOpt = std::visit(overloaded {
[&](AlreadyRegistered skippedFinalPath) -> std::optional<StorePathSet> {
finish(skippedFinalPath.path);
return std::nullopt;
},
[&](PerhapsNeedToRegister r) -> std::optional<StorePathSet> {
return r.refs;
},
}, outputReferencesIfUnregistered.at(outputName));
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!referencesOpt)
continue;
auto references = *referencesOpt;
Recursive Nix support This allows Nix builders to call Nix to build derivations, with some limitations. Example: let nixpkgs = fetchTarball channel:nixos-18.03; in with import <nixpkgs> {}; runCommand "foo" { buildInputs = [ nix jq ]; NIX_PATH = "nixpkgs=${nixpkgs}"; } '' hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })') $hello/bin/hello mkdir -p $out/bin ln -s $hello/bin/hello $out/bin/hello nix path-info -r --json $hello | jq . '' This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its $out, i.e. # ll ./result/bin/ lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello # nix-store -qR ./result /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131 /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5 /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo This is implemented as follows: * Before running the outer builder, Nix creates a Unix domain socket '.nix-socket' in the builder's temporary directory and sets $NIX_REMOTE to point to it. It starts a thread to process connections to this socket. (Thus you don't need to have nix-daemon running.) * The daemon thread uses a wrapper store (RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. for purity, queryPathInfo() won't return impure information such as signatures and timestamps). * After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to $hello. The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10 is forbidden unless /nix/store/kmwd... is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path. Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network. When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a pivot_root() in the builder to change the root directory of its mount namespace, and thus the host /nix/store is not visible in the mount namespace of the builder. To get around this, just before doing pivot_root(), we branch a second mount namespace that shares its /nix/store mountpoint with the parent. Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and in non-root mode (because setns() barfs).
2018-10-02 16:01:26 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto rewriteOutput = [&]() {
/* Apply hash rewriting if necessary. */
if (!outputRewrites.empty()) {
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
warn("rewriting hashes in '%1%'; cross fingers", actualPath);
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 15:15:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* FIXME: this is in-memory. */
StringSink sink;
dumpPath(actualPath, sink);
deletePath(actualPath);
sink.s = make_ref<std::string>(rewriteStrings(*sink.s, outputRewrites));
StringSource source(*sink.s);
restorePath(actualPath, source);
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 15:15:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* FIXME: set proper permissions in restorePath() so
we don't have to do another traversal. */
canonicalisePathMetaData(actualPath, -1, inodesSeen);
}
};
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 15:15:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto rewriteRefs = [&]() -> std::pair<bool, StorePathSet> {
/* In the CA case, we need the rewritten refs to calculate the
final path, therefore we look for a *non-rewritten
self-reference, and use a bool rather try to solve the
computationally intractable fixed point. */
std::pair<bool, StorePathSet> res {
false,
{},
};
for (auto & r : references) {
auto name = r.name();
auto origHash = std::string { r.hashPart() };
if (r == scratchPath)
res.first = true;
else if (outputRewrites.count(origHash) == 0)
res.second.insert(r);
else {
std::string newRef = outputRewrites.at(origHash);
newRef += '-';
newRef += name;
res.second.insert(StorePath { newRef });
}
}
return res;
};
nix-store --repair-path: Rebuild if there is no substituter
2016-01-06 22:07:59 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto newInfoFromCA = [&](const DerivationOutputCAFloating outputHash) -> ValidPathInfo {
auto & st = outputStats.at(outputName);
if (outputHash.method == FileIngestionMethod::Flat) {
/* The output path should be a regular file without execute permission. */
if (!S_ISREG(st.st_mode) || (st.st_mode & S_IXUSR) != 0)
throw BuildError(
"output path '%1%' should be a non-executable regular file "
"since recursive hashing is not enabled (outputHashMode=flat)",
actualPath);
}
rewriteOutput();
/* FIXME optimize and deduplicate with addToStore */
std::string oldHashPart { scratchPath.hashPart() };
HashModuloSink caSink { outputHash.hashType, oldHashPart };
switch (outputHash.method) {
case FileIngestionMethod::Recursive:
dumpPath(actualPath, caSink);
break;
case FileIngestionMethod::Flat:
readFile(actualPath, caSink);
break;
}
auto got = caSink.finish().first;
auto refs = rewriteRefs();
HashModuloSink narSink { htSHA256, oldHashPart };
dumpPath(actualPath, narSink);
auto narHashAndSize = narSink.finish();
ValidPathInfo newInfo0 {
worker.store.makeFixedOutputPath(
outputHash.method,
got,
outputPathName(drv->name, outputName),
refs.second,
refs.first),
narHashAndSize.first,
};
newInfo0.narSize = narHashAndSize.second;
newInfo0.ca = FixedOutputHash {
.method = outputHash.method,
.hash = got,
};
newInfo0.references = refs.second;
if (refs.first)
newInfo0.references.insert(newInfo0.path);
fix the hash rewriting for ca-derivations
2020-11-25 11:20:03 +01:00
if (scratchPath != newInfo0.path) {
// Also rewrite the output path
auto source = sinkToSource([&](Sink & nextSink) {
StringSink sink;
dumpPath(actualPath, sink);
RewritingSink rsink2(oldHashPart, std::string(newInfo0.path.hashPart()), nextSink);
Sink: Use std::string_view
2020-12-02 14:00:43 +01:00
rsink2(*sink.s);
fix the hash rewriting for ca-derivations
2020-11-25 11:20:03 +01:00
rsink2.flush();
});
Path tmpPath = actualPath + ".tmp";
restorePath(tmpPath, *source);
deletePath(actualPath);
movePath(tmpPath, actualPath);
}
Progress indicator: More improvements
2017-08-15 15:31:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
assert(newInfo0.ca);
return newInfo0;
};
Progress indicator: More improvements
2017-08-15 15:31:59 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
ValidPathInfo newInfo = std::visit(overloaded {
[&](DerivationOutputInputAddressed output) {
/* input-addressed case */
auto requiredFinalPath = output.path;
/* Preemptively add rewrite rule for final hash, as that is
what the NAR hash will use rather than normalized-self references */
if (scratchPath != requiredFinalPath)
outputRewrites.insert_or_assign(
std::string { scratchPath.hashPart() },
std::string { requiredFinalPath.hashPart() });
rewriteOutput();
auto narHashAndSize = hashPath(htSHA256, actualPath);
ValidPathInfo newInfo0 { requiredFinalPath, narHashAndSize.first };
newInfo0.narSize = narHashAndSize.second;
auto refs = rewriteRefs();
newInfo0.references = refs.second;
if (refs.first)
newInfo0.references.insert(newInfo0.path);
return newInfo0;
},
[&](DerivationOutputCAFixed dof) {
auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating {
.method = dof.hash.method,
.hashType = dof.hash.hash.type,
});
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 02:29:28 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Check wanted hash */
Hash & wanted = dof.hash.hash;
assert(newInfo0.ca);
auto got = getContentAddressHash(*newInfo0.ca);
if (wanted != got) {
/* Throw an error after registering the path as
valid. */
worker.hashMismatch = true;
delayedException = std::make_exception_ptr(
Alter "wanted:" to "specified:" in hash mismatch output This makes it even clearer which of the two hashes was specified in the nix files. Some may think that "wanted" and "got" is obvious, but: "got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
2020-10-29 00:33:14 +01:00
BuildError("hash mismatch in fixed-output derivation '%s':\n specified: %s\n got: %s",
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.store.printStorePath(drvPath),
wanted.to_string(SRI, true),
got.to_string(SRI, true)));
}
return newInfo0;
},
[&](DerivationOutputCAFloating dof) {
return newInfoFromCA(dof);
},
Allow non-CA derivations to depend on CA derivations
2020-09-23 16:30:42 +02:00
[&](DerivationOutputDeferred) {
// No derivation should reach that point without having been
// rewritten first
assert(false);
// Ugly, but the compiler insists on having this return a value
// of type `ValidPathInfo` despite the `assert(false)`, so
// let's provide it
return *(ValidPathInfo*)0;
},
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}, output.output);
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 15:15:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Calculate where we'll move the output files. In the checking case we
will leave leave them where they are, for now, rather than move to
their usual "final destination" */
auto finalDestPath = worker.store.printStorePath(newInfo.path);
Skip substituters with an incompatible store directory
2016-06-01 16:40:49 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Lock final output path, if not already locked. This happens with
floating CA derivations and hash-mismatching fixed-output
derivations. */
PathLocks dynamicOutputLock;
auto optFixedPath = output.path(worker.store, drv->name, outputName);
if (!optFixedPath ||
worker.store.printStorePath(*optFixedPath) != finalDestPath)
{
assert(newInfo.ca);
dynamicOutputLock.lockPaths({worker.store.toRealPath(finalDestPath)});
Handle queryPathInfo() failure from substituters when fallback is enabled Fixes #1990.
2018-09-07 16:35:48 +02:00
}
Eliminate the substituter mechanism Substitution is now simply a Store -> Store copy operation, most typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Move files, if needed */
if (worker.store.toRealPath(finalDestPath) != actualPath) {
if (buildMode == bmRepair) {
/* Path already exists, need to replace it */
replaceValidPath(worker.store.toRealPath(finalDestPath), actualPath);
actualPath = worker.store.toRealPath(finalDestPath);
} else if (buildMode == bmCheck) {
/* Path already exists, and we want to compare, so we leave out
new path in place. */
} else if (worker.store.isValidPath(newInfo.path)) {
/* Path already exists because CA path produced by something
else. No moving needed. */
assert(newInfo.ca);
} else {
auto destPath = worker.store.toRealPath(finalDestPath);
movePath(actualPath, destPath);
actualPath = destPath;
}
Correctly substitute from different storeDir Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.
2020-06-13 07:07:42 +02:00
}
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 18:08:52 +01:00
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
auto localStoreP = dynamic_cast<LocalStore *>(&worker.store);
if (!localStoreP)
Apply suggestions from code review Thanks! Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 16:21:12 +01:00
throw Unsupported("can only register outputs with local store, but this is %s", worker.store.getUri());
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
auto & localStore = *localStoreP;
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildMode == bmCheck) {
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!worker.store.isValidPath(newInfo.path)) continue;
ValidPathInfo oldInfo(*worker.store.queryPathInfo(newInfo.path));
if (newInfo.narHash != oldInfo.narHash) {
worker.checkMismatch = true;
if (settings.runDiffHook || settings.keepFailed) {
auto dst = worker.store.toRealPath(finalDestPath + checkSuffix);
deletePath(dst);
movePath(actualPath, dst);
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 18:08:52 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
handleDiffHook(
buildUser ? buildUser->getUID() : getuid(),
buildUser ? buildUser->getGID() : getgid(),
finalDestPath, dst, worker.store.printStorePath(drvPath), tmpDir);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
throw NotDeterministic("derivation '%s' may not be deterministic: output '%s' differs from '%s'",
worker.store.printStorePath(drvPath), worker.store.toRealPath(finalDestPath), dst);
} else
throw NotDeterministic("derivation '%s' may not be deterministic: output '%s' differs",
worker.store.printStorePath(drvPath), worker.store.toRealPath(finalDestPath));
}
* Substitutes should occupy a build slot.
2004-06-22 11:00:31 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Since we verified the build, it's now ultimately trusted. */
if (!oldInfo.ultimate) {
oldInfo.ultimate = true;
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
localStore.signPathInfo(oldInfo);
localStore.registerValidPaths({{oldInfo.path, oldInfo}});
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
* Substitutes should occupy a build slot.
2004-06-22 11:00:31 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
continue;
}
* Some more diagnostics changes.
2004-06-22 19:04:10 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* For debugging, print out the referenced and unreferenced paths. */
for (auto & i : inputPaths) {
auto j = references.find(i);
if (j == references.end())
debug("unreferenced input: '%1%'", worker.store.printStorePath(i));
else
debug("referenced input: '%1%'", worker.store.printStorePath(i));
}
* Substitutes should occupy a build slot.
2004-06-22 11:00:31 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (curRound == nrRounds) {
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
localStore.optimisePath(actualPath); // FIXME: combine with scanForReferences()
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.markContentsGood(newInfo.path);
}
Progress indicator: Show number of active items
2017-08-14 22:42:17 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
newInfo.deriver = drvPath;
newInfo.ultimate = true;
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
localStore.signPathInfo(newInfo);
Use vfork() instead of fork() if available Hopefully this reduces the chance of hitting ‘unable to fork: Cannot allocate memory’ errors. vfork() is used for everything except starting builders.
2012-11-09 18:00:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
finish(newInfo.path);
Use vfork() instead of fork() if available Hopefully this reduces the chance of hitting ‘unable to fork: Cannot allocate memory’ errors. vfork() is used for everything except starting builders.
2012-11-09 18:00:33 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If it's a CA path, register it right away. This is necessary if it
isn't statically known so that we can safely unlock the path before
the next iteration */
if (newInfo.ca)
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
localStore.registerValidPaths({{newInfo.path, newInfo}});
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
infos.emplace(outputName, std::move(newInfo));
}
Allow activities to be nested In particular, this allows more relevant activities ("substituting X") to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (buildMode == bmCheck) return;
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Apply output checks. */
checkOutputs(infos);
Let build.cc verify the expected hash of a substituter's output Since SubstitutionGoal::finished() in build.cc computes the hash anyway, we can prevent the inefficiency of computing the hash twice by letting the substituter tell Nix about the expected hash, which can then verify it.
2012-07-27 18:16:02 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Compare the result with the previous round, and report which
path is different, if any.*/
if (curRound > 1 && prevInfos != infos) {
assert(prevInfos.size() == infos.size());
for (auto i = prevInfos.begin(), j = infos.begin(); i != prevInfos.end(); ++i, ++j)
if (!(*i == *j)) {
result.isNonDeterministic = true;
Path prev = worker.store.printStorePath(i->second.path) + checkSuffix;
bool prevExists = keepPreviousRound && pathExists(prev);
hintformat hint = prevExists
? hintfmt("output '%s' of '%s' differs from '%s' from previous round",
worker.store.printStorePath(i->second.path), worker.store.printStorePath(drvPath), prev)
: hintfmt("output '%s' of '%s' differs from previous round",
worker.store.printStorePath(i->second.path), worker.store.printStorePath(drvPath));
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
handleDiffHook(
buildUser ? buildUser->getUID() : getuid(),
buildUser ? buildUser->getGID() : getgid(),
prev, worker.store.printStorePath(i->second.path),
worker.store.printStorePath(drvPath), tmpDir);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (settings.enforceDeterminism)
throw NotDeterministic(hint);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Improve error formatting Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 00:27:36 +01:00
printError(hint);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
curRound = nrRounds; // we know enough, bail out early
}
}
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If this is the first round of several, then move the output out of the way. */
if (nrRounds > 1 && curRound == 1 && curRound < nrRounds && keepPreviousRound) {
for (auto & [_, outputStorePath] : finalOutputs) {
auto path = worker.store.printStorePath(outputStorePath);
Path prev = path + checkSuffix;
deletePath(prev);
Path dst = path + checkSuffix;
if (rename(path.c_str(), dst.c_str()))
throw SysError("renaming '%s' to '%s'", path, dst);
Don't require --fallback to recover from disappeared binary cache NARs
2018-06-05 16:04:41 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (curRound < nrRounds) {
prevInfos = std::move(infos);
* Multiple and/or failing substitutes now work.
2004-06-24 15:40:38 +02:00
return;
}
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 21:17:54 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Remove the .check directories if we're done. FIXME: keep them
if the result was not determistic? */
if (curRound == nrRounds) {
for (auto & [_, outputStorePath] : finalOutputs) {
Path prev = worker.store.printStorePath(outputStorePath) + checkSuffix;
deletePath(prev);
}
Improve substitution progress indicator E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-14 20:14:55 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Register each output path as valid, and register the sets of
paths referenced by each of them. If there are cycles in the
outputs, this will fail. */
{
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
auto localStoreP = dynamic_cast<LocalStore *>(&worker.store);
if (!localStoreP)
Apply suggestions from code review Thanks! Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 16:21:12 +01:00
throw Unsupported("can only register outputs with local store, but this is %s", worker.store.getUri());
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
auto & localStore = *localStoreP;
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
ValidPathInfos infos2;
for (auto & [outputName, newInfo] : infos) {
LocalStore: Get rid of recursive_mutex
2020-11-03 12:52:57 +01:00
infos2.insert_or_assign(newInfo.path, newInfo);
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
localStore.registerValidPaths(infos2);
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
Improve substitution progress indicator E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-14 20:14:55 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* In case of a fixed-output derivation hash mismatch, throw an
exception now that we have registered the output as valid. */
if (delayedException)
std::rethrow_exception(delayedException);
Improve substitution progress indicator E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-14 20:14:55 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* If we made it this far, we are sure the output matches the derivation
(since the delayedException would be a fixed output CA mismatch). That
means it's safe to link the derivation to the output hash. We must do
that for floating CA derivations, which otherwise couldn't be cached,
but it's fine to do in all cases. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Use the hash modulo in the derivation outputs Rather than storing the derivation outputs as `drvPath!outputName` internally, store them as `drvHashModulo!outputName` (or `outputHash!outputName` for fixed-output derivations). This makes the storage slightly more opaque, but enables an earlier cutoff in cases where a fixed-output dependency changes (but keeps the same output hash) − same as what we already do for input-addressed derivations.
2020-12-09 16:56:56 +01:00
if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
auto outputHashes = staticOutputHashes(worker.store, *drv);
Rework the db schema for derivation outputs Add a new table for tracking the derivation output mappings. We used to hijack the `DerivationOutputs` table for that, but (despite its name), it isn't a really good fit: - Its entries depend on the drv being a valid path, making it play badly with garbage collection and preventing us to copy a drv output without copying the whole drv closure too; - It dosen't guaranty that the output path exists; By using a different table, we can experiment with a different schema better suited for tracking the output mappings of CA derivations. (incidentally, this also fixes #4138)
2020-10-20 15:14:02 +02:00
for (auto& [outputName, newInfo] : infos)
worker.store.registerDrvOutput(Realisation{
Use the hash modulo in the derivation outputs Rather than storing the derivation outputs as `drvPath!outputName` internally, store them as `drvHashModulo!outputName` (or `outputHash!outputName` for fixed-output derivations). This makes the storage slightly more opaque, but enables an earlier cutoff in cases where a fixed-output dependency changes (but keeps the same output hash) − same as what we already do for input-addressed derivations.
2020-12-09 16:56:56 +01:00
.id = DrvOutput{outputHashes.at(outputName), outputName},
Rework the db schema for derivation outputs Add a new table for tracking the derivation output mappings. We used to hijack the `DerivationOutputs` table for that, but (despite its name), it isn't a really good fit: - Its entries depend on the drv being a valid path, making it play badly with garbage collection and preventing us to copy a drv output without copying the whole drv closure too; - It dosen't guaranty that the output path exists; By using a different table, we can experiment with a different schema better suited for tracking the output mappings of CA derivations. (incidentally, this also fixes #4138)
2020-10-20 15:14:02 +02:00
.outPath = newInfo.path});
Use the hash modulo in the derivation outputs Rather than storing the derivation outputs as `drvPath!outputName` internally, store them as `drvHashModulo!outputName` (or `outputHash!outputName` for fixed-output derivations). This makes the storage slightly more opaque, but enables an earlier cutoff in cases where a fixed-output dependency changes (but keeps the same output hash) − same as what we already do for input-addressed derivations.
2020-12-09 16:56:56 +01:00
}
* Write build logs to disk again.
2004-06-29 11:41:50 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::checkOutputs(const std::map<Path, ValidPathInfo> & outputs)
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::map<Path, const ValidPathInfo &> outputsByPath;
for (auto & output : outputs)
outputsByPath.emplace(worker.store.printStorePath(output.second.path), output.second);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & output : outputs) {
auto & outputName = output.first;
auto & info = output.second;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
struct Checks
{
bool ignoreSelfRefs = false;
std::optional<uint64_t> maxSize, maxClosureSize;
std::optional<Strings> allowedReferences, allowedRequisites, disallowedReferences, disallowedRequisites;
};
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Compute the closure and closure size of some output. This
is slightly tricky because some of its references (namely
other outputs) may not be valid yet. */
auto getClosure = [&](const StorePath & path)
{
uint64_t closureSize = 0;
StorePathSet pathsDone;
std::queue<StorePath> pathsLeft;
pathsLeft.push(path);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
while (!pathsLeft.empty()) {
auto path = pathsLeft.front();
pathsLeft.pop();
if (!pathsDone.insert(path).second) continue;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto i = outputsByPath.find(worker.store.printStorePath(path));
if (i != outputsByPath.end()) {
closureSize += i->second.narSize;
for (auto & ref : i->second.references)
pathsLeft.push(ref);
} else {
auto info = worker.store.queryPathInfo(path);
closureSize += info->narSize;
for (auto & ref : info->references)
pathsLeft.push(ref);
}
}
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return std::make_pair(std::move(pathsDone), closureSize);
};
* In normaliseFState(), wrap registration of the output paths and the normal form in a single transaction to ensure that if we crash, either everything is registered or nothing is. This is for recoverability: unregistered paths in the store can be deleted arbitrarily, while registered paths can only be deleted by running the garbage collector.
2003-08-01 17:41:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto applyChecks = [&](const Checks & checks)
{
if (checks.maxSize && info.narSize > *checks.maxSize)
throw BuildError("path '%s' is too large at %d bytes; limit is %d bytes",
worker.store.printStorePath(info.path), info.narSize, *checks.maxSize);
Deduplicate basic derivation goals too See comments for security concerns. Also optimize goal creation by not traversing map twice.
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (checks.maxClosureSize) {
uint64_t closureSize = getClosure(info.path).second;
if (closureSize > *checks.maxClosureSize)
throw BuildError("closure of path '%s' is too large at %d bytes; limit is %d bytes",
worker.store.printStorePath(info.path), closureSize, *checks.maxClosureSize);
}
Deduplicate basic derivation goals too See comments for security concerns. Also optimize goal creation by not traversing map twice.
2020-08-22 22:44:47 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto checkRefs = [&](const std::optional<Strings> & value, bool allowed, bool recursive)
{
if (!value) return;
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Parse a list of reference specifiers. Each element must
either be a store path, or the symbolic name of the output
of the derivation (such as `out'). */
StorePathSet spec;
for (auto & i : *value) {
if (worker.store.isStorePath(i))
spec.insert(worker.store.parseStorePath(i));
else if (finalOutputs.count(i))
spec.insert(finalOutputs.at(i));
else throw BuildError("derivation contains an illegal reference specifier '%s'", i);
}
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto used = recursive
? getClosure(info.path).first
: info.references;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (recursive && checks.ignoreSelfRefs)
used.erase(info.path);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePathSet badPaths;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto & i : used)
if (allowed) {
if (!spec.count(i))
badPaths.insert(i);
} else {
if (spec.count(i))
badPaths.insert(i);
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!badPaths.empty()) {
string badPathsStr;
for (auto & i : badPaths) {
badPathsStr += "\n ";
badPathsStr += worker.store.printStorePath(i);
}
throw BuildError("output '%s' is not allowed to refer to the following paths:%s",
worker.store.printStorePath(info.path), badPathsStr);
}
};
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 13:36:17 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
checkRefs(checks.allowedReferences, true, false);
checkRefs(checks.allowedRequisites, true, true);
checkRefs(checks.disallowedReferences, false, false);
checkRefs(checks.disallowedRequisites, false, true);
};
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 13:36:17 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (auto structuredAttrs = parsedDrv->getStructuredAttrs()) {
auto outputChecks = structuredAttrs->find("outputChecks");
if (outputChecks != structuredAttrs->end()) {
auto output = outputChecks->find(outputName);
* Remove lock files after building.
2003-11-21 17:05:19 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (output != outputChecks->end()) {
Checks checks;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto maxSize = output->find("maxSize");
if (maxSize != output->end())
checks.maxSize = maxSize->get<uint64_t>();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 20:05:44 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto maxClosureSize = output->find("maxClosureSize");
if (maxClosureSize != output->end())
checks.maxClosureSize = maxClosureSize->get<uint64_t>();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto get = [&](const std::string & name) -> std::optional<Strings> {
auto i = output->find(name);
if (i != output->end()) {
Strings res;
for (auto j = i->begin(); j != i->end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' of derivation '%s' must be a list of strings", name, worker.store.printStorePath(drvPath));
res.push_back(j->get<std::string>());
}
checks.disallowedRequisites = res;
return res;
}
return {};
};
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
checks.allowedReferences = get("allowedReferences");
checks.allowedRequisites = get("allowedRequisites");
checks.disallowedReferences = get("disallowedReferences");
checks.disallowedRequisites = get("disallowedRequisites");
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
applyChecks(checks);
}
}
} else {
// legacy non-structured-attributes case
Checks checks;
checks.ignoreSelfRefs = true;
checks.allowedReferences = parsedDrv->getStringsAttr("allowedReferences");
checks.allowedRequisites = parsedDrv->getStringsAttr("allowedRequisites");
checks.disallowedReferences = parsedDrv->getStringsAttr("disallowedReferences");
checks.disallowedRequisites = parsedDrv->getStringsAttr("disallowedRequisites");
applyChecks(checks);
}
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
}
* Refactorings.
2003-07-20 21:29:38 +02:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path DerivationGoal::openLogFile()
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
logSize = 0;
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!settings.keepLog) return "";
* Re-enable build hooks.
2004-06-19 23:45:04 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto baseName = std::string(baseNameOf(worker.store.printStorePath(drvPath)));
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
/* Create a log file. */
Crudely make worker only provide a Store, not LocalStore We downcast in a few places, this will be refactored to be better later.
2020-12-20 18:36:52 +01:00
Path logDir;
if (auto localStore = dynamic_cast<LocalStore *>(&worker.store))
logDir = localStore->logDir;
else
logDir = settings.nixLogDir;
Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, string(baseName, 0, 2));
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
createDirs(dir);
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
Path logFileName = fmt("%s/%s%s", dir, string(baseName, 2),
settings.compressLog ? ".bz2" : "");
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 18:24:35 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
fdLogFile = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC | O_CLOEXEC, 0666);
if (!fdLogFile) throw SysError("creating log file '%1%'", logFileName);
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 18:29:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
logFileSink = std::make_shared<FdSink>(fdLogFile.get());
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 18:29:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (settings.compressLog)
logSink = std::shared_ptr<CompressionSink>(makeCompressionSink("bzip2", *logFileSink));
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
else
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
logSink = logFileSink;
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 18:29:57 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return logFileName;
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 13:36:17 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::closeLogFile()
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 02:05:54 +01:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
auto logSink2 = std::dynamic_pointer_cast<CompressionSink>(logSink);
if (logSink2) logSink2->finish();
if (logFileSink) logFileSink->flush();
logSink = logFileSink = 0;
fdLogFile = -1;
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 02:05:54 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::deleteTmpDir(bool force)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (tmpDir != "") {
/* Don't keep temporary directories for builtins because they
might have privileged stuff (like a copy of netrc). */
if (settings.keepFailed && !force && !drv->isBuiltin()) {
printError("note: keeping build directory '%s'", tmpDir);
chmod(tmpDir.c_str(), 0755);
* With `--max-jobs 0', print a nicer error message than "Assertion `!awake.empty()' failed."
2009-03-29 20:06:00 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
else
deletePath(tmpDir);
tmpDir = "";
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
}
* Refactorings.
2003-07-20 21:29:38 +02:00
}
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 16:44:00 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::handleChildOutput(int fd, const string & data)
{
if ((hook && fd == hook->builderOut.readSide.get()) ||
(!hook && fd == builderOut.readSide.get()))
{
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
killChild();
done(
BuildResult::LogLimitExceeded,
Error("%s killed after writing more than %d bytes of log output",
getName(), settings.maxLogSize));
return;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
}
* Refactorings.
2003-07-20 21:29:38 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
for (auto c : data)
if (c == '\r')
currentLogLinePos = 0;
else if (c == '\n')
flushLine();
else {
if (currentLogLinePos >= currentLogLine.size())
currentLogLine.resize(currentLogLinePos + 1);
currentLogLine[currentLogLinePos++] = c;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
}
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 17:19:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (logSink) (*logSink)(data);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
}
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 02:05:54 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (hook && fd == hook->fromHook.readSide.get()) {
for (auto c : data)
if (c == '\n') {
handleJSONLogMessage(currentHookLine, worker.act, hook->activities, true);
currentHookLine.clear();
} else
currentHookLine += c;
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 02:05:54 +01:00
}
* Refactorings.
2003-07-20 21:29:38 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::handleEOF(int fd)
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 17:53:23 +01:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!currentLogLine.empty()) flushLine();
worker.wakeUp(shared_from_this());
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 17:53:23 +01:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 17:33:24 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::flushLine()
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (handleJSONLogMessage(currentLogLine, *act, builderActivities, false))
;
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
else {
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
logTail.push_back(currentLogLine);
if (logTail.size() > settings.logLines) logTail.pop_front();
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
act->result(resBuildLogLine, currentLogLine);
}
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
currentLogLine = "";
currentLogLinePos = 0;
Make LocalStore thread-safe Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
std::map<std::string, std::optional<StorePath>> DerivationGoal::queryPartialDerivationOutputMap()
Call queryMissing() prior to building Without this, substitute info is fetched sequentially, which is superslow. In the old UI (e.g. nix-build), we call printMissing(), which calls queryMissing(), thereby preheating the binary cache cache. But the new UI doesn't do that.
2017-08-31 16:02:36 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!useDerivation || drv->type() != DerivationType::CAFloating) {
std::map<std::string, std::optional<StorePath>> res;
for (auto & [name, output] : drv->outputs)
res.insert_or_assign(name, output.path(worker.store, drv->name, name));
return res;
} else {
return worker.store.queryPartialDerivationOutputMap(drvPath);
}
Call queryMissing() prior to building Without this, substitute info is fetched sequentially, which is superslow. In the old UI (e.g. nix-build), we call printMissing(), which calls queryMissing(), thereby preheating the binary cache cache. But the new UI doesn't do that.
2017-08-31 16:02:36 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
OutputPathMap DerivationGoal::queryDerivationOutputMap()
* `fstateRefs' now works on derive expressions as well. TODO: make this more efficient. * A flag `-n' in 'nix --query' to normalise the argument. Default is not to normalise.
2003-07-29 12:43:12 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (!useDerivation || drv->type() != DerivationType::CAFloating) {
OutputPathMap res;
for (auto & [name, output] : drv->outputsAndOptPaths(worker.store))
res.insert_or_assign(name, *output.second);
return res;
} else {
return worker.store.queryDerivationOutputMap(drvPath);
Make "nix-build -A <derivation>.<output>" do the right thing For example, given a derivation with outputs "out", "man" and "bin": $ nix-build -A pkg produces ./result pointing to the "out" output; $ nix-build -A pkg.man produces ./result-man pointing to the "man" output; $ nix-build -A pkg.all produces ./result, ./result-man and ./result-bin; $ nix-build -A pkg.all -A pkg2 produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
}
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::checkPathValidity()
{
bool checkHash = buildMode == bmRepair;
for (auto & i : queryPartialDerivationOutputMap()) {
InitialOutput info {
.wanted = wantOutput(i.first, wantedOutputs),
};
if (i.second) {
auto outputPath = *i.second;
info.known = {
.path = outputPath,
.status = !worker.store.isValidPath(outputPath)
? PathStatus::Absent
: !checkHash || worker.pathContentsGood(outputPath)
? PathStatus::Valid
: PathStatus::Corrupt,
};
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
}
Register the realisations for unresolved drvs Once a build is done, get back to the original derivation, and register all the newly built outputs for this derivation. This allows Nix to work properly with derivations that don't have all their build inputs available − thus allowing garbage collection and (once it's implemented) binary substitution
2021-01-27 10:03:05 +01:00
if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
Derivation fullDrv = *drv;
if (auto upcasted = dynamic_cast<Derivation *>(drv.get()))
fullDrv = *upcasted;
auto outputHashes = staticOutputHashes(worker.store, fullDrv);
if (auto real = worker.store.queryRealisation(
DrvOutput{outputHashes.at(i.first), i.first})) {
info.known = {
.path = real->outPath,
.status = PathStatus::Valid,
};
}
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
initialOutputs.insert_or_assign(i.first, info);
Print only one error message if a build fails E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:25:35 +02:00
}
* `fstateRefs' now works on derive expressions as well. TODO: make this more efficient. * A flag `-n' in 'nix --query' to normalise the argument. Default is not to normalise.
2003-07-29 12:43:12 +02:00
}
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePath DerivationGoal::makeFallbackPath(std::string_view outputName)
{
return worker.store.makeStorePath(
"rewrite:" + std::string(drvPath.to_string()) + ":name:" + std::string(outputName),
Hash(htSHA256), outputPathName(drv->name, outputName));
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
}
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
StorePath DerivationGoal::makeFallbackPath(const StorePath & path)
* Refactorings.
2003-07-20 21:29:38 +02:00
{
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
return worker.store.makeStorePath(
"rewrite:" + std::string(drvPath.to_string()) + ":" + std::string(path.to_string()),
Hash(htSHA256), path.name());
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 20:09:32 +02:00
Call queryMissing() prior to building Without this, substitute info is fetched sequentially, which is superslow. In the old UI (e.g. nix-build), we call printMissing(), which calls queryMissing(), thereby preheating the binary cache cache. But the new UI doesn't do that.
2017-08-31 16:02:36 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
void DerivationGoal::done(BuildResult::Status status, std::optional<Error> ex)
{
result.status = status;
if (ex)
result.errorMsg = ex->what();
amDone(result.success() ? ecSuccess : ecFailed, ex);
if (result.status == BuildResult::TimedOut)
worker.timedOut = true;
if (result.status == BuildResult::PermanentFailure)
worker.permanentFailure = true;
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
mcExpectedBuilds.reset();
mcRunningBuilds.reset();
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 12:19:27 +01:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
if (result.success()) {
if (status == BuildResult::Built)
worker.doneBuilds++;
} else {
if (status != BuildResult::DependencyFailed)
worker.failedBuilds++;
Print only one error message if a build fails E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
2020-06-15 19:25:35 +02:00
}
Remove trailing whitespace / tabs
2012-07-27 15:59:18 +02:00
Trim build/derivation-goal.cc
2020-10-11 18:19:10 +02:00
worker.updateProgress();
Add operation ‘nix-store --repair-path’ This operation allows fixing corrupted or accidentally deleted store paths by redownloading them using substituters, if available. Since the corrupted path cannot be replaced atomically, there is a very small time window (one system call) during which neither the old (corrupted) nor the new (repaired) contents are available. So repairing should be used with some care on critical packages like Glibc.
2012-10-02 20:08:59 +02:00
}
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 23:06:23 +02:00
}
Reference in a new issue Copy permalink