Certain distributions (e.g. Arch Linux) require booting with initrd, as
they lack support for commonly used filesystems in the kernel (i.e. the
support is compiled in as modules)
The `mount` utility has an unexpected behavior when run with libasan,
causing false-positives during the integration testing.
For example, on Arch Linux with LD_PRELOAD pointing to libasan:
```
bash-5.0# mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
1
```
However:
```
bash-5.0# LD_PRELOAD= mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
0
```
Further investigation with strace shows a LeakSanitizer error:
```
bash-5.0# strace -s 512 mount -o remount,rw -v /
...
write(2, "==355==LeakSanitizer has encountered a fatal error.\n", 52) = -1 EBADF (Bad file descriptor)
write(2, "ReportFile::Write() can't output requested buffer!\n", 51) = -1 EBADF (Bad file descriptor)
exit_group(1) = ?
+++ exited with 1 +++
```
Let's workaround this by clearing the LD_PRELOAD variable for
systemd-remount-fs.service
We had all kinds of indentation: 2 sp, 3 sp, 4 sp, 8 sp, and mixed.
4 sp was the most common, in particular the majority of scripts under test/
used that. Let's standarize on 4 sp, because many commandlines are long and
there's a lot of nesting, and with 8sp indentation less stuff fits. 4 sp
also seems to be the default indentation, so this will make it less likely
that people will mess up if they don't load the editor config. (I think people
often use vi, and vi has no support to load project-wide configuration
automatically. We distribute a .vimrc file, but it is not loaded by default,
and even the instructions in it seem to discourage its use for security
reasons.)
Also remove the few vim config lines that were left. We should either have them
on all files, or none.
Also remove some strange stuff like '#!/bin/env bash', yikes.
By default the run_qemu() function enables KVM automatically
if it detects the /dev/kvm char device and if the machine is not
already a KVM one. Let's add a TEST_NO_KVM env variable to suppress
this detection.
Make the interactive debugging of (particularly QEMU) machines less
painful, by replacing the default vt220 TERM with linux one, and
by not shutting down the machine after running the test itself.
We'd install the service file, and then dbus-broker-launcher because it is
mentioned in ExecStart=, but not the main executable, so nothing would work.
Let's just install dbus-broker executables if found. They are small, so this
doesn't matter much, and is much easier than figuring the exact conditions
under which dbus-broker will be used instead of dbus-daemon.
Since systemd 206 the combination of systemd and mkinitcpio
causes, under certain conditions, the rootfs to be double fsck'd.
Symptoms:
```
:: performing fsck on '/dev/sda1'
systemd: clean, 3523/125488 files, 141738/501760 blocks
********************** WARNING **********************
* *
* The root device is not configured to be mounted *
* read-write! It may be fsck'd again later. *
* *
*****************************************************
<snip>
[ OK ] Started File System Check on Root Device
```
This occurs when neither 'ro' or 'rw', or only 'ro' is present
on the kernel command line. The solution is to mount the roofs
as read-write on the kernel command line, so systemd knows to not fsck
it again.
If the QEMU_SMP value has not been explicitly set, try to determine it
from the number of online CPUs using the nproc utility. If this approach
fails, fall back to the default value QEMU_SMP=1.
This change should significantly help when running integration tests
under QEMU on multicore systems.
Currently /boot/initramfs-linux.img is used as the default initrd for ArchLinux.
Although, since the kernel modules that are not necessary for the host environment are removed from
initramfs-linux.img by mkinitcpio 's autodetect hook, the kernel modules necessary for qemu may be missing.
(ata_piix, ext4, and so on in my case.)
As a result, the test environment may not be built properly and the test will be failed.
initramfs-linux-fallback.img will skip this autodetect hook, so the test will run successfully in more
environments.
Both initramfs-linux.img and initramfs-linux-fallback.img are generated by default.
rescue.service pulls in /bin/plymouth, which doesn't exist on some
distributions (e.g. Arch Linux). Let's mark it as optional, as it's not
even required by the referencing unit and causes unwanted fails in the
integration testsuite.
Fedora Rawhide renamed dbus.service to dbus-daemon.service - that
breaks tests which require working DBus (e.g. TEST-03-JOBS)
Excerpt from the dbus.spec:
The 'dbus' package is only retained for compatibility purposes. It will
eventually be removed and then replaced by 'Provides: dbus' in the
dbus-daemon package. It will then exclusively be used for other packages to
describe their dependency on a system and user bus. It does not pull in any
particular dbus *implementation*, nor any libraries. These should be pulled
in, if required, via explicit dependencies.
When parsing and installing binaries mentioned in Exec*= lines the
5ed0dcf4d5 commit added parsing logic to drop
prefixes, including handling duplicate exclamation marks. But this did not
handle arbitrary combination of multiple prefixes, ie. StartExec=+-/bin/sh was
parsed as -/bin/sh which then would fail to install.
Instead of using egrep and shell replacements, replace both with sed command
that does it all. This sed script extract a group of characters starting with a
/ up to the first space (if any) after the equals sign. This correctly handles
existing non-prefixed, prefixed, multiple-prefixed commands.
About half commands seem to repeat themself, thus sort -u cuts the list of
binaries to install about in half.
To validate change of behaviour both old and new functions were modified to
echo parsed binaries into separate files, and then diffed. The incorrect
-/bin/sh was missing in the new output.
Without this patch tests fail on default Ubuntu installs.
Nested KVM is very flaky as we learnt from our CI. Hence, let's avoid
KVM whenever we detect we are already running inside of KVM.
Maybe one day nested KVM is fixed, at which point we can turn this on
again, but for now let's simply avoid nested KVM, since reliable CI is
more important than quick CI, I guess.
And yes, avoiding KVM for our qemu runs does make things substantially
slower, but I think it's not a complete loss.
Inspired by @evverx' findings in:
https://github.com/systemd/systemd/pull/8701#issuecomment-380213302
This makes the script wait for the newly created partition to
show up before trying to put a filesystem on it, which should
prevent the tests from failing with the following error:
```
New situation:
Disklabel type: dos
Disk identifier: 0x3541a0ec
Device Boot Start End Sectors Size Id Type
/dev/loop6p1 2048 800767 798720 390M 83 Linux
/dev/loop6p2 800768 819199 18432 9M 83 Linux
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
The file /dev/loop6p1 does not exist and no size was specified.
make: *** [setup] Error 1
F: Failed to mkfs -t ext4
Makefile:4: recipe for target 'setup' failed
```
Let's be more restrictive when validating PID files and MAINPID=
messages: don't accept PIDs that make no sense, and if the configuration
source is not trusted, don't accept out-of-cgroup PIDs. A configuratin
source is considered trusted when the PID file is owned by root, or the
message was received from root.
This should lock things down a bit, in case service authors write out
PID files from unprivileged code or use NotifyAccess=all with
unprivileged code. Note that doing so was always problematic, just now
it's a bit less problematic.
When we open the PID file we'll now use the CHASE_SAFE chase_symlinks()
logic, to ensure that we won't follow an unpriviled-owned symlink to a
privileged-owned file thinking this was a valid privileged PID file,
even though it really isn't.
Fixes: #6632
This catches errors like "ninja not found", missing programs etc. early,
instead of silently ignoring them and trying to boot a broken VM.
In install_config_files(), allow some distro specific files to be absent
(such as /etc/sysconfig/init).
All test/TEST* but TEST-02-CRYPTSETUP share the same check_result_qemu()
and test_cleanup(), so move them into test_functions and only override
them in TEST-02-CRYPTSETUP.
Also provide a common test_run() which by default assumes that both QEMU
and nspawn tests are run. Particular tests which don't support either
need to explicitly opt out by setting $TEST_NO_{QEMU,NSPAWN}. Do it this
way around to avoid accidentally forgetting to opt in, and to encourage
test authors to at least always support nspawn.
Automatic rebuilding is removed: it doesn't play well with ninja, because
ninja always writes logs, and even if nothing needs to be built, it will
make the log file owned by root. So let's just remove this, and say that
the user must always do the build first.
I'm also keeping make for the tests, because ninja doesn't play well with
sudo.
Since the build directory is arbitrary, it needs to be specified, e.g.
sudo make BUILD_DIR=/home/zbyszek/src/systemd/build1 -C test/TEST-01-BASIC/
If run_qemu() exits with non-zero, this either meant that QEMU was not
available (which should be a SKIP) or that QEMU timed out if $QEMU_TIMEOUT was
set (which then should be a FAIL).
Limit the exit code of run_qemu() to QEMU availability only, and track timeouts
separately through the new $TIMED_OUT variable, which is then checked in
check_result_qemu().
Do the same for $NSPAWN_TIMEOUT and run_nspawn() so that nspawn and QEMU work
similarly.
There are many cgroups-related changes (thanks, @htejun!)
This commit will simplify testing a bit.
Use:
make run UNIFIED_CGROUP_HIERARCHY=yes to enable cgroup-v2
make run UNIFIED_CGROUP_HIERARCHY=no to enable cgroup-v1
systemd-udev generated an insane amount of log output at debug level.
It would break TEST-02-CRYPTSETUP by filling the overflowing the disk
(which seems to be a bug in itself!).
WARNING: Image format was not specified for
'/var/tmp/systemd-test.tGi3od/rootdisk.img' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write
operations on block 0 will be restricted. Specify the 'raw' format
explicitly to remove the restrictions.
Also use unsafe caching mode, we don't care about data integrity here.
Fixes:
$ cd test/TEST-07-ISSUE-1981/
$ sudo make clean setup run
...
timeout: failed to run command ‘systemd-nspawn’: No such file or directory
...
TEST RUN: https://github.com/systemd/systemd/issues/1981 [FAILED]
Makefile:10: recipe for target 'run' failed
make: *** [run] Error 1
* Use $ROOTLIBDIR/systemd always
* Don't pass $ROOTLIBDIR/systemd as the first argument:
$ cat /proc/1/cmdline
/lib/systemd/systemd/lib/systemd/systemd...
Fixes backward/forward incompatibility errors on spawning.
For example:
$ sudo make run
...
Failed to register machine: Cannot set property TasksMax, or unknown
property.
$ ../../systemd-nspawn --version
systemd 228
$ systemd-nspawn --version
systemd 225
Sempaphore containers are not booted with systemd, so machined is not
available, which makes nspawn bail. Just skip nspawn tests in such
environments.
[ -d /run/systemd/system ] is esentially what sd_booted(3) is doing,
but on Ubuntu 15.05, without 'systemd-container' installed, we also
need to check for the presence of the systemd-machined binary.
Fixes:
systemd-testsuite systemd[34]: PAM _pam_load_conf_file: unable to open /etc/pam.d/system-auth
systemd-testsuite systemd[34]: PAM _pam_load_conf_file: unable to open /etc/pam.d/system-auth
systemd-testsuite systemd[34]: user@0.service: Failed at step PAM spawning /lib/systemd/systemd: Operation not permitted
...
on Debian, Ubuntu
Fixes:
systemd-testsuite login[31]: cannot open login definitions /etc/login.defs [No such file or directory]
systemd-testsuite systemd[1]: Received SIGCHLD from PID 31 (login).
systemd-testsuite systemd[1]: Child 31 (login) died (code=exited, status=1/FAILURE)
systemd-testsuite systemd[1]: console-getty.service: Child 31 belongs to console-getty.service
systemd-testsuite systemd[1]: console-getty.service: Main process exited, code=exited, status=1/FAILURE
systemd-testsuite systemd[1]: console-getty.service: Changed running -> dead
on Debian/Ubuntu
Useful on other distros
libpam_modules installs modules into /lib/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/security
on Debian
Fixes:
systemd-testsuite login[36]: PAM unable to dlopen(pam_group.so): /lib/security/pam_group.so: cannot open shared object file: No such file or directory
systemd-testsuite login[36]: PAM adding faulty module: pam_group.so
systemd-testsuite login[36]: PAM unable to dlopen(pam_limits.so): /lib/security/pam_limits.so: cannot open shared object file: No such file or directory
systemd-testsuite login[36]: PAM adding faulty module: pam_limits.so
...
etc
Fixes:
systemd[39]: systemd-exit.service: Executing: /bin/kill -s 58 29
systemd[39]: systemd-exit.service: Failed at step EXEC spawning /bin/kill: No such file or directory
systemd[29]: Received SIGCHLD from PID 39 ((kill)).
systemd[29]: Child 39 ((kill)) died (code=exited, status=203/EXEC)
* remove journal flushing (systemd-journal-flush.service runs journalctl --flush on boot)
* use sh -c and PATH instead of @SYSTEMCTL@ expansion
* remove unnecessary semicolons etc
The basic setup for the well-known system and session buses is
now done in read-only files in ${datadir} (normally /usr/share).
See the NEWS entry for 1.9.18 for details.
http://cgit.freedesktop.org/dbus/dbus/tree/NEWS
Prefer asking pkg-config for the rootlibdir, to also support systems with
--enable-split-usr. Fall back to the hardcoded /usr/lib/systemd if that fails.
It tries to find a suitable QEMU binary and will use KVM if present.
We can now configure QEMU from outside with 4 variables :
- $QEMU_BIN : path to QEMU's binary
- $KERNEL_APPEND : arguments appended to kernel cmdline
- $KERNEL_BIN : path to a kernel
Default /boot/vmlinuz-$KERNEL_VER
- $INITRD : path to an initramfs
Default /boot/initramfs-${KERNEL_VER}.img
- $QEMU_SMP : number of CPU simulated by QEMU.
Default 1
(from Alexander Graf's script: http://www.spinics.net/lists/kvm/msg72389.html)
$ cd test
$ sudo make check
will run all tests in the TEST-* subdirectories
$ cd test/TEST-01-BASIC
$ sudo make clean setup run
will run the different stages of the test for debugging purposes
Frantisek Sumsal