We should never have used an unprefixed environment variable name.
All other systemd-nspawn variables have the "SYSTEMD_NSPAWN_" prefix,
and all other systemd variables have the "SYSTEMD_" prefix.
The new variable name takes precedence, but we fall back to checking the
old one. If only the old one is found, a warning is emitted.
In addition, SYSTEMD_NSPAWN_UNIFIED_HIERARCHY="" is accepted as an override
to avoid looking for the old variable name.
We have a variable with the same name ($UNIFIED_CGROUP_HIERARCHY) in tests,
which governs both systemd-nspawn and qemu behaviour. It is not renamed.
This avoids unnecessary noise in the stderr logs which dd always produces,
such as:
0+0 records in
0+0 records out
0 bytes copied, 0.000155284 s, 0.0 kB/s
Using truncate should not result in any functional change; the image will
still be created as a sparse file of the size specified.
In Ubuntu CI, we test binaries from the installed system, not from
$BUILD_DIR, so use the appropriate binary. Most of the calls to the
binaries are part of checking/processing asan-built binaries, and so
did not apply to Ubuntu CI, except for generating noise in the stderr
log like:
objdump: '/tmp/autopkgtest.83yGoI/build.fHB/src/test/TEST-01-BASIC/systemd-journald': No such file
However this also applies to the call to systemd-nspawn, which the debian
upstream test wrapper was sed-adjusting to use the installed binary
instead of the binary in $BUILD_DIR. This commit allows removing that
sed processing of the test-functions file during Ubuntu CI test.
This dir is created by create_empty_image_rootdir, as well as indirectly
by some other functions, but it should be created by import_initdir so
the newly-exported $initdir exists and can be used immediately without
relying on other functions to create it.
Only umount it during cleanup if the $TESTDIR/root dir is a mountpoint.
This avoids adding noise to the stderr log such as:
mountpoint: /var/tmp/systemd-test.waLOFT/root: No such file or directory
To make debugging much easier, especially for crashes in tests under
QEMU, let's store the entire coredump bundle in the systemd journal,
which is usually kept around by various CIs. Right now, we usually end
up with a journal, but without the coredump itself, which is pretty
useless.
without using -type f, the logs print an error such as:
E: E: modprobe: FATAL: Module asymmetric_keys not found in directory /lib/modules/4.15.0-54-generic
while this doesn't appear to cause problems, it can be extremely
distracting when trying to debug real failures.
Almost all tests were manually mounting/unmounting $TESTDIR/root
from the loopback image; this moves all that into test-functions
so the test setup functions are simplier.
Also add test_setup_cleanup() function, to cleanup what is mounted
by create_empty_image_rootdir()
In certain cases we might attempt to install a binary which is already
present in the test image, yet it's missing from the host system.
In such cases, let's check if the binary indeed exists in the image
before doing any other chcecks. If it does, immediately return with
success.
This was discovered during installation of
/usr/lib/systemd/systemd-bless-boot, which was not present in Ubuntu CI
(as the installed systemd was from the Ubuntu repositories), and the
binary itself was already in the image thanks to `ninja install`.
However, during extraction of binaries from the systemd service files,
another attempt to install this binary was made, which failed due to
`find_binary` being unable to find it.
Running integration tests with ASan is somewhat tricky to begin with, as
we need to pre-load the ASan runtime DSO for certain services (like
dbus), otherwise they won't start or behave as expected. In case of gcc
this is pretty easy, as we need the runtime DSO during compilation, so
it's already present on the host system. For clang things get more
complicated, as ASan is compiled in statically by default, thus to
enable the necessary dynamic-ish behavior one needs to compile with
-shared-libasan and then correctly set LD_PRELOAD_PATH, as the runtime
libraries are not in a standard library path.
Certain distributions (e.g. Arch Linux) require booting with initrd, as
they lack support for commonly used filesystems in the kernel (i.e. the
support is compiled in as modules)
The `mount` utility has an unexpected behavior when run with libasan,
causing false-positives during the integration testing.
For example, on Arch Linux with LD_PRELOAD pointing to libasan:
```
bash-5.0# mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
1
```
However:
```
bash-5.0# LD_PRELOAD= mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
0
```
Further investigation with strace shows a LeakSanitizer error:
```
bash-5.0# strace -s 512 mount -o remount,rw -v /
...
write(2, "==355==LeakSanitizer has encountered a fatal error.\n", 52) = -1 EBADF (Bad file descriptor)
write(2, "ReportFile::Write() can't output requested buffer!\n", 51) = -1 EBADF (Bad file descriptor)
exit_group(1) = ?
+++ exited with 1 +++
```
Let's workaround this by clearing the LD_PRELOAD variable for
systemd-remount-fs.service
We had all kinds of indentation: 2 sp, 3 sp, 4 sp, 8 sp, and mixed.
4 sp was the most common, in particular the majority of scripts under test/
used that. Let's standarize on 4 sp, because many commandlines are long and
there's a lot of nesting, and with 8sp indentation less stuff fits. 4 sp
also seems to be the default indentation, so this will make it less likely
that people will mess up if they don't load the editor config. (I think people
often use vi, and vi has no support to load project-wide configuration
automatically. We distribute a .vimrc file, but it is not loaded by default,
and even the instructions in it seem to discourage its use for security
reasons.)
Also remove the few vim config lines that were left. We should either have them
on all files, or none.
Also remove some strange stuff like '#!/bin/env bash', yikes.
By default the run_qemu() function enables KVM automatically
if it detects the /dev/kvm char device and if the machine is not
already a KVM one. Let's add a TEST_NO_KVM env variable to suppress
this detection.
Make the interactive debugging of (particularly QEMU) machines less
painful, by replacing the default vt220 TERM with linux one, and
by not shutting down the machine after running the test itself.
We'd install the service file, and then dbus-broker-launcher because it is
mentioned in ExecStart=, but not the main executable, so nothing would work.
Let's just install dbus-broker executables if found. They are small, so this
doesn't matter much, and is much easier than figuring the exact conditions
under which dbus-broker will be used instead of dbus-daemon.
Since systemd 206 the combination of systemd and mkinitcpio
causes, under certain conditions, the rootfs to be double fsck'd.
Symptoms:
```
:: performing fsck on '/dev/sda1'
systemd: clean, 3523/125488 files, 141738/501760 blocks
********************** WARNING **********************
* *
* The root device is not configured to be mounted *
* read-write! It may be fsck'd again later. *
* *
*****************************************************
<snip>
[ OK ] Started File System Check on Root Device
```
This occurs when neither 'ro' or 'rw', or only 'ro' is present
on the kernel command line. The solution is to mount the roofs
as read-write on the kernel command line, so systemd knows to not fsck
it again.
If the QEMU_SMP value has not been explicitly set, try to determine it
from the number of online CPUs using the nproc utility. If this approach
fails, fall back to the default value QEMU_SMP=1.
This change should significantly help when running integration tests
under QEMU on multicore systems.
Currently /boot/initramfs-linux.img is used as the default initrd for ArchLinux.
Although, since the kernel modules that are not necessary for the host environment are removed from
initramfs-linux.img by mkinitcpio 's autodetect hook, the kernel modules necessary for qemu may be missing.
(ata_piix, ext4, and so on in my case.)
As a result, the test environment may not be built properly and the test will be failed.
initramfs-linux-fallback.img will skip this autodetect hook, so the test will run successfully in more
environments.
Both initramfs-linux.img and initramfs-linux-fallback.img are generated by default.
rescue.service pulls in /bin/plymouth, which doesn't exist on some
distributions (e.g. Arch Linux). Let's mark it as optional, as it's not
even required by the referencing unit and causes unwanted fails in the
integration testsuite.
