Stochastic Fairness Queueing is a classless queueing discipline.
SFQ does not shape traffic but only schedules the transmission of packets, based on 'flows'.
The goal is to ensure fairness so that each flow is able to send data in turn,
thus preventing any single flow from drowning out the rest.
Meson appears to set the rpath only for some binaries it builds, but not
all. (The rules are not clear to me, but that's besides the point of
this commit).
Let's make sure if our test script operates on a binary that has no
rpath set we fall back preferably to the BUILD_DIR rather than directly
to the host.
This matters if a test uses a libsystemd symbol introduced in a version
newer than the one on the host. In that case "ldd" will not work on the
test binary if rpath is not set. With this fix that behaviour is
corrected, and "ldd" works correctly even in this case.
(Or in other words: before this fix on binaries lacking rpath we'd base
dependency info on the libraries of the host, not the buidl tree, if
they exist in both.)
In some containers unshare() is made unavailable entirely. Let's deal
with this that more gracefully and disable our sandboxing of services
then, so that we work in a container, under the assumption the container
manager is then responsible for sandboxing if we can't do it ourselves.
Previously, we'd insist on sandboxing as soon as any form of BindPath=
is used. With this change we only insist on it if we have a setting like
that where source and destination differ, i.e. there's a mapping
established that actually rearranges things, and thus would result in
systematically different behaviour if skipped (as opposed to mappings
that just make stuff read-only/writable that otherwise arent').
(Let's also update a test that intended to test for this behaviour with
a more specific configuration that still triggers the behaviour with
this change in place)
Fixes: #13955
(For testing purposes unshare() can easily be blocked with
systemd-nspawn --system-call-filter=~unshare.)
This sould make our test suite a bit more robust if it is slow running.
A few of our test services use StandardOutput=tty or StandardError=tty
in the tests in order to connect test services to the container console.
This gets into conflict with the container getty which wants exclusive
access to the console. Since the container getty is started with
Type=idle it typically gets started after a timeout only if the TTY is
already used, which hence introduces a race: if the test finishes
earlier all is good, if not, then the test gets kicked off the TTY which
then causes bash to abort since it cannot write any error messages
anymore.
Let's fix this hence: all tests that connect to the tty are now
synchronized to getty-pre.target, so they finish before any getty is
started.
We currently use the host's kernel and initramfs in our QEMU tests.
If the host is running on an encrypted LUKS partition, then the initramfs
will have a crypttab setup looking for the particular root disk it needs to
encrypt before booting into the system.
However, this disk obviously doesn't exist in our QEMU VM, so it turns out
our tests end up waiting for this device to become available, which will
never actually happen, and boot hangs for 90s until that service times out.
[*** ] A start job is running for /dev/disk/by-uuid/01234567-abcd-1234-abcd-0123456789ab (20s / 1min 30s)
In order to prevent this issue, let's pass "rd.luks=0" to disable LUKS in
the initramfs only as part of our default kernel command-line in our QEMU
tests.
This is enough to disable this behavior and prevent the timeout, while at
the same time doesn't conflict with our tests that actually check for LUKS
behavior in the systemd running under test (such as TEST-02-CRYPTSETUP).
Tested: `sudo make -C TEST-02-CRYPTSETUP/ clean setup run`
Discussed in #13743, the -.service semantic conflicts with the
existing root mount and slice names, making this feature not
uniformly extensible to all types. Change the name to be
<type>.d instead.
Updating to this format also extends the top-level dropin to
unit types.
journald assumes that getsockopt(SO_PEERCRED) correctly identifies the
process on the remote end of the socket. However, this is incorrect
according to man 7 socket:
The returned credentials are those that were in effect at the
time of the call to connect(2) or socketpair(2).
This becomes a problem when a new process inherits the stdout stream
from a parent. First, log messages from the child process will
be attributed to the parent. Second, the struct ucred used by journald
becomes invalid as soon as the parent exits. Further sendmsg calls then
fail with ENOENT. Logs for the child process then vanish from the journal.
Fix this by using recvmsg on the stdout stream, and refreshing the cached
struct ucred if SCM_CREDENTIALS indicate a new process.
Fixes#13708
The systemd-networkd-tests.py has some regex that uses non-capturing
groups, but there is no need to use that with assertRegex; the
groups aren't referenced so it doesn't matter if it's capturing or
non-capturing. However, there are a few places where optional groups
should have been used instead, so this changes that.
Specifically, groups like this:
(?:whatever |)
should actually be:
(whatever )?
Additionally, this is specifically needed for these tests to run on
Debian systems, because this assertRegex:
'Link File: (?:/usr)/lib/systemd/network/99-default.link'
needs to be:
'Link File: (/usr)?/lib/systemd/network/99-default.link'
The name with plural made more sense where multiple options could be specified
in one line. After changes in the pull request, this option only accepts one
value, so from users' POV it should be singular.
(The field in the data structure remains plural, because it actually stores
multiple values.)
Just as `RuntimeMaxSec=` is supported for service units, add support for
it to scope units. This will gracefully kill a scope after the timeout
expires from the moment the scope enters the running state.
This could be used for time-limited login sessions, for example.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Fixes: #12035
TEST-41 verifies that the StartLimitBurst property will correctly
limit the number of unit restarts, but the test currently doesn't
adjust the StartLimitIntervalSec which defaults to 10 seconds.
On Ubuntu CI, running under un-accelerated qemu, it can take more than
10 seconds to perform all 3 restarts, which avoids the burst limit,
and fails the test.
Instead, specify a long StartLimitIntervalSec in the test, so we can
be sure to correctly test StartLimitBurst even on slow testbeds.
Fixes#13794.
Many tests were also masking systemd-machined.service. But machined
should only start when activated, so having it not masked shouldn't be
noticable. TEST-25-IMPORT needs it.
I *think* this was originally added to make it easier to see what was happening
in tests. Later we added the functionality to print the journal on failure, so
this redirection has stopped being useful.
In https://github.com/systemd/systemd/pull/13719#issuecomment-539292650
@filbranden shows that grep tries to write to stdout and fails. In general,
we should not assume that writing to the console it always possible. We have
special code to handle this in pid1 after all:
99 19:22:10.731965 fstat(1, <unfinished ...>
99 19:22:10.731993 <... fstat resumed>{st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
99 19:22:10.732070 write(1, "ExecStartPost={ path=/bin/echo ; argv[]=/bin/echo ${4_four_ex} ; ignore_errors=no ; start_time=[Mon 2019-10-07 19:22:10 PDT] ; stop_time=[Mon 209-10-07 19:22:10 PDT] ; pid=97 ; code=exited ; status=0 }\n", 203) = -1 EIO (Input/output error)
99 19:22:10.732174 write(2, "grep: ", 6) = -1 EIO (Input/output error)
99 19:22:10.732226 write(2, "write error", 11) = -1 EIO (Input/output error)
99 19:22:10.732263 write(2, ": Input/output error", 20) = -1 EIO (Input/output error)
99 19:22:10.732298 write(2, "\n", 1 <unfinished ...>
99 19:22:10.732325 <... write resumed>) = -1 EIO (Input/output error)
99 19:22:10.732349 exit_group(2) = ?
99 19:22:10.732424 +++ exited with 2 +++
Removing the redirection should make the tests less flakey.
Replaces #13719.
While at it, also drop NotifyAccess=all. I think it was added purposefully in
TEST-20-MAINPIDGAMES, and then cargo culted to newer tests.
This temporarily blacklists some tests when run under Ubuntu CI.
This is the upstream side of the Debian 'upstream' test MR:
https://salsa.debian.org/systemd-team/systemd/merge_requests/52
The tests blacklisted here should only be temporarily blacklisted
until they can be fixed; the intention is that these blacklist files
will be added and removed over time while debugging/fixing flaky
and/or regressed tests, without causing test failure noise for other
PRs.
v2:
- if RestartKillSignal= is not specified, fall back to KillSignal=. This is necessary
to preserve backwards compatibility (and keep KillSignal= generally useful).
We should never have used an unprefixed environment variable name.
All other systemd-nspawn variables have the "SYSTEMD_NSPAWN_" prefix,
and all other systemd variables have the "SYSTEMD_" prefix.
The new variable name takes precedence, but we fall back to checking the
old one. If only the old one is found, a warning is emitted.
In addition, SYSTEMD_NSPAWN_UNIFIED_HIERARCHY="" is accepted as an override
to avoid looking for the old variable name.
We have a variable with the same name ($UNIFIED_CGROUP_HIERARCHY) in tests,
which governs both systemd-nspawn and qemu behaviour. It is not renamed.
IPServiceType set to CS6 (network control) causes problems on some old
network setups that continue to interpret the field as IP TOS.
Make DHCP work on such networks by allowing this field to be set to
CS4 (Realtime) instead, as this maps to IPTOS_LOWDELAY.
Signed-off-by: Siddharth Chandrasekaran <csiddharth@vmware.com>
The "Ex" variant was originally only added for ExecStartXYZ= but it makes
sense to have feature parity for the rest of the exec command properties
as well (e.g. ExecReload=, ExecStop=, etc).
When nspawn container with private network starts, networkd creates
the default route for the interface. The route may cause problem on
the host side, and it can be created with DefaultRouteOnDevice= now.
Hence, this makes networkd not create the route implicitly any more.
Closes#13418.
Add a fido_id program meant to be run for devices in the hidraw
subsystem via an IMPORT directive. The program parses the HID report
descriptor and assigns the ID_SECURITY_TOKEN environment variable if a
declared usage matches the FIDO_CTAPHID_USAGE declared in the FIDO CTAP
specification. This replaces the previous approach of whitelisting all
known security token models manually.
This commit is accompanied by a test suite and a fuzzer target for the
descriptor parsing routine.
Fixes: #11996.
*We* control the sysctl setting. If the user configured IPv6, then we apply the
settings, and just make sure that at some point during the configuration the
sysctl is disabled (i.e. ipv6 enabled) if we have IPv6 configured.
Replaces #13283.
This test runs under qemu, which may run on some testbeds without
acceleration; in those cases, a 10s timeout is frequently too short.
Simply removing the timeout to allow the default timeoutsec should
be enough time for the test to finish, even on very slow testbeds.
This avoids unnecessary noise in the stderr logs which dd always produces,
such as:
0+0 records in
0+0 records out
0 bytes copied, 0.000155284 s, 0.0 kB/s
Using truncate should not result in any functional change; the image will
still be created as a sparse file of the size specified.
In Ubuntu CI, we test binaries from the installed system, not from
$BUILD_DIR, so use the appropriate binary. Most of the calls to the
binaries are part of checking/processing asan-built binaries, and so
did not apply to Ubuntu CI, except for generating noise in the stderr
log like:
objdump: '/tmp/autopkgtest.83yGoI/build.fHB/src/test/TEST-01-BASIC/systemd-journald': No such file
However this also applies to the call to systemd-nspawn, which the debian
upstream test wrapper was sed-adjusting to use the installed binary
instead of the binary in $BUILD_DIR. This commit allows removing that
sed processing of the test-functions file during Ubuntu CI test.
These tests runs under qemu, and on some testbeds, without acceleration.
On those systems, the current 180 second overall test timeout is too
short to run the test.
Increasing the timeout to 600s should be enough, even for slow
non-accelerated qemu testbeds.
This dir is created by create_empty_image_rootdir, as well as indirectly
by some other functions, but it should be created by import_initdir so
the newly-exported $initdir exists and can be used immediately without
relying on other functions to create it.
Only umount it during cleanup if the $TESTDIR/root dir is a mountpoint.
This avoids adding noise to the stderr log such as:
mountpoint: /var/tmp/systemd-test.waLOFT/root: No such file or directory
To make debugging much easier, especially for crashes in tests under
QEMU, let's store the entire coredump bundle in the systemd journal,
which is usually kept around by various CIs. Right now, we usually end
up with a journal, but without the coredump itself, which is pretty
useless.
Otherwise, changing the default gateway doesn't purge old gateway routes
left on the system during daemon restart. This also fixes removing other
foreign gateway routes that don't match the expected configuration.
Tested:
Changed gateway addresses prior to the patch and they lingered on
the system during each reconfiguration. Applied this patch and
reconfigured gateways and other routes multiple times and it removed
the foreign routes that had gateways that didn't match.
Signed-off-by: William A. Kennington III <william@wkennington.com>
The `coproc` implementation seems to be a little bit different in older
bash versions, so the `strace` is sometimes started AFTER `systemctl
daemon-reload`, which causes unexpected fails. Let's help it a little by
sleeping for a bit.
The MPOL_LOCAL constant is not recognized in current strace versions.
Let's match at least the numerical value of this constant until the
strace patch is approved & merged.
If networkctl crashes, like recently with SIGABRT, it returns absolutely
no output, which may be confusing during debugging. Help it a little
with a short informative message.
`adduser` is in certain cases a standalone package which provides a
better user experience. In other cases it's just a symlink to `useradd`.
And some distributions don't have `adduser` at all, like Arch Linux.
Let's use the `useradd` binary instead, which should provide the same
functionality everywhere.
This reworks how we load units from disk. Instead of chasing symlinks every
time we are asked to load a unit by name, we slurp all symlinks from disk
and build two hashmaps:
1. from unit name to either alias target, or fragment on disk
(if an alias, we put just the target name in the hashmap, if a fragment
we put an absolute path, so we can distinguish both).
2. from a unit name to all aliases
Reading all this data can be pretty costly (40 ms) on my machine, so we keep it
around for reuse.
The advantage is that we can reliably know what all the aliases of a given unit
are. This means we can reliably load dropins under all names. This fixes#11972.
I adjusted the tests to pass. I don't think the behaviour makes much sense,
even if we ignore the issue with "lazy loading" of aliases. E.g. in the
last section, the fact that dropins for yup@.service and yup@3.service are
not loaded seems to be a plain old bug.
This option is only used on reboot, not on other types of shutdown
modes, so it is misleading.
Keep the old name working for backward compatibility, but remove it
from the documentation.
Rather than always enabling the shutdown WD on kexec, which might be
dangerous in case the kernel driver and/or the hardware implementation
does not reset the wd on kexec, add a new timer, disabled by default,
to let users optionally enable the shutdown WD on kexec separately
from the runtime and reboot ones. Advise in the documentation to
also use the runtime WD in conjunction with it.
Fixes: a637d0f9ec ("core: set shutdown watchdog on kexec too")
It seems that old kernels do not support prefix routes with
non-default route tables. This adds a fallback logic when adding route
fails. In that case, prefix route is created by kernel and the default
route table is used.
Some distros install nologin as /usr/sbin/nologin, others as
/sbin/nologin.
Since we can't really on merged-usr everywhere (where the path wouldn't
matter), make the path build time configurable via -Dnologin-path=.
Closes#13028
without using -type f, the logs print an error such as:
E: E: modprobe: FATAL: Module asymmetric_keys not found in directory /lib/modules/4.15.0-54-generic
while this doesn't appear to cause problems, it can be extremely
distracting when trying to debug real failures.
Almost all tests were manually mounting/unmounting $TESTDIR/root
from the loopback image; this moves all that into test-functions
so the test setup functions are simplier.
Also add test_setup_cleanup() function, to cleanup what is mounted
by create_empty_image_rootdir()
Even if addresses provided by DHCP is assigned, the state file may not
be written yet, or resolved may not receive the state change signal
yet, or resolved may not process the signal yet...
Previously, event if link's setup state is in failed, tests may pass,
as systemd-networkd-wait-online success if the state is in failed state.
This makes tests be checked more strictly.
In Ubuntu CI, udev-test.pl is run from the debian/test/udev script,
in a test dir created for it; but udev-test.pl setup mounts a
dir, so if it doesn't cleanup/unmount before exiting, the test dir
autopkgtest created for it can't be removed, and autopkgtest
aborts the entire test suite, for example this output (from a
test run inside an armhf container):
autopkgtest [12:45:36]: test udev: [-----------------------
umount: test/tmpfs: no mount point specified.
mknod: test/tmpfs/dev/null: Operation not permitted
unable to create test/tmpfs/dev/null at ./udev-test.pl line 1611.
Failed to set up the environment, skipping the test at ./udev-test.pl line 1731.
autopkgtest [12:45:41]: test udev: -----------------------]
autopkgtest [12:45:44]: test udev: - - - - - - - - - - results - - - - - - - - - -
udev FAIL non-zero exit status 77
rm: cannot remove '/tmp/autopkgtest.ocPFA6/autopkgtest_tmp/test/tmpfs': Device or resource busy
autopkgtest [12:46:22]: ERROR: "rm -rf /tmp/autopkgtest.ocPFA6/udev-artifacts /tmp/autopkgtest.ocPFA6/autopkgtest_tmp" failed with stderr "rm:
The `set -e` option is incompatible with a subshell/compound command,
which is followed by || <EXPR>. In such case, the -e option is ignored
in all affected subshells/functions (see man bash(1) for command `set`).
In certain cases we might attempt to install a binary which is already
present in the test image, yet it's missing from the host system.
In such cases, let's check if the binary indeed exists in the image
before doing any other chcecks. If it does, immediately return with
success.
This was discovered during installation of
/usr/lib/systemd/systemd-bless-boot, which was not present in Ubuntu CI
(as the installed systemd was from the Ubuntu repositories), and the
binary itself was already in the image thanks to `ninja install`.
However, during extraction of binaries from the systemd service files,
another attempt to install this binary was made, which failed due to
`find_binary` being unable to find it.
Before only one comparison was allowed. Let's make this more flexible:
ConditionKernelVersion = ">=4.0" "<=4.5"
Fixes#12881.
This also fixes expressions like "ConditionKernelVersion=>" which would
evaluate as true.
job_compare return value is undefined in case the jobs have a loop
between them, so better make a test to make sure transaction cycle
detection catches it.
The test-engine Test2 tests the cycle detection when units a, b and d
all start at once
,-------------------after-----------------,
v |
a/start ---after---> d/start ---after---> b/start
Extend the test with Test11 that adds i.service which causes a and d
stop (by unordered Conflicts=) while starting b. Because stops precede
starts, we effectively eliminate the job cycle and all transaction jobs
should be applicable.
,-------------------after-----------------,
v |
a/stop <---after--- d/stop <---after--- b/start
. . ^
. . |
'. . . . . . . . . i/start ---after------'
A class was used to put some utility functions under a namespace. But then this
namespace was inherited into classes, so the namespace split was undone. Let's
just define those functions in the module namespace.
(The Utilities class with a few functions still remain, because of the
unittest-style self.assert* craziness.)
For normal shell calls, python syntax is quite verbose. We don't need to punish
ourselves like that. In some places we would fork a shell to do argument splitting
for us. We know that our arguments can be safely split on whitespace, so let's do
that ourselves in all cases, without forking a shell.
We also expect command output to be valid text, so let's always set
universal_newlines=True.
This makes things shorter and easier to read. Development is also easier because
it's possible to paste many of the commands directly to/from a shell.
Adds the resumeflags= kernel command line option to allow setting a
custom device timeout for the resume device (defaults to the same as the
root device).
In certain situations, the systemctl commands may fail (e.g. due to
missing shared libraries), but the 'script' continues and creates a
/testok file, marking the test incorrectly as passed. Let's fix this and
bail out immediately when a command exits with a non-zero exit code.
Running integration tests with ASan is somewhat tricky to begin with, as
we need to pre-load the ASan runtime DSO for certain services (like
dbus), otherwise they won't start or behave as expected. In case of gcc
this is pretty easy, as we need the runtime DSO during compilation, so
it's already present on the host system. For clang things get more
complicated, as ASan is compiled in statically by default, thus to
enable the necessary dynamic-ish behavior one needs to compile with
-shared-libasan and then correctly set LD_PRELOAD_PATH, as the runtime
libraries are not in a standard library path.
These services are likely to coredump, and we expect that but aren't
interested in the coredump. Hence let's turn off processing by setting
RLIMIT_CORE to 0/0.
This patch adds netdev ipvtap that is based on the
IP-VLAN network interface, called ipvtap. An ipvtap device can be created
in the same way as an ipvlan device, using 'kind ipvtap', and then accessed
using the tap user space interface.
Certain distributions (e.g. Arch Linux) require booting with initrd, as
they lack support for commonly used filesystems in the kernel (i.e. the
support is compiled in as modules)
The `mount` utility has an unexpected behavior when run with libasan,
causing false-positives during the integration testing.
For example, on Arch Linux with LD_PRELOAD pointing to libasan:
```
bash-5.0# mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
1
```
However:
```
bash-5.0# LD_PRELOAD= mount -o remount,rw -v /
mount: /dev/sda1 mounted on /.
bash-5.0# echo $?
0
```
Further investigation with strace shows a LeakSanitizer error:
```
bash-5.0# strace -s 512 mount -o remount,rw -v /
...
write(2, "==355==LeakSanitizer has encountered a fatal error.\n", 52) = -1 EBADF (Bad file descriptor)
write(2, "ReportFile::Write() can't output requested buffer!\n", 51) = -1 EBADF (Bad file descriptor)
exit_group(1) = ?
+++ exited with 1 +++
```
Let's workaround this by clearing the LD_PRELOAD variable for
systemd-remount-fs.service
Allow users to set the IPv4 DF bit in outgoing packets, or to inherit its
value from the IPv4 inner header. If the encapsulated protocol is IPv6 and
DF is configured to be inherited, always set it.
It makes more sense to call VXLAN ID as
1. the VXLAN Network Identifier (VNI) (or VXLAN Segment ID)
2. test-network: rename VXLAN Id to VNI
3. fuzzer: Add VXLAN VNI directive to fuzzer
