Users might end up with more than one of those service enabled, through
admin mistake, or broken installation scriptlets, or whatever. On my machine,
I had both chronyd and timesyncd happilly running at the same time. If
more than one is enabled, it's better to have just one running. Adding
Conflicts will make the issue more visible in logs too.
This patch introduces the systemd pstore service which will archive the
contents of the Linux persistent storage filesystem, pstore, to other storage,
thus preserving the existing information contained in the pstore, and clearing
pstore storage for future error events.
Linux provides a persistent storage file system, pstore[1], that can store
error records when the kernel dies (or reboots or powers-off). These records in
turn can be referenced to debug kernel problems (currently the kernel stuffs
the tail of the dmesg, which also contains a stack backtrace, into pstore).
The pstore file system supports a variety of backends that map onto persistent
storage, such as the ACPI ERST[2, Section 18.5 Error Serialization] and UEFI
variables[3 Appendix N Common Platform Error Record]. The pstore backends
typically offer a relatively small amount of persistent storage, e.g. 64KiB,
which can quickly fill up and thus prevent subsequent kernel crashes from
recording errors. Thus there is a need to monitor and extract the pstore
contents so that future kernel problems can also record information in the
pstore.
The pstore service is independent of the kdump service. In cloud environments
specifically, host and guest filesystems are on remote filesystems (eg. iSCSI
or NFS), thus kdump relies [implicitly and/or explicitly] upon proper operation
of networking software *and* hardware *and* infrastructure. Thus it may not be
possible to capture a kernel coredump to a file since writes over the network
may not be possible.
The pstore backend, on the other hand, is completely local and provides a path
to store error records which will survive a reboot and aid in post-mortem
debugging.
Usage Notes:
This tool moves files from /sys/fs/pstore into /var/lib/systemd/pstore.
To enable kernel recording of error records into pstore, one must either pass
crash_kexec_post_notifiers[4] to the kernel command line or enable via 'echo Y
> /sys/module/kernel/parameters/crash_kexec_post_notifiers'. This option
invokes the recording of errors into pstore *before* an attempt to kexec/kdump
on a kernel crash.
Optionally, to record reboots and shutdowns in the pstore, one can either pass
the printk.always_kmsg_dump[4] to the kernel command line or enable via 'echo Y >
/sys/module/printk/parameters/always_kmsg_dump'. This option enables code on the
shutdown path to record information via pstore.
This pstore service is a oneshot service. When run, the service invokes
systemd-pstore which is a tool that performs the following:
- reads the pstore.conf configuration file
- collects the lists of files in the pstore (eg. /sys/fs/pstore)
- for certain file types (eg. dmesg) a handler is invoked
- for all other files, the file is moved from pstore
- In the case of dmesg handler, final processing occurs as such:
- files processed in reverse lexigraphical order to faciliate
reconstruction of original dmesg
- the filename is examined to determine which dmesg it is a part
- the file is appended to the reconstructed dmesg
For example, the following pstore contents:
root@vm356:~# ls -al /sys/fs/pstore
total 0
drwxr-x--- 2 root root 0 May 9 09:50 .
drwxr-xr-x 7 root root 0 May 9 09:50 ..
-r--r--r-- 1 root root 1610 May 9 09:49 dmesg-efi-155741337601001
-r--r--r-- 1 root root 1778 May 9 09:49 dmesg-efi-155741337602001
-r--r--r-- 1 root root 1726 May 9 09:49 dmesg-efi-155741337603001
-r--r--r-- 1 root root 1746 May 9 09:49 dmesg-efi-155741337604001
-r--r--r-- 1 root root 1686 May 9 09:49 dmesg-efi-155741337605001
-r--r--r-- 1 root root 1690 May 9 09:49 dmesg-efi-155741337606001
-r--r--r-- 1 root root 1775 May 9 09:49 dmesg-efi-155741337607001
-r--r--r-- 1 root root 1811 May 9 09:49 dmesg-efi-155741337608001
-r--r--r-- 1 root root 1817 May 9 09:49 dmesg-efi-155741337609001
-r--r--r-- 1 root root 1795 May 9 09:49 dmesg-efi-155741337710001
-r--r--r-- 1 root root 1770 May 9 09:49 dmesg-efi-155741337711001
-r--r--r-- 1 root root 1796 May 9 09:49 dmesg-efi-155741337712001
-r--r--r-- 1 root root 1787 May 9 09:49 dmesg-efi-155741337713001
-r--r--r-- 1 root root 1808 May 9 09:49 dmesg-efi-155741337714001
-r--r--r-- 1 root root 1754 May 9 09:49 dmesg-efi-155741337715001
results in the following:
root@vm356:~# ls -al /var/lib/systemd/pstore/155741337/
total 92
drwxr-xr-x 2 root root 4096 May 9 09:50 .
drwxr-xr-x 4 root root 40 May 9 09:50 ..
-rw-r--r-- 1 root root 1610 May 9 09:50 dmesg-efi-155741337601001
-rw-r--r-- 1 root root 1778 May 9 09:50 dmesg-efi-155741337602001
-rw-r--r-- 1 root root 1726 May 9 09:50 dmesg-efi-155741337603001
-rw-r--r-- 1 root root 1746 May 9 09:50 dmesg-efi-155741337604001
-rw-r--r-- 1 root root 1686 May 9 09:50 dmesg-efi-155741337605001
-rw-r--r-- 1 root root 1690 May 9 09:50 dmesg-efi-155741337606001
-rw-r--r-- 1 root root 1775 May 9 09:50 dmesg-efi-155741337607001
-rw-r--r-- 1 root root 1811 May 9 09:50 dmesg-efi-155741337608001
-rw-r--r-- 1 root root 1817 May 9 09:50 dmesg-efi-155741337609001
-rw-r--r-- 1 root root 1795 May 9 09:50 dmesg-efi-155741337710001
-rw-r--r-- 1 root root 1770 May 9 09:50 dmesg-efi-155741337711001
-rw-r--r-- 1 root root 1796 May 9 09:50 dmesg-efi-155741337712001
-rw-r--r-- 1 root root 1787 May 9 09:50 dmesg-efi-155741337713001
-rw-r--r-- 1 root root 1808 May 9 09:50 dmesg-efi-155741337714001
-rw-r--r-- 1 root root 1754 May 9 09:50 dmesg-efi-155741337715001
-rw-r--r-- 1 root root 26754 May 9 09:50 dmesg.txt
where dmesg.txt is reconstructed from the group of related
dmesg-efi-155741337* files.
Configuration file:
The pstore.conf configuration file has four settings, described below.
- Storage : one of "none", "external", or "journal". With "none", this
tool leaves the contents of pstore untouched. With "external", the
contents of the pstore are moved into the /var/lib/systemd/pstore,
as well as logged into the journal. With "journal", the contents of
the pstore are recorded only in the systemd journal. The default is
"external".
- Unlink : is a boolean. When "true", the default, then files in the
pstore are removed once processed. When "false", processing of the
pstore occurs normally, but the pstore files remain.
References:
[1] "Persistent storage for a kernel's dying breath",
March 23, 2011.
https://lwn.net/Articles/434821/
[2] "Advanced Configuration and Power Interface Specification",
version 6.2, May 2017.
https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
[3] "Unified Extensible Firmware Interface Specification",
version 2.8, March 2019.
https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_final.pdf
[4] "The kernel’s command-line parameters",
https://static.lwn.net/kerneldoc/admin-guide/kernel-parameters.html
We use that on all other services, and hence should here too. Otherwise
the service will be killed with SIGSYS when doing something not
whitelisted, which is a bit crass.
While the need for access to character devices can be tricky to determine for
the general case, it's obvious that most of our services have no need to access
block devices. For logind and timedated this can be tightened further.
We had all kinds of indentation: 2 sp, 3 sp, 4 sp, 8 sp, and mixed.
4 sp was the most common, in particular the majority of scripts under test/
used that. Let's standarize on 4 sp, because many commandlines are long and
there's a lot of nesting, and with 8sp indentation less stuff fits. 4 sp
also seems to be the default indentation, so this will make it less likely
that people will mess up if they don't load the editor config. (I think people
often use vi, and vi has no support to load project-wide configuration
automatically. We distribute a .vimrc file, but it is not loaded by default,
and even the instructions in it seem to discourage its use for security
reasons.)
Also remove the few vim config lines that were left. We should either have them
on all files, or none.
Also remove some strange stuff like '#!/bin/env bash', yikes.
time-sync.target is supposed to indicate system clock is synchronized
with a remote clock, but as used through 241 it only provided a system
clock that was updated based on a locally-maintained timestamp. Systems
that are powered off for extended periods would not come up with
accurate time.
Retain the existing behavior using a new time-set.target leaving
time-sync.target for cases where accuracy is required.
Closes#8861
Let's be safe, rather than sorry. This way DynamicUser=yes services can
neither take benefit of, nor create SUID/SGID binaries.
Given that DynamicUser= is a recent addition only we should be able to
get away with turning this on, even though this is strictly speaking a
binary compatibility breakage.
This patch was initially prompted by a report on a Fedora update [1], that the
upgrade causes systemd-resolved.service and systemd-networkd.service to be
re-enabled. We generally want to preserve the enablement of all services during
upgrades, so a reset like this is not expected.
Both services declare two symlinks in their [Install] sections, for their dbus
names and for multi-user.target.wants/. It turns out that both services were
only partially enabled, because their dbus unit symlinks
/etc/systemd/system/dbus-org.freedesktop.{resolve1,network1}.service were
created, by the symlinks in /etc/systemd/system/multi-user.target.wants/ were
not. This means that the units could be activated by dbus, but not in usual
fashion using systemctl start. Our tools make it rather hard to figure out when
something like this happens, and it is definitely an area for improvement on its
own. The symlink in .wants/ was filtered out by during packaging, but the dbus
symlink was left in (I assume by mistake).
Let's simplify things by not creating the symlinks statically during 'ninja
install'. This means that the units shipped by systemd have to be enabled in
the usual fashion, which in turns means that [Install] section and presets
become the "single source of truth" and we don't have two sets of conflicting
configuration.
Let's consider a few cases:
- developer: a developer installs systemd from git on a running system, and they
don't want the installation to reset enablement of anything. So this change is
either positive for them, or has no effect (if they have everything at
defaults).
- package creation: we want to create symlinks using 'preset-all' and 'preset'
on upgraded packages, we don't want to have any static symlinks. This change
will remove the need to filter out symlinks in packaging and of course fix
the original report.
- installation of systemd from scratch: this change means that without
'preset-all' the system will not be functional. This case could be affected
negatively by this change, but I think it's enough of a corner case to accept
this. In practice I expect people to build a package, not installl directly
into the file system, so this might not even matter in practice.
Creating those symlinks was probably the right thing in the beginning, but
nowadays the preset system is very well established and people expect it to
be honoured. Ignoring the presets and doing static configuration is not welcome
anymore.
Note: during package installation, either 'preset-all' or 'preset getty@.service
machines.target remote-cryptsetup.target remote-fs.target
systemd-networkd.service systemd-resolved.service
systemd-networkd-wait-online.service systemd-timesyncd.service' should be called.
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-616045ca76
A couple of API VFS we mount via .mount units. Let's set the three flags
for those too, just in case.
This is just paranoia, nothing else, but shouldn't hurt.
This service uses PAM anyway, hence let pam_keyring set things up for
us. Moreover, this way we ensure that the invocation ID is not set for
this service as key, and thus can't confuse the user service's
invocation ID.
Fixes: #11649
`systemd-journal-catalog-update.service` writes to `/var`. However, it's
not explicitly ordered wrt `systemd-tmpfiles-setup.service`, which means
that it may run before or after.
This is an issue for Fedora CoreOS, which uses Ignition. We want to be
able to prepare `/var` on first boot from the initrd, where the SELinux
policy is not loaded yet. This means that the hierarchy under `/var` is
not correctly labeled. We add a `Z /var - - -` tmpfiles entry so that it
gets relabeled once `/var` gets mounted post-switchroot.
So any service that tries to access `/var` before `systemd-tmpfiles`
relabels it is likely to hit `EACCES`.
Fix this by simply ordering `systemd-journal-catalog-update.service`
after `systemd-tmpfiles-setup.service`. This is also clearer since the
tmpfiles entries are the canonical source of how `/var` should be
populated.
For more context on this, see:
https://github.com/coreos/ignition/issues/635#issuecomment-446620297
ProtectHostname= turns off hostname change propagation from host to
service. This means for services that care about the hostname and need
to be able to notice changes to it it's not suitable (though it is
useful for most other cases still).
Let's turn it off hence for journald (which logs the current hostname)
for networkd (which optionally sends the current hostname to dhcp
servers) and resolved (which announces the current hostname via
llmnr/mdns).
This behaves similar to the "boot into firmware" logic, and also allows
either direct EFI operation (which sd-boot supports and others might
support eventually too) or override through env var.
This was an overzealous setting from commit 99894b867f. Without this,
`hostnamectl set-hostname` fails with
Could not set property: Access denied
as `sethostname()` fails with `EPERM`.
Linux can be run on a device meant to act as a USB peripheral. In order
for a machine to act as such a USB device it has to be equipped with
a UDC - USB Device Controller.
This patch adds a target reached when UDC becomes available. It can be used
for activating e.g. a service unit which composes a USB gadget with
configfs and activates it.
A follow-up for commit a8cb1dc3e0.
Commit a8cb1dc3e0 made sure that initrd-cleanup.service won't be stopped
when initrd-switch-root.target is isolated.
However even with this change, it might happen that initrd-cleanup.service
survives the switch to rootfs (since it has no ordering constraints against
initrd-switch-root.target) and is stopped right after when default.target is
isolated. This led to initrd-cleanup.service entering in failed state as it
happens when oneshot services are stopped.
This patch along with a8cb1dc3e0 should fix issue #4343.
Fixes: #4343
Currently, tmpfiles runs in two separate services at boot. /dev is
populated by systemd-tmpfiles-setup-dev.service and everything else by
systemd-tmpfiles-setup.service. The former was so far conditionalized by
CAP_SYS_MODULES. The reasoning was that the primary purpose of
populating /dev was to create device nodes based on the static device
node info exported in kernel modules through MODALIAS. And without the
privs to load kernel modules doing so is unnecessary. That thinking is
incomplete however, as there might be reason to create stuff in /dev
outside of the static modalias usecase. Thus, let's drop the
conditionalization to ensure that tmpfiles.d rules are always executed
at least once under all conditions.
Fixes: #11544
Instead of enabling it unconditionally and then using ConditionPathExists=/etc/fstab,
and possibly masking this condition if it should be enabled for auto gpt stuff,
just pull it in explicitly when required.
We already *install* those as real files since de78fa9ba0.
Meson will start to copy symlinks as-is, so we would get dangling symlinks in
/usr/lib/systemd/user/.
I considered the layout in our sources to match the layout in the installation
filesystem (i.e. creating units/system/ and moving all files from units/ to
units/system/), but that seems overkill. By using normal files for both we get
some duplication, but those files change rarely, so it's not a big downside in
practice.
Fixes#9906.
Let's simplify things and drop the logic that /var/lib/machines is setup
as auto-growing btrfs loopback file /var/lib/machines.raw.
THis was done in order to make quota available for machine management,
but quite frankly never really worked properly, as we couldn't grow the
file system in sync with its use properly. Moreover philosophically it's
problematic overriding the admin's choice of file system like this.
Let's hence drop this, and simplify things. Deleting code is a good
feeling.
Now that regular file systems provide project quota we could probably
add per-machine quota support based on that, hence the btrfs quota
argument is not that interesting anymore (though btrfs quota is a bit
more powerful as it allows recursive quota, i.e. that the machine pool
gets an overall quota in addition to per-machine quota).
Otherwise we might install the socket unit early, but the service
backing it late, and then end up in strange loops when we enter rescue
mode, because we saw an event on /dev/rfkill but really can't dispatch
it nor flush it.
Fixes: #9171
now that logind doesn't mount $XDG_RUNTIME_DIR anymore we can lock down
the service using fs namespacing (as we don't need the mount to
propagate to the host namespace anymore).
Zbigniew Jędrzejewski-Szmek