Chen Jie observed and analyzed a deadlock. Assuming systemd-kmsg-syslogd
is already stopped, but rsyslogd is not started yet:
1. systemd makes a synchronous call to dbus-daemon.
2. dbus-daemon wants to write something to syslog.
3. syslog needs to be started by systemd.
... but cannot be, because systemd is waiting in 1.
Solve this by avoiding synchronous D-Bus calls. I had to write an async
bus registration call. Interestingly, D-Bus authors anticipated this, in
documentation to dbus_bus_set_unique_name():
> The only reason to use this function is to re-implement the equivalent
> of dbus_bus_register() yourself. One (probably unusual) reason to do
> that might be to do the bus registration call asynchronously instead
> of synchronously.
Lennart's comments from IRC:
> though I think this doesn't fix the problem in its entirety
> simply because dbus_connection_open_private() itself is still synchronous
> i.e. the connect() call behind it is not async
> I think I listed that issue actually on some D-Bus todo list
> i.e. to make dbus_connection_get() fully async
> but that's going to be hard
> so your patch looks good
So it may not be perfect, but it's clearly an improvement.
I did not manage to reproduce the original deadlock with the patch.
When running on a kernel without audit support, systemd currently
writes a mysterious-sounding error to its log:
systemd[1]: Failed to connect to audit log: Protocol not supported
Better to suppress the audit_open() failure message when (and only
when) it is due to running on a kernel without audit support, since in
this case the admin probably does not mind systemd not writing to the
audit log. This way, more serious errors like ENOMEM and EACCES will
stand out more.
HASHMAP_FOREACH is safe against the removal of the current entry, but
not against the removal of other entries. job_finish_and_invalidate()
can recursively remove other entries.
It triggered an assertion failure:
Assertion 'j->installed' failed at src/manager.c:1218, function
transaction_apply(). Aborting.
Fix the crash by iterating from the beginning when there is a
possibility that the iterator could be invalid.
It is O(n^2) in the worst case, but that's better than a crash.
https://bugzilla.redhat.com/show_bug.cgi?id=717325
signal_to_string:
Produce names for SIGRTMIN+n.
Never give an "n/a" result. In the worst case give the number itself as
a string.
signal_from_string:
Parse "RTMIN+n".
Parse any valid signal number.
Instead of having individual counters n_serializing and n_deserializing
have a single one n_reloading, which should be sufficient.
Set n_reloading when we are about to go down for reexecution to avoid
cgroup trimming when we free the units for reexecution.
In some cases systemd forgets to start enabled services, e.g. in this scenario:
1. The job "sendmail.service/start" is installed.
2. Before systemd proceeds with this job, a process requests a reload of
sendmail. The job "sendmail.service/reload" is enqueued.
3. The original job is silently discarded. The service is not started.
The reload job proceeds by doing nothing.
The fix is to allow merging transaction jobs with the active job.
With the fix the resulting merged job "sendmail.service/reload-or-start" is
installed and the service works as expected.
https://bugzilla.redhat.com/show_bug.cgi?id=633774
We simply keep open copies of the dbus connections across the
reexecution and close them as last step of it. A client can thus simply
wait until its connection is dropped to know when the reexecution is
finished.
https://bugzilla.redhat.com/show_bug.cgi?id=698198
Instead of the /dev/.run trick we have currently implemented, we decided
to move the early-boot runtime dir to /run.
An existing /var/run directory is bind-mounted to /run. If /var/run is
already a symlink, no action is taken.
An existing /var/lock directory is bind-mounted to /run/lock.
If /var/lock is already a symlink, no action is taken.
To implement the directory vs. symlink logic, we have a:
ConditionPathIsDirectory=
now, which is used in the mount units.
Skipped mount unit in case of symlink:
$ systemctl status var-run.mount
var-run.mount - Runtime Directory
Loaded: loaded (/lib/systemd/system/var-run.mount)
Active: inactive (dead)
start condition failed at Fri, 25 Mar 2011 04:51:41 +0100; 6min ago
Where: /var/run
What: /run
CGroup: name=systemd:/system/var-run.mount
The systemd rpm needs to make sure to add something like:
%pre
mkdir -p -m0755 /run >/dev/null 2>&1 || :
or it needs to be added to filesystem.rpm.
Udev -git already uses /run if that exists, and is writable at bootup.
Otherwise it falls back to the current /dev/.udev.
Dracut and plymouth need to be adopted to switch from /dev/.run to run
too.
Cheers,
Kay
Now that we have /dev/.run there's no need to use abstract namespace
sockets. So, let's move things to /dev/.run, to make things more easily
discoverable and improve compat with chroot() and fs namespacing.
