We _always_ return NULL from destructors to allow direct assignments to
the variable holding the object. Especially on hashmaps, which treat NULL
as empty hashmap, this is pretty neat.
This patch removes includes that are not used. The removals were found with
include-what-you-use which checks if any of the symbols from a header is
in use.
There will be more debugging options later.
--enable-debug will enable them all.
--enable-debug=hashmap will enable only hashmap debugging.
Also rename the C #define to ENABLE_DEBUG_* pattern.
This is a rewrite of the hashmap implementation. Its advantage is lower
memory usage.
It uses open addressing (entries are stored in an array, as opposed to
linked lists). Hash collisions are resolved with linear probing and
Robin Hood displacement policy. See the references in hashmap.c.
Some fun empirical findings about hashmap usage in systemd on my laptop:
- 98 % of allocated hashmaps are Sets.
- Sets contain 78 % of all entries, plain Hashmaps 17 %, and
OrderedHashmaps 5 %.
- 60 % of allocated hashmaps contain only 1 entry.
- 90 % of allocated hashmaps contain 5 or fewer entries.
- 75 % of all entries are in hashmaps that use trivial_hash_ops.
Clearly it makes sense to:
- store entries in distinct entry types. Especially for Sets - their
entries are the most numerous and they require the least information
to store an entry.
- have a way to store small numbers of entries directly in the hashmap
structs, and only allocate the usual entry arrays when the direct
storage is full.
The implementation has an optional debugging feature (enabled by
defining the ENABLE_HASHMAP_DEBUG macro), where it:
- tracks all allocated hashmaps in a linked list so that one can
easily find them in gdb,
- tracks which function/line allocated a given hashmap, and
- checks for invalid mixing of hashmap iteration and modification.
Since entries are not allocated one-by-one anymore, mempools are not
used for entries. Originally I meant to drop mempools entirely, but it's
still worth it to use them for the hashmap structs. My testing indicates
that it makes loading of units about 5 % faster (a test with 10000 units
where more than 200000 hashmaps are allocated - pure malloc: 449±4 ms,
mempools: 427±7 ms).
Here are some memory usage numbers, taken on my laptop with a more or
less normal Fedora setup after booting with SELinux disabled (SELinux
increases systemd's memory usage significantly):
systemd (PID 1) Original New Change
dirty memory (from pmap -x 1) [KiB] 2152 1264 -41 %
total heap allocations (from gdb-heap) [KiB] 1623 756 -53 %
It cannot fail in the current hashmap implementation, but it may fail in
alternative implementations (unless a sufficiently large reservation has
been placed beforehand).
With the current hashmap implementation that uses chaining, placing a
reservation can serve two purposes:
- To optimize putting of entries if the number of entries to put is
known. The reservation allocates buckets, so later resizing can be
avoided.
- To avoid having very long bucket chains after using
hashmap_move(_one).
In an alternative hashmap implementation it will serve an additional
purpose:
- To guarantee a subsequent hashmap_move(_one) will not fail with
-ENOMEM (this never happens in the current implementation).
-ENOENT is the same return value as if 'other' were an allocated hashmap
that does not contain the key. A NULL hashmap is a possible way of
expressing a hashmap that contains no key.
When hashmap_replace detects no such key exists yet, it calls hashmap_put that
performs the same check again. Avoid that by splitting the core of hashmap_put
into a separate function.
The following hashmap_* and set_* functions/macros have never had any
users in systemd's history:
*_iterate_backwards
*_iterate_skip
*_last
*_FOREACH_BACKWARDS
Remove this dead code.
It is redundant to store 'hash' and 'compare' function pointers in
struct Hashmap separately. The functions always comprise a pair.
Store a single pointer to struct hash_ops instead.
systemd keeps hundreds of hashmaps, so this saves a little bit of
memory.
In trying to track down a stupid linker bug, I noticed a bunch of
memset() calls that should be using memzero() to make it more "obvious"
that the options are correct (i.e. 0 is not the length, but the data to
set). So fix up all current calls to memset(foo, 0, length) to
memzero(foo, length).
Instead of fixing the hashmap bucket array to 127 entries dynamically
size it, starting with a smaller one of 31. As soon as a fill level of
75% is reached, quadruple the size, and so on.
This should siginficantly optimize the lookup time in large tables
(from O(n) back to O(1)), and save memory on smaller tables (which most
are).
This unifies much of the logic behind them:
- All four will now ofllow the rule that the earlier file and earlier
assignment in the .d/ directories wins. Before, sysctl was the only
outlier, where the later setting always won.
- All four now support getopt() and --help on the command line.
- All four can now handle specification of configuration file names on
the command line to apply. The tools will automatically find them, and
apply them. Previously only tmpfiles could do that. This is useful for
%post scripts in RPMs and suchlike.
- This fixes various error path issues in conf_files_list()
When traversing entry array chains for a bisection or for retrieving an
item by index we previously always started at the beginning of the
chain. Since we tend to look at the same chains repeatedly, let's cache
where we have been the last time, and maybe we can skip ahead with this
the next time.
This turns most bisections and index lookups from O(log(n)*log(n)) into
O(log(n)). More importantly however, we seek around on disk much less,
which is good to reduce buffer cache and seek times on rotational disks.
The new 'unique' API allows listing all unique field values that a field
specified by a field name can take in all entries of the journal. This
allows answering queries such as "What units logged to the journal?",
"What hosts have logged into the journal?", "Which boot IDs have logged
into the journal?".
Ultimately this allows implementation of tools similar to lastlog based
on journal data.
Note that listing these field values will not work for journal files
created with older journald, as the field values are not indexed in
older files.
In some cases, like wrong configuration, restarting after error
does not help, so administrator can specify statuses by RestartPreventExitStatus
which will not cause restart of a service.
Sometimes you have non-standart exit status, so this can be specified
by SuccessfulExitStatus.
We finally got the OK from all contributors with non-trivial commits to
relicense systemd from GPL2+ to LGPL2.1+.
Some udev bits continue to be GPL2+ for now, but we are looking into
relicensing them too, to allow free copy/paste of all code within
systemd.
The bits that used to be MIT continue to be MIT.
The big benefit of the relicensing is that closed source code may now
link against libsystemd-login.so and friends.
