This introduces a new data threshold setting for sd_journal objects
which controls the maximum size of objects to decompress. This is
relieves the library from having to decompress full data objects even
if a client program is only interested in the initial part of them.
This speeds up "systemd-coredumpctl" drastically when invoked without
parameters.
When traversing entry array chains for a bisection or for retrieving an
item by index we previously always started at the beginning of the
chain. Since we tend to look at the same chains repeatedly, let's cache
where we have been the last time, and maybe we can skip ahead with this
the next time.
This turns most bisections and index lookups from O(log(n)*log(n)) into
O(log(n)). More importantly however, we seek around on disk much less,
which is good to reduce buffer cache and seek times on rotational disks.
The new 'unique' API allows listing all unique field values that a field
specified by a field name can take in all entries of the journal. This
allows answering queries such as "What units logged to the journal?",
"What hosts have logged into the journal?", "Which boot IDs have logged
into the journal?".
Ultimately this allows implementation of tools similar to lastlog based
on journal data.
Note that listing these field values will not work for journal files
created with older journald, as the field values are not indexed in
older files.
Let's clean up our terminology a bit. New terminology:
FSS = Forward Secure Sealing
FSPRG = Forward Secure Pseudo-Random Generator
FSS is the combination of FSPRG and a HMAC.
Sealing = process of adding authentication tags to the journal.
Verification = process of checking authentication tags to the journal.
Sealing Key = The key used for adding authentication tags to the journal.
Verification Key = The key used for checking authentication tags of the journal.
Key pair = The pair of Sealing Key and Verification Key
Internally, the Sealing Key is the combination of the FSPRG State plus
change interval/start time.
Internally, the Verification Key is the combination of the FSPRG Seed
plus change interval/start time.
instead of having one simple per-file cache implement an more
comprehensive one that works for multiple files and can actually
maintain multiple maps per file and per object type.
This adds forward-secure authentication of journal files. This patch
includes key generation as well as tagging of journal files,
Verification of journal files will be added in a later patch.
The default of 2047 hash table entries turned out to result in way too
many collisions for bigger files, hence scale the hash table size by the
estimated maximum file size.
Previously, when the main data hash table grows too full the performance
simply started to decrease drastically. Instead, now simply rotate to a
new journal file as the hash table gets to full, so that we can start
with a new fresh empty hash table.
Lennart Poettering