log.h really should only include the bare minimum of other headers, as
it is really pulled into pretty much everything else and already in
itself one of the most basic pieces of code we have.
Let's hence drop inclusion of:
1. sd-id128.h because it's entirely unneeded in current log.h
2. errno.h, dito.
3. sys/signalfd.h which we can replace by a simple struct forward
declaration
4. process-util.h which was needed for getpid_cached() which we now hide
in a funciton log_emergency_level() instead, which nicely abstracts
the details away.
5. sys/socket.h which was needed for struct iovec, but a simple struct
forward declaration suffices for that too.
Ultimately this actually makes our source tree larger (since users of
the functionality above must now include it themselves, log.h won't do
that for them), but I think it helps to untangle our web of includes a
tiny bit.
(Background: I'd like to isolate the generic bits of src/basic/ enough
so that we can do a git submodule import into casync for it)
This adds a new safe_fork() wrapper around fork() and makes use of it
everywhere. The new wrapper does a couple of things we previously did
manually and separately in a safer, more correct and automatic way:
1. Optionally resets signal handlers/mask in the child
2. Sets a name on all processes we fork off right after forking off (and
the patch assigns useful names for all processes we fork off now,
following a systematic naming scheme: always enclosed in () – in order
to indicate that these are not proper, exec()ed processes, but only
forked off children, and if the process is long-running with only our
own code, without execve()'ing something else, it gets am "sd-" prefix.)
3. Optionally closes all file descriptors in the child
4. Optionally sets a PR_SET_DEATHSIG to SIGTERM in the child, in a safe
way so that the parent dying before this happens being handled
safely.
5. Optionally reopens the logs
6. Optionally connects stdin/stdout/stderr to /dev/null
7. Debug logs about the forked off processes.
The advantage is that is the name is mispellt, cpp will warn us.
$ git grep -Ee "conf.set\('(HAVE|ENABLE)_" -l|xargs sed -r -i "s/conf.set\('(HAVE|ENABLE)_/conf.set10('\1_/"
$ git grep -Ee '#ifn?def (HAVE|ENABLE)' -l|xargs sed -r -i 's/#ifdef (HAVE|ENABLE)/#if \1/; s/#ifndef (HAVE|ENABLE)/#if ! \1/;'
$ git grep -Ee 'if.*defined\(HAVE' -l|xargs sed -i -r 's/defined\((HAVE_[A-Z0-9_]*)\)/\1/g'
$ git grep -Ee 'if.*defined\(ENABLE' -l|xargs sed -i -r 's/defined\((ENABLE_[A-Z0-9_]*)\)/\1/g'
+ manual changes to meson.build
squash! build-sys: use #if Y instead of #ifdef Y everywhere
v2:
- fix incorrect setting of HAVE_LIBIDN2
`journalctl -o export | systemd-journal-remote -o /tmp/dir -`
gives the following error messages.
```
Failed to open output journal /tmp/dir: Invalid argument
Failed to get writer for source stdin: Invalid argument
Failed to create source for fd:0 (stdin): Invalid argument
```
And these are hard to understand what is the problem.
This commit makes journal-remote check whether the output file name
ends with .journal suffix or not, and if not, output error message.
The option MHD_OPTION_STRICT_FOR_CLIENT is provided since libmicrohttpd-0.9.54, and
MHD_USE_PEDANTIC_CHECKS will be deprecated in future.
This makes support both option.
This moves pretty much all uses of getpid() over to getpid_raw(). I
didn't specifically check whether the optimization is worth it for each
replacement, but in order to keep things simple and systematic I
switched over everything at once.
We are going to add this child as a source to our event loop so we don't
want to block when reading data from it as this will prevent us from
processing other events. Specifically this will block the signalfds
which means if we are waiting for data from curl we won't handle SIGTERM
or SIGINT until we happen to get more data.
errno value is not protected (it is undefined after this function returns).
Various mhd_* functions are not documented to protect errno, so this could not
guaranteed anyway.
Throughout the tree there's spurious use of spaces separating ++ and --
operators from their respective operands. Make ++ and -- operator
consistent with the majority of existing uses; discard the spaces.
When we rotate journals, we must set offline and close the current one,
but don't generally need to wait for this to complete.
Instead, we'll initiate an asynchronous offline via
journal_file_set_offline(oldfile, false), and add the file to a
per-server set of deferred closes to be closed later when they
won't block.
There's one complication however; journal_file_open() via
journal_file_verify_header() assumes that any writable journal in the
online state is the product of an unclean shutdown or other form of
corruption.
Thus there's a need for journal_file_open() to be aware of deferred
closes and synchronize with their completion when opening preexisting
journals for writing. To facilitate this the deferred closes set is
supplied to the journal_file_open() function where the deferred closes
may be closed synchronously before verifying the header in such
circumstances.
Set the MHD_OPTION_CONNECTION_MEMORY_LIMIT to 128KB. The precious value was DATA_SIZE_MAX, which was defined as 1024*1024*768. This caused journal-remote to allocate 756MB for each journal-upload connection, thus exhausting the available memory.
This commit fixes the following broken --getter option:
when systemd-journal-remote is called with --getter option,
it causes the error meesage "Zero sources specified" and
the getter command will not be called.
When --url option is specified, e.g. --url='http://some.host:19531/entries'
retrieved remote journal entries will be stored to
/var/log/journal/remote/remote-some.host.journal
Currently, --url option supports the only form like http(s)://some.host:19531.
This commit adds support to call systemd-journal-remote as follwos:
systemd-journal-remote --url='http://some.host:19531'
systemd-journal-remote --url='http://some.host:19531/'
systemd-journal-remote --url='http://some.host:19531/entries'
systemd-journal-remote --url='http://some.host:19531/entries?boot&follow'
The first three example result the same and retrieve all entries.
The last example retrieves only current boot entries and wait new events.
core: Add flexible way to provide socket type
the socket type should be a diffrent argumet
in make_socket_fd . In this way we can set the socket
type like SOCK_STREAM SOCK_DGRAM in the address.
journal-remote: modify make_socket_fd
src/journal-remote/journal-remote.c:590:13: warning: Value MHD_HTTP_METHOD_NOT_ACCEPTABLE is deprecated, use MHD_HTTP_NOT_ACCEPTABLE
return mhd_respond(connection, MHD_HTTP_METHOD_NOT_ACCEPTABLE,
^
The new define was added in 0.9.38. Instead of requiring the new
libmicrohttpd version, provide the fallback, it is trivial.
While journal received remotely can be sealed, it can only be done
on the command line using --seal, so for consistency, we will
also permit to set it in the configuration file.
When constructing the journal filename to store logs from a remote host, remove the port of the tcp connection, as the port will change with every reboot/connection loss between sender/reveiver machines. Having the port in the filename will cause a new journal file to be created for every reboot or connection loss.
For the implementation, a new argument "bool include_port" is added to the getpeername_pretty() function. This is passed to the sockaddr_pretty() function. The value of the include_port argument is set to true in all calls of getpeername_pretty(), except for 2 calls in journal-remote.c, where it is set to false.
The macro is generically useful for putting together search paths, hence
let's make it truly generic, by dropping the implicit ".d" appending it
does, and leave that to the caller. Also rename it from
CONF_DIRS_NULSTR() to CONF_PATHS_NULSTR(), since it's not strictly about
dirs that way, but any kind of file system path.
Also, mark CONF_DIR_SPLIT_USR() as internal macro by renaming it to
_CONF_PATHS_SPLIT_USR() so that the leading underscore indicates that
it's internal.
Explicitly set MHD_OPTION_CONNECTION_MEMORY_LIMIT to a larger value,
when setting up microhttpd, to give more memory per HTTP(S) connection.
This way systemd-journal-remote can now prevent microhttpd from failing
in creating response headers with messages like "Not enough memory for
write", especially when lots of HTTPS requests arrive. That's precisely
because MHD_OPTION_CONNECTION_MEMORY_LIMIT in libmicrohttpd defaults to
32768, which is in practice insufficient in this case.
See also https://gnunet.org/bugs/view.php?id=4007 for more details.
Fixes: https://github.com/coreos/bugs/issues/927
