https://sourceware.org/glibc/wiki/Proposals/GroupMerging
== Justification ==
It is common today for users to rely on centrally-managed user stores for
handling their user accounts. However, much software existing today does
not have an innate understanding of such accounts. Instead, they commonly
rely on membership in known groups for managing access-control (for
example the "wheel" group on Fedora and RHEL systems or the "adm" group
on Debian-derived systems). In the present incarnation of nsswitch, the
only way to have such groups managed by a remote user store such as
FreeIPA or Active Directory would be to manually remove the groups from
/etc/group on the clients so that nsswitch would then move past nss_files
and into the SSSD, nss-ldap or other remote user database.
== Solution ==
With this patch, a new action is introduced for nsswitch:
NSS_ACTION_MERGE. To take advantage of it, one will add [SUCCESS=merge]
between two database entries in the nsswitch.conf file. When a group is
located in the first of the two group entries, processing will continue
on to the next one. If the group is also found in the next entry (and the
group name and GID are an exact match), the member list of the second
entry will be added to the group object to be returned.
== Implementation ==
After each DL_LOOKUP_FN() returns, the next action is checked. If the
function returned NSS_STATUS_SUCCESS and the next action is
NSS_ACTION_MERGE, a copy of the result buffer is saved for the next pass
through the loop. If on this next pass through the loop the database
returns another instance of a group matching both the group name and GID,
the member list is added to the previous list and it is returned as a
single object. If the following database does not contain the same group,
then the original is copied back into the destination buffer.
This patch implements merge functionality only for the group database.
For other databases, there is a default implementation that will return
the EINVAL errno if a merge is requested. The merge functionality can be
implemented for other databases at a later time if such is needed. Each
database must provide a unique implementation of the deep-copy and merge
functions.
If [SUCCESS=merge] is present in nsswitch.conf for a glibc version that
does not support it, glibc will process results up until that operation,
at which time it will return results if it has found them or else will
simply return an error. In practical terms, this ends up behaving like
the remainder of the nsswitch.conf line does not exist.
== Iterators ==
This feature does not modify the iterator functionality from its current
behavior. If getgrnam() or getgrgid() is called, glibc will iterate
through all entries in the `group` line in nsswitch.conf and display the
list of members without attempting to merge them. This is consistent with
the behavior of nss_files where if two separate lines are specified for
the same group in /etc/groups, getgrnam()/getgrgid() will display both.
Clients are already expected to handle this gracefully.
== No Premature Optimizations ==
The following is a list of places that might be eligible for
optimization, but were not overengineered for this initial contribution:
* Any situation where a merge may occur will result in one malloc() of
the same size as the input buffer.
* Any situation where a merge does occur will result in a second
malloc() to hold the list of pointers to member name strings.
* The list of members is simply concatenated together and is not tested
for uniqueness (which is identical to the behavior for nss_files,
which will simply return identical values if they both exist on the
line in the file. This could potentially be optimized to reduce space
usage in the buffer, but it is both complex and computationally
expensive to do so.
== Testing ==
I performed testing by running the getent utility against my newly-built
glibc and configuring /etc/nsswitch.conf with the following entry:
group: group: files [SUCCESS=merge] sss
In /etc/group I included the line:
wheel❌10:sgallagh
I then configured my local SSSD using the id_provider=local to respond
with:
wheel:*:10:localuser,localuser2
I then ran `getent group wheel` against the newly-built glibc in
multiple situations and received the expected output as described
above:
* When SSSD was running.
* When SSSD was configured in nsswitch.conf but the daemon was not
running.
* When SSSD was configured in nsswitch.conf but nss_sss.so.2 was not
installed on the system.
* When the order of 'sss' and 'files' was reversed.
* All of the above with the [SUCCESS=merge] removed (to ensure no
regressions).
* All of the above with `getent group 10`.
* All of the above with `getent group` with and without
`enumerate=true` set in SSSD.
* All of the above with and without nscd enabled on the system.
The fmemopen implementation does not account the file position correctly in
append mode. The following example shows the failure:
===
int main ()
{
char buf[10] = "test";
FILE *fp = fmemopen (buf, 10, "a+");
fseek (fp, 0, SEEK_SET);
int gr;
if ((gr = getc (fp)) != 't' ||
(gr = getc (fp)) != 'e' ||
(gr = getc (fp)) != 's' ||
(gr = getc (fp)) != 't' ||
(gr = getc (fp)) != EOF)
{
printf ("%s: getc failed returned %i\n", __FUNCTION__, gr);
return 1;
}
return 0;
}
===
This is due both how read and write operation update the buffer position,
taking in consideration buffer lenght instead of maximum position defined
by the open mode. This patch fixes it and also fixes fseek not returning
EINVAL for invalid whence modes.
Tested on x86_64 and i686.
[BZ #20012]
* libio/fmemopen.c (fmemopen_read): Use buffer maximum position, not
length to calculate the buffer to read.
(fmemopen_write): Set the buffer position based on bytes written.
(fmemopen_seek): Return EINVAL for invalid whence modes.
As discussed in libc-alpha [1] current clone with CLONE_VM (without
CLONE_THREAD set) will reset the pthread pid/tid fields to -1. The
issue is since memory is shared between the parent and child it will
clobber parent's cached pid/tid leading to internal inconsistencies
if the value is not restored.
And even it is restored it may lead to racy conditions when between
set/restore a thread might invoke pthread function that validate the
pthread with INVALID_TD_P/INVALID_NOT_TERMINATED_TD_P and thus get
wrong results.
As stated in BZ19957, previously reports of this behaviour was close
with EWONTFIX due the fact usage of clone outside glibc is tricky
since glibc requires consistent internal pthread, while using clone
directly may not provide it. However since now posix_spawn uses
clone (CLONE_VM) to fixes various issues related to previous vfork
usage this issue requires fixing.
The vfork implementation also does something similar, but instead
it negates and restores only the *pid* field and functions that
might access its value know to handle such case (getpid, raise
and pthread ones that uses INVALID_TD_P/INVALID_NOT_TERMINATED_TD_P
macros that check only *tid* field). Also vfork does not call
__clone directly, instead calling either __NR_vfork or __NR_clone
directly.
So this patch removes this clone behavior by avoiding setting
the pthread pid/tid field for CLONE_VM. There is no need to
check for CLONE_THREAD, since the minimum supported kernel in all
architecture implies that CLONE_VM must be used with CLONE_THREAD,
otherwise clone returns EINVAL.
Instead of current approach of:
int clone(int (*fn)(void *), void *child_stack, int flags, ...)
[...]
if (flags & CLONE_THREAD)
goto do_syscall;
pid_t new_value;
if (flags & CLONE_VM)
new_value = -1;
else
new_value = getpid ();
THREAD_SETMEM (THREAD_SELF, pid, new_value);
THREAD_SETMEM (THREAD_SELF, tid, new_value);
do_syscall:
[...]
The new approach uses:
int clone(int (*fn)(void *), void *child_stack, int flags, ...)
[...]
if (flags & CLONE_VM)
goto do_syscall;
pid_t new_value = getpid ();
THREAD_SETMEM (THREAD_SELF, pid, new_value);
THREAD_SETMEM (THREAD_SELF, tid, new_value);
do_syscall:
[...]
It also removes the linux tst-getpid2.c test which expects the previous
behavior and instead add another clone test.
Tested on x86_64, i686, x32, powerpc64le, aarch64, armhf, s390, and
s390x. I also did limited check on mips32 and sparc64 (using the new
added test).
I also got reviews from both m68k, hppa, and tile. So I presume for
these architecture the patch works.
The fixes for alpha, microblaze, sh, ia64, and nio2 have not been
tested.
[1] https://sourceware.org/ml/libc-alpha/2016-04/msg00307.html
* sysdeps/unix/sysv/linux/Makefile [$(subdir) == nptl] (test): Remove
tst-getpid2.
(test): Add tst-clone2.
* sysdeps/unix/sysv/linux/tst-clone2.c: New file.
* sysdeps/unix/sysv/linux/aarch64/clone.S (__clone): Do not change
pid/tid fields for CLONE_VM.
* sysdeps/unix/sysv/linux/arm/clone.S: Likewise.
* sysdeps/unix/sysv/linux/i386/clone.S: Likewise.
* sysdeps/unix/sysv/linux/mips/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/clone.S: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
* sysdeps/unix/sysv/linux/tst-getpid2.c: Remove file.
Call __memset_power8 to pad, with zeros, the remaining bytes in the
dest string on __strncpy_power8 and __stpncpy_power8. This improves
performance when n is larger than the input string, giving ~30% gain for
larger strings without impacting much shorter strings.
When converting a struct hostent response to struct gaih_addrtuple, the
gethosts macro (which is called from gaih_inet) used alloca, without
malloc fallback for large responses. This commit changes this code to
use calloc unconditionally.
This commit also consolidated a second hostent-to-gaih_addrtuple
conversion loop (in gaih_inet) to use the new conversion function.
Previously, application code had to set up the d_namlen member if
the target supported it, involving conditional compilation. After
this change, glob will use the length of the string in d_name instead
of d_namlen to determine the file name length. All glibc targets
provide the d_type and d_ino members, and setting them as needed for
gl_readdir is straightforward.
Changing the behavior with regards to d_ino is left to a future
cleanup.
stdio.h declares flockfile, ftrylockfile, funlockfile, getc_unlocked,
getchar_unlocked, putc_unlocked and putchar_unlocked if __USE_POSIX,
with comments "These are defined in POSIX.1:1996.". But __USE_POSIX
is actually POSIX.1:1990, and these functions should not be declared
for 1990 / 1992 / 1993 POSIX, XPG3 or XPG4. This patch fixes stdio.h
to use __USE_POSIX199506 instead for those conditionals, as that is
the correct conditional for the version of POSIX that introduced
threads, and with threads those functions.
Tested for x86_64 and x86 (testsuite, and that installed shared
libraries are unchanged by the patch).
[BZ #20014]
* libio/stdio.h (getc_unlocked): Declare if [__USE_POSIX199506],
not [__USE_POSIX].
(getchar_unlocked): Likewise.
(putc_unlocked): Likewise.
(putchar_unlocked): Likewise.
(flockfile): Likewise.
(ftrylockfile): Likewise.
(funlockfile): Likewise.
* conform/Makefile (test-xfail-XPG3/stdio.h/conform): Remove
variable.
(test-xfail-XPG4/stdio.h/conform): Likewise.
The conformtest expectations for langinfo.h fail to include the YESSTR
and NOSTR constants that were present in UNIX98 and earlier XPG
standards. This patch adds those expectations, so fixing three
XFAILs.
Tested for x86_64 and x86.
* conform/data/langinfo.h-data [XPG3 || XPG4 || UNIX98] (YESSTR):
Expect constant.
[XPG3 || XPG4 || UNIX98] (NOSTR): Likewise.
* conform/Makefile (test-xfail-XPG3/langinfo.h/conform): Remove
variable.
(test-xfail-XPG4/langinfo.h/conform): Likewise.
(test-xfail-UNIX98/langinfo.h/conform): Likewise.
Similar to my previous fix for XOPEN2K
<https://sourceware.org/ml/libc-alpha/2016-04/msg00631.html>, now that
bugs in the conformtest expectations for stdio.h for UNIX98 have been
corrected, that case too fails because fseeko and ftello are now
correctly expected, but off_t is not defined. As in that fix, it
seems appropriate to define off_t in stdio.h for this standard as
well, and this patch does so.
Tested for x86_64 and x86 (testsuite, and that installed shared
libraries are unchanged by the patch).
* libio/stdio.h (off_t): Also define if [__USE_UNIX98].
[__USE_LARGEFILE64] (off64_t): Likewise.
* conform/Makefile (test-xfail-UNIX98/stdio.h/conform): Remove
variable.
The conform/ test of stdio.h wrongly does not expect fdopen for XPG3
and XPG4. fdopen is in those standards; this patch corrects the
expectations.
Tested for x86_64 and x86.
* conform/data/stdio.h-data (fdopen): Expect also for
[XPG3 || XPG4].
The conform/ test of stdio.h for UNIX98 fails with surious namespace
errors for functions that are correctly declared for that standard.
This patch fixes the expectations to expect those functions also for
UNIX98. (This does not by itself fix the XFAIL of that test, and is
not based a full review of the header expectations so there could
still be other bugs in the expectations for this header for UNIX98.)
Tested for x86_64 and x86.
* conform/data/stdio.h-data (flockfile): Also expect for [UNIX98].
(fseeko): Likewise.
(ftello): Likewise.
(ftrylockfile): Likewise.
(funlockfile): Likewise.
(getc_unlocked): Likewise.
(getchar_unlocked): Likewise.
(putc_unlocked): Likewise.
(putchar_unlocked): Likewise.
Previously, we allocated room in the result space before the check,
leaving uninitialized data there in case the check failed.
This also consolidates the behavior between single (A or AAAA) and
dual (A and AAAA in parallel) queries. Single queries checked
the record length against the QTYPE, not the RRTYPE.
The conformtest expectations for signal.h have various declarations
that are expected for POSIX (1996) and all later standards, except,
wrongly, for XOPEN2K. This shows up as failures of tests for two
other headers, which are allowed to make visible symbols from
signal.h, because of an incorrect namespace failure for sigval
(required in signal.h in XOPEN2K, so should be allowed for those other
headers); signal.h tests for various standards fail anyway because of
other problems in the header. This patch fixes the incorrect
expectations and removes the two XFAILs that this fixes.
Tested for x86_64 and x86.
* conform/data/signal.h-data (union sigval): Expect also if
[XOPEN2K].
(struct sigevent): Likewise.
(SIGEV_NONE): Likewise.
(SIGEV_SIGNAL): Likewise.
(SIGEV_THREAD): Likewise.
(SIGRTMIN): Likewise.
(SIGRTMAX): Likewise.
* conform/Makefile (test-xfail-XOPEN2K/aio.h/conform): Remove
variable.
(test-xfail-XOPEN2K/mqueue.h/conform): Likewise.
In various error scenarios (for example, if the server closes the
TCP connection before sending the full response), send_vc can return
without resetting the *resplen2 value. This can pass uninitialized
or unexpected data to the caller.
this patch adds the missing SOL_IUCV socket level definition
and socket options SO_IPRMDATA_MSG, SO_MSGLIMIT, SO_MSGSIZE
which can be used with get/setsockopt().
SCM_IUCV_TRGCLS is needed to send/receive ancillary data with send/recvmsg().
The defines are copied from kernel-source:
include/net/iucv/af_iucv.h
include/linux/socket.h
Current GLIBC fmemopen fails with a simple testcase:
char buffer[500] = "x";
FILE *stream;
stream = fmemopen(buffer, 500, "r+");
fwrite("fish",sizeof(char),5,stream);
printf("pos-1:%ld\n",ftell(stream));
fflush(stream);
printf("pos-2:%ld\n",ftell(stream));
It returns:
pos-1:5
pos-2:0
Where it should return:
pos-1:5
pos-2:5
This is due the internal write function does not correctly update the internal
object position state and then the seek operation returns a wrong value. This
patch fixes it.
It fixes both BZ #20005 and BZ #19230 (marked as duplicated). A new test is
added to check for such case.
Tested on x86_64 and i686.
* libio/fmemopen.c (fmemopen_write): Update internal position after
write.
* stdio-common/Makefile (tests): Add tst-fmemopen4.c.
* stdio-common/tst-fmemopen4.c: New file..
langinfo.h declares nl_langinfo_l if __USE_XOPEN2K. But this function
was new in the 2008 edition of POSIX. This patch fixes the condition
accordingly.
Tested for x86_64 and x86 (testsuite, and that installed shared
libraries are unchanged by the patch).
[BZ #19996]
* locale/langinfo.h (nl_langinfo_l): Declare if [__USE_XOPEN2K8],
not [__USE_XOPEN2K].
* conform/Makefile (test-xfail-XOPEN2K/langinfo.h/conform): Remove
variable.
The conform/ test expectations for stdarg.h were wrongly missing an
expectation of va_copy for XOPEN2K (based on C99, so including that
macro). This patch fixes this.
Tested for x86_64 and x86.
* conform/data/stdarg.h-data [XOPEN2K] (va_copy): Require macro.
* conform/Makefile (test-xfail-XOPEN2K/stdarg.h/conform): Remove
variable.
The header conformance test for stdio.h for XOPEN2K fails because the
header does not define the off_t type, used in the expected
declarations for fseeko and ftello.
The absence of this type is not actually strictly a bug (hence no bug
report being filed in Bugzilla), since POSIX didn't require the type
to be declared in this header until the 2008 edition. However, the
glibc convention in such cases - where the type falls under the
general *_t POSIX reservation, and so it's OK to define it for all
POSIX versions - is to make the headers self-contained in this regard
even for the older POSIX versions not requiring the type to be defined
despite including other declarations depending on the type. Thus,
this patch adjusts the condition in the header and removes the XFAIL
(rather than adapting the expectation to work when the functions are
declared using __off_t without off_t being defined).
Tested for x86_64 and x86 (testsuite, and that installed stripped
shared libraries are unchanged by the patch).
* libio/stdio.h (off_t): Define if [__USE_XOPEN2K], not
[__USE_XOPEN2K8].
[__USE_LARGEFILE64] (off64_t): Likewise.
* conform/Makefile (test-xfail-XOPEN2K/stdio.h/conform): Remove
variable.
stdio.h declares cuserid if __USE_XOPEN. But this was removed in the
2001 edition of POSIX.
The #endif comment "Use X/Open, but not issue 6." reflects the correct
logic, but does not correspond to the #ifdef. The use of a correct
libc-hacker. The online archives for libc-hacker in August 2000 are
broken, but the messages can be found in the qmail archives in
/sourceware1/qmail/lists-sourceware/libc-hacker/archive/26 if you have
shell access to sourceware.
The issue showed up in August 2000 because of a warning about a
non-prototype definition in sysdeps/posix/cuserid.c when there was no
previous prototype declaration. Since we've now eliminated
non-prototype function definitions, that issue does not apply. The
other points from that discussion were about whether it should be
included in _GNU_SOURCE; whether _GNU_SOURCE should include
"everything"; whether deprecated interfaces such as this should be
excluded from it; and whether, even given exclusion of deprecated
interfaces, it should apply for deprecations in a version of POSIX
that at that time had not been released.
This patch follows the more conservative approach to a fix of keeping
the interface in _GNU_SOURCE. That matches how L_cuserid is handled.
I think there is a strong case for eliminating this interface from
_GNU_SOURCE (but this may not automatically be the case for every
interface removed in newer POSIX versions), but then L_cuserid should
also be removed from _GNU_SOURCE (in stdio-common/stdio_lim.h.in) at
the same time.
Tested for x86_64 and x86 (testsuite, and that installed shared
libraries are unchanged by the patch).
[BZ #19989]
* libio/stdio.h (cuserid): Do not declare if
[__USE_XOPEN2K && !__USE_GNU].
* conform/Makefile (test-xfail-XOPEN2K8/stdio.h/conform): Remove
variable.
RFC2292 macros were obsoleted by RFC3542, and should not be exposed
any more. Notably since IPV6_PKTINFO has been reintroduced with a
completely different API.
* bits/in.h (IPV6_PKTINFO): Rename to IPV6_2292PKTINFO.
(IPV6_HOPOPTS): Rename to IPV6_2292HOPOPTS.
(IPV6_DSTOPTS): Rename to IPV6_2292DSTOPTS.
(IPV6_RTHDR): Rename to IPV6_2292RTHDR.
(IPV6_PKTOPTIONS): Rename to IPV6_2292PKTOPTIONS.
(IPV6_HOPLIMIT): Rename to IPV6_2292HOPLIMIT.
(IPV6_RECVPKTINFO): New macro.
(IPV6_PKTINFO): New macro.
__libc_memalign in ld.so allocates one page at a time and tries to
optimize consecutive __libc_memalign calls by hoping that the next
mmap is after the current memory allocation.
However, the kernel hands out mmap addresses in top-down order, so
this optimization in practice never happens, with the result that we
have more mmap calls and waste a bunch of space for each __libc_memalign.
This change makes __libc_memalign to mmap one page extra. Worst case,
the kernel never puts a backing page behind it, but best case it allows
__libc_memalign to operate much much better. For elf/tst-align --direct,
it reduces number of mmap calls from 12 to 9.
* elf/dl-minimal.c (__libc_memalign): Mmap one extra page.
The ISO 14652/30112 specs say the defaults for the week keyword are:
7, 19971130, 7
The localedef has been using those defaults for the first two, but
0 for the last one.
This patch optimizes strcasestr function for power >= 8 systems. The average
improvement of this optimization is ~40% and compares 16 bytes at a time
using vector instructions. This patch is tested on powerpc64 and powerpc64le.
This patch adds full support for cross-building benchmarks. Some
benchmarks like those that need locales to be generated cannot be
built and are hence skipped for cross builds.
Tested by cross building for aarch64 on x86_64 and then running the
generated benchmark on aarch64.
* benchtests/Makefile (wcsmbs-benchset): Include only for
native builds and runs.
(LOCALES): Likewise.
(bench-build): Build timing-type here instead of the bench
target. Generate locale only for native builds.
* benchtests/README: Add note for cross-building.
