Support a new directive 'codepoint_collation' in the LC_COLLATE
section of a locale source file. This new directive causes all
collation rules to be dropped and instead STRCMP (strcmp or
wcscmp) is used for collation of the input character set. This
is required to allow for a C.UTF-8 that contains zero collation
rules (minimal size) and sorts using code point sorting.
To date the only implementation of a locale with zero collation
rules is the C/POSIX locale. The C/POSIX locale provides
identity tables for _NL_COLLATE_COLLSEQMB and
_NL_COLLATE_COLLSEQWC that map to ASCII even though it has zero
rules. This has lead to existing fnmatch, regexec, and regcomp
implementations that require these tables. It is not correct
to use these tables when nrules == 0, but the conservative fix
is to provide these tables when nrules == 0. This assures that
existing static applications using a new C.UTF-8 locale with
'codepoint_collation' at least have functional range expressions
with ASCII e.g. [0-9] or [a-z]. Such static applications would
not have the fixes to fnmatch, regexec and regcomp that avoid
the use of the tables when nrules == 0. Future fixes to fnmatch,
regexec, and regcomp would allow range expressions to use the
full set of code points for such ranges.
Tested on x86_64 and i686 without regression.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
We stopped adding "Contributed by" or similar lines in sources in 2012
in favour of git logs and keeping the Contributors section of the
glibc manual up to date. Removing these lines makes the license
header a bit more consistent across files and also removes the
possibility of error in attribution when license blocks or files are
copied across since the contributed-by lines don't actually reflect
reality in those cases.
Move all "Contributed by" and similar lines (Written by, Test by,
etc.) into a new file CONTRIBUTED-BY to retain record of these
contributions. These contributors are also mentioned in
manual/contrib.texi, so we just maintain this additional record as a
courtesy to the earlier developers.
The following scripts were used to filter a list of files to edit in
place and to clean up the CONTRIBUTED-BY file respectively. These
were not added to the glibc sources because they're not expected to be
of any use in future given that this is a one time task:
https://gist.github.com/siddhesh/b5ecac94eabfd72ed2916d6d8157e7dchttps://gist.github.com/siddhesh/15ea1f5e435ace9774f485030695ee02
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
For the legacy ABI with supports 32-bit time_t it calls the 64-bit
time directly, since the LFS symbols calls the 64-bit time_t ones
internally.
Checked on i686-linux-gnu and x86_64-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Fix trivial leak identified by coverity. The program runs to exit and
the leak doesn't grow, but it's just cleaner to free the allocated
memory.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
During ellipsis processing the collation cursor was not correctly
moved to the end of the ellipsis after processing.
The code inserted the new entry after the cursor, but before the
real end of the ellipsis:
[cursor]
... element_t <-> element_t <-> element_t <-> element_t
"<U0000>" "<U0001>" "<U007F>"
startp endp
At the end of the function we have:
[cursor]
... element_t <-> element_t <-> element_t
"<U007E>" "<U007F>"
endp
The cursor should be pointing at endp, the last element in the
doubly-linked list, otherwise when execution returns to the
caller we will start inserting the next line after <U007E>.
Subsequent operations end up unlinking the ellipsis end entry or
just leaving it in the list dangling from the end. This kind of
dangling is immediately visible in C.UTF-8 with the following
sorting from strcoll:
<U0010FFFF>
<U0000FFFF>
<U000007FF>
<U0000007F>
With the cursor correctly adjusted the end entry is correctly given
the right location and thus the right weight.
Retested and no regressions on x86_64 and i686.
Co-authored-by: Carlos O'Donell <carlos@redhat.com>
These symbol usages are not definitions, so compat_symbol_reference is
more appropriate than compat_symbol. compat_symbol_reference is also
safe to emit multiple times (in case the inline assembly is
duplicated; this is possible because it is nested in a function).
compat_symbol does not necessarily have this property because it is
intended to provide a symbol definition.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
I've updated copyright dates in glibc for 2021. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files. As well as the usual annual
updates, mainly dates in --version output (minus csu/version.c which
previously had to be handled manually but is now successfully updated
by update-copyrights), there is a small change to the copyright notice
in NEWS which should let NEWS get updated automatically next year.
Please remember to include 2021 in the dates for any new files added
in future (which means updating any existing uncommitted patches you
have that add new files to use the new copyright dates in them).
I used these shell commands:
../glibc/scripts/update-copyrights $PWD/../gnulib/build-aux/update-copyright
(cd ../glibc && git commit -am"[this commit message]")
and then ignored the output, which consisted lines saying "FOO: warning:
copyright statement not found" for each of 6694 files FOO.
I then removed trailing white space from benchtests/bench-pthread-locks.c
and iconvdata/tst-iconv-big5-hkscs-to-2ucs4.c, to work around this
diagnostic from Savannah:
remote: *** pre-commit check failed ...
remote: *** error: lines with trailing whitespace found
remote: error: hook declined to update refs/heads/master
Among the warnings causing a glibc build with GCC 11 to fail is one
for a call new_composite_name in setlocale.c. The newnames argument
is declared as an array with __LC_LAST elements, but when the category
argument is not LC_ALL, it actually only has one element. Since the
number of elements depends on the first argument to the function, it
seems clearer to declare the argument as a pointer.
Tested with build-many-glibcs.py for arm-linux-gnueabi, where this
allows the build to get further.
Reviewed-by: DJ Delorie <dj@redhat.com>
It replaces the internal usage of __{f,l}xstat{at}{64} with the
__{f,l}stat{at}{64}. It should not change the generate code since
sys/stat.h explicit defines redirections to internal calls back to
xstat* symbols.
Checked with a build for all affected ABIs. I also check on
x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Lukasz Majewski <lukma@denx.de>
ISO-8859-8-based locales will need this transliteration if the locale
files contain these Unicode characters.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
We automatically create $(complocaledir) in the testroot.root
now and so we don't need to create it in the test.
Tested on x86_64.
Reviewed-by: DJ Delorie <dj@redhat.com>
On the current AArch64 buildbot the default 20s timeout is not
enough to run this test, it seems
make test t=locale/tst-localedef-path-norm
takes about 25s, so i increased the timeout to 30s.
During testing of localedef running in a minimal container
there were several error cases which were hard to diagnose
since they appeared as strerror (errno) values printed by the
higher level functions. This change adds three new verbose
messages for potential failure paths. The new messages give
the user the opportunity to use -v and display additional
information about why localedef might be failing. I found
these messages useful myself while writing a localedef
container test for --no-hard-links.
Since the changes cleanup the code that handle codeset
normalization we add tst-localedef-path-norm which contains
many sub-tests to verify the correct expected normalization of
codeset strings both when installing to default paths (the
only time normalization is enabled) and installing to absolute
paths. During the refactoring I created at least one
buffer-overflow which valgrind caught, but these tests did not
catch because the exec in the container had a very clean heap
with zero-initialized memory. However, between valgrind and
the tests the results are clean.
The new tst-localedef-path-norm passes without regression on
x86_64.
Change-Id: I28b9f680711ff00252a2cb15625b774cc58ecb9d
The locale/tst-locale-locpath test unsets LANG, then runs a test with
test_wrapper_env and expects LANG to remain unset for that test. This
does not work for cross-testing with cross-test-ssh.sh when sshd (on
the system specified as an argument to cross-test-ssh.sh) is
configured to have a default LANG setting.
The general design used in cross testing, after commit
8540f6d2a7 ("Don't require test wrappers
to preserve environment variables, use more consistent environment.",
6 June 2014), is that environment settings required by tests should be
passed explicitly to $(test-wrapper-env). This patch changes
tst-locale-locpath.sh to pass an explicit LANG= rather than expecting
"unset LANG" to be in effect for the program run under
test_wrapper_env. Note that this does slightly change the environment
in which the test is run natively (empty LANG instead of unset LANG)
but that difference does not appear relevant to what it is trying to
test.
Tested for Arm that this fixes the failure seen for that test in
cross-testing.
I've updated copyright dates in glibc for 2020. This is the patch for
the changes not generated by scripts/update-copyrights and subsequent
build / regeneration of generated files. As well as the usual annual
updates, mainly dates in --version output (minus libc.texinfo which
previously had to be handled manually but is now successfully updated
by update-copyrights), there is a fix to
sysdeps/unix/sysv/linux/powerpc/bits/termios-c_lflag.h where a typo in
the copyright notice meant it failed to be updated automatically.
Please remember to include 2020 in the dates for any new files added
in future (which means updating any existing uncommitted patches you
have that add new files to use the new copyright dates in them).
_nl_load_locale_from_archive() checks for a zero size, but
divides by both (size) and (size-2). Extend the check to
guard against a size of two or less.
Tested by manually corrupting locale-archive and running a program
that calls setlocale() with LOCPATH unset (size is typically very
large).
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This propagates the recent http->https URL changes.
Since I used gperf 3.1 to regenerate, this is also a minor
internal-to-localedef API change.
URL problem reported by Joseph Myers in:
https://www.sourceware.org/ml/libc-alpha/2019-09/msg00143.html
* locale/programs/charmap-kw.h, locale/programs/locfile-kw.h:
Regenerate with gperf 3.1.
* locale/programs/linereader.h (kw_hash_fct_t):
* locale/programs/repertoire.c (repertoiremap_hash):
2nd arg is now size_t not unsigned, for compatibility with gperf 3.1.
struct charseq used a zero-length array instead of a flexible array
member. This required a strange construct to initialize struct
charseq objects, and GCC 10 warns about that:
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
In file included from programs/repertoire.h:24,
from programs/localedef.h:32,
from programs/ld-ctype.c:35:
programs/charmap.h:63:17: note: destination object declared here
63 | unsigned char bytes[0];
| ^~~~~
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
cc1: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
programs/charmap.h:63:17: note: destination object declared here
The change makes the object physically const, but it is not expected
to be modified.
This patch adds Cyrillic to plain ASCII transliteration table according
to GOST 7.79-2000 System B standard to the C locale.
[BZ #2872]
* locale/C-translit.h.in: Add Cyrillic transliteration.
The arguments passed by Makefile are missing match with the arguments
taken by locale/tst-locale-locpath.sh. Without this patch
cross-test-ssh.sh will be called twice in a single command while
doing the make check test wish ssh test wrapper.
* locale/Makefile: Fix tst-locale-locpath arguments passing.
This patch fixes the gcc warnings seen with gcc 9 -march>=z13 on s390x:
programs/ld-ctype.c: In function ‘ctype_read’:
programs/ld-ctype.c:1392:13: error: ‘wch’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1392 | uint32_t wch;
| ^~~
programs/ld-ctype.c:1401:7: error: ‘seq’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1401 | if (seq != NULL && seq->nbytes == 1)
| ^
programs/ld-ctype.c:1391:20: note: ‘seq’ was declared here
1391 | struct charseq *seq;
| ^~~
Both seq and wch are uninitialized if get_character fails.
Thus we are now returning with an error.
ChangeLog:
* locale/programs/ld-ctype.c (charclass_symbolic_ellipsis):
Return error if get_character fails.
When -Werror=parentheses is in use, iconvconfig.c builds fail with:
iconvconfig.c: In function ‘write_output’:
iconvconfig.c:1084:34: error: suggest parentheses around ‘+’ inside ‘>>’ [-Werror=parentheses]
hash_size = next_prime (nnames + nnames >> 1);
~~~~~~~^~~~~~~~
This patch adds parentheses to the expression. Not where suggested by
the compiler warning, but where it produces the expected result, i.e.:
where it has the effect of multiplying nnames by 1.5.
Likewise for elem_size in ld-collate.c.
Tested for powerpc64le.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Two cases of "int * 1.4" may result in imprecise results, which
in at least one case resulted in i686 and x86-64 producing
different locale files. This replaced that floating point multiply
with integer operations. While the hash table margin is increased
from 40% to 50%, testing shows only 2% increase in overall size
of the locale archive.
https://bugzilla.redhat.com/show_bug.cgi?id=1311954
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This patch adds fall-through comments in some cases where -Wextra
produces implicit-fallthrough warnings.
The patch is non-exhaustive. Apart from architecture-specific code
for non-x86_64 architectures, it does not change sunrpc/xdr.c (legacy
code, probably should have such changes, but left to be dealt with
separately), or places that already had comments about the
fall-through but not matching the form expected by
-Wimplicit-fallthrough=3 (the default level with -Wextra; my
inclination is to adjust those comments to match rather than
downgrading to -Wimplicit-fallthrough=1 to allow any comment), or one
place where I thought the implicit fallthrough was not correct and so
should be handled separately as a bug fix. I think the key thing to
consider in review of this patch is whether the fall-through is indeed
intended and correct in each place where such a comment is added.
Tested for x86_64.
* elf/dl-exception.c (_dl_exception_create_format): Add
fall-through comments.
* elf/ldconfig.c (parse_conf_include): Likewise.
* elf/rtld.c (print_statistics): Likewise.
* locale/programs/charmap.c (parse_charmap): Likewise.
* misc/mntent_r.c (__getmntent_r): Likewise.
* posix/wordexp.c (parse_arith): Likewise.
(parse_backtick): Likewise.
* resolv/ns_ttl.c (ns_parse_ttl): Likewise.
* sysdeps/x86/cpu-features.c (init_cpu_features): Likewise.
* sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise.
Downstream distributions need consistent sets of hardlinks in
order for rpm to operate effectively. This means that even if
locales are built with a high level of parallelism that the
resulting files need to have consistent hardlink counts. The only
way to achieve this is with a post-install hardlink pass using a
program like 'hardlink' (shipped in Fedora).
If the downstream distro wants to post-process the hardlinks then
the time spent in localedef looking up sibling directories and
processing hardlinks is wasted effort.
To optimize the build and install pass we add a --no-hard-links
option to localedef to avoid doing the hardlink optimziation for
size.
Tested on x86_64 with 'make localedata/install-locale-files'
before and after. Without the patch we have files with 100+
hardlink counts. After the patch and running with --no-hard-links
all link counts are 1. This patch also alters the convenience
target 'make localedata/install-locale-files' to use the new
option.
Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Continuing fixes for -Os build issues shown with build-many-glibcs.py,
this patch adds uses of DIAG_* to disable -Wmaybe-uninitialized in two
more places where code inlined from strcoll / wcscoll is wrongly
diagnosed as possibly using uninitialized structure fields. (All
these warnings in different places for these functions are I think
essentially the same bug.)
Tested with build-many-glibcs.py for alpha-linux-gnu and
mips-linux-gnu that this fixes the -Os build failures for those
configurations with GCC 7.
* locale/weightwc.h (findidx): Ignore -Wmaybe-uninitialized for
-Os in two more places.
The findidx functions used in implementing strcoll / wcscoll already
use DIAG_IGNORE_Os_NEEDS_COMMENT for spurious -Wmaybe-uninitialized
warnings that appear with -Os. In building with GCC 7 for x86_64 with
-Os, I find there are additional such warnings, for the same structure
elements, which are spurious for the same reasons given in the
existing comments (and this was also reported for MIPS with GCC 5 in
bug 21313). This patch adds corresponding uses of DIAG_* in the
places that get the additional warnings.
Tested for x86_64 with -Os that this eliminates those warnings and so
allows the build to progress further.
[BZ #21313]
* locale/weight.h (findidx): Disable -Wmaybe-uninitialized for -Os
in another place.
* locale/weightwc.h (findidx): Likewise.
All the previous changes also repeated to support abbreviated
alternative month names. In most languages which have declension and
need nominative/genitive month names the abbreviated forms for both
cases are the same. An example where they do differ is May in Russian:
this name is too short to be abbreviated so even the abbreviated form
features the declension suffixes.
[BZ #10871]
* locale/C-time.c (_nl_C_LC_TIME): Add abbreviated alternative month
names, define them as the same as abbreviated month names explicitly.
* locale/categories.def (LC_TIME): Add ab_alt_mon and wide-ab_alt_mon.
* locale/langinfo.h: (_NL_ABALTMON_1, _NL_ABALTMON_2, _NL_ABALTMON_3,
_NL_ABALTMON_4, _NL_ABALTMON_5, _NL_ABALTMON_6, _NL_ABALTMON_7,
_NL_ABALTMON_8, _NL_ABALTMON_9, _NL_ABALTMON_10, _NL_ABALTMON_11,
_NL_ABALTMON_12, _NL_WABALTMON_1, _NL_WABALTMON_2, _NL_WABALTMON_3,
_NL_WABALTMON_4, _NL_WABALTMON_5, _NL_WABALTMON_6, _NL_WABALTMON_7,
_NL_WABALTMON_8, _NL_WABALTMON_9, _NL_WABALTMON_10, _NL_WABALTMON_11,
_NL_WABALTMON_12): New enum constants.
* locale/programs/ld-time.c (struct locale_time_t): Add ab_alt_mon,
wab_alt_mon, and ab_alt_mon_defined members.
(time_output): Output ab_alt_mon and wab_alt_mon members.
(time_read): Read them, initialize them as copies of abmon and wabmon
respectively if they are missing, initialize ab_alt_mon_defined.
* locale/programs/locfile-kw.gperf (ab_alt_mon): Define.
* locale/programs/locfile-kw.h: Regenerate.
* locale/programs/locfile-token.h (tok_ab_alt_mon): New enum constant.
* time/Makefile [$(run-built-tests) = yes] (LOCALES): Add es_ES.UTF-8
and ru_RU.UTF-8.
* time/strftime_l.c (a_altmonth, aam_len): New macros.
[!COMPILE_WIDE] (ABALTMON_1): New macro.
(__strftime_internal): Handle %Ob and %Oh formats.
* time/strptime_l.c [_LIBC] (ab_alt_month_name): New macro.
(__strptime_internal): Handle %Ob and %Oh formats.
* time/tst-strptime.c (day_tests): Add more tests to parse different
forms of month names including the new %Ob format specifier.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Some languages (Slavic, Baltic, etc.) require a genitive case of the
month name when formatting a full date (with the day number) while
they require a nominative case when referring to the month standalone.
This requirement cannot be fulfilled without providing two forms for
each month name. From now it is specified that nl_langinfo(MON_1)
series (up to MON_12) and strftime("%B") generate the month names in
the grammatical form used when the month is a part of a complete date.
If the grammatical form used when the month is named by itself is needed,
the new values nl_langinfo(ALTMON_1) (up to ALTMON_12) and
strftime("%OB") are supported. This new feature is optional so the
languages which do not need it or do not yet provide the updated
locales simply do not use it and their behaviour is unchanged.
[BZ #10871]
* locale/C-time.c (_nl_C_LC_TIME): Add alternative month names,
define them as the same as primary full month names explicitly.
* locale/categories.def (LC_TIME): Add alt_mon and wide-alt_mon.
* locale/langinfo.h (__ALTMON_1, __ALTMON_2, __ALTMON_3, __ALTMON_4,
__ALTMON_5, __ALTMON_6, __ALTMON_7, __ALTMON_8, __ALTMON_9, __ALTMON_10,
__ALTMON_11, __ALTMON_12, _NL_WALTMON_1, _NL_WALTMON_2, _NL_WALTMON_3,
_NL_WALTMON_4, _NL_WALTMON_5, _NL_WALTMON_6, _NL_WALTMON_7,
_NL_WALTMON_8, _NL_WALTMON_9, _NL_WALTMON_10, _NL_WALTMON_11,
_NL_WALTMON_12): New enum constants.
[__USE_GNU] (ALTMON_1, ALTMON_2, ALTMON_3, ALTMON_4, ALTMON_5, ALTMON_6,
ALTMON_7, ALTMON_8, ALTMON_9, ALTMON_10, ALTMON_11, ALTMON_12): New
macros.
* locale/programs/ld-time.c (struct locale_time_t): Add alt_mon,
walt_mon, and alt_mon_defined members.
(time_output): Output alt_mon and walt_mon members.
(time_read): Read them, initialize them as copies of mon and wmon
respectively if they are missing, initialize alt_mon_defined.
* locale/programs/locfile-kw.gperf (alt_mon): Define.
* locale/programs/locfile-kw.h: Regenerate.
* locale/programs/locfile-token.h (tok_alt_mon): New enum constant.
* localedata/tst-langinfo.c (map): Add tests for the new constants
ALTMON_1 .. ALTMON_12.
* time/Makefile [$(run-built-tests) = yes] (LOCALES): Add fr_FR.UTF-8
and pl_PL.UTF-8.
* time/strftime_l.c (f_altmonth): New macro.
(__strftime_internal): Handle %OB format.
* time/strptime_l.c [_LIBC] (alt_month_name): New macro.
(__strptime_internal): Handle %OB format.
* time/tst-strptime.c (day_tests): Add tests to parse different forms
of month names including the new %OB format specifier.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
