In commit 9479b6d5e0 we updated all of
the collation data to harmonize with the new version of ISO 14651
which is derived from Unicode 9.0.0. This collation update brought
with it some changes to locales which were not desirable by some
users, in particular it altered the meaning of the
locale-dependent-range regular expression, namely [a-z] and [A-Z], and
for en_US it caused uppercase letters to be matched by [a-z] for the
first time. The matching of uppercase letters by [a-z] is something
which is already known to users of other locales which have this
property, but this change could cause significant problems to en_US
and other similar locales that had never had this change before.
Whether this behaviour is desirable or not is contentious and GNU Awk
has this to say on the topic:
https://www.gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html
While the POSIX standard also has this further to say: "RE Bracket
Expression":
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap09.html
"The current standard leaves unspecified the behavior of a range
expression outside the POSIX locale. ... As noted above, efforts were
made to resolve the differences, but no solution has been found that
would be specific enough to allow for portable software while not
invalidating existing implementations."
In glibc we implement the requirement of ISO POSIX-2:1993 and use
collation element order (CEO) to construct the range expression, the
API internally is __collseq_table_lookup(). The fact that we use CEO
and also have 4-level weights on each collation rule means that we can
in practice reorder the collation rules in iso14651_t1_common (the new
data) to provide consistent range expression resolution *and* the
weights should maintain the expected total order. Therefore this
patch does three things:
* Reorder the collation rules for the LATIN script in
iso14651_t1_common to deinterlace uppercase and lowercase letters in
the collation element orders.
* Adds new test data en_US.UTF-8.in for sort-test.sh which exercises
strcoll* and strxfrm* and ensures the ISO 14651 collation remains.
* Add back tests to tst-fnmatch.input and tst-regexloc.c which
exercise that [a-z] does not match A or Z.
The reordering of the ISO 14651 data is done in an entirely mechanical
fashion using the following program attached to the bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28
It is up for discussion if the iso14651_t1_common data should be
refined further to have 3 very tight collation element ranges that
include only a-z, A-Z, and 0-9, which would implement the solution
sought after in:
https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c12
and implemented here:
https://www.sourceware.org/ml/libc-alpha/2018-07/msg00854.html
No regressions on x86_64.
Verified that removal of the iso14651_t1_common change causes tst-fnmatch
to regress with:
422: fnmatch ("[a-z]", "A", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
...
425: fnmatch ("[A-Z]", "z", 0) = 0 (FAIL, expected FNM_NOMATCH) ***
POSIX locale descriptions
and
POSIX character set descriptions
Ulrich Drepper Time-stamp: <2004/11/27 13:06:54 drepper>
drepper@redhat.com
This directory contains the data needed to build the locale data files
to use the internationalization features of the GNU libc.
POSIX.2 describes the `localedef' utility which is part of the GNU libc.
You need this program to "compile" the locale description in a form
suitable for fast access by the GNU libc functions. Any compilation is
based on a given character set.
Once you run `make install' for the GNU libc the data files are
automatically installed in the right place, ready for use by the
`localedef' program.
To compile the locale data files you simply have to decide which locale
(based on the location and the language) and which character set you
use. E.g., French speaking Canadians would use the locale `fr_CA' and
the character set `ISO_8859-1,1987'. Calling `localedef' to get the
desired data should happen like this:
localedef -i fr_CA -f ISO-8859-1 fr_CA
This will place the 6 output files in the appropriate directory where
the GNU libc functions can find them. Please note that you need
permission to write to this directory ($(prefix)/share/locale, where
$(prefix) is the value you specified while configuring GNU libc). If
you do not have the necessary permissions, you can write the files into an
arbitrary directory by giving a path including a '/' character instead
of `fr_CA'. E.g., to put the new files in a subdirectory of the
current directory simply use
localedef -i fr_CA -f ISO-8859-1 ./fr_CA
How to use these data files is described in the GNU libc manual,
especially in the section describing the `setlocale' function.
All problems should be reported using
https://sourceware.org/bugzilla/
One more note: the `POSIX' locale definition is not meant to be used
as an input file for `localedef'. It is rather there to show the
values with are built in the libc binaries as default values when no
legal locale is found or the "C" or "POSIX" locale is selected.
The collation test suite
########################
This package also contains a (beginning of a) test suite for the
collation functions in the GNU libc. The files are provided sorted.
The test program shuffles the lines and sort them afterwards.
Some of the files are provided in 8bit form, i.e., not only ASCII
characters. So the tools you use to process the files should be 8bit
clean.
To run the test program the appropriate locale information must be
installed. Therefore the localedef program is used to generate this
data used the locale and charmap description files contained here.
Since we cannot run the localedef program in case of cross-compilation
no tests at all are performed.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local Variables:
mode:text
eval:(load-library "time-stamp")
eval:(make-local-variable 'write-file-hooks)
eval:(add-hook 'write-file-hooks 'time-stamp)
eval:(setq time-stamp-format '(time-stamp-yyyy/mm/dd time-stamp-hh:mm:ss user-login-name))
End:
7cd7d36f1f