2001-11-05 09:11:26 +01:00
|
|
|
@node Character Set Handling, Locales, String and Array Utilities, Top
|
|
|
|
@c %MENU% Support for extended character sets
|
|
|
|
@chapter Character Set Handling
|
|
|
|
|
|
|
|
@ifnottex
|
|
|
|
@macro cal{text}
|
|
|
|
\text\
|
|
|
|
@end macro
|
|
|
|
@end ifnottex
|
|
|
|
|
|
|
|
Character sets used in the early days of computing had only six, seven,
|
|
|
|
or eight bits for each character: there was never a case where more than
|
|
|
|
eight bits (one byte) were used to represent a single character. The
|
|
|
|
limitations of this approach became more apparent as more people
|
|
|
|
grappled with non-Roman character sets, where not all the characters
|
|
|
|
that make up a language's character set can be represented by @math{2^8}
|
|
|
|
choices. This chapter shows the functionality that was added to the C
|
|
|
|
library to support multiple character sets.
|
|
|
|
|
|
|
|
@menu
|
|
|
|
* Extended Char Intro:: Introduction to Extended Characters.
|
|
|
|
* Charset Function Overview:: Overview about Character Handling
|
|
|
|
Functions.
|
|
|
|
* Restartable multibyte conversion:: Restartable multibyte conversion
|
|
|
|
Functions.
|
|
|
|
* Non-reentrant Conversion:: Non-reentrant Conversion Function.
|
|
|
|
* Generic Charset Conversion:: Generic Charset Conversion.
|
|
|
|
@end menu
|
|
|
|
|
|
|
|
|
|
|
|
@node Extended Char Intro
|
|
|
|
@section Introduction to Extended Characters
|
|
|
|
|
|
|
|
A variety of solutions is available to overcome the differences between
|
|
|
|
character sets with a 1:1 relation between bytes and characters and
|
|
|
|
character sets with ratios of 2:1 or 4:1. The remainder of this
|
|
|
|
section gives a few examples to help understand the design decisions
|
|
|
|
made while developing the functionality of the @w{C library}.
|
|
|
|
|
|
|
|
@cindex internal representation
|
|
|
|
A distinction we have to make right away is between internal and
|
|
|
|
external representation. @dfn{Internal representation} means the
|
|
|
|
representation used by a program while keeping the text in memory.
|
|
|
|
External representations are used when text is stored or transmitted
|
|
|
|
through some communication channel. Examples of external
|
|
|
|
representations include files waiting in a directory to be
|
|
|
|
read and parsed.
|
|
|
|
|
|
|
|
Traditionally there has been no difference between the two representations.
|
|
|
|
It was equally comfortable and useful to use the same single-byte
|
|
|
|
representation internally and externally. This comfort level decreases
|
|
|
|
with more and larger character sets.
|
|
|
|
|
|
|
|
One of the problems to overcome with the internal representation is
|
|
|
|
handling text that is externally encoded using different character
|
|
|
|
sets. Assume a program that reads two texts and compares them using
|
|
|
|
some metric. The comparison can be usefully done only if the texts are
|
|
|
|
internally kept in a common format.
|
|
|
|
|
|
|
|
@cindex wide character
|
|
|
|
For such a common format (@math{=} character set) eight bits are certainly
|
|
|
|
no longer enough. So the smallest entity will have to grow: @dfn{wide
|
|
|
|
characters} will now be used. Instead of one byte per character, two or
|
|
|
|
four will be used instead. (Three are not good to address in memory and
|
|
|
|
more than four bytes seem not to be necessary).
|
|
|
|
|
|
|
|
@cindex Unicode
|
|
|
|
@cindex ISO 10646
|
|
|
|
As shown in some other part of this manual,
|
|
|
|
@c !!! Ahem, wide char string functions are not yet covered -- drepper
|
|
|
|
a completely new family has been created of functions that can handle wide
|
|
|
|
character texts in memory. The most commonly used character sets for such
|
|
|
|
internal wide character representations are Unicode and @w{ISO 10646}
|
|
|
|
(also known as UCS for Universal Character Set). Unicode was originally
|
|
|
|
planned as a 16-bit character set; whereas, @w{ISO 10646} was designed to
|
|
|
|
be a 31-bit large code space. The two standards are practically identical.
|
|
|
|
They have the same character repertoire and code table, but Unicode specifies
|
|
|
|
added semantics. At the moment, only characters in the first @code{0x10000}
|
|
|
|
code positions (the so-called Basic Multilingual Plane, BMP) have been
|
|
|
|
assigned, but the assignment of more specialized characters outside this
|
|
|
|
16-bit space is already in progress. A number of encodings have been
|
|
|
|
defined for Unicode and @w{ISO 10646} characters:
|
|
|
|
@cindex UCS-2
|
|
|
|
@cindex UCS-4
|
|
|
|
@cindex UTF-8
|
|
|
|
@cindex UTF-16
|
|
|
|
UCS-2 is a 16-bit word that can only represent characters
|
|
|
|
from the BMP, UCS-4 is a 32-bit word than can represent any Unicode
|
|
|
|
and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where
|
|
|
|
ASCII characters are represented by ASCII bytes and non-ASCII characters
|
|
|
|
by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension
|
|
|
|
of UCS-2 in which pairs of certain UCS-2 words can be used to encode
|
|
|
|
non-BMP characters up to @code{0x10ffff}.
|
|
|
|
|
|
|
|
To represent wide characters the @code{char} type is not suitable. For
|
|
|
|
this reason the @w{ISO C} standard introduces a new type that is
|
|
|
|
designed to keep one character of a wide character string. To maintain
|
|
|
|
the similarity there is also a type corresponding to @code{int} for
|
|
|
|
those functions that take a single wide character.
|
|
|
|
|
|
|
|
@comment stddef.h
|
|
|
|
@comment ISO
|
|
|
|
@deftp {Data type} wchar_t
|
|
|
|
This data type is used as the base type for wide character strings.
|
2001-11-07 08:21:22 +01:00
|
|
|
In other words, arrays of objects of this type are the equivalent of
|
|
|
|
@code{char[]} for multibyte character strings. The type is defined in
|
2001-11-05 09:11:26 +01:00
|
|
|
@file{stddef.h}.
|
|
|
|
|
|
|
|
The @w{ISO C90} standard, where @code{wchar_t} was introduced, does not
|
|
|
|
say anything specific about the representation. It only requires that
|
|
|
|
this type is capable of storing all elements of the basic character set.
|
|
|
|
Therefore it would be legitimate to define @code{wchar_t} as @code{char},
|
|
|
|
which might make sense for embedded systems.
|
|
|
|
|
|
|
|
But for GNU systems @code{wchar_t} is always 32 bits wide and, therefore,
|
|
|
|
capable of representing all UCS-4 values and, therefore, covering all of
|
|
|
|
@w{ISO 10646}. Some Unix systems define @code{wchar_t} as a 16-bit type
|
|
|
|
and thereby follow Unicode very strictly. This definition is perfectly
|
|
|
|
fine with the standard, but it also means that to represent all
|
|
|
|
characters from Unicode and @w{ISO 10646} one has to use UTF-16 surrogate
|
|
|
|
characters, which is in fact a multi-wide-character encoding. But
|
|
|
|
resorting to multi-wide-character encoding contradicts the purpose of the
|
|
|
|
@code{wchar_t} type.
|
|
|
|
@end deftp
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftp {Data type} wint_t
|
|
|
|
@code{wint_t} is a data type used for parameters and variables that
|
|
|
|
contain a single wide character. As the name suggests this type is the
|
|
|
|
equivalent of @code{int} when using the normal @code{char} strings. The
|
|
|
|
types @code{wchar_t} and @code{wint_t} often have the same
|
|
|
|
representation if their size is 32 bits wide but if @code{wchar_t} is
|
|
|
|
defined as @code{char} the type @code{wint_t} must be defined as
|
|
|
|
@code{int} due to the parameter promotion.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
This type is defined in @file{wchar.h} and was introduced in
|
|
|
|
@w{Amendment 1} to @w{ISO C90}.
|
|
|
|
@end deftp
|
|
|
|
|
|
|
|
As there are for the @code{char} data type macros are available for
|
|
|
|
specifying the minimum and maximum value representable in an object of
|
|
|
|
type @code{wchar_t}.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypevr Macro wint_t WCHAR_MIN
|
|
|
|
The macro @code{WCHAR_MIN} evaluates to the minimum value representable
|
|
|
|
by an object of type @code{wint_t}.
|
|
|
|
|
|
|
|
This macro was introduced in @w{Amendment 1} to @w{ISO C90}.
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypevr Macro wint_t WCHAR_MAX
|
|
|
|
The macro @code{WCHAR_MAX} evaluates to the maximum value representable
|
|
|
|
by an object of type @code{wint_t}.
|
|
|
|
|
|
|
|
This macro was introduced in @w{Amendment 1} to @w{ISO C90}.
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
Another special wide character value is the equivalent to @code{EOF}.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypevr Macro wint_t WEOF
|
|
|
|
The macro @code{WEOF} evaluates to a constant expression of type
|
|
|
|
@code{wint_t} whose value is different from any member of the extended
|
|
|
|
character set.
|
|
|
|
|
|
|
|
@code{WEOF} need not be the same value as @code{EOF} and unlike
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{EOF} it also need @emph{not} be negative. In other words, sloppy
|
2001-11-05 09:11:26 +01:00
|
|
|
code like
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
@{
|
|
|
|
int c;
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
while ((c = getc (fp)) < 0)
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
has to be rewritten to use @code{WEOF} explicitly when wide characters
|
|
|
|
are used:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
@{
|
|
|
|
wint_t c;
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
while ((c = wgetc (fp)) != WEOF)
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is
|
|
|
|
defined in @file{wchar.h}.
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
|
|
|
|
These internal representations present problems when it comes to storing
|
|
|
|
and transmittal. Because each single wide character consists of more
|
|
|
|
than one byte, they are effected by byte-ordering. Thus, machines with
|
|
|
|
different endianesses would see different values when accessing the same
|
|
|
|
data. This byte ordering concern also applies for communication protocols
|
[BZ #2510, BZ #2830, BZ #3137, BZ #3313, BZ #3426, BZ #3465, BZ #3480, BZ #3483, BZ #3493, BZ #3514, BZ #3515, BZ #3664, BZ #3673, BZ #3674]
2007-01-11 Jakub Jelinek <jakub@redhat.com>
* sysdeps/i386/soft-fp/sfp-machine.h: Remove.
* sysdeps/x86_64/soft-fp/sfp-machine.h: Likewise.
2007-01-10 Ulrich Drepper <drepper@redhat.com>
* io/fts.c: Make sure fts_cur is always valid after return from
fts_read.
Patch by Miloslav Trmac <mitr@redhat.com>.
2006-10-27 Richard Sandiford <richard@codesourcery.com>
* elf/elf.h (R_MIPS_GLOB_DAT): Define.
(R_MIPS_NUM): Bump by 1.
2007-01-03 Jakub Jelinek <jakub@redhat.com>
* posix/execvp.c: Include alloca.h.
(allocate_scripts_argv): Renamed to...
(scripts_argv): ... this. Don't allocate buffer here nor count
arguments.
(execvp): Use alloca if possible.
* posix/Makefile: Add rules to build and run tst-vfork3 test.
* posix/tst-vfork3.c: New test.
* stdlib/Makefile (tst-strtod3-ENV): Define.
2007-01-02 Ulrich Drepper <drepper@redhat.com>
* posix/getconf.c: Update copyright year.
* nss/getent.c: Likewise.
* iconv/iconvconfig.c: Likewise.
* iconv/iconv_prog.c: Likewise.
* elf/ldconfig.c: Likewise.
* catgets/gencat.c: Likewise.
* csu/version.c: Likewise.
* elf/ldd.bash.in: Likewise.
* elf/sprof.c (print_version): Likewise.
* locale/programs/locale.c: Likewise.
* locale/programs/localedef.c: Likewise.
* nscd/nscd.c (print_version): Likewise.
* debug/xtrace.sh: Likewise.
* malloc/memusage.sh: Likewise.
* malloc/mtrace.pl: Likewise.
* debug/catchsegv.sh: Likewise.
2006-12-24 Ulrich Drepper <drepper@redhat.com>
* malloc/malloc.c (sYSMALLOc): Remove some unnecessary alignment
attempts.
2006-12-23 Ulrich Drepper <drepper@redhat.com>
* posix/wordexp.c: Remove some unnecessary tests.
2006-12-20 SUGIOKA Toshinobu <sugioka@itonet.co.jp>
* sysdeps/unix/sysv/linux/sh/bits/shm.h: New file.
* nss/getXXbyYY_r.c: Include atomic.h.
(INTERNAL (REENTRANT_NAME)): Write startp after start_fct,
add atomic_write_barrier () in between.
2006-11-28 Jakub Jelinek <jakub@redhat.com>
* elf/dl-support.c: Include dl-procinfo.h.
* sysdeps/powerpc/dl-procinfo.h (PPC_PLATFORM_POWER4,
PPC_PLATFORM_PPC970, PPC_PLATFORM_POWER5, PPC_PLATFORM_POWER5_PLUS,
PPC_PLATFORM_POWER6, PPC_PLATFORM_CELL_BE, PPC_PLATFORM_POWER6X):
Define.
(_dl_string_platform): Use PPC_PLATFORM_* macros instead of
hardcoded constants.
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_platform): Use
PPC_PLATFORM_* macros for array designators.
2006-11-11 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_cap_flags): Add 3 new cap
names to the beginning.
(_dl_powerpc_platforms): Add "power6x".
* sysdeps/powerpc/dl-procinfo.h (_DL_HWCAP_FIRST): Decrease.
(HWCAP_IMPORTANT): Add PPC_FEATURE_HAS_DFP.
(_DL_PLATFORMS_COUNT): Increase.
(_dl_string_platform): Handle power6x case.
* sysdeps/powerpc/sysdep.h (PPC_FEATURE_PA6T, PPC_FEATURE_HAS_DFP,
PPC_FEATURE_POWER6_EXT): Define.
(PPC_FEATURE_POWER5, PPC_FEATURE_POWER5_PLUS): Correct Comment.
[-2^31 .. 2^31) range.
* sysdeps/unix/sysv/linux/bits/statvfs.h: Define ST_RELATIME.
* sysdeps/unix/sysv/linux/internal_statvfs.c (__statvfs_getflags):
Handle relatime mount option.
2006-12-13 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/powerpc/powerpc32/setcontext.S: Include
kernel-features.h.
2006-12-11 Ulrich Drepper <drepper@redhat.com>
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Parse thousand
separators also if no non-zero digits found.
* stdlib/Makefile (tests): Add tst-strtod3.
[BZ #3664]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix test to recognize
empty parsed strings.
* stdlib/Makefile (tests): Add tst-strtod2.
* stdlib/tst-strtod2.c: New file.
[BZ #3673]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix exp_limit
computation.
* stdlib/Makefile (tests): Add tst-atof2.
* stdlib/tst-atof2.c: New file.
[BZ #3674]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Adjust exponent value
correctly if removing trailing zero of hex-float.
* stdlib/Makefile (tests): Add tst-atof1.
* stdlib/tst-atof1.c: New file.
* misc/mntent_r.c (__hasmntopt): Check p[optlen] even when p == rest.
Start searching for next comma at p rather than rest.
* misc/Makefile (tests): Add tst-mntent2.
* misc/tst-mntent2.c: New test.
2006-12-08 Ulrich Drepper <drepper@redhat.com>
* malloc/memusage.c: Handle realloc with new size of zero and
non-NULL pointer correctly.
(me): Really write first record twice.
(struct entry): Make format bi-arch safe.
(dest): Write out more realloc statistics.
* malloc/memusagestat.c (struct entry): Make format bi-arch safe.
2006-12-05 Jakub Jelinek <jakub@redhat.com>
* nis/nis_subr.c (nis_getnames): Revert last change.
2006-12-03 Kaz Kojima <kkojima@rr.iij4u.or.jp>
* sysdeps/unix/sysv/linux/sh/sys/io.h: Removed.
2006-11-30 H.J. Lu <hongjiu.lu@intel.com>
* sysdeps/i386/i686/memcmp.S: Use jump table as the base of
jump table entries.
2006-11-30 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/i386/clone.S: Provide CFI for the outermost
`clone' function to ensure proper unwinding stop of gdb.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
2006-12-01 Ulrich Drepper <drepper@redhat.com>
* nscd/nscd.init: Remove obsolete and commented-out -S option
handling.
2006-11-23 Jakub Jelinek <jakub@redhat.com>
[BZ #3514]
* manual/string.texi (strncmp): Fix pastos from wcscmp description.
[BZ #3515]
* manual/string.texi (strtok): Remove duplicate paragraph.
2006-12-01 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Fix compatibility with
libgcc not supporting `rflags' unwinding (register # >= 17).
2006-11-30 Jakub Jelinek <jakub@redhat.com>
* sunrpc/svc_run.c (svc_run): Set my_pollfd to new_pollfd if realloc
succeeded.
2006-11-29 Daniel Jacobowitz <dan@codesourcery.com>
Jakub Jelinek <jakub@redhat.com>
Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c (restore_rt): Add correct
unwind information.
* sysdeps/unix/sysv/linux/x86_64/Makefile: Provide symbols for
'restore_rt' even in the 'signal' directory.
* sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym: Extend the regs list.
malloc crashed. Don't allocate memory unnecessarily in each
loop.
2006-10-21 Jakub Jelinek <jakub@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Fix last change.
2006-11-20 Ulrich Drepper <drepper@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Optimize a bit.
2006-11-18 Bruno Haible <bruno@clisp.org>
* sysdeps/unix/sysv/linux/i386/getgroups.c (__getgroups): Invoke
__sysconf only after having tried to call getgroups32.
2006-11-19 Ulrich Drepper <drepper@redhat.com>
* nss/nss_files/files-hosts.c (LINE_PARSER): Support IPv6-style
addresses for IPv4 queries if they can be mapped.
2006-11-16 Jakub Jelinek <jakub@redhat.com>
* sysdeps/x86_64/fpu/s_copysignf.S (__copysignf): Switch to .text.
* sysdeps/x86_64/fpu/s_copysign.S (__copysign): Likewise.
(signmask): Add .size directive.
(othermask): Add .type directive.
2006-11-14 Ulrich Drepper <drepper@redhat.com>
* po/nl.po: Update from translation team.
* timezone/zdump.c: Redo fix for BZ #3137.
2006-11-14 Jakub Jelinek <jakub@redhat.com>
* nss/nss_files/files-alias.c (get_next_alias): Set line back
to first_unused after parsing :include: file.
* timezone/africa: Update from tzdata2006o.
* timezone/antarctica: Likewise.
* timezone/asia: Likewise.
* timezone/australasia: Likewise.
* timezone/backward: Likewise.
* timezone/europe: Likewise.
* timezone/iso3166.tab: Likewise.
* timezone/northamerica: Likewise.
* timezone/southamerica: Likewise.
* timezone/zone.tab: Likewise.
* time/tzfile.c (__tzfile_read): Extend to handle new file format
on machines with 64-bit time_t.
* timezone/checktab.awk: Update from tzcode2006o.
* timezone/ialloc.c: Likewise.
* timezone/private.h: Likewise.
* timezone/scheck.c: Likewise.
* timezone/tzfile.h: Likewise.
* timezone/tzselect.ksh: Likewise.
* timezone/zdump.c: Likewise.
* timezone/zic.c: Likewise.
[BZ #3483]
* elf/ldconfig.c (main): Call setlocale and textdomain.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
[BZ #3480]
* manual/argp.texi: Fix typos.
* manual/charset.texi: Likewise.
* manual/errno.texi: Likewise.
* manual/filesys.texi: Likewise.
* manual/lang.texi: Likewise.
* manual/maint.texi: Likewise.
* manual/memory.texi: Likewise.
* manual/message.texi: Likewise.
* manual/resource.texi: Likewise.
* manual/search.texi: Likewise.
* manual/signal.texi: Likewise.
* manual/startup.texi: Likewise.
* manual/stdio.texi: Likewise.
* manual/sysinfo.texi: Likewise.
* manual/syslog.texi: Likewise.
* manual/time.texi: Likewise.
Patch by Ralf Wildenhues <Ralf.Wildenhues@gmx.de>.
[BZ #3465]
* sunrpc/clnt_raw.c: Minimal message improvements.
* sunrpc/pm_getmaps.c: Likewise.
* nis/nss_nisplus/nisplus-publickey.c: Likewise.
* nis/nis_print_group_entry.c: Likewise.
* locale/programs/repertoire.c: Likewise.
* locale/programs/charmap.c: Likewise.
* malloc/memusage.sh: Likewise.
* elf/dl-deps.c: Likewise.
* locale/programs/ld-collate.c: Likewise.
* libio/vswprintf.c: Likewise.
* malloc/memusagestat.c: Likewise.
* sunrpc/auth_unix.c: Likewise.
* sunrpc/rpc_main.c: Likewise.
* nscd/cache.c: Likewise.
* locale/programs/repertoire.c: Unify output messages.
* locale/programs/charmap.c: Likewise.
* locale/programs/ld-ctype.c: Likewise.
* locale/programs/ld-monetary.c: Likewise.
* locale/programs/ld-numeric.c: Likewise.
* locale/programs/ld-time.c: Likewise.
* elf/ldconfig.c: Likewise.
* nscd/selinux.c: Likewise.
* elf/cache.c: Likewise.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
2006-11-10 Jakub Jelinek <jakub@redhat.com>
* string/strxfrm_l.c (STRXFRM): Fix trailing \1 optimization
if N is one bigger than return value.
* string/tst-strxfrm2.c (do_test): Also test strxfrm with l1 + 1
and l1 last arguments, if buf is defined, verify the return value
equals to strlen (buf) and verify no byte beyond passed length
is modified.
2006-11-10 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
* sysdeps/gnu/siglist.c (__old_sys_siglist, __old_sys_sigabbrev):
Use __new_sys_siglist instead of _sys_siglist_internal as
second macro argument.
(_old_sys_siglist): Use declare_symbol_alias macro instead of
strong_alias.
2006-11-09 Ulrich Drepper <drepper@redhat.com>
[BZ #3493]
* posix/unistd.h (sysconf): Remove const attribute.
* sysdeps/posix/getaddrinfo.c (getaddrinfo): Fix test for
temporary or deprecated addresses.
Patch by Sridhar Samudrala <sri@us.ibm.com>.
* string/Makefile (tests): Add tst-strxfrm2.
* string/tst-strxfrm2.c: New file.
2006-10-09 Jakub Jelinek <jakub@redhat.com>
* elf/dl-debug.c (_dl_debug_initialize): Check r->r_map for 0
rather than r->r_brk.
* string/strxfrm_l.c (STRXFRM): Do the trailing \1 removal
optimization even if needed > n.
2006-11-07 Jakub Jelinek <jakub@redhat.com>
* include/libc-symbols.h (declare_symbol): Rename to...
(declare_symbol_alias): ... this. Add ORIGINAL argument, imply
strong_alias (ORIGINAL, SYMBOL) in asm to make sure it preceedes
.size directive.
* sysdeps/gnu/errlist-compat.awk: Adjust for declare_symbol_alias
changes.
* sysdeps/gnu/siglist.c: Likewise.
2006-11-03 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/fpu/bits/mathinline.h
[__LIBC_INTERNAL_MATH_INLINES]: Moved to ...
* sysdeps/powerpc/fpu/math_private.h: ...here. New file.
2006-11-05 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/i386/sysconf.c (intel_check_word):
Update handling of cache descriptor 0x49 for new models.
* sysdeps/unix/sysv/linux/x86_64/sysconf.c (intel_check_word):
Likewise.
2006-11-02 Ulrich Drepper <drepper@redhat.com>
* configure.in: Work around ld --help change and avoid -z relro
test completely if the architecture doesn't care about security.
2006-11-01 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
2006-10-31 Ulrich Drepper <drepper@redhat.com>
* stdlib/atexit.c (atexit): Don't mark as hidden when used to
generate compatibility version.
2006-10-29 Ulrich Drepper <drepper@redhat.com>
* configure.in: Relax -z relro requirement a bit.
* po/sv.po: Update from translation team.
2006-10-29 Jakub Jelinek <jakub@redhat.com>
* elf/dl-sym.c (do_sym): Use RTLD_SINGLE_THREAD_P.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Likewise.
* elf/dl-close.c (_dl_close_worker): Likewise.
* elf/dl-open.c (_dl_open_worker): Likewise.
* sysdeps/generic/sysdep-cancel.h (RTLD_SINGLE_THREAD_P): Define.
* configure.in: Require assembler support for visibility, compiler
support for visibility and aliases, linker support for various -z
options.
* Makeconfig: Remove conditional code which now is unnecessary.
* config.h.in: Likewise.
* config.make.in: Likewise.
* dlfcn/Makefile: Likewise.
* elf/Makefile: Likewise.
* elf/dl-load.c: Likewise.
* elf/rtld.c: Likewise.
* include/libc-symbols.h: Likewise.
* include/stdio.h: Likewise.
* io/Makefile: Likewise.
* io/fstat.c: Likewise.
* io/fstat64.c: Likewise.
* io/fstatat.c: Likewise.
* io/fstatat64.c: Likewise.
* io/lstat.c: Likewise.
* io/lstat64.c: Likewise.
* io/mknod.c: Likewise.
* io/mknodat.c: Likewise.
* io/stat.c: Likewise.
* io/stat64.c: Likewise.
* libio/stdio.c: Likewise.
* nscd/Makefile: Likewise.
* stdlib/Makefile: Likewise.
* stdlib/atexit.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/i386/sysdep.h: Likewise.
* sysdeps/i386/i686/memcmp.S: Likewise.
* sysdeps/powerpc/powerpc32/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/i386/sigaction.c: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Likewise.
* Makerules: USE_TLS support is now default.
* tls.make.c: Likewise.
* csu/Versions: Likewise.
* csu/libc-start.c: Likewise.
* csu/libc-tls.c: Likewise.
* csu/version.c: Likewise.
* dlfcn/dlinfo.c: Likewise.
* elf/dl-addr.c: Likewise.
* elf/dl-cache.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-iteratephdr.c: Likewise.
* elf/dl-load.c: Likewise.
* elf/dl-lookup.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-reloc.c: Likewise.
* elf/dl-support.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-sysdep.c: Likewise.
* elf/dl-tls.c: Likewise.
* elf/ldconfig.c: Likewise.
* elf/rtld.c: Likewise.
* elf/tst-tls-dlinfo.c: Likewise.
* elf/tst-tls1.c: Likewise.
* elf/tst-tls10.h: Likewise.
* elf/tst-tls14.c: Likewise.
* elf/tst-tls2.c: Likewise.
* elf/tst-tls3.c: Likewise.
* elf/tst-tls4.c: Likewise.
* elf/tst-tls5.c: Likewise.
* elf/tst-tls6.c: Likewise.
* elf/tst-tls7.c: Likewise.
* elf/tst-tls8.c: Likewise.
* elf/tst-tls9.c: Likewise.
* elf/tst-tlsmod1.c: Likewise.
* elf/tst-tlsmod13.c: Likewise.
* elf/tst-tlsmod13a.c: Likewise.
* elf/tst-tlsmod14a.c: Likewise.
* elf/tst-tlsmod2.c: Likewise.
* elf/tst-tlsmod3.c: Likewise.
* elf/tst-tlsmod4.c: Likewise.
* elf/tst-tlsmod5.c: Likewise.
* elf/tst-tlsmod6.c: Likewise.
* include/errno.h: Likewise.
* include/link.h: Likewise.
* include/tls.h: Likewise.
* locale/global-locale.c: Likewise.
* locale/localeinfo.h: Likewise.
* malloc/arena.c: Likewise.
* malloc/hooks.c: Likewise.
* malloc/malloc.c: Likewise.
* resolv/Versions: Likewise.
* sysdeps/alpha/dl-machine.h: Likewise.
* sysdeps/alpha/libc-tls.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/generic/tls.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/ia64/dl-machine.h: Likewise.
* sysdeps/ia64/libc-tls.c: Likewise.
* sysdeps/mach/hurd/fork.c: Likewise.
* sysdeps/mach/hurd/i386/tls.h: Likewise.
* sysdeps/powerpc/powerpc32/dl-machine.c: Likwise.
* sysdeps/powerpc/powerpc32/dl-machine.h: Likewise.
* sysdeps/powerpc/powerpc64/dl-machine.h: Likewise.
* sysdeps/s390/libc-tls.c: Likewise.
* sysdeps/s390/s390-32/dl-machine.h: Likewise.
* sysdeps/s390/s390-64/dl-machine.h: Likewise.
* sysdeps/sh/dl-machine.h: Likewise.
* sysdeps/sparc/sparc32/dl-machine.h: Likewise.
* sysdeps/sparc/sparc64/dl-machine.h: Likewise.
* sysdeps/x86_64/dl-machine.h: Likewise.
[BZ #3426]
* stdlib/stdlib.h: Adjust comment for canonicalize_file_name to
reality.
2006-10-27 Jakub Jelinek <jakub@redhat.com>
* elf/dl-lookup.c (_dl_debug_bindings): Remove unused symbol_scope
argument.
(_dl_lookup_symbol_x): Adjust caller.
* sysdeps/generic/ldsodefs.h (struct link_namespaces): Remove
_ns_global_scope.
* elf/rtld.c (dl_main): Don't initialize _ns_global_scope.
* elf/dl-libc.c: Revert l_scope name changes.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* elf/dl-close.c (_dl_close): Likewise.
* elf/dl-open.c (dl_open_worker): Likewise. If not SINGLE_THREAD_P,
always use __rtld_mrlock_{change,done}. Always free old scope list
here if not l_scope_mem.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Revert l_scope name
change. Never free scope list here. Just __rtld_mrlock_lock before
the lookup and __rtld_mrlock_unlock it after the lookup.
* elf/dl-sym.c: Likewise.
* include/link.h (struct r_scoperec): Remove.
(struct link_map): Replace l_scoperec with l_scope, l_scoperec_mem
with l_scope_mem and l_scoperec_lock with l_scope_lock.
2006-10-25 Ulrich Drepper <drepper@redhat.com>
* sysdeps/gnu/netinet/tcp.h: Define TCP_CONGESTION.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* configure.in: Disable building profile libraries by default.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* elf/dl-lookup.c (_dl_lookup_symbol_x): Add warning to
_dl_lookup_symbol_x code.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* elf/dl-runtime.c: Include sysdep-cancel.h.
(_dl_fixup, _dl_profile_fixup): Use __rtld_mrlock_* and
scoperec->nusers only if !SINGLE_THREAD_P. Use atomic_*
instead of catomic_* macros.
* elf/dl-sym.c: Include sysdep-cancel.h.
(do_sym): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-close.c: Include sysdep-cancel.h.
(_dl_close): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-open.c: Include sysdep-cancel.h.
(dl_open_worker): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Set maxfb to address of last
fastbin rather than end of fastbin array.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* sysdeps/i386/i486/bits/atomic.h (catomic_decrement): Use correct
body macro.
* sysdeps/x86_64/bits/atomic.h
(__arch_c_compare_and_exchange_val_64_acq): Add missing casts.
(catomic_decrement): Use correct body macro.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* include/atomic.h: Add a unique prefix to all local variables
in macros.
* csu/tst-atomic.c (do_test): Test also catomic_* macros.
2006-10-14 Ulrich Drepper <drepper@redhat.com>
* resolv/arpa/nameser.h: Document that ns_t_a6 is deprecated.
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Don't use get_fast_max to
determine highest fast bin to consolidate, always look into all of
them.
(do_check_malloc_state): Only require for empty bins for large
sizes in main arena.
* libio/stdio.h: Add more __wur attributes.
2006-11-12 Andreas Jaeger <aj@suse.de>
[BZ #2510]
* manual/search.texi (Hash Search Function): Clarify.
(Array Search Function): Clarify.
2006-11-12 Joseph Myers <joseph@codesourcery.com>
[BZ #2830]
* math/atest-exp.c (main): Cast hex value to mp_limb_t before
shifting.
* math/atest-exp2.c (read_mpn_hex): Likewise.
* math/atest-sincos.c (main): Likewise.
* sysdeps/unix/sysv/linux/syscalls.list: Add epoll_pwait.
* sysdeps/unix/sysv/linux/sys/epoll.h: Declare epoll_pwait.
* sysdeps/unix/sysv/linux/Versions (libc): Add epoll_pwait for
version GLIBC_2.6.
* Versions.def: Add GLIBC_2.6 for libc.
* sysdeps/i386/i486/bits/atomic.h: Add catomic_* support.
2006-10-11 Jakub Jelinek <jakub@redhat.com>
* malloc/malloc.c (_int_malloc): Remove unused any_larger variable.
* nis/nis_defaults.c (__nis_default_access): Don't call getenv twice.
* nis/nis_subr.c (nis_getnames): Use __secure_getenv instead of getenv.
* sysdeps/generic/unsecvars.h: Add NIS_PATH.
2006-10-11 Ulrich Drepper <drepper@redhat.com>
* include/atomic.c: Define catomic_* operations.
* sysdeps/x86_64/bits/atomic.h: Likewise. Fix a few minor problems.
* stdlib/cxa_finalize.c: Use catomic_* operations instead of atomic_*.
* malloc/memusage.c: Likewise.
* gmon/mcount.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-profile.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-runtime.c: Likewise.
* elf/dl-fptr.c: Likewise.
* resolv/res_libc.c: Likewise.
2006-10-10 Roland McGrath <roland@frob.com>
* sysdeps/mach/hurd/utimes.c: Use a union to avoid an improper cast.
* sysdeps/mach/hurd/futimes.c: Likewise.
* sysdeps/mach/hurd/lutimes.c: Likewise.
2006-10-09 Ulrich Drepper <drepper@redhat.com>
Jakub Jelinek <jakub@redhat.com>
Implement reference counting of scope records.
* elf/dl-close.c (_dl_close): Remove all scopes from removed objects
from the list in objects which remain. Always allocate new scope
record.
* elf/dl-open.c (dl_open_worker): When growing array for scopes,
don't resize, allocate a new one.
* elf/dl-runtime.c: Update reference counters before using a scope
array.
* elf/dl-sym.c: Likewise.
* elf/dl-libc.c: Adjust for l_scope name change.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* include/link.h: Include <rtld-lowlevel.h>. Define struct
r_scoperec. Replace r_scope with pointer to r_scoperec structure.
Add l_scoperec_lock.
* sysdeps/generic/ldsodefs.h: Include <rtld-lowlevel.h>.
* sysdeps/generic/rtld-lowlevel.h: New file.
* include/atomic.h: Rename atomic_and to atomic_and_val and
atomic_or to atomic_or_val. Define new macros atomic_and and
atomic_or which do not return values.
* sysdeps/x86_64/bits/atomic.h: Define atomic_and and atomic_or.
Various cleanups.
* sysdeps/i386/i486/bits/atomic.h: Likewise.
* po/sv.po: Update from translation team.
2006-10-07 Ulrich Drepper <drepper@redhat.com>
* Versions.def: Add GLIBC_2.6 to libpthread.
* include/shlib-compat.h (SHLIB_COMPAT): Expand parameters before use.
(versioned_symbol): Likewise.
(compat_symbol): Likewise.
* po/tr.po: Update from translation team.
* nis/Banner: Removed. It's been integral part forever and the
author info is incomplete anyway.
* libio/Banner: Likewise.
2006-10-06 Ulrich Drepper <drepper@redhat.com>
* version.h (VERSION): Bump to 2.5.90 for new development tree.
2007-01-11 22:51:07 +01:00
|
|
|
that are all byte-based and therefore require that the sender has to
|
2001-11-05 09:11:26 +01:00
|
|
|
decide about splitting the wide character in bytes. A last (but not least
|
|
|
|
important) point is that wide characters often require more storage space
|
|
|
|
than a customized byte-oriented character set.
|
|
|
|
|
|
|
|
@cindex multibyte character
|
|
|
|
@cindex EBCDIC
|
2001-11-07 08:21:22 +01:00
|
|
|
For all the above reasons, an external encoding that is different from
|
|
|
|
the internal encoding is often used if the latter is UCS-2 or UCS-4.
|
2001-11-05 09:11:26 +01:00
|
|
|
The external encoding is byte-based and can be chosen appropriately for
|
|
|
|
the environment and for the texts to be handled. A variety of different
|
|
|
|
character sets can be used for this external encoding (information that
|
|
|
|
will not be exhaustively presented here--instead, a description of the
|
|
|
|
major groups will suffice). All of the ASCII-based character sets
|
2001-11-07 08:21:22 +01:00
|
|
|
fulfill one requirement: they are "filesystem safe." This means that
|
|
|
|
the character @code{'/'} is used in the encoding @emph{only} to
|
2001-11-05 09:11:26 +01:00
|
|
|
represent itself. Things are a bit different for character sets like
|
|
|
|
EBCDIC (Extended Binary Coded Decimal Interchange Code, a character set
|
|
|
|
family used by IBM), but if the operation system does not understand
|
2001-11-07 08:21:22 +01:00
|
|
|
EBCDIC directly the parameters-to-system calls have to be converted
|
|
|
|
first anyhow.
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
@itemize @bullet
|
2001-11-07 08:21:22 +01:00
|
|
|
@item
|
|
|
|
The simplest character sets are single-byte character sets. There can
|
|
|
|
be only up to 256 characters (for @w{8 bit} character sets), which is
|
|
|
|
not sufficient to cover all languages but might be sufficient to handle
|
|
|
|
a specific text. Handling of a @w{8 bit} character sets is simple. This
|
|
|
|
is not true for other kinds presented later, and therefore, the
|
2001-11-05 09:11:26 +01:00
|
|
|
application one uses might require the use of @w{8 bit} character sets.
|
|
|
|
|
|
|
|
@cindex ISO 2022
|
|
|
|
@item
|
|
|
|
The @w{ISO 2022} standard defines a mechanism for extended character
|
|
|
|
sets where one character @emph{can} be represented by more than one
|
|
|
|
byte. This is achieved by associating a state with the text.
|
|
|
|
Characters that can be used to change the state can be embedded in the
|
|
|
|
text. Each byte in the text might have a different interpretation in each
|
|
|
|
state. The state might even influence whether a given byte stands for a
|
|
|
|
character on its own or whether it has to be combined with some more
|
|
|
|
bytes.
|
|
|
|
|
|
|
|
@cindex EUC
|
|
|
|
@cindex Shift_JIS
|
|
|
|
@cindex SJIS
|
|
|
|
In most uses of @w{ISO 2022} the defined character sets do not allow
|
|
|
|
state changes that cover more than the next character. This has the
|
|
|
|
big advantage that whenever one can identify the beginning of the byte
|
|
|
|
sequence of a character one can interpret a text correctly. Examples of
|
|
|
|
character sets using this policy are the various EUC character sets
|
|
|
|
(used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN)
|
|
|
|
or Shift_JIS (SJIS, a Japanese encoding).
|
|
|
|
|
|
|
|
But there are also character sets using a state that is valid for more
|
|
|
|
than one character and has to be changed by another byte sequence.
|
|
|
|
Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN.
|
|
|
|
|
|
|
|
@item
|
|
|
|
@cindex ISO 6937
|
|
|
|
Early attempts to fix 8 bit character sets for other languages using the
|
|
|
|
Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes
|
|
|
|
representing characters like the acute accent do not produce output
|
|
|
|
themselves: one has to combine them with other characters to get the
|
|
|
|
desired result. For example, the byte sequence @code{0xc2 0x61}
|
|
|
|
(non-spacing acute accent, followed by lower-case `a') to get the ``small
|
|
|
|
a with acute'' character. To get the acute accent character on its own,
|
|
|
|
one has to write @code{0xc2 0x20} (the non-spacing acute followed by a
|
|
|
|
space).
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Character sets like @w{ISO 6937} are used in some embedded systems such
|
2001-11-05 09:11:26 +01:00
|
|
|
as teletex.
|
|
|
|
|
|
|
|
@item
|
|
|
|
@cindex UTF-8
|
|
|
|
Instead of converting the Unicode or @w{ISO 10646} text used internally,
|
|
|
|
it is often also sufficient to simply use an encoding different than
|
|
|
|
UCS-2/UCS-4. The Unicode and @w{ISO 10646} standards even specify such an
|
|
|
|
encoding: UTF-8. This encoding is able to represent all of @w{ISO
|
|
|
|
10646} 31 bits in a byte string of length one to six.
|
|
|
|
|
|
|
|
@cindex UTF-7
|
|
|
|
There were a few other attempts to encode @w{ISO 10646} such as UTF-7,
|
|
|
|
but UTF-8 is today the only encoding that should be used. In fact, with
|
|
|
|
any luck UTF-8 will soon be the only external encoding that has to be
|
|
|
|
supported. It proves to be universally usable and its only disadvantage
|
|
|
|
is that it favors Roman languages by making the byte string
|
|
|
|
representation of other scripts (Cyrillic, Greek, Asian scripts) longer
|
|
|
|
than necessary if using a specific character set for these scripts.
|
|
|
|
Methods like the Unicode compression scheme can alleviate these
|
|
|
|
problems.
|
|
|
|
@end itemize
|
|
|
|
|
|
|
|
The question remaining is: how to select the character set or encoding
|
|
|
|
to use. The answer: you cannot decide about it yourself, it is decided
|
|
|
|
by the developers of the system or the majority of the users. Since the
|
|
|
|
goal is interoperability one has to use whatever the other people one
|
|
|
|
works with use. If there are no constraints, the selection is based on
|
|
|
|
the requirements the expected circle of users will have. In other words,
|
|
|
|
if a project is expected to be used in only, say, Russia it is fine to use
|
|
|
|
KOI8-R or a similar character set. But if at the same time people from,
|
|
|
|
say, Greece are participating one should use a character set that allows
|
|
|
|
all people to collaborate.
|
|
|
|
|
|
|
|
The most widely useful solution seems to be: go with the most general
|
|
|
|
character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding
|
|
|
|
and problems about users not being able to use their own language
|
|
|
|
adequately are a thing of the past.
|
|
|
|
|
|
|
|
One final comment about the choice of the wide character representation
|
|
|
|
is necessary at this point. We have said above that the natural choice
|
|
|
|
is using Unicode or @w{ISO 10646}. This is not required, but at least
|
|
|
|
encouraged, by the @w{ISO C} standard. The standard defines at least a
|
|
|
|
macro @code{__STDC_ISO_10646__} that is only defined on systems where
|
|
|
|
the @code{wchar_t} type encodes @w{ISO 10646} characters. If this
|
|
|
|
symbol is not defined one should avoid making assumptions about the wide
|
|
|
|
character representation. If the programmer uses only the functions
|
|
|
|
provided by the C library to handle wide character strings there should
|
|
|
|
be no compatibility problems with other systems.
|
|
|
|
|
|
|
|
@node Charset Function Overview
|
|
|
|
@section Overview about Character Handling Functions
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
A Unix @w{C library} contains three different sets of functions in two
|
|
|
|
families to handle character set conversion. One of the function families
|
|
|
|
(the most commonly used) is specified in the @w{ISO C90} standard and,
|
|
|
|
therefore, is portable even beyond the Unix world. Unfortunately this
|
|
|
|
family is the least useful one. These functions should be avoided
|
|
|
|
whenever possible, especially when developing libraries (as opposed to
|
|
|
|
applications).
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
The second family of functions got introduced in the early Unix standards
|
|
|
|
(XPG2) and is still part of the latest and greatest Unix standard:
|
|
|
|
@w{Unix 98}. It is also the most powerful and useful set of functions.
|
|
|
|
But we will start with the functions defined in @w{Amendment 1} to
|
|
|
|
@w{ISO C90}.
|
|
|
|
|
|
|
|
@node Restartable multibyte conversion
|
|
|
|
@section Restartable Multibyte Conversion Functions
|
|
|
|
|
|
|
|
The @w{ISO C} standard defines functions to convert strings from a
|
|
|
|
multibyte representation to wide character strings. There are a number
|
|
|
|
of peculiarities:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
|
|
|
The character set assumed for the multibyte encoding is not specified
|
|
|
|
as an argument to the functions. Instead the character set specified by
|
|
|
|
the @code{LC_CTYPE} category of the current locale is used; see
|
|
|
|
@ref{Locale Categories}.
|
|
|
|
|
|
|
|
@item
|
|
|
|
The functions handling more than one character at a time require NUL
|
|
|
|
terminated strings as the argument (i.e., converting blocks of text
|
2001-11-07 08:21:22 +01:00
|
|
|
does not work unless one can add a NUL byte at an appropriate place).
|
2001-11-05 09:11:26 +01:00
|
|
|
The GNU C library contains some extensions to the standard that allow
|
|
|
|
specifying a size, but basically they also expect terminated strings.
|
|
|
|
@end itemize
|
|
|
|
|
|
|
|
Despite these limitations the @w{ISO C} functions can be used in many
|
|
|
|
contexts. In graphical user interfaces, for instance, it is not
|
|
|
|
uncommon to have functions that require text to be displayed in a wide
|
2001-11-07 08:21:22 +01:00
|
|
|
character string if the text is not simple ASCII. The text itself might
|
2001-11-05 09:11:26 +01:00
|
|
|
come from a file with translations and the user should decide about the
|
|
|
|
current locale, which determines the translation and therefore also the
|
|
|
|
external encoding used. In such a situation (and many others) the
|
|
|
|
functions described here are perfect. If more freedom while performing
|
|
|
|
the conversion is necessary take a look at the @code{iconv} functions
|
|
|
|
(@pxref{Generic Charset Conversion}).
|
|
|
|
|
|
|
|
@menu
|
|
|
|
* Selecting the Conversion:: Selecting the conversion and its properties.
|
|
|
|
* Keeping the state:: Representing the state of the conversion.
|
|
|
|
* Converting a Character:: Converting Single Characters.
|
|
|
|
* Converting Strings:: Converting Multibyte and Wide Character
|
|
|
|
Strings.
|
|
|
|
* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
|
|
|
|
@end menu
|
|
|
|
|
|
|
|
@node Selecting the Conversion
|
|
|
|
@subsection Selecting the conversion and its properties
|
|
|
|
|
|
|
|
We already said above that the currently selected locale for the
|
|
|
|
@code{LC_CTYPE} category decides about the conversion that is performed
|
|
|
|
by the functions we are about to describe. Each locale uses its own
|
|
|
|
character set (given as an argument to @code{localedef}) and this is the
|
|
|
|
one assumed as the external multibyte encoding. The wide character
|
2008-06-04 11:46:50 +02:00
|
|
|
set is always UCS-4, at least on GNU systems.
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
A characteristic of each multibyte character set is the maximum number
|
|
|
|
of bytes that can be necessary to represent one character. This
|
|
|
|
information is quite important when writing code that uses the
|
|
|
|
conversion functions (as shown in the examples below).
|
|
|
|
The @w{ISO C} standard defines two macros that provide this information.
|
|
|
|
|
|
|
|
|
|
|
|
@comment limits.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypevr Macro int MB_LEN_MAX
|
|
|
|
@code{MB_LEN_MAX} specifies the maximum number of bytes in the multibyte
|
|
|
|
sequence for a single character in any of the supported locales. It is
|
|
|
|
a compile-time constant and is defined in @file{limits.h}.
|
|
|
|
@pindex limits.h
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypevr Macro int MB_CUR_MAX
|
|
|
|
@code{MB_CUR_MAX} expands into a positive integer expression that is the
|
|
|
|
maximum number of bytes in a multibyte character in the current locale.
|
|
|
|
The value is never greater than @code{MB_LEN_MAX}. Unlike
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{MB_LEN_MAX} this macro need not be a compile-time constant, and in
|
2001-11-05 09:11:26 +01:00
|
|
|
the GNU C library it is not.
|
|
|
|
|
|
|
|
@pindex stdlib.h
|
|
|
|
@code{MB_CUR_MAX} is defined in @file{stdlib.h}.
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
Two different macros are necessary since strictly @w{ISO C90} compilers
|
|
|
|
do not allow variable length array definitions, but still it is desirable
|
|
|
|
to avoid dynamic allocation. This incomplete piece of code shows the
|
|
|
|
problem:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
@{
|
|
|
|
char buf[MB_LEN_MAX];
|
|
|
|
ssize_t len = 0;
|
|
|
|
|
|
|
|
while (! feof (fp))
|
|
|
|
@{
|
|
|
|
fread (&buf[len], 1, MB_CUR_MAX - len, fp);
|
2002-06-30 06:04:20 +02:00
|
|
|
/* @r{@dots{} process} buf */
|
2001-11-05 09:11:26 +01:00
|
|
|
len -= used;
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The code in the inner loop is expected to have always enough bytes in
|
|
|
|
the array @var{buf} to convert one multibyte character. The array
|
|
|
|
@var{buf} has to be sized statically since many compilers do not allow a
|
2001-11-07 08:21:22 +01:00
|
|
|
variable size. The @code{fread} call makes sure that @code{MB_CUR_MAX}
|
2001-11-05 09:11:26 +01:00
|
|
|
bytes are always available in @var{buf}. Note that it isn't
|
|
|
|
a problem if @code{MB_CUR_MAX} is not a compile-time constant.
|
|
|
|
|
|
|
|
|
|
|
|
@node Keeping the state
|
|
|
|
@subsection Representing the state of the conversion
|
|
|
|
|
|
|
|
@cindex stateful
|
|
|
|
In the introduction of this chapter it was said that certain character
|
2001-11-07 08:21:22 +01:00
|
|
|
sets use a @dfn{stateful} encoding. That is, the encoded values depend
|
2001-11-05 09:11:26 +01:00
|
|
|
in some way on the previous bytes in the text.
|
|
|
|
|
|
|
|
Since the conversion functions allow converting a text in more than one
|
|
|
|
step we must have a way to pass this information from one call of the
|
|
|
|
functions to another.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftp {Data type} mbstate_t
|
|
|
|
@cindex shift state
|
|
|
|
A variable of type @code{mbstate_t} can contain all the information
|
|
|
|
about the @dfn{shift state} needed from one call to a conversion
|
|
|
|
function to another.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{mbstate_t} is defined in @file{wchar.h}. It was introduced in
|
|
|
|
@w{Amendment 1} to @w{ISO C90}.
|
|
|
|
@end deftp
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
To use objects of type @code{mbstate_t} the programmer has to define such
|
|
|
|
objects (normally as local variables on the stack) and pass a pointer to
|
2001-11-05 09:11:26 +01:00
|
|
|
the object to the conversion functions. This way the conversion function
|
|
|
|
can update the object if the current multibyte character set is stateful.
|
|
|
|
|
|
|
|
There is no specific function or initializer to put the state object in
|
|
|
|
any specific state. The rules are that the object should always
|
|
|
|
represent the initial state before the first use, and this is achieved by
|
|
|
|
clearing the whole variable with code such as follows:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
@{
|
|
|
|
mbstate_t state;
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
/* @r{from now on @var{state} can be used.} */
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
When using the conversion functions to generate output it is often
|
|
|
|
necessary to test whether the current state corresponds to the initial
|
|
|
|
state. This is necessary, for example, to decide whether to emit
|
|
|
|
escape sequences to set the state to the initial state at certain
|
|
|
|
sequence points. Communication protocols often require this.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun int mbsinit (const mbstate_t *@var{ps})
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{mbsinit} function determines whether the state object pointed
|
|
|
|
to by @var{ps} is in the initial state. If @var{ps} is a null pointer or
|
|
|
|
the object is in the initial state the return value is nonzero. Otherwise
|
2001-11-05 09:11:26 +01:00
|
|
|
it is zero.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{mbsinit} was introduced in @w{Amendment 1} to @w{ISO C90} and is
|
2001-11-05 09:11:26 +01:00
|
|
|
declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Code using @code{mbsinit} often looks similar to this:
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
@c Fix the example to explicitly say how to generate the escape sequence
|
|
|
|
@c to restore the initial state.
|
|
|
|
@smallexample
|
|
|
|
@{
|
|
|
|
mbstate_t state;
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
/* @r{Use @var{state}.} */
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
if (! mbsinit (&state))
|
|
|
|
@{
|
|
|
|
/* @r{Emit code to return to initial state.} */
|
|
|
|
const wchar_t empty[] = L"";
|
|
|
|
const wchar_t *srcp = empty;
|
|
|
|
wcsrtombs (outbuf, &srcp, outbuflen, &state);
|
|
|
|
@}
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The code to emit the escape sequence to get back to the initial state is
|
|
|
|
interesting. The @code{wcsrtombs} function can be used to determine the
|
|
|
|
necessary output code (@pxref{Converting Strings}). Please note that on
|
|
|
|
GNU systems it is not necessary to perform this extra action for the
|
|
|
|
conversion from multibyte text to wide character text since the wide
|
|
|
|
character encoding is not stateful. But there is nothing mentioned in
|
|
|
|
any standard that prohibits making @code{wchar_t} using a stateful
|
|
|
|
encoding.
|
|
|
|
|
|
|
|
@node Converting a Character
|
|
|
|
@subsection Converting Single Characters
|
|
|
|
|
|
|
|
The most fundamental of the conversion functions are those dealing with
|
|
|
|
single characters. Please note that this does not always mean single
|
|
|
|
bytes. But since there is very often a subset of the multibyte
|
|
|
|
character set that consists of single byte sequences, there are
|
2001-11-07 08:21:22 +01:00
|
|
|
functions to help with converting bytes. Frequently, ASCII is a subpart
|
|
|
|
of the multibyte character set. In such a scenario, each ASCII character
|
|
|
|
stands for itself, and all other characters have at least a first byte
|
2001-11-05 09:11:26 +01:00
|
|
|
that is beyond the range @math{0} to @math{127}.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun wint_t btowc (int @var{c})
|
|
|
|
The @code{btowc} function (``byte to wide character'') converts a valid
|
|
|
|
single byte character @var{c} in the initial shift state into the wide
|
|
|
|
character equivalent using the conversion rules from the currently
|
|
|
|
selected locale of the @code{LC_CTYPE} category.
|
|
|
|
|
|
|
|
If @code{(unsigned char) @var{c}} is no valid single byte multibyte
|
|
|
|
character or if @var{c} is @code{EOF}, the function returns @code{WEOF}.
|
|
|
|
|
|
|
|
Please note the restriction of @var{c} being tested for validity only in
|
|
|
|
the initial shift state. No @code{mbstate_t} object is used from
|
|
|
|
which the state information is taken, and the function also does not use
|
|
|
|
any static state.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{btowc} function was introduced in @w{Amendment 1} to @w{ISO C90}
|
2001-11-05 09:11:26 +01:00
|
|
|
and is declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
2008-06-04 11:46:50 +02:00
|
|
|
Despite the limitation that the single byte value is always interpreted
|
|
|
|
in the initial state, this function is actually useful most of the time.
|
2001-11-05 09:11:26 +01:00
|
|
|
Most characters are either entirely single-byte character sets or they
|
|
|
|
are extension to ASCII. But then it is possible to write code like this
|
|
|
|
(not that this specific example is very useful):
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
wchar_t *
|
|
|
|
itow (unsigned long int val)
|
|
|
|
@{
|
|
|
|
static wchar_t buf[30];
|
|
|
|
wchar_t *wcp = &buf[29];
|
|
|
|
*wcp = L'\0';
|
|
|
|
while (val != 0)
|
|
|
|
@{
|
|
|
|
*--wcp = btowc ('0' + val % 10);
|
|
|
|
val /= 10;
|
|
|
|
@}
|
|
|
|
if (wcp == &buf[29])
|
|
|
|
*--wcp = L'0';
|
|
|
|
return wcp;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
Why is it necessary to use such a complicated implementation and not
|
|
|
|
simply cast @code{'0' + val % 10} to a wide character? The answer is
|
|
|
|
that there is no guarantee that one can perform this kind of arithmetic
|
|
|
|
on the character of the character set used for @code{wchar_t}
|
|
|
|
representation. In other situations the bytes are not constant at
|
|
|
|
compile time and so the compiler cannot do the work. In situations like
|
2008-06-04 11:46:50 +02:00
|
|
|
this, using @code{btowc} is required.
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
@noindent
|
2008-06-04 11:46:50 +02:00
|
|
|
There is also a function for the conversion in the other direction.
|
2001-11-05 09:11:26 +01:00
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun int wctob (wint_t @var{c})
|
|
|
|
The @code{wctob} function (``wide character to byte'') takes as the
|
|
|
|
parameter a valid wide character. If the multibyte representation for
|
|
|
|
this character in the initial state is exactly one byte long, the return
|
|
|
|
value of this function is this character. Otherwise the return value is
|
|
|
|
@code{EOF}.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{wctob} was introduced in @w{Amendment 1} to @w{ISO C90} and
|
|
|
|
is declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
There are more general functions to convert single character from
|
|
|
|
multibyte representation to wide characters and vice versa. These
|
|
|
|
functions pose no limit on the length of the multibyte representation
|
|
|
|
and they also do not require it to be in the initial state.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps})
|
|
|
|
@cindex stateful
|
|
|
|
The @code{mbrtowc} function (``multibyte restartable to wide
|
|
|
|
character'') converts the next multibyte character in the string pointed
|
|
|
|
to by @var{s} into a wide character and stores it in the wide character
|
|
|
|
string pointed to by @var{pwc}. The conversion is performed according
|
|
|
|
to the locale currently selected for the @code{LC_CTYPE} category. If
|
|
|
|
the conversion for the character set used in the locale requires a state,
|
|
|
|
the multibyte string is interpreted in the state represented by the
|
|
|
|
object pointed to by @var{ps}. If @var{ps} is a null pointer, a static,
|
|
|
|
internal state variable used only by the @code{mbrtowc} function is
|
|
|
|
used.
|
|
|
|
|
|
|
|
If the next multibyte character corresponds to the NUL wide character,
|
|
|
|
the return value of the function is @math{0} and the state object is
|
|
|
|
afterwards in the initial state. If the next @var{n} or fewer bytes
|
|
|
|
form a correct multibyte character, the return value is the number of
|
|
|
|
bytes starting from @var{s} that form the multibyte character. The
|
|
|
|
conversion state is updated according to the bytes consumed in the
|
|
|
|
conversion. In both cases the wide character (either the @code{L'\0'}
|
|
|
|
or the one found in the conversion) is stored in the string pointed to
|
|
|
|
by @var{pwc} if @var{pwc} is not null.
|
|
|
|
|
|
|
|
If the first @var{n} bytes of the multibyte string possibly form a valid
|
|
|
|
multibyte character but there are more than @var{n} bytes needed to
|
|
|
|
complete it, the return value of the function is @code{(size_t) -2} and
|
|
|
|
no value is stored. Please note that this can happen even if @var{n}
|
2001-11-07 08:21:22 +01:00
|
|
|
has a value greater than or equal to @code{MB_CUR_MAX} since the input
|
2001-11-05 09:11:26 +01:00
|
|
|
might contain redundant shift sequences.
|
|
|
|
|
|
|
|
If the first @code{n} bytes of the multibyte string cannot possibly form
|
|
|
|
a valid multibyte character, no value is stored, the global variable
|
|
|
|
@code{errno} is set to the value @code{EILSEQ}, and the function returns
|
|
|
|
@code{(size_t) -1}. The conversion state is afterwards undefined.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{mbrtowc} was introduced in @w{Amendment 1} to @w{ISO C90} and
|
|
|
|
is declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
Use of @code{mbrtowc} is straightforward. A function that copies a
|
|
|
|
multibyte string into a wide character string while at the same time
|
|
|
|
converting all lowercase characters into uppercase could look like this
|
|
|
|
(this is not the final version, just an example; it has no error
|
|
|
|
checking, and sometimes leaks memory):
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
wchar_t *
|
|
|
|
mbstouwcs (const char *s)
|
|
|
|
@{
|
|
|
|
size_t len = strlen (s);
|
|
|
|
wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
|
|
|
|
wchar_t *wcp = result;
|
|
|
|
wchar_t tmp[1];
|
|
|
|
mbstate_t state;
|
|
|
|
size_t nbytes;
|
|
|
|
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
|
|
|
|
@{
|
|
|
|
if (nbytes >= (size_t) -2)
|
|
|
|
/* Invalid input string. */
|
|
|
|
return NULL;
|
2003-07-22 22:05:55 +02:00
|
|
|
*wcp++ = towupper (tmp[0]);
|
2001-11-05 09:11:26 +01:00
|
|
|
len -= nbytes;
|
|
|
|
s += nbytes;
|
|
|
|
@}
|
|
|
|
return result;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The use of @code{mbrtowc} should be clear. A single wide character is
|
|
|
|
stored in @code{@var{tmp}[0]}, and the number of consumed bytes is stored
|
2001-11-07 08:21:22 +01:00
|
|
|
in the variable @var{nbytes}. If the conversion is successful, the
|
|
|
|
uppercase variant of the wide character is stored in the @var{result}
|
|
|
|
array and the pointer to the input string and the number of available
|
2001-11-05 09:11:26 +01:00
|
|
|
bytes is adjusted.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The only non-obvious thing about @code{mbrtowc} might be the way memory
|
|
|
|
is allocated for the result. The above code uses the fact that there
|
2001-11-05 09:11:26 +01:00
|
|
|
can never be more wide characters in the converted results than there are
|
2001-11-07 08:21:22 +01:00
|
|
|
bytes in the multibyte input string. This method yields a pessimistic
|
|
|
|
guess about the size of the result, and if many wide character strings
|
|
|
|
have to be constructed this way or if the strings are long, the extra
|
|
|
|
memory required to be allocated because the input string contains
|
|
|
|
multibyte characters might be significant. The allocated memory block can
|
|
|
|
be resized to the correct size before returning it, but a better solution
|
|
|
|
might be to allocate just the right amount of space for the result right
|
|
|
|
away. Unfortunately there is no function to compute the length of the wide
|
|
|
|
character string directly from the multibyte string. There is, however, a
|
2001-11-05 09:11:26 +01:00
|
|
|
function that does part of the work.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps})
|
|
|
|
The @code{mbrlen} function (``multibyte restartable length'') computes
|
|
|
|
the number of at most @var{n} bytes starting at @var{s}, which form the
|
|
|
|
next valid and complete multibyte character.
|
|
|
|
|
|
|
|
If the next multibyte character corresponds to the NUL wide character,
|
|
|
|
the return value is @math{0}. If the next @var{n} bytes form a valid
|
|
|
|
multibyte character, the number of bytes belonging to this multibyte
|
|
|
|
character byte sequence is returned.
|
|
|
|
|
[BZ #2510, BZ #2830, BZ #3137, BZ #3313, BZ #3426, BZ #3465, BZ #3480, BZ #3483, BZ #3493, BZ #3514, BZ #3515, BZ #3664, BZ #3673, BZ #3674]
2007-01-11 Jakub Jelinek <jakub@redhat.com>
* sysdeps/i386/soft-fp/sfp-machine.h: Remove.
* sysdeps/x86_64/soft-fp/sfp-machine.h: Likewise.
2007-01-10 Ulrich Drepper <drepper@redhat.com>
* io/fts.c: Make sure fts_cur is always valid after return from
fts_read.
Patch by Miloslav Trmac <mitr@redhat.com>.
2006-10-27 Richard Sandiford <richard@codesourcery.com>
* elf/elf.h (R_MIPS_GLOB_DAT): Define.
(R_MIPS_NUM): Bump by 1.
2007-01-03 Jakub Jelinek <jakub@redhat.com>
* posix/execvp.c: Include alloca.h.
(allocate_scripts_argv): Renamed to...
(scripts_argv): ... this. Don't allocate buffer here nor count
arguments.
(execvp): Use alloca if possible.
* posix/Makefile: Add rules to build and run tst-vfork3 test.
* posix/tst-vfork3.c: New test.
* stdlib/Makefile (tst-strtod3-ENV): Define.
2007-01-02 Ulrich Drepper <drepper@redhat.com>
* posix/getconf.c: Update copyright year.
* nss/getent.c: Likewise.
* iconv/iconvconfig.c: Likewise.
* iconv/iconv_prog.c: Likewise.
* elf/ldconfig.c: Likewise.
* catgets/gencat.c: Likewise.
* csu/version.c: Likewise.
* elf/ldd.bash.in: Likewise.
* elf/sprof.c (print_version): Likewise.
* locale/programs/locale.c: Likewise.
* locale/programs/localedef.c: Likewise.
* nscd/nscd.c (print_version): Likewise.
* debug/xtrace.sh: Likewise.
* malloc/memusage.sh: Likewise.
* malloc/mtrace.pl: Likewise.
* debug/catchsegv.sh: Likewise.
2006-12-24 Ulrich Drepper <drepper@redhat.com>
* malloc/malloc.c (sYSMALLOc): Remove some unnecessary alignment
attempts.
2006-12-23 Ulrich Drepper <drepper@redhat.com>
* posix/wordexp.c: Remove some unnecessary tests.
2006-12-20 SUGIOKA Toshinobu <sugioka@itonet.co.jp>
* sysdeps/unix/sysv/linux/sh/bits/shm.h: New file.
* nss/getXXbyYY_r.c: Include atomic.h.
(INTERNAL (REENTRANT_NAME)): Write startp after start_fct,
add atomic_write_barrier () in between.
2006-11-28 Jakub Jelinek <jakub@redhat.com>
* elf/dl-support.c: Include dl-procinfo.h.
* sysdeps/powerpc/dl-procinfo.h (PPC_PLATFORM_POWER4,
PPC_PLATFORM_PPC970, PPC_PLATFORM_POWER5, PPC_PLATFORM_POWER5_PLUS,
PPC_PLATFORM_POWER6, PPC_PLATFORM_CELL_BE, PPC_PLATFORM_POWER6X):
Define.
(_dl_string_platform): Use PPC_PLATFORM_* macros instead of
hardcoded constants.
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_platform): Use
PPC_PLATFORM_* macros for array designators.
2006-11-11 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_cap_flags): Add 3 new cap
names to the beginning.
(_dl_powerpc_platforms): Add "power6x".
* sysdeps/powerpc/dl-procinfo.h (_DL_HWCAP_FIRST): Decrease.
(HWCAP_IMPORTANT): Add PPC_FEATURE_HAS_DFP.
(_DL_PLATFORMS_COUNT): Increase.
(_dl_string_platform): Handle power6x case.
* sysdeps/powerpc/sysdep.h (PPC_FEATURE_PA6T, PPC_FEATURE_HAS_DFP,
PPC_FEATURE_POWER6_EXT): Define.
(PPC_FEATURE_POWER5, PPC_FEATURE_POWER5_PLUS): Correct Comment.
[-2^31 .. 2^31) range.
* sysdeps/unix/sysv/linux/bits/statvfs.h: Define ST_RELATIME.
* sysdeps/unix/sysv/linux/internal_statvfs.c (__statvfs_getflags):
Handle relatime mount option.
2006-12-13 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/powerpc/powerpc32/setcontext.S: Include
kernel-features.h.
2006-12-11 Ulrich Drepper <drepper@redhat.com>
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Parse thousand
separators also if no non-zero digits found.
* stdlib/Makefile (tests): Add tst-strtod3.
[BZ #3664]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix test to recognize
empty parsed strings.
* stdlib/Makefile (tests): Add tst-strtod2.
* stdlib/tst-strtod2.c: New file.
[BZ #3673]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix exp_limit
computation.
* stdlib/Makefile (tests): Add tst-atof2.
* stdlib/tst-atof2.c: New file.
[BZ #3674]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Adjust exponent value
correctly if removing trailing zero of hex-float.
* stdlib/Makefile (tests): Add tst-atof1.
* stdlib/tst-atof1.c: New file.
* misc/mntent_r.c (__hasmntopt): Check p[optlen] even when p == rest.
Start searching for next comma at p rather than rest.
* misc/Makefile (tests): Add tst-mntent2.
* misc/tst-mntent2.c: New test.
2006-12-08 Ulrich Drepper <drepper@redhat.com>
* malloc/memusage.c: Handle realloc with new size of zero and
non-NULL pointer correctly.
(me): Really write first record twice.
(struct entry): Make format bi-arch safe.
(dest): Write out more realloc statistics.
* malloc/memusagestat.c (struct entry): Make format bi-arch safe.
2006-12-05 Jakub Jelinek <jakub@redhat.com>
* nis/nis_subr.c (nis_getnames): Revert last change.
2006-12-03 Kaz Kojima <kkojima@rr.iij4u.or.jp>
* sysdeps/unix/sysv/linux/sh/sys/io.h: Removed.
2006-11-30 H.J. Lu <hongjiu.lu@intel.com>
* sysdeps/i386/i686/memcmp.S: Use jump table as the base of
jump table entries.
2006-11-30 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/i386/clone.S: Provide CFI for the outermost
`clone' function to ensure proper unwinding stop of gdb.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
2006-12-01 Ulrich Drepper <drepper@redhat.com>
* nscd/nscd.init: Remove obsolete and commented-out -S option
handling.
2006-11-23 Jakub Jelinek <jakub@redhat.com>
[BZ #3514]
* manual/string.texi (strncmp): Fix pastos from wcscmp description.
[BZ #3515]
* manual/string.texi (strtok): Remove duplicate paragraph.
2006-12-01 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Fix compatibility with
libgcc not supporting `rflags' unwinding (register # >= 17).
2006-11-30 Jakub Jelinek <jakub@redhat.com>
* sunrpc/svc_run.c (svc_run): Set my_pollfd to new_pollfd if realloc
succeeded.
2006-11-29 Daniel Jacobowitz <dan@codesourcery.com>
Jakub Jelinek <jakub@redhat.com>
Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c (restore_rt): Add correct
unwind information.
* sysdeps/unix/sysv/linux/x86_64/Makefile: Provide symbols for
'restore_rt' even in the 'signal' directory.
* sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym: Extend the regs list.
malloc crashed. Don't allocate memory unnecessarily in each
loop.
2006-10-21 Jakub Jelinek <jakub@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Fix last change.
2006-11-20 Ulrich Drepper <drepper@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Optimize a bit.
2006-11-18 Bruno Haible <bruno@clisp.org>
* sysdeps/unix/sysv/linux/i386/getgroups.c (__getgroups): Invoke
__sysconf only after having tried to call getgroups32.
2006-11-19 Ulrich Drepper <drepper@redhat.com>
* nss/nss_files/files-hosts.c (LINE_PARSER): Support IPv6-style
addresses for IPv4 queries if they can be mapped.
2006-11-16 Jakub Jelinek <jakub@redhat.com>
* sysdeps/x86_64/fpu/s_copysignf.S (__copysignf): Switch to .text.
* sysdeps/x86_64/fpu/s_copysign.S (__copysign): Likewise.
(signmask): Add .size directive.
(othermask): Add .type directive.
2006-11-14 Ulrich Drepper <drepper@redhat.com>
* po/nl.po: Update from translation team.
* timezone/zdump.c: Redo fix for BZ #3137.
2006-11-14 Jakub Jelinek <jakub@redhat.com>
* nss/nss_files/files-alias.c (get_next_alias): Set line back
to first_unused after parsing :include: file.
* timezone/africa: Update from tzdata2006o.
* timezone/antarctica: Likewise.
* timezone/asia: Likewise.
* timezone/australasia: Likewise.
* timezone/backward: Likewise.
* timezone/europe: Likewise.
* timezone/iso3166.tab: Likewise.
* timezone/northamerica: Likewise.
* timezone/southamerica: Likewise.
* timezone/zone.tab: Likewise.
* time/tzfile.c (__tzfile_read): Extend to handle new file format
on machines with 64-bit time_t.
* timezone/checktab.awk: Update from tzcode2006o.
* timezone/ialloc.c: Likewise.
* timezone/private.h: Likewise.
* timezone/scheck.c: Likewise.
* timezone/tzfile.h: Likewise.
* timezone/tzselect.ksh: Likewise.
* timezone/zdump.c: Likewise.
* timezone/zic.c: Likewise.
[BZ #3483]
* elf/ldconfig.c (main): Call setlocale and textdomain.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
[BZ #3480]
* manual/argp.texi: Fix typos.
* manual/charset.texi: Likewise.
* manual/errno.texi: Likewise.
* manual/filesys.texi: Likewise.
* manual/lang.texi: Likewise.
* manual/maint.texi: Likewise.
* manual/memory.texi: Likewise.
* manual/message.texi: Likewise.
* manual/resource.texi: Likewise.
* manual/search.texi: Likewise.
* manual/signal.texi: Likewise.
* manual/startup.texi: Likewise.
* manual/stdio.texi: Likewise.
* manual/sysinfo.texi: Likewise.
* manual/syslog.texi: Likewise.
* manual/time.texi: Likewise.
Patch by Ralf Wildenhues <Ralf.Wildenhues@gmx.de>.
[BZ #3465]
* sunrpc/clnt_raw.c: Minimal message improvements.
* sunrpc/pm_getmaps.c: Likewise.
* nis/nss_nisplus/nisplus-publickey.c: Likewise.
* nis/nis_print_group_entry.c: Likewise.
* locale/programs/repertoire.c: Likewise.
* locale/programs/charmap.c: Likewise.
* malloc/memusage.sh: Likewise.
* elf/dl-deps.c: Likewise.
* locale/programs/ld-collate.c: Likewise.
* libio/vswprintf.c: Likewise.
* malloc/memusagestat.c: Likewise.
* sunrpc/auth_unix.c: Likewise.
* sunrpc/rpc_main.c: Likewise.
* nscd/cache.c: Likewise.
* locale/programs/repertoire.c: Unify output messages.
* locale/programs/charmap.c: Likewise.
* locale/programs/ld-ctype.c: Likewise.
* locale/programs/ld-monetary.c: Likewise.
* locale/programs/ld-numeric.c: Likewise.
* locale/programs/ld-time.c: Likewise.
* elf/ldconfig.c: Likewise.
* nscd/selinux.c: Likewise.
* elf/cache.c: Likewise.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
2006-11-10 Jakub Jelinek <jakub@redhat.com>
* string/strxfrm_l.c (STRXFRM): Fix trailing \1 optimization
if N is one bigger than return value.
* string/tst-strxfrm2.c (do_test): Also test strxfrm with l1 + 1
and l1 last arguments, if buf is defined, verify the return value
equals to strlen (buf) and verify no byte beyond passed length
is modified.
2006-11-10 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
* sysdeps/gnu/siglist.c (__old_sys_siglist, __old_sys_sigabbrev):
Use __new_sys_siglist instead of _sys_siglist_internal as
second macro argument.
(_old_sys_siglist): Use declare_symbol_alias macro instead of
strong_alias.
2006-11-09 Ulrich Drepper <drepper@redhat.com>
[BZ #3493]
* posix/unistd.h (sysconf): Remove const attribute.
* sysdeps/posix/getaddrinfo.c (getaddrinfo): Fix test for
temporary or deprecated addresses.
Patch by Sridhar Samudrala <sri@us.ibm.com>.
* string/Makefile (tests): Add tst-strxfrm2.
* string/tst-strxfrm2.c: New file.
2006-10-09 Jakub Jelinek <jakub@redhat.com>
* elf/dl-debug.c (_dl_debug_initialize): Check r->r_map for 0
rather than r->r_brk.
* string/strxfrm_l.c (STRXFRM): Do the trailing \1 removal
optimization even if needed > n.
2006-11-07 Jakub Jelinek <jakub@redhat.com>
* include/libc-symbols.h (declare_symbol): Rename to...
(declare_symbol_alias): ... this. Add ORIGINAL argument, imply
strong_alias (ORIGINAL, SYMBOL) in asm to make sure it preceedes
.size directive.
* sysdeps/gnu/errlist-compat.awk: Adjust for declare_symbol_alias
changes.
* sysdeps/gnu/siglist.c: Likewise.
2006-11-03 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/fpu/bits/mathinline.h
[__LIBC_INTERNAL_MATH_INLINES]: Moved to ...
* sysdeps/powerpc/fpu/math_private.h: ...here. New file.
2006-11-05 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/i386/sysconf.c (intel_check_word):
Update handling of cache descriptor 0x49 for new models.
* sysdeps/unix/sysv/linux/x86_64/sysconf.c (intel_check_word):
Likewise.
2006-11-02 Ulrich Drepper <drepper@redhat.com>
* configure.in: Work around ld --help change and avoid -z relro
test completely if the architecture doesn't care about security.
2006-11-01 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
2006-10-31 Ulrich Drepper <drepper@redhat.com>
* stdlib/atexit.c (atexit): Don't mark as hidden when used to
generate compatibility version.
2006-10-29 Ulrich Drepper <drepper@redhat.com>
* configure.in: Relax -z relro requirement a bit.
* po/sv.po: Update from translation team.
2006-10-29 Jakub Jelinek <jakub@redhat.com>
* elf/dl-sym.c (do_sym): Use RTLD_SINGLE_THREAD_P.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Likewise.
* elf/dl-close.c (_dl_close_worker): Likewise.
* elf/dl-open.c (_dl_open_worker): Likewise.
* sysdeps/generic/sysdep-cancel.h (RTLD_SINGLE_THREAD_P): Define.
* configure.in: Require assembler support for visibility, compiler
support for visibility and aliases, linker support for various -z
options.
* Makeconfig: Remove conditional code which now is unnecessary.
* config.h.in: Likewise.
* config.make.in: Likewise.
* dlfcn/Makefile: Likewise.
* elf/Makefile: Likewise.
* elf/dl-load.c: Likewise.
* elf/rtld.c: Likewise.
* include/libc-symbols.h: Likewise.
* include/stdio.h: Likewise.
* io/Makefile: Likewise.
* io/fstat.c: Likewise.
* io/fstat64.c: Likewise.
* io/fstatat.c: Likewise.
* io/fstatat64.c: Likewise.
* io/lstat.c: Likewise.
* io/lstat64.c: Likewise.
* io/mknod.c: Likewise.
* io/mknodat.c: Likewise.
* io/stat.c: Likewise.
* io/stat64.c: Likewise.
* libio/stdio.c: Likewise.
* nscd/Makefile: Likewise.
* stdlib/Makefile: Likewise.
* stdlib/atexit.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/i386/sysdep.h: Likewise.
* sysdeps/i386/i686/memcmp.S: Likewise.
* sysdeps/powerpc/powerpc32/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/i386/sigaction.c: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Likewise.
* Makerules: USE_TLS support is now default.
* tls.make.c: Likewise.
* csu/Versions: Likewise.
* csu/libc-start.c: Likewise.
* csu/libc-tls.c: Likewise.
* csu/version.c: Likewise.
* dlfcn/dlinfo.c: Likewise.
* elf/dl-addr.c: Likewise.
* elf/dl-cache.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-iteratephdr.c: Likewise.
* elf/dl-load.c: Likewise.
* elf/dl-lookup.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-reloc.c: Likewise.
* elf/dl-support.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-sysdep.c: Likewise.
* elf/dl-tls.c: Likewise.
* elf/ldconfig.c: Likewise.
* elf/rtld.c: Likewise.
* elf/tst-tls-dlinfo.c: Likewise.
* elf/tst-tls1.c: Likewise.
* elf/tst-tls10.h: Likewise.
* elf/tst-tls14.c: Likewise.
* elf/tst-tls2.c: Likewise.
* elf/tst-tls3.c: Likewise.
* elf/tst-tls4.c: Likewise.
* elf/tst-tls5.c: Likewise.
* elf/tst-tls6.c: Likewise.
* elf/tst-tls7.c: Likewise.
* elf/tst-tls8.c: Likewise.
* elf/tst-tls9.c: Likewise.
* elf/tst-tlsmod1.c: Likewise.
* elf/tst-tlsmod13.c: Likewise.
* elf/tst-tlsmod13a.c: Likewise.
* elf/tst-tlsmod14a.c: Likewise.
* elf/tst-tlsmod2.c: Likewise.
* elf/tst-tlsmod3.c: Likewise.
* elf/tst-tlsmod4.c: Likewise.
* elf/tst-tlsmod5.c: Likewise.
* elf/tst-tlsmod6.c: Likewise.
* include/errno.h: Likewise.
* include/link.h: Likewise.
* include/tls.h: Likewise.
* locale/global-locale.c: Likewise.
* locale/localeinfo.h: Likewise.
* malloc/arena.c: Likewise.
* malloc/hooks.c: Likewise.
* malloc/malloc.c: Likewise.
* resolv/Versions: Likewise.
* sysdeps/alpha/dl-machine.h: Likewise.
* sysdeps/alpha/libc-tls.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/generic/tls.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/ia64/dl-machine.h: Likewise.
* sysdeps/ia64/libc-tls.c: Likewise.
* sysdeps/mach/hurd/fork.c: Likewise.
* sysdeps/mach/hurd/i386/tls.h: Likewise.
* sysdeps/powerpc/powerpc32/dl-machine.c: Likwise.
* sysdeps/powerpc/powerpc32/dl-machine.h: Likewise.
* sysdeps/powerpc/powerpc64/dl-machine.h: Likewise.
* sysdeps/s390/libc-tls.c: Likewise.
* sysdeps/s390/s390-32/dl-machine.h: Likewise.
* sysdeps/s390/s390-64/dl-machine.h: Likewise.
* sysdeps/sh/dl-machine.h: Likewise.
* sysdeps/sparc/sparc32/dl-machine.h: Likewise.
* sysdeps/sparc/sparc64/dl-machine.h: Likewise.
* sysdeps/x86_64/dl-machine.h: Likewise.
[BZ #3426]
* stdlib/stdlib.h: Adjust comment for canonicalize_file_name to
reality.
2006-10-27 Jakub Jelinek <jakub@redhat.com>
* elf/dl-lookup.c (_dl_debug_bindings): Remove unused symbol_scope
argument.
(_dl_lookup_symbol_x): Adjust caller.
* sysdeps/generic/ldsodefs.h (struct link_namespaces): Remove
_ns_global_scope.
* elf/rtld.c (dl_main): Don't initialize _ns_global_scope.
* elf/dl-libc.c: Revert l_scope name changes.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* elf/dl-close.c (_dl_close): Likewise.
* elf/dl-open.c (dl_open_worker): Likewise. If not SINGLE_THREAD_P,
always use __rtld_mrlock_{change,done}. Always free old scope list
here if not l_scope_mem.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Revert l_scope name
change. Never free scope list here. Just __rtld_mrlock_lock before
the lookup and __rtld_mrlock_unlock it after the lookup.
* elf/dl-sym.c: Likewise.
* include/link.h (struct r_scoperec): Remove.
(struct link_map): Replace l_scoperec with l_scope, l_scoperec_mem
with l_scope_mem and l_scoperec_lock with l_scope_lock.
2006-10-25 Ulrich Drepper <drepper@redhat.com>
* sysdeps/gnu/netinet/tcp.h: Define TCP_CONGESTION.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* configure.in: Disable building profile libraries by default.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* elf/dl-lookup.c (_dl_lookup_symbol_x): Add warning to
_dl_lookup_symbol_x code.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* elf/dl-runtime.c: Include sysdep-cancel.h.
(_dl_fixup, _dl_profile_fixup): Use __rtld_mrlock_* and
scoperec->nusers only if !SINGLE_THREAD_P. Use atomic_*
instead of catomic_* macros.
* elf/dl-sym.c: Include sysdep-cancel.h.
(do_sym): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-close.c: Include sysdep-cancel.h.
(_dl_close): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-open.c: Include sysdep-cancel.h.
(dl_open_worker): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Set maxfb to address of last
fastbin rather than end of fastbin array.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* sysdeps/i386/i486/bits/atomic.h (catomic_decrement): Use correct
body macro.
* sysdeps/x86_64/bits/atomic.h
(__arch_c_compare_and_exchange_val_64_acq): Add missing casts.
(catomic_decrement): Use correct body macro.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* include/atomic.h: Add a unique prefix to all local variables
in macros.
* csu/tst-atomic.c (do_test): Test also catomic_* macros.
2006-10-14 Ulrich Drepper <drepper@redhat.com>
* resolv/arpa/nameser.h: Document that ns_t_a6 is deprecated.
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Don't use get_fast_max to
determine highest fast bin to consolidate, always look into all of
them.
(do_check_malloc_state): Only require for empty bins for large
sizes in main arena.
* libio/stdio.h: Add more __wur attributes.
2006-11-12 Andreas Jaeger <aj@suse.de>
[BZ #2510]
* manual/search.texi (Hash Search Function): Clarify.
(Array Search Function): Clarify.
2006-11-12 Joseph Myers <joseph@codesourcery.com>
[BZ #2830]
* math/atest-exp.c (main): Cast hex value to mp_limb_t before
shifting.
* math/atest-exp2.c (read_mpn_hex): Likewise.
* math/atest-sincos.c (main): Likewise.
* sysdeps/unix/sysv/linux/syscalls.list: Add epoll_pwait.
* sysdeps/unix/sysv/linux/sys/epoll.h: Declare epoll_pwait.
* sysdeps/unix/sysv/linux/Versions (libc): Add epoll_pwait for
version GLIBC_2.6.
* Versions.def: Add GLIBC_2.6 for libc.
* sysdeps/i386/i486/bits/atomic.h: Add catomic_* support.
2006-10-11 Jakub Jelinek <jakub@redhat.com>
* malloc/malloc.c (_int_malloc): Remove unused any_larger variable.
* nis/nis_defaults.c (__nis_default_access): Don't call getenv twice.
* nis/nis_subr.c (nis_getnames): Use __secure_getenv instead of getenv.
* sysdeps/generic/unsecvars.h: Add NIS_PATH.
2006-10-11 Ulrich Drepper <drepper@redhat.com>
* include/atomic.c: Define catomic_* operations.
* sysdeps/x86_64/bits/atomic.h: Likewise. Fix a few minor problems.
* stdlib/cxa_finalize.c: Use catomic_* operations instead of atomic_*.
* malloc/memusage.c: Likewise.
* gmon/mcount.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-profile.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-runtime.c: Likewise.
* elf/dl-fptr.c: Likewise.
* resolv/res_libc.c: Likewise.
2006-10-10 Roland McGrath <roland@frob.com>
* sysdeps/mach/hurd/utimes.c: Use a union to avoid an improper cast.
* sysdeps/mach/hurd/futimes.c: Likewise.
* sysdeps/mach/hurd/lutimes.c: Likewise.
2006-10-09 Ulrich Drepper <drepper@redhat.com>
Jakub Jelinek <jakub@redhat.com>
Implement reference counting of scope records.
* elf/dl-close.c (_dl_close): Remove all scopes from removed objects
from the list in objects which remain. Always allocate new scope
record.
* elf/dl-open.c (dl_open_worker): When growing array for scopes,
don't resize, allocate a new one.
* elf/dl-runtime.c: Update reference counters before using a scope
array.
* elf/dl-sym.c: Likewise.
* elf/dl-libc.c: Adjust for l_scope name change.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* include/link.h: Include <rtld-lowlevel.h>. Define struct
r_scoperec. Replace r_scope with pointer to r_scoperec structure.
Add l_scoperec_lock.
* sysdeps/generic/ldsodefs.h: Include <rtld-lowlevel.h>.
* sysdeps/generic/rtld-lowlevel.h: New file.
* include/atomic.h: Rename atomic_and to atomic_and_val and
atomic_or to atomic_or_val. Define new macros atomic_and and
atomic_or which do not return values.
* sysdeps/x86_64/bits/atomic.h: Define atomic_and and atomic_or.
Various cleanups.
* sysdeps/i386/i486/bits/atomic.h: Likewise.
* po/sv.po: Update from translation team.
2006-10-07 Ulrich Drepper <drepper@redhat.com>
* Versions.def: Add GLIBC_2.6 to libpthread.
* include/shlib-compat.h (SHLIB_COMPAT): Expand parameters before use.
(versioned_symbol): Likewise.
(compat_symbol): Likewise.
* po/tr.po: Update from translation team.
* nis/Banner: Removed. It's been integral part forever and the
author info is incomplete anyway.
* libio/Banner: Likewise.
2006-10-06 Ulrich Drepper <drepper@redhat.com>
* version.h (VERSION): Bump to 2.5.90 for new development tree.
2007-01-11 22:51:07 +01:00
|
|
|
If the first @var{n} bytes possibly form a valid multibyte
|
2001-11-07 08:21:22 +01:00
|
|
|
character but the character is incomplete, the return value is
|
|
|
|
@code{(size_t) -2}. Otherwise the multibyte character sequence is invalid
|
2001-11-05 09:11:26 +01:00
|
|
|
and the return value is @code{(size_t) -1}.
|
|
|
|
|
|
|
|
The multibyte sequence is interpreted in the state represented by the
|
|
|
|
object pointed to by @var{ps}. If @var{ps} is a null pointer, a state
|
|
|
|
object local to @code{mbrlen} is used.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{mbrlen} was introduced in @w{Amendment 1} to @w{ISO C90} and
|
|
|
|
is declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The attentive reader now will note that @code{mbrlen} can be implemented
|
2001-11-05 09:11:26 +01:00
|
|
|
as
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
This is true and in fact is mentioned in the official specification.
|
|
|
|
How can this function be used to determine the length of the wide
|
|
|
|
character string created from a multibyte character string? It is not
|
|
|
|
directly usable, but we can define a function @code{mbslen} using it:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
size_t
|
|
|
|
mbslen (const char *s)
|
|
|
|
@{
|
|
|
|
mbstate_t state;
|
|
|
|
size_t result = 0;
|
|
|
|
size_t nbytes;
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
|
|
|
|
@{
|
|
|
|
if (nbytes >= (size_t) -2)
|
|
|
|
/* @r{Something is wrong.} */
|
|
|
|
return (size_t) -1;
|
|
|
|
s += nbytes;
|
|
|
|
++result;
|
|
|
|
@}
|
|
|
|
return result;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
This function simply calls @code{mbrlen} for each multibyte character
|
|
|
|
in the string and counts the number of function calls. Please note that
|
|
|
|
we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen}
|
2001-11-07 08:21:22 +01:00
|
|
|
call. This is acceptable since a) this value is larger then the length of
|
|
|
|
the longest multibyte character sequence and b) we know that the string
|
|
|
|
@var{s} ends with a NUL byte, which cannot be part of any other multibyte
|
|
|
|
character sequence but the one representing the NUL wide character.
|
2001-11-05 09:11:26 +01:00
|
|
|
Therefore, the @code{mbrlen} function will never read invalid memory.
|
|
|
|
|
|
|
|
Now that this function is available (just to make this clear, this
|
|
|
|
function is @emph{not} part of the GNU C library) we can compute the
|
|
|
|
number of wide character required to store the converted multibyte
|
|
|
|
character string @var{s} using
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
Please note that the @code{mbslen} function is quite inefficient. The
|
2001-11-07 08:21:22 +01:00
|
|
|
implementation of @code{mbstouwcs} with @code{mbslen} would have to
|
|
|
|
perform the conversion of the multibyte character input string twice, and
|
|
|
|
this conversion might be quite expensive. So it is necessary to think
|
|
|
|
about the consequences of using the easier but imprecise method before
|
2001-11-05 09:11:26 +01:00
|
|
|
doing the work twice.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps})
|
|
|
|
The @code{wcrtomb} function (``wide character restartable to
|
|
|
|
multibyte'') converts a single wide character into a multibyte string
|
|
|
|
corresponding to that wide character.
|
|
|
|
|
|
|
|
If @var{s} is a null pointer, the function resets the state stored in
|
|
|
|
the objects pointed to by @var{ps} (or the internal @code{mbstate_t}
|
|
|
|
object) to the initial state. This can also be achieved by a call like
|
|
|
|
this:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
wcrtombs (temp_buf, L'\0', ps)
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
since, if @var{s} is a null pointer, @code{wcrtomb} performs as if it
|
|
|
|
writes into an internal buffer, which is guaranteed to be large enough.
|
|
|
|
|
|
|
|
If @var{wc} is the NUL wide character, @code{wcrtomb} emits, if
|
|
|
|
necessary, a shift sequence to get the state @var{ps} into the initial
|
2001-11-07 08:21:22 +01:00
|
|
|
state followed by a single NUL byte, which is stored in the string
|
2001-11-05 09:11:26 +01:00
|
|
|
@var{s}.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Otherwise a byte sequence (possibly including shift sequences) is written
|
|
|
|
into the string @var{s}. This only happens if @var{wc} is a valid wide
|
|
|
|
character (i.e., it has a multibyte representation in the character set
|
|
|
|
selected by locale of the @code{LC_CTYPE} category). If @var{wc} is no
|
|
|
|
valid wide character, nothing is stored in the strings @var{s},
|
|
|
|
@code{errno} is set to @code{EILSEQ}, the conversion state in @var{ps}
|
2001-11-05 09:11:26 +01:00
|
|
|
is undefined and the return value is @code{(size_t) -1}.
|
|
|
|
|
|
|
|
If no error occurred the function returns the number of bytes stored in
|
|
|
|
the string @var{s}. This includes all bytes representing shift
|
|
|
|
sequences.
|
|
|
|
|
|
|
|
One word about the interface of the function: there is no parameter
|
|
|
|
specifying the length of the array @var{s}. Instead the function
|
|
|
|
assumes that there are at least @code{MB_CUR_MAX} bytes available since
|
|
|
|
this is the maximum length of any byte sequence representing a single
|
|
|
|
character. So the caller has to make sure that there is enough space
|
|
|
|
available, otherwise buffer overruns can occur.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{wcrtomb} was introduced in @w{Amendment 1} to @w{ISO C90} and is
|
|
|
|
declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
Using @code{wcrtomb} is as easy as using @code{mbrtowc}. The following
|
|
|
|
example appends a wide character string to a multibyte character string.
|
|
|
|
Again, the code is not really useful (or correct), it is simply here to
|
|
|
|
demonstrate the use and some problems.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
char *
|
|
|
|
mbscatwcs (char *s, size_t len, const wchar_t *ws)
|
|
|
|
@{
|
|
|
|
mbstate_t state;
|
|
|
|
/* @r{Find the end of the existing string.} */
|
|
|
|
char *wp = strchr (s, '\0');
|
|
|
|
len -= wp - s;
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
do
|
|
|
|
@{
|
|
|
|
size_t nbytes;
|
|
|
|
if (len < MB_CUR_LEN)
|
|
|
|
@{
|
|
|
|
/* @r{We cannot guarantee that the next}
|
|
|
|
@r{character fits into the buffer, so}
|
|
|
|
@r{return an error.} */
|
|
|
|
errno = E2BIG;
|
|
|
|
return NULL;
|
|
|
|
@}
|
|
|
|
nbytes = wcrtomb (wp, *ws, &state);
|
|
|
|
if (nbytes == (size_t) -1)
|
|
|
|
/* @r{Error in the conversion.} */
|
|
|
|
return NULL;
|
|
|
|
len -= nbytes;
|
|
|
|
wp += nbytes;
|
|
|
|
@}
|
|
|
|
while (*ws++ != L'\0');
|
|
|
|
return s;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
First the function has to find the end of the string currently in the
|
|
|
|
array @var{s}. The @code{strchr} call does this very efficiently since a
|
|
|
|
requirement for multibyte character representations is that the NUL byte
|
|
|
|
is never used except to represent itself (and in this context, the end
|
|
|
|
of the string).
|
|
|
|
|
|
|
|
After initializing the state object the loop is entered where the first
|
|
|
|
task is to make sure there is enough room in the array @var{s}. We
|
|
|
|
abort if there are not at least @code{MB_CUR_LEN} bytes available. This
|
|
|
|
is not always optimal but we have no other choice. We might have less
|
|
|
|
than @code{MB_CUR_LEN} bytes available but the next multibyte character
|
|
|
|
might also be only one byte long. At the time the @code{wcrtomb} call
|
2001-11-07 08:21:22 +01:00
|
|
|
returns it is too late to decide whether the buffer was large enough. If
|
|
|
|
this solution is unsuitable, there is a very slow but more accurate
|
2001-11-05 09:11:26 +01:00
|
|
|
solution.
|
|
|
|
|
|
|
|
@smallexample
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
if (len < MB_CUR_LEN)
|
|
|
|
@{
|
|
|
|
mbstate_t temp_state;
|
|
|
|
memcpy (&temp_state, &state, sizeof (state));
|
|
|
|
if (wcrtomb (NULL, *ws, &temp_state) > len)
|
|
|
|
@{
|
|
|
|
/* @r{We cannot guarantee that the next}
|
|
|
|
@r{character fits into the buffer, so}
|
|
|
|
@r{return an error.} */
|
|
|
|
errno = E2BIG;
|
|
|
|
return NULL;
|
|
|
|
@}
|
|
|
|
@}
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@end smallexample
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Here we perform the conversion that might overflow the buffer so that
|
|
|
|
we are afterwards in the position to make an exact decision about the
|
|
|
|
buffer size. Please note the @code{NULL} argument for the destination
|
|
|
|
buffer in the new @code{wcrtomb} call; since we are not interested in the
|
|
|
|
converted text at this point, this is a nice way to express this. The
|
|
|
|
most unusual thing about this piece of code certainly is the duplication
|
|
|
|
of the conversion state object, but if a change of the state is necessary
|
|
|
|
to emit the next multibyte character, we want to have the same shift state
|
|
|
|
change performed in the real conversion. Therefore, we have to preserve
|
2001-11-05 09:11:26 +01:00
|
|
|
the initial shift state information.
|
|
|
|
|
|
|
|
There are certainly many more and even better solutions to this problem.
|
|
|
|
This example is only provided for educational purposes.
|
|
|
|
|
|
|
|
@node Converting Strings
|
|
|
|
@subsection Converting Multibyte and Wide Character Strings
|
|
|
|
|
|
|
|
The functions described in the previous section only convert a single
|
|
|
|
character at a time. Most operations to be performed in real-world
|
|
|
|
programs include strings and therefore the @w{ISO C} standard also
|
|
|
|
defines conversions on entire strings. However, the defined set of
|
|
|
|
functions is quite limited; therefore, the GNU C library contains a few
|
|
|
|
extensions that can help in some important situations.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
|
|
|
|
The @code{mbsrtowcs} function (``multibyte string restartable to wide
|
|
|
|
character string'') converts an NUL-terminated multibyte character
|
|
|
|
string at @code{*@var{src}} into an equivalent wide character string,
|
|
|
|
including the NUL wide character at the end. The conversion is started
|
|
|
|
using the state information from the object pointed to by @var{ps} or
|
|
|
|
from an internal object of @code{mbsrtowcs} if @var{ps} is a null
|
2001-11-07 08:21:22 +01:00
|
|
|
pointer. Before returning, the state object is updated to match the state
|
2001-11-05 09:11:26 +01:00
|
|
|
after the last converted character. The state is the initial state if the
|
|
|
|
terminating NUL byte is reached and converted.
|
|
|
|
|
|
|
|
If @var{dst} is not a null pointer, the result is stored in the array
|
|
|
|
pointed to by @var{dst}; otherwise, the conversion result is not
|
|
|
|
available since it is stored in an internal buffer.
|
|
|
|
|
|
|
|
If @var{len} wide characters are stored in the array @var{dst} before
|
|
|
|
reaching the end of the input string, the conversion stops and @var{len}
|
|
|
|
is returned. If @var{dst} is a null pointer, @var{len} is never checked.
|
|
|
|
|
|
|
|
Another reason for a premature return from the function call is if the
|
|
|
|
input string contains an invalid multibyte sequence. In this case the
|
|
|
|
global variable @code{errno} is set to @code{EILSEQ} and the function
|
|
|
|
returns @code{(size_t) -1}.
|
|
|
|
|
|
|
|
@c XXX The ISO C9x draft seems to have a problem here. It says that PS
|
|
|
|
@c is not updated if DST is NULL. This is not said straightforward and
|
|
|
|
@c none of the other functions is described like this. It would make sense
|
|
|
|
@c to define the function this way but I don't think it is meant like this.
|
|
|
|
|
|
|
|
In all other cases the function returns the number of wide characters
|
|
|
|
converted during this call. If @var{dst} is not null, @code{mbsrtowcs}
|
2001-11-07 08:21:22 +01:00
|
|
|
stores in the pointer pointed to by @var{src} either a null pointer (if
|
2001-11-05 09:11:26 +01:00
|
|
|
the NUL byte in the input string was reached) or the address of the byte
|
|
|
|
following the last converted multibyte character.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
|
|
|
@code{mbsrtowcs} was introduced in @w{Amendment 1} to @w{ISO C90} and is
|
|
|
|
declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The definition of the @code{mbsrtowcs} function has one important
|
|
|
|
limitation. The requirement that @var{dst} has to be a NUL-terminated
|
2001-11-05 09:11:26 +01:00
|
|
|
string provides problems if one wants to convert buffers with text. A
|
|
|
|
buffer is normally no collection of NUL-terminated strings but instead a
|
|
|
|
continuous collection of lines, separated by newline characters. Now
|
|
|
|
assume that a function to convert one line from a buffer is needed. Since
|
|
|
|
the line is not NUL-terminated, the source pointer cannot directly point
|
|
|
|
into the unmodified text buffer. This means, either one inserts the NUL
|
|
|
|
byte at the appropriate place for the time of the @code{mbsrtowcs}
|
|
|
|
function call (which is not doable for a read-only buffer or in a
|
|
|
|
multi-threaded application) or one copies the line in an extra buffer
|
2001-11-07 08:21:22 +01:00
|
|
|
where it can be terminated by a NUL byte. Note that it is not in general
|
|
|
|
possible to limit the number of characters to convert by setting the
|
|
|
|
parameter @var{len} to any specific value. Since it is not known how
|
|
|
|
many bytes each multibyte character sequence is in length, one can only
|
2001-11-05 09:11:26 +01:00
|
|
|
guess.
|
|
|
|
|
|
|
|
@cindex stateful
|
|
|
|
There is still a problem with the method of NUL-terminating a line right
|
|
|
|
after the newline character, which could lead to very strange results.
|
|
|
|
As said in the description of the @code{mbsrtowcs} function above the
|
|
|
|
conversion state is guaranteed to be in the initial shift state after
|
|
|
|
processing the NUL byte at the end of the input string. But this NUL
|
|
|
|
byte is not really part of the text (i.e., the conversion state after
|
|
|
|
the newline in the original text could be something different than the
|
|
|
|
initial shift state and therefore the first character of the next line
|
|
|
|
is encoded using this state). But the state in question is never
|
|
|
|
accessible to the user since the conversion stops after the NUL byte
|
|
|
|
(which resets the state). Most stateful character sets in use today
|
|
|
|
require that the shift state after a newline be the initial state--but
|
|
|
|
this is not a strict guarantee. Therefore, simply NUL-terminating a
|
2001-11-07 08:21:22 +01:00
|
|
|
piece of a running text is not always an adequate solution and,
|
2001-11-05 09:11:26 +01:00
|
|
|
therefore, should never be used in generally used code.
|
|
|
|
|
|
|
|
The generic conversion interface (@pxref{Generic Charset Conversion})
|
|
|
|
does not have this limitation (it simply works on buffers, not
|
|
|
|
strings), and the GNU C library contains a set of functions that take
|
|
|
|
additional parameters specifying the maximal number of bytes that are
|
|
|
|
consumed from the input string. This way the problem of
|
|
|
|
@code{mbsrtowcs}'s example above could be solved by determining the line
|
|
|
|
length and passing this length to the function.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
|
|
|
|
The @code{wcsrtombs} function (``wide character string restartable to
|
|
|
|
multibyte string'') converts the NUL-terminated wide character string at
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{*@var{src}} into an equivalent multibyte character string and
|
2001-11-05 09:11:26 +01:00
|
|
|
stores the result in the array pointed to by @var{dst}. The NUL wide
|
|
|
|
character is also converted. The conversion starts in the state
|
|
|
|
described in the object pointed to by @var{ps} or by a state object
|
|
|
|
locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If
|
|
|
|
@var{dst} is a null pointer, the conversion is performed as usual but the
|
|
|
|
result is not available. If all characters of the input string were
|
2001-11-07 08:21:22 +01:00
|
|
|
successfully converted and if @var{dst} is not a null pointer, the
|
2001-11-05 09:11:26 +01:00
|
|
|
pointer pointed to by @var{src} gets assigned a null pointer.
|
|
|
|
|
|
|
|
If one of the wide characters in the input string has no valid multibyte
|
|
|
|
character equivalent, the conversion stops early, sets the global
|
|
|
|
variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}.
|
|
|
|
|
|
|
|
Another reason for a premature stop is if @var{dst} is not a null
|
|
|
|
pointer and the next converted character would require more than
|
|
|
|
@var{len} bytes in total to the array @var{dst}. In this case (and if
|
|
|
|
@var{dest} is not a null pointer) the pointer pointed to by @var{src} is
|
|
|
|
assigned a value pointing to the wide character right after the last one
|
|
|
|
successfully converted.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Except in the case of an encoding error the return value of the
|
|
|
|
@code{wcsrtombs} function is the number of bytes in all the multibyte
|
|
|
|
character sequences stored in @var{dst}. Before returning the state in
|
|
|
|
the object pointed to by @var{ps} (or the internal object in case
|
|
|
|
@var{ps} is a null pointer) is updated to reflect the state after the
|
|
|
|
last conversion. The state is the initial shift state in case the
|
2001-11-05 09:11:26 +01:00
|
|
|
terminating NUL wide character was converted.
|
|
|
|
|
|
|
|
@pindex wchar.h
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{wcsrtombs} function was introduced in @w{Amendment 1} to
|
2001-11-05 09:11:26 +01:00
|
|
|
@w{ISO C90} and is declared in @file{wchar.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
The restriction mentioned above for the @code{mbsrtowcs} function applies
|
|
|
|
here also. There is no possibility of directly controlling the number of
|
2001-11-07 08:21:22 +01:00
|
|
|
input characters. One has to place the NUL wide character at the correct
|
|
|
|
place or control the consumed input indirectly via the available output
|
2001-11-05 09:11:26 +01:00
|
|
|
array size (the @var{len} parameter).
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment GNU
|
|
|
|
@deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps})
|
|
|
|
The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs}
|
|
|
|
function. All the parameters are the same except for @var{nmc}, which is
|
|
|
|
new. The return value is the same as for @code{mbsrtowcs}.
|
|
|
|
|
|
|
|
This new parameter specifies how many bytes at most can be used from the
|
2001-11-07 08:21:22 +01:00
|
|
|
multibyte character string. In other words, the multibyte character
|
|
|
|
string @code{*@var{src}} need not be NUL-terminated. But if a NUL byte
|
|
|
|
is found within the @var{nmc} first bytes of the string, the conversion
|
2001-11-05 09:11:26 +01:00
|
|
|
stops here.
|
|
|
|
|
|
|
|
This function is a GNU extension. It is meant to work around the
|
|
|
|
problems mentioned above. Now it is possible to convert a buffer with
|
|
|
|
multibyte character text piece for piece without having to care about
|
|
|
|
inserting NUL bytes and the effect of NUL bytes on the conversion state.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
A function to convert a multibyte string into a wide character string
|
|
|
|
and display it could be written like this (this is not a really useful
|
|
|
|
example):
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
void
|
|
|
|
showmbs (const char *src, FILE *fp)
|
|
|
|
@{
|
|
|
|
mbstate_t state;
|
|
|
|
int cnt = 0;
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
while (1)
|
|
|
|
@{
|
|
|
|
wchar_t linebuf[100];
|
|
|
|
const char *endp = strchr (src, '\n');
|
|
|
|
size_t n;
|
|
|
|
|
|
|
|
/* @r{Exit if there is no more line.} */
|
|
|
|
if (endp == NULL)
|
|
|
|
break;
|
|
|
|
|
|
|
|
n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
|
|
|
|
linebuf[n] = L'\0';
|
|
|
|
fprintf (fp, "line %d: \"%S\"\n", linebuf);
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
There is no problem with the state after a call to @code{mbsnrtowcs}.
|
|
|
|
Since we don't insert characters in the strings that were not in there
|
|
|
|
right from the beginning and we use @var{state} only for the conversion
|
|
|
|
of the given buffer, there is no problem with altering the state.
|
|
|
|
|
|
|
|
@comment wchar.h
|
|
|
|
@comment GNU
|
|
|
|
@deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps})
|
|
|
|
The @code{wcsnrtombs} function implements the conversion from wide
|
|
|
|
character strings to multibyte character strings. It is similar to
|
|
|
|
@code{wcsrtombs} but, just like @code{mbsnrtowcs}, it takes an extra
|
|
|
|
parameter, which specifies the length of the input string.
|
|
|
|
|
|
|
|
No more than @var{nwc} wide characters from the input string
|
|
|
|
@code{*@var{src}} are converted. If the input string contains a NUL
|
|
|
|
wide character in the first @var{nwc} characters, the conversion stops at
|
|
|
|
this place.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{wcsnrtombs} function is a GNU extension and just like
|
|
|
|
@code{mbsnrtowcs} helps in situations where no NUL-terminated input
|
2001-11-05 09:11:26 +01:00
|
|
|
strings are available.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
|
|
|
|
@node Multibyte Conversion Example
|
|
|
|
@subsection A Complete Multibyte Conversion Example
|
|
|
|
|
|
|
|
The example programs given in the last sections are only brief and do
|
|
|
|
not contain all the error checking, etc. Presented here is a complete
|
|
|
|
and documented example. It features the @code{mbrtowc} function but it
|
|
|
|
should be easy to derive versions using the other functions.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
int
|
|
|
|
file_mbsrtowcs (int input, int output)
|
|
|
|
@{
|
|
|
|
/* @r{Note the use of @code{MB_LEN_MAX}.}
|
|
|
|
@r{@code{MB_CUR_MAX} cannot portably be used here.} */
|
|
|
|
char buffer[BUFSIZ + MB_LEN_MAX];
|
|
|
|
mbstate_t state;
|
|
|
|
int filled = 0;
|
|
|
|
int eof = 0;
|
|
|
|
|
|
|
|
/* @r{Initialize the state.} */
|
|
|
|
memset (&state, '\0', sizeof (state));
|
|
|
|
|
|
|
|
while (!eof)
|
|
|
|
@{
|
|
|
|
ssize_t nread;
|
|
|
|
ssize_t nwrite;
|
|
|
|
char *inp = buffer;
|
|
|
|
wchar_t outbuf[BUFSIZ];
|
|
|
|
wchar_t *outp = outbuf;
|
|
|
|
|
|
|
|
/* @r{Fill up the buffer from the input file.} */
|
|
|
|
nread = read (input, buffer + filled, BUFSIZ);
|
|
|
|
if (nread < 0)
|
|
|
|
@{
|
|
|
|
perror ("read");
|
|
|
|
return 0;
|
|
|
|
@}
|
|
|
|
/* @r{If we reach end of file, make a note to read no more.} */
|
|
|
|
if (nread == 0)
|
|
|
|
eof = 1;
|
|
|
|
|
|
|
|
/* @r{@code{filled} is now the number of bytes in @code{buffer}.} */
|
|
|
|
filled += nread;
|
|
|
|
|
|
|
|
/* @r{Convert those bytes to wide characters--as many as we can.} */
|
|
|
|
while (1)
|
|
|
|
@{
|
|
|
|
size_t thislen = mbrtowc (outp, inp, filled, &state);
|
|
|
|
/* @r{Stop converting at invalid character;}
|
|
|
|
@r{this can mean we have read just the first part}
|
|
|
|
@r{of a valid character.} */
|
|
|
|
if (thislen == (size_t) -1)
|
|
|
|
break;
|
|
|
|
/* @r{We want to handle embedded NUL bytes}
|
|
|
|
@r{but the return value is 0. Correct this.} */
|
|
|
|
if (thislen == 0)
|
|
|
|
thislen = 1;
|
|
|
|
/* @r{Advance past this character.} */
|
|
|
|
inp += thislen;
|
|
|
|
filled -= thislen;
|
|
|
|
++outp;
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{Write the wide characters we just made.} */
|
|
|
|
nwrite = write (output, outbuf,
|
|
|
|
(outp - outbuf) * sizeof (wchar_t));
|
|
|
|
if (nwrite < 0)
|
|
|
|
@{
|
|
|
|
perror ("write");
|
|
|
|
return 0;
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{See if we have a @emph{real} invalid character.} */
|
|
|
|
if ((eof && filled > 0) || filled >= MB_CUR_MAX)
|
|
|
|
@{
|
|
|
|
error (0, 0, "invalid multibyte character");
|
|
|
|
return 0;
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{If any characters must be carried forward,}
|
|
|
|
@r{put them at the beginning of @code{buffer}.} */
|
|
|
|
if (filled > 0)
|
2009-06-01 20:43:26 +02:00
|
|
|
memmove (buffer, inp, filled);
|
2001-11-05 09:11:26 +01:00
|
|
|
@}
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
|
|
|
|
@node Non-reentrant Conversion
|
|
|
|
@section Non-reentrant Conversion Function
|
|
|
|
|
|
|
|
The functions described in the previous chapter are defined in
|
2001-11-07 08:21:22 +01:00
|
|
|
@w{Amendment 1} to @w{ISO C90}, but the original @w{ISO C90} standard
|
|
|
|
also contained functions for character set conversion. The reason that
|
|
|
|
these original functions are not described first is that they are almost
|
2001-11-05 09:11:26 +01:00
|
|
|
entirely useless.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The problem is that all the conversion functions described in the
|
|
|
|
original @w{ISO C90} use a local state. Using a local state implies that
|
|
|
|
multiple conversions at the same time (not only when using threads)
|
|
|
|
cannot be done, and that you cannot first convert single characters and
|
|
|
|
then strings since you cannot tell the conversion functions which state
|
2001-11-05 09:11:26 +01:00
|
|
|
to use.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
These original functions are therefore usable only in a very limited set
|
2001-11-05 09:11:26 +01:00
|
|
|
of situations. One must complete converting the entire string before
|
|
|
|
starting a new one, and each string/text must be converted with the same
|
|
|
|
function (there is no problem with the library itself; it is guaranteed
|
|
|
|
that no library function changes the state of any of these functions).
|
|
|
|
@strong{For the above reasons it is highly requested that the functions
|
2001-11-07 08:21:22 +01:00
|
|
|
described in the previous section be used in place of non-reentrant
|
2001-11-05 09:11:26 +01:00
|
|
|
conversion functions.}
|
|
|
|
|
|
|
|
@menu
|
|
|
|
* Non-reentrant Character Conversion:: Non-reentrant Conversion of Single
|
|
|
|
Characters.
|
|
|
|
* Non-reentrant String Conversion:: Non-reentrant Conversion of Strings.
|
|
|
|
* Shift State:: States in Non-reentrant Functions.
|
|
|
|
@end menu
|
|
|
|
|
|
|
|
@node Non-reentrant Character Conversion
|
|
|
|
@subsection Non-reentrant Conversion of Single Characters
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size})
|
|
|
|
The @code{mbtowc} (``multibyte to wide character'') function when called
|
|
|
|
with non-null @var{string} converts the first multibyte character
|
|
|
|
beginning at @var{string} to its corresponding wide character code. It
|
|
|
|
stores the result in @code{*@var{result}}.
|
|
|
|
|
|
|
|
@code{mbtowc} never examines more than @var{size} bytes. (The idea is
|
|
|
|
to supply for @var{size} the number of bytes of data you have in hand.)
|
|
|
|
|
|
|
|
@code{mbtowc} with non-null @var{string} distinguishes three
|
|
|
|
possibilities: the first @var{size} bytes at @var{string} start with
|
|
|
|
valid multibyte characters, they start with an invalid byte sequence or
|
|
|
|
just part of a character, or @var{string} points to an empty string (a
|
|
|
|
null character).
|
|
|
|
|
|
|
|
For a valid multibyte character, @code{mbtowc} converts it to a wide
|
|
|
|
character and stores that in @code{*@var{result}}, and returns the
|
|
|
|
number of bytes in that character (always at least @math{1} and never
|
|
|
|
more than @var{size}).
|
|
|
|
|
|
|
|
For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an
|
|
|
|
empty string, it returns @math{0}, also storing @code{'\0'} in
|
|
|
|
@code{*@var{result}}.
|
|
|
|
|
|
|
|
If the multibyte character code uses shift characters, then
|
|
|
|
@code{mbtowc} maintains and updates a shift state as it scans. If you
|
|
|
|
call @code{mbtowc} with a null pointer for @var{string}, that
|
|
|
|
initializes the shift state to its standard initial value. It also
|
|
|
|
returns nonzero if the multibyte character code in use actually has a
|
|
|
|
shift state. @xref{Shift State}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar})
|
|
|
|
The @code{wctomb} (``wide character to multibyte'') function converts
|
|
|
|
the wide character code @var{wchar} to its corresponding multibyte
|
|
|
|
character sequence, and stores the result in bytes starting at
|
|
|
|
@var{string}. At most @code{MB_CUR_MAX} characters are stored.
|
|
|
|
|
|
|
|
@code{wctomb} with non-null @var{string} distinguishes three
|
|
|
|
possibilities for @var{wchar}: a valid wide character code (one that can
|
2001-11-07 08:21:22 +01:00
|
|
|
be translated to a multibyte character), an invalid code, and
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{L'\0'}.
|
|
|
|
|
|
|
|
Given a valid code, @code{wctomb} converts it to a multibyte character,
|
|
|
|
storing the bytes starting at @var{string}. Then it returns the number
|
|
|
|
of bytes in that character (always at least @math{1} and never more
|
|
|
|
than @code{MB_CUR_MAX}).
|
|
|
|
|
|
|
|
If @var{wchar} is an invalid wide character code, @code{wctomb} returns
|
|
|
|
@math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also
|
|
|
|
storing @code{'\0'} in @code{*@var{string}}.
|
|
|
|
|
|
|
|
If the multibyte character code uses shift characters, then
|
|
|
|
@code{wctomb} maintains and updates a shift state as it scans. If you
|
|
|
|
call @code{wctomb} with a null pointer for @var{string}, that
|
|
|
|
initializes the shift state to its standard initial value. It also
|
|
|
|
returns nonzero if the multibyte character code in use actually has a
|
|
|
|
shift state. @xref{Shift State}.
|
|
|
|
|
|
|
|
Calling this function with a @var{wchar} argument of zero when
|
|
|
|
@var{string} is not null has the side-effect of reinitializing the
|
|
|
|
stored shift state @emph{as well as} storing the multibyte character
|
|
|
|
@code{'\0'} and returning @math{0}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
Similar to @code{mbrlen} there is also a non-reentrant function that
|
|
|
|
computes the length of a multibyte character. It can be defined in
|
|
|
|
terms of @code{mbtowc}.
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun int mblen (const char *@var{string}, size_t @var{size})
|
|
|
|
The @code{mblen} function with a non-null @var{string} argument returns
|
|
|
|
the number of bytes that make up the multibyte character beginning at
|
|
|
|
@var{string}, never examining more than @var{size} bytes. (The idea is
|
|
|
|
to supply for @var{size} the number of bytes of data you have in hand.)
|
|
|
|
|
|
|
|
The return value of @code{mblen} distinguishes three possibilities: the
|
|
|
|
first @var{size} bytes at @var{string} start with valid multibyte
|
|
|
|
characters, they start with an invalid byte sequence or just part of a
|
|
|
|
character, or @var{string} points to an empty string (a null character).
|
|
|
|
|
|
|
|
For a valid multibyte character, @code{mblen} returns the number of
|
|
|
|
bytes in that character (always at least @code{1} and never more than
|
2001-11-07 08:21:22 +01:00
|
|
|
@var{size}). For an invalid byte sequence, @code{mblen} returns
|
2001-11-05 09:11:26 +01:00
|
|
|
@math{-1}. For an empty string, it returns @math{0}.
|
|
|
|
|
|
|
|
If the multibyte character code uses shift characters, then @code{mblen}
|
|
|
|
maintains and updates a shift state as it scans. If you call
|
|
|
|
@code{mblen} with a null pointer for @var{string}, that initializes the
|
|
|
|
shift state to its standard initial value. It also returns a nonzero
|
|
|
|
value if the multibyte character code in use actually has a shift state.
|
|
|
|
@xref{Shift State}.
|
|
|
|
|
|
|
|
@pindex stdlib.h
|
|
|
|
The function @code{mblen} is declared in @file{stdlib.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
|
|
|
|
@node Non-reentrant String Conversion
|
|
|
|
@subsection Non-reentrant Conversion of Strings
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
For convenience the @w{ISO C90} standard also defines functions to
|
2001-11-05 09:11:26 +01:00
|
|
|
convert entire strings instead of single characters. These functions
|
|
|
|
suffer from the same problems as their reentrant counterparts from
|
|
|
|
@w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}.
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size})
|
|
|
|
The @code{mbstowcs} (``multibyte string to wide character string'')
|
|
|
|
function converts the null-terminated string of multibyte characters
|
|
|
|
@var{string} to an array of wide character codes, storing not more than
|
|
|
|
@var{size} wide characters into the array beginning at @var{wstring}.
|
|
|
|
The terminating null character counts towards the size, so if @var{size}
|
|
|
|
is less than the actual number of wide characters resulting from
|
|
|
|
@var{string}, no terminating null character is stored.
|
|
|
|
|
|
|
|
The conversion of characters from @var{string} begins in the initial
|
|
|
|
shift state.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
If an invalid multibyte character sequence is found, the @code{mbstowcs}
|
|
|
|
function returns a value of @math{-1}. Otherwise, it returns the number
|
|
|
|
of wide characters stored in the array @var{wstring}. This number does
|
|
|
|
not include the terminating null character, which is present if the
|
2001-11-05 09:11:26 +01:00
|
|
|
number is less than @var{size}.
|
|
|
|
|
|
|
|
Here is an example showing how to convert a string of multibyte
|
|
|
|
characters, allocating enough space for the result.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
wchar_t *
|
|
|
|
mbstowcs_alloc (const char *string)
|
|
|
|
@{
|
|
|
|
size_t size = strlen (string) + 1;
|
|
|
|
wchar_t *buf = xmalloc (size * sizeof (wchar_t));
|
|
|
|
|
|
|
|
size = mbstowcs (buf, string, size);
|
|
|
|
if (size == (size_t) -1)
|
|
|
|
return NULL;
|
|
|
|
buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
|
|
|
|
return buf;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
@comment stdlib.h
|
|
|
|
@comment ISO
|
|
|
|
@deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size})
|
|
|
|
The @code{wcstombs} (``wide character string to multibyte string'')
|
|
|
|
function converts the null-terminated wide character array @var{wstring}
|
|
|
|
into a string containing multibyte characters, storing not more than
|
|
|
|
@var{size} bytes starting at @var{string}, followed by a terminating
|
|
|
|
null character if there is room. The conversion of characters begins in
|
|
|
|
the initial shift state.
|
|
|
|
|
|
|
|
The terminating null character counts towards the size, so if @var{size}
|
|
|
|
is less than or equal to the number of bytes needed in @var{wstring}, no
|
|
|
|
terminating null character is stored.
|
|
|
|
|
|
|
|
If a code that does not correspond to a valid multibyte character is
|
2001-11-07 08:21:22 +01:00
|
|
|
found, the @code{wcstombs} function returns a value of @math{-1}.
|
|
|
|
Otherwise, the return value is the number of bytes stored in the array
|
|
|
|
@var{string}. This number does not include the terminating null character,
|
2001-11-05 09:11:26 +01:00
|
|
|
which is present if the number is less than @var{size}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
@node Shift State
|
|
|
|
@subsection States in Non-reentrant Functions
|
|
|
|
|
|
|
|
In some multibyte character codes, the @emph{meaning} of any particular
|
|
|
|
byte sequence is not fixed; it depends on what other sequences have come
|
2001-11-07 08:21:22 +01:00
|
|
|
earlier in the same string. Typically there are just a few sequences that
|
|
|
|
can change the meaning of other sequences; these few are called
|
2001-11-05 09:11:26 +01:00
|
|
|
@dfn{shift sequences} and we say that they set the @dfn{shift state} for
|
|
|
|
other sequences that follow.
|
|
|
|
|
|
|
|
To illustrate shift state and shift sequences, suppose we decide that
|
|
|
|
the sequence @code{0200} (just one byte) enters Japanese mode, in which
|
|
|
|
pairs of bytes in the range from @code{0240} to @code{0377} are single
|
|
|
|
characters, while @code{0201} enters Latin-1 mode, in which single bytes
|
|
|
|
in the range from @code{0240} to @code{0377} are characters, and
|
|
|
|
interpreted according to the ISO Latin-1 character set. This is a
|
|
|
|
multibyte code that has two alternative shift states (``Japanese mode''
|
|
|
|
and ``Latin-1 mode''), and two shift sequences that specify particular
|
|
|
|
shift states.
|
|
|
|
|
|
|
|
When the multibyte character code in use has shift states, then
|
|
|
|
@code{mblen}, @code{mbtowc}, and @code{wctomb} must maintain and update
|
|
|
|
the current shift state as they scan the string. To make this work
|
|
|
|
properly, you must follow these rules:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
|
|
|
Before starting to scan a string, call the function with a null pointer
|
|
|
|
for the multibyte character address---for example, @code{mblen (NULL,
|
|
|
|
0)}. This initializes the shift state to its standard initial value.
|
|
|
|
|
|
|
|
@item
|
|
|
|
Scan the string one character at a time, in order. Do not ``back up''
|
|
|
|
and rescan characters already scanned, and do not intersperse the
|
|
|
|
processing of different strings.
|
|
|
|
@end itemize
|
|
|
|
|
|
|
|
Here is an example of using @code{mblen} following these rules:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
void
|
|
|
|
scan_string (char *s)
|
|
|
|
@{
|
|
|
|
int length = strlen (s);
|
|
|
|
|
|
|
|
/* @r{Initialize shift state.} */
|
|
|
|
mblen (NULL, 0);
|
|
|
|
|
|
|
|
while (1)
|
|
|
|
@{
|
|
|
|
int thischar = mblen (s, length);
|
|
|
|
/* @r{Deal with end of string and invalid characters.} */
|
|
|
|
if (thischar == 0)
|
|
|
|
break;
|
|
|
|
if (thischar == -1)
|
|
|
|
@{
|
|
|
|
error ("invalid multibyte character");
|
|
|
|
break;
|
|
|
|
@}
|
|
|
|
/* @r{Advance past this character.} */
|
|
|
|
s += thischar;
|
|
|
|
length -= thischar;
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not
|
|
|
|
reentrant when using a multibyte code that uses a shift state. However,
|
|
|
|
no other library functions call these functions, so you don't have to
|
|
|
|
worry that the shift state will be changed mysteriously.
|
|
|
|
|
|
|
|
|
|
|
|
@node Generic Charset Conversion
|
|
|
|
@section Generic Charset Conversion
|
|
|
|
|
|
|
|
The conversion functions mentioned so far in this chapter all had in
|
|
|
|
common that they operate on character sets that are not directly
|
|
|
|
specified by the functions. The multibyte encoding used is specified by
|
|
|
|
the currently selected locale for the @code{LC_CTYPE} category. The
|
|
|
|
wide character set is fixed by the implementation (in the case of GNU C
|
|
|
|
library it is always UCS-4 encoded @w{ISO 10646}.
|
|
|
|
|
|
|
|
This has of course several problems when it comes to general character
|
|
|
|
conversion:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
2001-11-07 08:21:22 +01:00
|
|
|
For every conversion where neither the source nor the destination
|
|
|
|
character set is the character set of the locale for the @code{LC_CTYPE}
|
|
|
|
category, one has to change the @code{LC_CTYPE} locale using
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{setlocale}.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Changing the @code{LC_TYPE} locale introduces major problems for the rest
|
|
|
|
of the programs since several more functions (e.g., the character
|
|
|
|
classification functions, @pxref{Classification of Characters}) use the
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{LC_CTYPE} category.
|
|
|
|
|
|
|
|
@item
|
|
|
|
Parallel conversions to and from different character sets are not
|
|
|
|
possible since the @code{LC_CTYPE} selection is global and shared by all
|
|
|
|
threads.
|
|
|
|
|
|
|
|
@item
|
|
|
|
If neither the source nor the destination character set is the character
|
|
|
|
set used for @code{wchar_t} representation, there is at least a two-step
|
2001-11-07 08:21:22 +01:00
|
|
|
process necessary to convert a text using the functions above. One would
|
|
|
|
have to select the source character set as the multibyte encoding,
|
2001-11-05 09:11:26 +01:00
|
|
|
convert the text into a @code{wchar_t} text, select the destination
|
|
|
|
character set as the multibyte encoding, and convert the wide character
|
|
|
|
text to the multibyte (@math{=} destination) character set.
|
|
|
|
|
|
|
|
Even if this is possible (which is not guaranteed) it is a very tiring
|
|
|
|
work. Plus it suffers from the other two raised points even more due to
|
|
|
|
the steady changing of the locale.
|
|
|
|
@end itemize
|
|
|
|
|
|
|
|
The XPG2 standard defines a completely new set of functions, which has
|
|
|
|
none of these limitations. They are not at all coupled to the selected
|
|
|
|
locales, and they have no constraints on the character sets selected for
|
2001-11-07 08:21:22 +01:00
|
|
|
source and destination. Only the set of available conversions limits
|
|
|
|
them. The standard does not specify that any conversion at all must be
|
|
|
|
available. Such availability is a measure of the quality of the
|
2001-11-05 09:11:26 +01:00
|
|
|
implementation.
|
|
|
|
|
|
|
|
In the following text first the interface to @code{iconv} and then the
|
|
|
|
conversion function, will be described. Comparisons with other
|
|
|
|
implementations will show what obstacles stand in the way of portable
|
2001-11-07 08:21:22 +01:00
|
|
|
applications. Finally, the implementation is described in so far as might
|
2001-11-05 09:11:26 +01:00
|
|
|
interest the advanced user who wants to extend conversion capabilities.
|
|
|
|
|
|
|
|
@menu
|
|
|
|
* Generic Conversion Interface:: Generic Character Set Conversion Interface.
|
|
|
|
* iconv Examples:: A complete @code{iconv} example.
|
|
|
|
* Other iconv Implementations:: Some Details about other @code{iconv}
|
|
|
|
Implementations.
|
|
|
|
* glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C
|
|
|
|
library.
|
|
|
|
@end menu
|
|
|
|
|
|
|
|
@node Generic Conversion Interface
|
|
|
|
@subsection Generic Character Set Conversion Interface
|
|
|
|
|
|
|
|
This set of functions follows the traditional cycle of using a resource:
|
|
|
|
open--use--close. The interface consists of three functions, each of
|
|
|
|
which implements one step.
|
|
|
|
|
|
|
|
Before the interfaces are described it is necessary to introduce a
|
|
|
|
data type. Just like other open--use--close interfaces the functions
|
|
|
|
introduced here work using handles and the @file{iconv.h} header
|
|
|
|
defines a special type for the handles used.
|
|
|
|
|
|
|
|
@comment iconv.h
|
|
|
|
@comment XPG2
|
|
|
|
@deftp {Data Type} iconv_t
|
|
|
|
This data type is an abstract type defined in @file{iconv.h}. The user
|
|
|
|
must not assume anything about the definition of this type; it must be
|
|
|
|
completely opaque.
|
|
|
|
|
|
|
|
Objects of this type can get assigned handles for the conversions using
|
|
|
|
the @code{iconv} functions. The objects themselves need not be freed, but
|
|
|
|
the conversions for which the handles stand for have to.
|
|
|
|
@end deftp
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
The first step is the function to create a handle.
|
|
|
|
|
|
|
|
@comment iconv.h
|
|
|
|
@comment XPG2
|
|
|
|
@deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode})
|
|
|
|
The @code{iconv_open} function has to be used before starting a
|
|
|
|
conversion. The two parameters this function takes determine the
|
|
|
|
source and destination character set for the conversion, and if the
|
|
|
|
implementation has the possibility to perform such a conversion, the
|
|
|
|
function returns a handle.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
If the wanted conversion is not available, the @code{iconv_open} function
|
|
|
|
returns @code{(iconv_t) -1}. In this case the global variable
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{errno} can have the following values:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item EMFILE
|
|
|
|
The process already has @code{OPEN_MAX} file descriptors open.
|
|
|
|
@item ENFILE
|
|
|
|
The system limit of open file is reached.
|
|
|
|
@item ENOMEM
|
|
|
|
Not enough memory to carry out the operation.
|
|
|
|
@item EINVAL
|
|
|
|
The conversion from @var{fromcode} to @var{tocode} is not supported.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
It is not possible to use the same descriptor in different threads to
|
|
|
|
perform independent conversions. The data structures associated
|
|
|
|
with the descriptor include information about the conversion state.
|
|
|
|
This must not be messed up by using it in different conversions.
|
|
|
|
|
|
|
|
An @code{iconv} descriptor is like a file descriptor as for every use a
|
|
|
|
new descriptor must be created. The descriptor does not stand for all
|
|
|
|
of the conversions from @var{fromset} to @var{toset}.
|
|
|
|
|
|
|
|
The GNU C library implementation of @code{iconv_open} has one
|
|
|
|
significant extension to other implementations. To ease the extension
|
|
|
|
of the set of available conversions, the implementation allows storing
|
2001-11-07 08:21:22 +01:00
|
|
|
the necessary files with data and code in an arbitrary number of
|
2001-11-05 09:11:26 +01:00
|
|
|
directories. How this extension must be written will be explained below
|
|
|
|
(@pxref{glibc iconv Implementation}). Here it is only important to say
|
|
|
|
that all directories mentioned in the @code{GCONV_PATH} environment
|
|
|
|
variable are considered only if they contain a file @file{gconv-modules}.
|
|
|
|
These directories need not necessarily be created by the system
|
|
|
|
administrator. In fact, this extension is introduced to help users
|
2001-11-07 08:21:22 +01:00
|
|
|
writing and using their own, new conversions. Of course, this does not
|
2001-11-05 09:11:26 +01:00
|
|
|
work for security reasons in SUID binaries; in this case only the system
|
2001-11-07 08:21:22 +01:00
|
|
|
directory is considered and this normally is
|
|
|
|
@file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment
|
|
|
|
variable is examined exactly once at the first call of the
|
|
|
|
@code{iconv_open} function. Later modifications of the variable have no
|
2001-11-05 09:11:26 +01:00
|
|
|
effect.
|
|
|
|
|
|
|
|
@pindex iconv.h
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{iconv_open} function was introduced early in the X/Open
|
|
|
|
Portability Guide, @w{version 2}. It is supported by all commercial
|
|
|
|
Unices as it is required for the Unix branding. However, the quality and
|
|
|
|
completeness of the implementation varies widely. The @code{iconv_open}
|
2001-11-05 09:11:26 +01:00
|
|
|
function is declared in @file{iconv.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
The @code{iconv} implementation can associate large data structure with
|
2001-11-07 08:21:22 +01:00
|
|
|
the handle returned by @code{iconv_open}. Therefore, it is crucial to
|
|
|
|
free all the resources once all conversions are carried out and the
|
2001-11-05 09:11:26 +01:00
|
|
|
conversion is not needed anymore.
|
|
|
|
|
|
|
|
@comment iconv.h
|
|
|
|
@comment XPG2
|
|
|
|
@deftypefun int iconv_close (iconv_t @var{cd})
|
|
|
|
The @code{iconv_close} function frees all resources associated with the
|
|
|
|
handle @var{cd}, which must have been returned by a successful call to
|
|
|
|
the @code{iconv_open} function.
|
|
|
|
|
|
|
|
If the function call was successful the return value is @math{0}.
|
|
|
|
Otherwise it is @math{-1} and @code{errno} is set appropriately.
|
|
|
|
Defined error are:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item EBADF
|
|
|
|
The conversion descriptor is invalid.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
@pindex iconv.h
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{iconv_close} function was introduced together with the rest
|
2001-11-05 09:11:26 +01:00
|
|
|
of the @code{iconv} functions in XPG2 and is declared in @file{iconv.h}.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
The standard defines only one actual conversion function. This has,
|
|
|
|
therefore, the most general interface: it allows conversion from one
|
|
|
|
buffer to another. Conversion from a file to a buffer, vice versa, or
|
|
|
|
even file to file can be implemented on top of it.
|
|
|
|
|
|
|
|
@comment iconv.h
|
|
|
|
@comment XPG2
|
|
|
|
@deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
|
|
|
|
@cindex stateful
|
|
|
|
The @code{iconv} function converts the text in the input buffer
|
|
|
|
according to the rules associated with the descriptor @var{cd} and
|
|
|
|
stores the result in the output buffer. It is possible to call the
|
|
|
|
function for the same text several times in a row since for stateful
|
|
|
|
character sets the necessary state information is kept in the data
|
|
|
|
structures associated with the descriptor.
|
|
|
|
|
|
|
|
The input buffer is specified by @code{*@var{inbuf}} and it contains
|
|
|
|
@code{*@var{inbytesleft}} bytes. The extra indirection is necessary for
|
|
|
|
communicating the used input back to the caller (see below). It is
|
|
|
|
important to note that the buffer pointer is of type @code{char} and the
|
|
|
|
length is measured in bytes even if the input text is encoded in wide
|
|
|
|
characters.
|
|
|
|
|
|
|
|
The output buffer is specified in a similar way. @code{*@var{outbuf}}
|
|
|
|
points to the beginning of the buffer with at least
|
|
|
|
@code{*@var{outbytesleft}} bytes room for the result. The buffer
|
|
|
|
pointer again is of type @code{char} and the length is measured in
|
|
|
|
bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer, the
|
|
|
|
conversion is performed but no output is available.
|
|
|
|
|
|
|
|
If @var{inbuf} is a null pointer, the @code{iconv} function performs the
|
|
|
|
necessary action to put the state of the conversion into the initial
|
|
|
|
state. This is obviously a no-op for non-stateful encodings, but if the
|
|
|
|
encoding has a state, such a function call might put some byte sequences
|
|
|
|
in the output buffer, which perform the necessary state changes. The
|
|
|
|
next call with @var{inbuf} not being a null pointer then simply goes on
|
|
|
|
from the initial state. It is important that the programmer never makes
|
2001-11-07 08:21:22 +01:00
|
|
|
any assumption as to whether the conversion has to deal with states.
|
|
|
|
Even if the input and output character sets are not stateful, the
|
2001-11-05 09:11:26 +01:00
|
|
|
implementation might still have to keep states. This is due to the
|
|
|
|
implementation chosen for the GNU C library as it is described below.
|
|
|
|
Therefore an @code{iconv} call to reset the state should always be
|
|
|
|
performed if some protocol requires this for the output text.
|
|
|
|
|
|
|
|
The conversion stops for one of three reasons. The first is that all
|
|
|
|
characters from the input buffer are converted. This actually can mean
|
|
|
|
two things: either all bytes from the input buffer are consumed or
|
|
|
|
there are some bytes at the end of the buffer that possibly can form a
|
|
|
|
complete character but the input is incomplete. The second reason for a
|
|
|
|
stop is that the output buffer is full. And the third reason is that
|
|
|
|
the input contains invalid characters.
|
|
|
|
|
|
|
|
In all of these cases the buffer pointers after the last successful
|
|
|
|
conversion, for input and output buffer, are stored in @var{inbuf} and
|
|
|
|
@var{outbuf}, and the available room in each buffer is stored in
|
|
|
|
@var{inbytesleft} and @var{outbytesleft}.
|
|
|
|
|
|
|
|
Since the character sets selected in the @code{iconv_open} call can be
|
|
|
|
almost arbitrary, there can be situations where the input buffer contains
|
|
|
|
valid characters, which have no identical representation in the output
|
|
|
|
character set. The behavior in this situation is undefined. The
|
|
|
|
@emph{current} behavior of the GNU C library in this situation is to
|
|
|
|
return with an error immediately. This certainly is not the most
|
|
|
|
desirable solution; therefore, future versions will provide better ones,
|
|
|
|
but they are not yet finished.
|
|
|
|
|
|
|
|
If all input from the input buffer is successfully converted and stored
|
|
|
|
in the output buffer, the function returns the number of non-reversible
|
|
|
|
conversions performed. In all other cases the return value is
|
|
|
|
@code{(size_t) -1} and @code{errno} is set appropriately. In such cases
|
|
|
|
the value pointed to by @var{inbytesleft} is nonzero.
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item EILSEQ
|
|
|
|
The conversion stopped because of an invalid byte sequence in the input.
|
|
|
|
After the call, @code{*@var{inbuf}} points at the first byte of the
|
|
|
|
invalid byte sequence.
|
|
|
|
|
|
|
|
@item E2BIG
|
|
|
|
The conversion stopped because it ran out of space in the output buffer.
|
|
|
|
|
|
|
|
@item EINVAL
|
|
|
|
The conversion stopped because of an incomplete byte sequence at the end
|
|
|
|
of the input buffer.
|
|
|
|
|
|
|
|
@item EBADF
|
|
|
|
The @var{cd} argument is invalid.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
@pindex iconv.h
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{iconv} function was introduced in the XPG2 standard and is
|
2001-11-05 09:11:26 +01:00
|
|
|
declared in the @file{iconv.h} header.
|
|
|
|
@end deftypefun
|
|
|
|
|
|
|
|
The definition of the @code{iconv} function is quite good overall. It
|
|
|
|
provides quite flexible functionality. The only problems lie in the
|
|
|
|
boundary cases, which are incomplete byte sequences at the end of the
|
|
|
|
input buffer and invalid input. A third problem, which is not really
|
|
|
|
a design problem, is the way conversions are selected. The standard
|
|
|
|
does not say anything about the legitimate names, a minimal set of
|
|
|
|
available conversions. We will see how this negatively impacts other
|
|
|
|
implementations, as demonstrated below.
|
|
|
|
|
|
|
|
@node iconv Examples
|
|
|
|
@subsection A complete @code{iconv} example
|
|
|
|
|
|
|
|
The example below features a solution for a common problem. Given that
|
|
|
|
one knows the internal encoding used by the system for @code{wchar_t}
|
|
|
|
strings, one often is in the position to read text from a file and store
|
|
|
|
it in wide character buffers. One can do this using @code{mbsrtowcs},
|
|
|
|
but then we run into the problems discussed above.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
int
|
|
|
|
file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
|
|
|
|
@{
|
|
|
|
char inbuf[BUFSIZ];
|
|
|
|
size_t insize = 0;
|
|
|
|
char *wrptr = (char *) outbuf;
|
|
|
|
int result = 0;
|
|
|
|
iconv_t cd;
|
|
|
|
|
|
|
|
cd = iconv_open ("WCHAR_T", charset);
|
|
|
|
if (cd == (iconv_t) -1)
|
|
|
|
@{
|
|
|
|
/* @r{Something went wrong.} */
|
|
|
|
if (errno == EINVAL)
|
|
|
|
error (0, 0, "conversion from '%s' to wchar_t not available",
|
|
|
|
charset);
|
|
|
|
else
|
|
|
|
perror ("iconv_open");
|
|
|
|
|
|
|
|
/* @r{Terminate the output string.} */
|
|
|
|
*outbuf = L'\0';
|
|
|
|
|
|
|
|
return -1;
|
|
|
|
@}
|
|
|
|
|
|
|
|
while (avail > 0)
|
|
|
|
@{
|
|
|
|
size_t nread;
|
|
|
|
size_t nconv;
|
|
|
|
char *inptr = inbuf;
|
|
|
|
|
|
|
|
/* @r{Read more input.} */
|
|
|
|
nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
|
|
|
|
if (nread == 0)
|
|
|
|
@{
|
|
|
|
/* @r{When we come here the file is completely read.}
|
|
|
|
@r{This still could mean there are some unused}
|
|
|
|
@r{characters in the @code{inbuf}. Put them back.} */
|
|
|
|
if (lseek (fd, -insize, SEEK_CUR) == -1)
|
|
|
|
result = -1;
|
|
|
|
|
|
|
|
/* @r{Now write out the byte sequence to get into the}
|
|
|
|
@r{initial state if this is necessary.} */
|
|
|
|
iconv (cd, NULL, NULL, &wrptr, &avail);
|
|
|
|
|
|
|
|
break;
|
|
|
|
@}
|
|
|
|
insize += nread;
|
|
|
|
|
|
|
|
/* @r{Do the conversion.} */
|
|
|
|
nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
|
|
|
|
if (nconv == (size_t) -1)
|
|
|
|
@{
|
|
|
|
/* @r{Not everything went right. It might only be}
|
|
|
|
@r{an unfinished byte sequence at the end of the}
|
|
|
|
@r{buffer. Or it is a real problem.} */
|
|
|
|
if (errno == EINVAL)
|
|
|
|
/* @r{This is harmless. Simply move the unused}
|
|
|
|
@r{bytes to the beginning of the buffer so that}
|
|
|
|
@r{they can be used in the next round.} */
|
|
|
|
memmove (inbuf, inptr, insize);
|
|
|
|
else
|
|
|
|
@{
|
|
|
|
/* @r{It is a real problem. Maybe we ran out of}
|
|
|
|
@r{space in the output buffer or we have invalid}
|
|
|
|
@r{input. In any case back the file pointer to}
|
|
|
|
@r{the position of the last processed byte.} */
|
|
|
|
lseek (fd, -insize, SEEK_CUR);
|
|
|
|
result = -1;
|
|
|
|
break;
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{Terminate the output string.} */
|
|
|
|
if (avail >= sizeof (wchar_t))
|
|
|
|
*((wchar_t *) wrptr) = L'\0';
|
|
|
|
|
|
|
|
if (iconv_close (cd) != 0)
|
|
|
|
perror ("iconv_close");
|
|
|
|
|
|
|
|
return (wchar_t *) wrptr - outbuf;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@cindex stateful
|
|
|
|
This example shows the most important aspects of using the @code{iconv}
|
|
|
|
functions. It shows how successive calls to @code{iconv} can be used to
|
|
|
|
convert large amounts of text. The user does not have to care about
|
|
|
|
stateful encodings as the functions take care of everything.
|
|
|
|
|
|
|
|
An interesting point is the case where @code{iconv} returns an error and
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{errno} is set to @code{EINVAL}. This is not really an error in the
|
|
|
|
transformation. It can happen whenever the input character set contains
|
|
|
|
byte sequences of more than one byte for some character and texts are not
|
|
|
|
processed in one piece. In this case there is a chance that a multibyte
|
|
|
|
sequence is cut. The caller can then simply read the remainder of the
|
|
|
|
takes and feed the offending bytes together with new character from the
|
|
|
|
input to @code{iconv} and continue the work. The internal state kept in
|
|
|
|
the descriptor is @emph{not} unspecified after such an event as is the
|
2001-11-05 09:11:26 +01:00
|
|
|
case with the conversion functions from the @w{ISO C} standard.
|
|
|
|
|
|
|
|
The example also shows the problem of using wide character strings with
|
|
|
|
@code{iconv}. As explained in the description of the @code{iconv}
|
|
|
|
function above, the function always takes a pointer to a @code{char}
|
|
|
|
array and the available space is measured in bytes. In the example, the
|
|
|
|
output buffer is a wide character buffer; therefore, we use a local
|
|
|
|
variable @var{wrptr} of type @code{char *}, which is used in the
|
|
|
|
@code{iconv} calls.
|
|
|
|
|
|
|
|
This looks rather innocent but can lead to problems on platforms that
|
2001-11-07 08:21:22 +01:00
|
|
|
have tight restriction on alignment. Therefore the caller of @code{iconv}
|
|
|
|
has to make sure that the pointers passed are suitable for access of
|
2001-11-05 09:11:26 +01:00
|
|
|
characters from the appropriate character set. Since, in the
|
|
|
|
above case, the input parameter to the function is a @code{wchar_t}
|
|
|
|
pointer, this is the case (unless the user violates alignment when
|
|
|
|
computing the parameter). But in other situations, especially when
|
|
|
|
writing generic functions where one does not know what type of character
|
|
|
|
set one uses and, therefore, treats text as a sequence of bytes, it might
|
|
|
|
become tricky.
|
|
|
|
|
|
|
|
@node Other iconv Implementations
|
|
|
|
@subsection Some Details about other @code{iconv} Implementations
|
|
|
|
|
|
|
|
This is not really the place to discuss the @code{iconv} implementation
|
|
|
|
of other systems but it is necessary to know a bit about them to write
|
|
|
|
portable programs. The above mentioned problems with the specification
|
|
|
|
of the @code{iconv} functions can lead to portability issues.
|
|
|
|
|
|
|
|
The first thing to notice is that, due to the large number of character
|
|
|
|
sets in use, it is certainly not practical to encode the conversions
|
|
|
|
directly in the C library. Therefore, the conversion information must
|
|
|
|
come from files outside the C library. This is usually done in one or
|
|
|
|
both of the following ways:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
|
|
|
The C library contains a set of generic conversion functions that can
|
|
|
|
read the needed conversion tables and other information from data files.
|
|
|
|
These files get loaded when necessary.
|
|
|
|
|
|
|
|
This solution is problematic as it requires a great deal of effort to
|
2001-11-07 08:21:22 +01:00
|
|
|
apply to all character sets (potentially an infinite set). The
|
2001-11-05 09:11:26 +01:00
|
|
|
differences in the structure of the different character sets is so large
|
|
|
|
that many different variants of the table-processing functions must be
|
2001-11-07 08:21:22 +01:00
|
|
|
developed. In addition, the generic nature of these functions make them
|
2001-11-05 09:11:26 +01:00
|
|
|
slower than specifically implemented functions.
|
|
|
|
|
|
|
|
@item
|
|
|
|
The C library only contains a framework that can dynamically load
|
|
|
|
object files and execute the conversion functions contained therein.
|
|
|
|
|
|
|
|
This solution provides much more flexibility. The C library itself
|
|
|
|
contains only very little code and therefore reduces the general memory
|
|
|
|
footprint. Also, with a documented interface between the C library and
|
|
|
|
the loadable modules it is possible for third parties to extend the set
|
|
|
|
of available conversion modules. A drawback of this solution is that
|
|
|
|
dynamic loading must be available.
|
|
|
|
@end itemize
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Some implementations in commercial Unices implement a mixture of these
|
|
|
|
possibilities; the majority implement only the second solution. Using
|
|
|
|
loadable modules moves the code out of the library itself and keeps
|
2001-11-05 09:11:26 +01:00
|
|
|
the door open for extensions and improvements, but this design is also
|
|
|
|
limiting on some platforms since not many platforms support dynamic
|
|
|
|
loading in statically linked programs. On platforms without this
|
|
|
|
capability it is therefore not possible to use this interface in
|
|
|
|
statically linked programs. The GNU C library has, on ELF platforms, no
|
|
|
|
problems with dynamic loading in these situations; therefore, this
|
2001-11-07 08:21:22 +01:00
|
|
|
point is moot. The danger is that one gets acquainted with this
|
2001-11-05 09:11:26 +01:00
|
|
|
situation and forgets about the restrictions on other systems.
|
|
|
|
|
|
|
|
A second thing to know about other @code{iconv} implementations is that
|
|
|
|
the number of available conversions is often very limited. Some
|
|
|
|
implementations provide, in the standard release (not special
|
|
|
|
international or developer releases), at most 100 to 200 conversion
|
|
|
|
possibilities. This does not mean 200 different character sets are
|
2001-11-07 08:21:22 +01:00
|
|
|
supported; for example, conversions from one character set to a set of 10
|
2001-11-05 09:11:26 +01:00
|
|
|
others might count as 10 conversions. Together with the other direction
|
2001-11-07 08:21:22 +01:00
|
|
|
this makes 20 conversion possibilities used up by one character set. One
|
|
|
|
can imagine the thin coverage these platform provide. Some Unix vendors
|
|
|
|
even provide only a handful of conversions, which renders them useless for
|
2001-11-05 09:11:26 +01:00
|
|
|
almost all uses.
|
|
|
|
|
|
|
|
This directly leads to a third and probably the most problematic point.
|
|
|
|
The way the @code{iconv} conversion functions are implemented on all
|
|
|
|
known Unix systems and the availability of the conversion functions from
|
|
|
|
character set @math{@cal{A}} to @math{@cal{B}} and the conversion from
|
|
|
|
@math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the
|
|
|
|
conversion from @math{@cal{A}} to @math{@cal{C}} is available.
|
|
|
|
|
|
|
|
This might not seem unreasonable and problematic at first, but it is a
|
|
|
|
quite big problem as one will notice shortly after hitting it. To show
|
|
|
|
the problem we assume to write a program that has to convert from
|
|
|
|
@math{@cal{A}} to @math{@cal{C}}. A call like
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}");
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
fails according to the assumption above. But what does the program
|
|
|
|
do now? The conversion is necessary; therefore, simply giving up is not
|
|
|
|
an option.
|
|
|
|
|
|
|
|
This is a nuisance. The @code{iconv} function should take care of this.
|
2001-11-07 08:21:22 +01:00
|
|
|
But how should the program proceed from here on? If it tries to convert
|
2001-11-05 09:11:26 +01:00
|
|
|
to character set @math{@cal{B}}, first the two @code{iconv_open}
|
|
|
|
calls
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}");
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
and
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}");
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
@noindent
|
|
|
|
will succeed, but how to find @math{@cal{B}}?
|
|
|
|
|
|
|
|
Unfortunately, the answer is: there is no general solution. On some
|
|
|
|
systems guessing might help. On those systems most character sets can
|
2001-11-07 08:21:22 +01:00
|
|
|
convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text. Beside
|
|
|
|
this only some very system-specific methods can help. Since the
|
2001-11-05 09:11:26 +01:00
|
|
|
conversion functions come from loadable modules and these modules must
|
|
|
|
be stored somewhere in the filesystem, one @emph{could} try to find them
|
|
|
|
and determine from the available file which conversions are available
|
|
|
|
and whether there is an indirect route from @math{@cal{A}} to
|
|
|
|
@math{@cal{C}}.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
This example shows one of the design errors of @code{iconv} mentioned
|
2001-11-05 09:11:26 +01:00
|
|
|
above. It should at least be possible to determine the list of available
|
2001-11-07 08:21:22 +01:00
|
|
|
conversion programmatically so that if @code{iconv_open} says there is no
|
2001-11-05 09:11:26 +01:00
|
|
|
such conversion, one could make sure this also is true for indirect
|
|
|
|
routes.
|
|
|
|
|
|
|
|
@node glibc iconv Implementation
|
|
|
|
@subsection The @code{iconv} Implementation in the GNU C library
|
|
|
|
|
|
|
|
After reading about the problems of @code{iconv} implementations in the
|
|
|
|
last section it is certainly good to note that the implementation in
|
|
|
|
the GNU C library has none of the problems mentioned above. What
|
|
|
|
follows is a step-by-step analysis of the points raised above. The
|
|
|
|
evaluation is based on the current state of the development (as of
|
|
|
|
January 1999). The development of the @code{iconv} functions is not
|
|
|
|
complete, but basic functionality has solidified.
|
|
|
|
|
|
|
|
The GNU C library's @code{iconv} implementation uses shared loadable
|
|
|
|
modules to implement the conversions. A very small number of
|
|
|
|
conversions are built into the library itself but these are only rather
|
|
|
|
trivial conversions.
|
|
|
|
|
|
|
|
All the benefits of loadable modules are available in the GNU C library
|
|
|
|
implementation. This is especially appealing since the interface is
|
|
|
|
well documented (see below), and it, therefore, is easy to write new
|
|
|
|
conversion modules. The drawback of using loadable objects is not a
|
|
|
|
problem in the GNU C library, at least on ELF systems. Since the
|
|
|
|
library is able to load shared objects even in statically linked
|
2001-11-07 08:21:22 +01:00
|
|
|
binaries, static linking need not be forbidden in case one wants to use
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{iconv}.
|
|
|
|
|
|
|
|
The second mentioned problem is the number of supported conversions.
|
|
|
|
Currently, the GNU C library supports more than 150 character sets. The
|
|
|
|
way the implementation is designed the number of supported conversions
|
|
|
|
is greater than 22350 (@math{150} times @math{149}). If any conversion
|
|
|
|
from or to a character set is missing, it can be added easily.
|
|
|
|
|
|
|
|
Particularly impressive as it may be, this high number is due to the
|
|
|
|
fact that the GNU C library implementation of @code{iconv} does not have
|
|
|
|
the third problem mentioned above (i.e., whenever there is a conversion
|
|
|
|
from a character set @math{@cal{A}} to @math{@cal{B}} and from
|
|
|
|
@math{@cal{B}} to @math{@cal{C}} it is always possible to convert from
|
|
|
|
@math{@cal{A}} to @math{@cal{C}} directly). If the @code{iconv_open}
|
2001-11-07 08:21:22 +01:00
|
|
|
returns an error and sets @code{errno} to @code{EINVAL}, there is no
|
2001-11-05 09:11:26 +01:00
|
|
|
known way, directly or indirectly, to perform the wanted conversion.
|
|
|
|
|
|
|
|
@cindex triangulation
|
2001-11-07 08:21:22 +01:00
|
|
|
Triangulation is achieved by providing for each character set a
|
|
|
|
conversion from and to UCS-4 encoded @w{ISO 10646}. Using @w{ISO 10646}
|
2001-11-05 09:11:26 +01:00
|
|
|
as an intermediate representation it is possible to @dfn{triangulate}
|
|
|
|
(i.e., convert with an intermediate representation).
|
|
|
|
|
|
|
|
There is no inherent requirement to provide a conversion to @w{ISO
|
|
|
|
10646} for a new character set, and it is also possible to provide other
|
|
|
|
conversions where neither source nor destination character set is @w{ISO
|
2001-11-07 08:21:22 +01:00
|
|
|
10646}. The existing set of conversions is simply meant to cover all
|
2001-11-05 09:11:26 +01:00
|
|
|
conversions that might be of interest.
|
|
|
|
|
|
|
|
@cindex ISO-2022-JP
|
|
|
|
@cindex EUC-JP
|
|
|
|
All currently available conversions use the triangulation method above,
|
2001-11-07 08:21:22 +01:00
|
|
|
making conversion run unnecessarily slow. If, for example, somebody
|
2001-11-05 09:11:26 +01:00
|
|
|
often needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution
|
|
|
|
would involve direct conversion between the two character sets, skipping
|
|
|
|
the input to @w{ISO 10646} first. The two character sets of interest
|
|
|
|
are much more similar to each other than to @w{ISO 10646}.
|
|
|
|
|
|
|
|
In such a situation one easily can write a new conversion and provide it
|
|
|
|
as a better alternative. The GNU C library @code{iconv} implementation
|
|
|
|
would automatically use the module implementing the conversion if it is
|
|
|
|
specified to be more efficient.
|
|
|
|
|
|
|
|
@subsubsection Format of @file{gconv-modules} files
|
|
|
|
|
|
|
|
All information about the available conversions comes from a file named
|
|
|
|
@file{gconv-modules}, which can be found in any of the directories along
|
|
|
|
the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented
|
|
|
|
text files, where each of the lines has one of the following formats:
|
|
|
|
|
|
|
|
@itemize @bullet
|
|
|
|
@item
|
2001-11-07 08:21:22 +01:00
|
|
|
If the first non-whitespace character is a @kbd{#} the line contains only
|
2001-11-05 09:11:26 +01:00
|
|
|
comments and is ignored.
|
|
|
|
|
|
|
|
@item
|
2001-11-07 08:21:22 +01:00
|
|
|
Lines starting with @code{alias} define an alias name for a character
|
|
|
|
set. Two more words are expected on the line. The first word
|
2001-11-05 09:11:26 +01:00
|
|
|
defines the alias name, and the second defines the original name of the
|
|
|
|
character set. The effect is that it is possible to use the alias name
|
|
|
|
in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and
|
|
|
|
achieve the same result as when using the real character set name.
|
|
|
|
|
|
|
|
This is quite important as a character set has often many different
|
2001-11-07 08:21:22 +01:00
|
|
|
names. There is normally an official name but this need not correspond to
|
|
|
|
the most popular name. Beside this many character sets have special
|
|
|
|
names that are somehow constructed. For example, all character sets
|
|
|
|
specified by the ISO have an alias of the form @code{ISO-IR-@var{nnn}}
|
|
|
|
where @var{nnn} is the registration number. This allows programs that
|
|
|
|
know about the registration number to construct character set names and
|
|
|
|
use them in @code{iconv_open} calls. More on the available names and
|
2001-11-05 09:11:26 +01:00
|
|
|
aliases follows below.
|
|
|
|
|
|
|
|
@item
|
|
|
|
Lines starting with @code{module} introduce an available conversion
|
|
|
|
module. These lines must contain three or four more words.
|
|
|
|
|
|
|
|
The first word specifies the source character set, the second word the
|
2001-11-07 08:21:22 +01:00
|
|
|
destination character set of conversion implemented in this module, and
|
2001-11-05 09:11:26 +01:00
|
|
|
the third word is the name of the loadable module. The filename is
|
|
|
|
constructed by appending the usual shared object suffix (normally
|
|
|
|
@file{.so}) and this file is then supposed to be found in the same
|
2001-11-07 08:21:22 +01:00
|
|
|
directory the @file{gconv-modules} file is in. The last word on the line,
|
2001-11-05 09:11:26 +01:00
|
|
|
which is optional, is a numeric value representing the cost of the
|
|
|
|
conversion. If this word is missing, a cost of @math{1} is assumed. The
|
|
|
|
numeric value itself does not matter that much; what counts are the
|
|
|
|
relative values of the sums of costs for all possible conversion paths.
|
|
|
|
Below is a more precise description of the use of the cost value.
|
|
|
|
@end itemize
|
|
|
|
|
|
|
|
Returning to the example above where one has written a module to directly
|
|
|
|
convert from ISO-2022-JP to EUC-JP and back. All that has to be done is
|
|
|
|
to put the new module, let its name be ISO2022JP-EUCJP.so, in a directory
|
|
|
|
and add a file @file{gconv-modules} with the following content in the
|
|
|
|
same directory:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1
|
|
|
|
module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
To see why this is sufficient, it is necessary to understand how the
|
|
|
|
conversion used by @code{iconv} (and described in the descriptor) is
|
|
|
|
selected. The approach to this problem is quite simple.
|
|
|
|
|
|
|
|
At the first call of the @code{iconv_open} function the program reads
|
|
|
|
all available @file{gconv-modules} files and builds up two tables: one
|
|
|
|
containing all the known aliases and another that contains the
|
|
|
|
information about the conversions and which shared object implements
|
|
|
|
them.
|
|
|
|
|
|
|
|
@subsubsection Finding the conversion path in @code{iconv}
|
|
|
|
|
|
|
|
The set of available conversions form a directed graph with weighted
|
|
|
|
edges. The weights on the edges are the costs specified in the
|
|
|
|
@file{gconv-modules} files. The @code{iconv_open} function uses an
|
|
|
|
algorithm suitable for search for the best path in such a graph and so
|
|
|
|
constructs a list of conversions that must be performed in succession
|
|
|
|
to get the transformation from the source to the destination character
|
|
|
|
set.
|
|
|
|
|
|
|
|
Explaining why the above @file{gconv-modules} files allows the
|
|
|
|
@code{iconv} implementation to resolve the specific ISO-2022-JP to
|
|
|
|
EUC-JP conversion module instead of the conversion coming with the
|
|
|
|
library itself is straightforward. Since the latter conversion takes two
|
|
|
|
steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to
|
|
|
|
EUC-JP), the cost is @math{1+1 = 2}. The above @file{gconv-modules}
|
|
|
|
file, however, specifies that the new conversion modules can perform this
|
|
|
|
conversion with only the cost of @math{1}.
|
|
|
|
|
|
|
|
A mysterious item about the @file{gconv-modules} file above (and also
|
|
|
|
the file coming with the GNU C library) are the names of the character
|
|
|
|
sets specified in the @code{module} lines. Why do almost all the names
|
|
|
|
end in @code{//}? And this is not all: the names can actually be
|
|
|
|
regular expressions. At this point in time this mystery should not be
|
|
|
|
revealed, unless you have the relevant spell-casting materials: ashes
|
|
|
|
from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix
|
|
|
|
blessed by St.@: Emacs, assorted herbal roots from Central America, sand
|
|
|
|
from Cebu, etc. Sorry! @strong{The part of the implementation where
|
|
|
|
this is used is not yet finished. For now please simply follow the
|
|
|
|
existing examples. It'll become clearer once it is. --drepper}
|
|
|
|
|
|
|
|
A last remark about the @file{gconv-modules} is about the names not
|
2001-11-07 08:21:22 +01:00
|
|
|
ending with @code{//}. A character set named @code{INTERNAL} is often
|
|
|
|
mentioned. From the discussion above and the chosen name it should have
|
|
|
|
become clear that this is the name for the representation used in the
|
|
|
|
intermediate step of the triangulation. We have said that this is UCS-4
|
|
|
|
but actually that is not quite right. The UCS-4 specification also
|
|
|
|
includes the specification of the byte ordering used. Since a UCS-4 value
|
|
|
|
consists of four bytes, a stored value is effected by byte ordering. The
|
|
|
|
internal representation is @emph{not} the same as UCS-4 in case the byte
|
|
|
|
ordering of the processor (or at least the running process) is not the
|
|
|
|
same as the one required for UCS-4. This is done for performance reasons
|
|
|
|
as one does not want to perform unnecessary byte-swapping operations if
|
|
|
|
one is not interested in actually seeing the result in UCS-4. To avoid
|
[BZ #2510, BZ #2830, BZ #3137, BZ #3313, BZ #3426, BZ #3465, BZ #3480, BZ #3483, BZ #3493, BZ #3514, BZ #3515, BZ #3664, BZ #3673, BZ #3674]
2007-01-11 Jakub Jelinek <jakub@redhat.com>
* sysdeps/i386/soft-fp/sfp-machine.h: Remove.
* sysdeps/x86_64/soft-fp/sfp-machine.h: Likewise.
2007-01-10 Ulrich Drepper <drepper@redhat.com>
* io/fts.c: Make sure fts_cur is always valid after return from
fts_read.
Patch by Miloslav Trmac <mitr@redhat.com>.
2006-10-27 Richard Sandiford <richard@codesourcery.com>
* elf/elf.h (R_MIPS_GLOB_DAT): Define.
(R_MIPS_NUM): Bump by 1.
2007-01-03 Jakub Jelinek <jakub@redhat.com>
* posix/execvp.c: Include alloca.h.
(allocate_scripts_argv): Renamed to...
(scripts_argv): ... this. Don't allocate buffer here nor count
arguments.
(execvp): Use alloca if possible.
* posix/Makefile: Add rules to build and run tst-vfork3 test.
* posix/tst-vfork3.c: New test.
* stdlib/Makefile (tst-strtod3-ENV): Define.
2007-01-02 Ulrich Drepper <drepper@redhat.com>
* posix/getconf.c: Update copyright year.
* nss/getent.c: Likewise.
* iconv/iconvconfig.c: Likewise.
* iconv/iconv_prog.c: Likewise.
* elf/ldconfig.c: Likewise.
* catgets/gencat.c: Likewise.
* csu/version.c: Likewise.
* elf/ldd.bash.in: Likewise.
* elf/sprof.c (print_version): Likewise.
* locale/programs/locale.c: Likewise.
* locale/programs/localedef.c: Likewise.
* nscd/nscd.c (print_version): Likewise.
* debug/xtrace.sh: Likewise.
* malloc/memusage.sh: Likewise.
* malloc/mtrace.pl: Likewise.
* debug/catchsegv.sh: Likewise.
2006-12-24 Ulrich Drepper <drepper@redhat.com>
* malloc/malloc.c (sYSMALLOc): Remove some unnecessary alignment
attempts.
2006-12-23 Ulrich Drepper <drepper@redhat.com>
* posix/wordexp.c: Remove some unnecessary tests.
2006-12-20 SUGIOKA Toshinobu <sugioka@itonet.co.jp>
* sysdeps/unix/sysv/linux/sh/bits/shm.h: New file.
* nss/getXXbyYY_r.c: Include atomic.h.
(INTERNAL (REENTRANT_NAME)): Write startp after start_fct,
add atomic_write_barrier () in between.
2006-11-28 Jakub Jelinek <jakub@redhat.com>
* elf/dl-support.c: Include dl-procinfo.h.
* sysdeps/powerpc/dl-procinfo.h (PPC_PLATFORM_POWER4,
PPC_PLATFORM_PPC970, PPC_PLATFORM_POWER5, PPC_PLATFORM_POWER5_PLUS,
PPC_PLATFORM_POWER6, PPC_PLATFORM_CELL_BE, PPC_PLATFORM_POWER6X):
Define.
(_dl_string_platform): Use PPC_PLATFORM_* macros instead of
hardcoded constants.
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_platform): Use
PPC_PLATFORM_* macros for array designators.
2006-11-11 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/dl-procinfo.c (_dl_powerpc_cap_flags): Add 3 new cap
names to the beginning.
(_dl_powerpc_platforms): Add "power6x".
* sysdeps/powerpc/dl-procinfo.h (_DL_HWCAP_FIRST): Decrease.
(HWCAP_IMPORTANT): Add PPC_FEATURE_HAS_DFP.
(_DL_PLATFORMS_COUNT): Increase.
(_dl_string_platform): Handle power6x case.
* sysdeps/powerpc/sysdep.h (PPC_FEATURE_PA6T, PPC_FEATURE_HAS_DFP,
PPC_FEATURE_POWER6_EXT): Define.
(PPC_FEATURE_POWER5, PPC_FEATURE_POWER5_PLUS): Correct Comment.
[-2^31 .. 2^31) range.
* sysdeps/unix/sysv/linux/bits/statvfs.h: Define ST_RELATIME.
* sysdeps/unix/sysv/linux/internal_statvfs.c (__statvfs_getflags):
Handle relatime mount option.
2006-12-13 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/powerpc/powerpc32/setcontext.S: Include
kernel-features.h.
2006-12-11 Ulrich Drepper <drepper@redhat.com>
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Parse thousand
separators also if no non-zero digits found.
* stdlib/Makefile (tests): Add tst-strtod3.
[BZ #3664]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix test to recognize
empty parsed strings.
* stdlib/Makefile (tests): Add tst-strtod2.
* stdlib/tst-strtod2.c: New file.
[BZ #3673]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Fix exp_limit
computation.
* stdlib/Makefile (tests): Add tst-atof2.
* stdlib/tst-atof2.c: New file.
[BZ #3674]
* stdlib/strtod_l.c (____STRTOF_INTERNAL): Adjust exponent value
correctly if removing trailing zero of hex-float.
* stdlib/Makefile (tests): Add tst-atof1.
* stdlib/tst-atof1.c: New file.
* misc/mntent_r.c (__hasmntopt): Check p[optlen] even when p == rest.
Start searching for next comma at p rather than rest.
* misc/Makefile (tests): Add tst-mntent2.
* misc/tst-mntent2.c: New test.
2006-12-08 Ulrich Drepper <drepper@redhat.com>
* malloc/memusage.c: Handle realloc with new size of zero and
non-NULL pointer correctly.
(me): Really write first record twice.
(struct entry): Make format bi-arch safe.
(dest): Write out more realloc statistics.
* malloc/memusagestat.c (struct entry): Make format bi-arch safe.
2006-12-05 Jakub Jelinek <jakub@redhat.com>
* nis/nis_subr.c (nis_getnames): Revert last change.
2006-12-03 Kaz Kojima <kkojima@rr.iij4u.or.jp>
* sysdeps/unix/sysv/linux/sh/sys/io.h: Removed.
2006-11-30 H.J. Lu <hongjiu.lu@intel.com>
* sysdeps/i386/i686/memcmp.S: Use jump table as the base of
jump table entries.
2006-11-30 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/i386/clone.S: Provide CFI for the outermost
`clone' function to ensure proper unwinding stop of gdb.
* sysdeps/unix/sysv/linux/x86_64/clone.S: Likewise.
2006-12-01 Ulrich Drepper <drepper@redhat.com>
* nscd/nscd.init: Remove obsolete and commented-out -S option
handling.
2006-11-23 Jakub Jelinek <jakub@redhat.com>
[BZ #3514]
* manual/string.texi (strncmp): Fix pastos from wcscmp description.
[BZ #3515]
* manual/string.texi (strtok): Remove duplicate paragraph.
2006-12-01 Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Fix compatibility with
libgcc not supporting `rflags' unwinding (register # >= 17).
2006-11-30 Jakub Jelinek <jakub@redhat.com>
* sunrpc/svc_run.c (svc_run): Set my_pollfd to new_pollfd if realloc
succeeded.
2006-11-29 Daniel Jacobowitz <dan@codesourcery.com>
Jakub Jelinek <jakub@redhat.com>
Jan Kratochvil <jan.kratochvil@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sigaction.c (restore_rt): Add correct
unwind information.
* sysdeps/unix/sysv/linux/x86_64/Makefile: Provide symbols for
'restore_rt' even in the 'signal' directory.
* sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym: Extend the regs list.
malloc crashed. Don't allocate memory unnecessarily in each
loop.
2006-10-21 Jakub Jelinek <jakub@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Fix last change.
2006-11-20 Ulrich Drepper <drepper@redhat.com>
* resolv/mapv4v6addr.h (map_v4v6_address): Optimize a bit.
2006-11-18 Bruno Haible <bruno@clisp.org>
* sysdeps/unix/sysv/linux/i386/getgroups.c (__getgroups): Invoke
__sysconf only after having tried to call getgroups32.
2006-11-19 Ulrich Drepper <drepper@redhat.com>
* nss/nss_files/files-hosts.c (LINE_PARSER): Support IPv6-style
addresses for IPv4 queries if they can be mapped.
2006-11-16 Jakub Jelinek <jakub@redhat.com>
* sysdeps/x86_64/fpu/s_copysignf.S (__copysignf): Switch to .text.
* sysdeps/x86_64/fpu/s_copysign.S (__copysign): Likewise.
(signmask): Add .size directive.
(othermask): Add .type directive.
2006-11-14 Ulrich Drepper <drepper@redhat.com>
* po/nl.po: Update from translation team.
* timezone/zdump.c: Redo fix for BZ #3137.
2006-11-14 Jakub Jelinek <jakub@redhat.com>
* nss/nss_files/files-alias.c (get_next_alias): Set line back
to first_unused after parsing :include: file.
* timezone/africa: Update from tzdata2006o.
* timezone/antarctica: Likewise.
* timezone/asia: Likewise.
* timezone/australasia: Likewise.
* timezone/backward: Likewise.
* timezone/europe: Likewise.
* timezone/iso3166.tab: Likewise.
* timezone/northamerica: Likewise.
* timezone/southamerica: Likewise.
* timezone/zone.tab: Likewise.
* time/tzfile.c (__tzfile_read): Extend to handle new file format
on machines with 64-bit time_t.
* timezone/checktab.awk: Update from tzcode2006o.
* timezone/ialloc.c: Likewise.
* timezone/private.h: Likewise.
* timezone/scheck.c: Likewise.
* timezone/tzfile.h: Likewise.
* timezone/tzselect.ksh: Likewise.
* timezone/zdump.c: Likewise.
* timezone/zic.c: Likewise.
[BZ #3483]
* elf/ldconfig.c (main): Call setlocale and textdomain.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
[BZ #3480]
* manual/argp.texi: Fix typos.
* manual/charset.texi: Likewise.
* manual/errno.texi: Likewise.
* manual/filesys.texi: Likewise.
* manual/lang.texi: Likewise.
* manual/maint.texi: Likewise.
* manual/memory.texi: Likewise.
* manual/message.texi: Likewise.
* manual/resource.texi: Likewise.
* manual/search.texi: Likewise.
* manual/signal.texi: Likewise.
* manual/startup.texi: Likewise.
* manual/stdio.texi: Likewise.
* manual/sysinfo.texi: Likewise.
* manual/syslog.texi: Likewise.
* manual/time.texi: Likewise.
Patch by Ralf Wildenhues <Ralf.Wildenhues@gmx.de>.
[BZ #3465]
* sunrpc/clnt_raw.c: Minimal message improvements.
* sunrpc/pm_getmaps.c: Likewise.
* nis/nss_nisplus/nisplus-publickey.c: Likewise.
* nis/nis_print_group_entry.c: Likewise.
* locale/programs/repertoire.c: Likewise.
* locale/programs/charmap.c: Likewise.
* malloc/memusage.sh: Likewise.
* elf/dl-deps.c: Likewise.
* locale/programs/ld-collate.c: Likewise.
* libio/vswprintf.c: Likewise.
* malloc/memusagestat.c: Likewise.
* sunrpc/auth_unix.c: Likewise.
* sunrpc/rpc_main.c: Likewise.
* nscd/cache.c: Likewise.
* locale/programs/repertoire.c: Unify output messages.
* locale/programs/charmap.c: Likewise.
* locale/programs/ld-ctype.c: Likewise.
* locale/programs/ld-monetary.c: Likewise.
* locale/programs/ld-numeric.c: Likewise.
* locale/programs/ld-time.c: Likewise.
* elf/ldconfig.c: Likewise.
* nscd/selinux.c: Likewise.
* elf/cache.c: Likewise.
Patch mostly by Benno Schulenberg <bensberg@justemail.net>.
2006-11-10 Jakub Jelinek <jakub@redhat.com>
* string/strxfrm_l.c (STRXFRM): Fix trailing \1 optimization
if N is one bigger than return value.
* string/tst-strxfrm2.c (do_test): Also test strxfrm with l1 + 1
and l1 last arguments, if buf is defined, verify the return value
equals to strlen (buf) and verify no byte beyond passed length
is modified.
2006-11-10 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
* sysdeps/gnu/siglist.c (__old_sys_siglist, __old_sys_sigabbrev):
Use __new_sys_siglist instead of _sys_siglist_internal as
second macro argument.
(_old_sys_siglist): Use declare_symbol_alias macro instead of
strong_alias.
2006-11-09 Ulrich Drepper <drepper@redhat.com>
[BZ #3493]
* posix/unistd.h (sysconf): Remove const attribute.
* sysdeps/posix/getaddrinfo.c (getaddrinfo): Fix test for
temporary or deprecated addresses.
Patch by Sridhar Samudrala <sri@us.ibm.com>.
* string/Makefile (tests): Add tst-strxfrm2.
* string/tst-strxfrm2.c: New file.
2006-10-09 Jakub Jelinek <jakub@redhat.com>
* elf/dl-debug.c (_dl_debug_initialize): Check r->r_map for 0
rather than r->r_brk.
* string/strxfrm_l.c (STRXFRM): Do the trailing \1 removal
optimization even if needed > n.
2006-11-07 Jakub Jelinek <jakub@redhat.com>
* include/libc-symbols.h (declare_symbol): Rename to...
(declare_symbol_alias): ... this. Add ORIGINAL argument, imply
strong_alias (ORIGINAL, SYMBOL) in asm to make sure it preceedes
.size directive.
* sysdeps/gnu/errlist-compat.awk: Adjust for declare_symbol_alias
changes.
* sysdeps/gnu/siglist.c: Likewise.
2006-11-03 Steven Munroe <sjmunroe@us.ibm.com>
* sysdeps/powerpc/fpu/bits/mathinline.h
[__LIBC_INTERNAL_MATH_INLINES]: Moved to ...
* sysdeps/powerpc/fpu/math_private.h: ...here. New file.
2006-11-05 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/i386/sysconf.c (intel_check_word):
Update handling of cache descriptor 0x49 for new models.
* sysdeps/unix/sysv/linux/x86_64/sysconf.c (intel_check_word):
Likewise.
2006-11-02 Ulrich Drepper <drepper@redhat.com>
* configure.in: Work around ld --help change and avoid -z relro
test completely if the architecture doesn't care about security.
2006-11-01 Ulrich Drepper <drepper@redhat.com>
* po/sv.po: Update from translation team.
2006-10-31 Ulrich Drepper <drepper@redhat.com>
* stdlib/atexit.c (atexit): Don't mark as hidden when used to
generate compatibility version.
2006-10-29 Ulrich Drepper <drepper@redhat.com>
* configure.in: Relax -z relro requirement a bit.
* po/sv.po: Update from translation team.
2006-10-29 Jakub Jelinek <jakub@redhat.com>
* elf/dl-sym.c (do_sym): Use RTLD_SINGLE_THREAD_P.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Likewise.
* elf/dl-close.c (_dl_close_worker): Likewise.
* elf/dl-open.c (_dl_open_worker): Likewise.
* sysdeps/generic/sysdep-cancel.h (RTLD_SINGLE_THREAD_P): Define.
* configure.in: Require assembler support for visibility, compiler
support for visibility and aliases, linker support for various -z
options.
* Makeconfig: Remove conditional code which now is unnecessary.
* config.h.in: Likewise.
* config.make.in: Likewise.
* dlfcn/Makefile: Likewise.
* elf/Makefile: Likewise.
* elf/dl-load.c: Likewise.
* elf/rtld.c: Likewise.
* include/libc-symbols.h: Likewise.
* include/stdio.h: Likewise.
* io/Makefile: Likewise.
* io/fstat.c: Likewise.
* io/fstat64.c: Likewise.
* io/fstatat.c: Likewise.
* io/fstatat64.c: Likewise.
* io/lstat.c: Likewise.
* io/lstat64.c: Likewise.
* io/mknod.c: Likewise.
* io/mknodat.c: Likewise.
* io/stat.c: Likewise.
* io/stat64.c: Likewise.
* libio/stdio.c: Likewise.
* nscd/Makefile: Likewise.
* stdlib/Makefile: Likewise.
* stdlib/atexit.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/i386/sysdep.h: Likewise.
* sysdeps/i386/i686/memcmp.S: Likewise.
* sysdeps/powerpc/powerpc32/sysdep.h: Likewise.
* sysdeps/unix/sysv/linux/i386/sigaction.c: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sigaction.c: Likewise.
* Makerules: USE_TLS support is now default.
* tls.make.c: Likewise.
* csu/Versions: Likewise.
* csu/libc-start.c: Likewise.
* csu/libc-tls.c: Likewise.
* csu/version.c: Likewise.
* dlfcn/dlinfo.c: Likewise.
* elf/dl-addr.c: Likewise.
* elf/dl-cache.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-iteratephdr.c: Likewise.
* elf/dl-load.c: Likewise.
* elf/dl-lookup.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-reloc.c: Likewise.
* elf/dl-support.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-sysdep.c: Likewise.
* elf/dl-tls.c: Likewise.
* elf/ldconfig.c: Likewise.
* elf/rtld.c: Likewise.
* elf/tst-tls-dlinfo.c: Likewise.
* elf/tst-tls1.c: Likewise.
* elf/tst-tls10.h: Likewise.
* elf/tst-tls14.c: Likewise.
* elf/tst-tls2.c: Likewise.
* elf/tst-tls3.c: Likewise.
* elf/tst-tls4.c: Likewise.
* elf/tst-tls5.c: Likewise.
* elf/tst-tls6.c: Likewise.
* elf/tst-tls7.c: Likewise.
* elf/tst-tls8.c: Likewise.
* elf/tst-tls9.c: Likewise.
* elf/tst-tlsmod1.c: Likewise.
* elf/tst-tlsmod13.c: Likewise.
* elf/tst-tlsmod13a.c: Likewise.
* elf/tst-tlsmod14a.c: Likewise.
* elf/tst-tlsmod2.c: Likewise.
* elf/tst-tlsmod3.c: Likewise.
* elf/tst-tlsmod4.c: Likewise.
* elf/tst-tlsmod5.c: Likewise.
* elf/tst-tlsmod6.c: Likewise.
* include/errno.h: Likewise.
* include/link.h: Likewise.
* include/tls.h: Likewise.
* locale/global-locale.c: Likewise.
* locale/localeinfo.h: Likewise.
* malloc/arena.c: Likewise.
* malloc/hooks.c: Likewise.
* malloc/malloc.c: Likewise.
* resolv/Versions: Likewise.
* sysdeps/alpha/dl-machine.h: Likewise.
* sysdeps/alpha/libc-tls.c: Likewise.
* sysdeps/generic/ldsodefs.h: Likewise.
* sysdeps/generic/tls.h: Likewise.
* sysdeps/i386/dl-machine.h: Likewise.
* sysdeps/ia64/dl-machine.h: Likewise.
* sysdeps/ia64/libc-tls.c: Likewise.
* sysdeps/mach/hurd/fork.c: Likewise.
* sysdeps/mach/hurd/i386/tls.h: Likewise.
* sysdeps/powerpc/powerpc32/dl-machine.c: Likwise.
* sysdeps/powerpc/powerpc32/dl-machine.h: Likewise.
* sysdeps/powerpc/powerpc64/dl-machine.h: Likewise.
* sysdeps/s390/libc-tls.c: Likewise.
* sysdeps/s390/s390-32/dl-machine.h: Likewise.
* sysdeps/s390/s390-64/dl-machine.h: Likewise.
* sysdeps/sh/dl-machine.h: Likewise.
* sysdeps/sparc/sparc32/dl-machine.h: Likewise.
* sysdeps/sparc/sparc64/dl-machine.h: Likewise.
* sysdeps/x86_64/dl-machine.h: Likewise.
[BZ #3426]
* stdlib/stdlib.h: Adjust comment for canonicalize_file_name to
reality.
2006-10-27 Jakub Jelinek <jakub@redhat.com>
* elf/dl-lookup.c (_dl_debug_bindings): Remove unused symbol_scope
argument.
(_dl_lookup_symbol_x): Adjust caller.
* sysdeps/generic/ldsodefs.h (struct link_namespaces): Remove
_ns_global_scope.
* elf/rtld.c (dl_main): Don't initialize _ns_global_scope.
* elf/dl-libc.c: Revert l_scope name changes.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* elf/dl-close.c (_dl_close): Likewise.
* elf/dl-open.c (dl_open_worker): Likewise. If not SINGLE_THREAD_P,
always use __rtld_mrlock_{change,done}. Always free old scope list
here if not l_scope_mem.
* elf/dl-runtime.c (_dl_fixup, _dl_profile_fixup): Revert l_scope name
change. Never free scope list here. Just __rtld_mrlock_lock before
the lookup and __rtld_mrlock_unlock it after the lookup.
* elf/dl-sym.c: Likewise.
* include/link.h (struct r_scoperec): Remove.
(struct link_map): Replace l_scoperec with l_scope, l_scoperec_mem
with l_scope_mem and l_scoperec_lock with l_scope_lock.
2006-10-25 Ulrich Drepper <drepper@redhat.com>
* sysdeps/gnu/netinet/tcp.h: Define TCP_CONGESTION.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* configure.in: Disable building profile libraries by default.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* elf/dl-lookup.c (_dl_lookup_symbol_x): Add warning to
_dl_lookup_symbol_x code.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* elf/dl-runtime.c: Include sysdep-cancel.h.
(_dl_fixup, _dl_profile_fixup): Use __rtld_mrlock_* and
scoperec->nusers only if !SINGLE_THREAD_P. Use atomic_*
instead of catomic_* macros.
* elf/dl-sym.c: Include sysdep-cancel.h.
(do_sym): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-close.c: Include sysdep-cancel.h.
(_dl_close): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
* elf/dl-open.c: Include sysdep-cancel.h.
(dl_open_worker): Use __rtld_mrlock_* and scoperec->nusers only
if !SINGLE_THREAD_P. Use atomic_* instead of catomic_* macros.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Set maxfb to address of last
fastbin rather than end of fastbin array.
2006-10-18 Ulrich Drepper <drepper@redhat.com>
* sysdeps/i386/i486/bits/atomic.h (catomic_decrement): Use correct
body macro.
* sysdeps/x86_64/bits/atomic.h
(__arch_c_compare_and_exchange_val_64_acq): Add missing casts.
(catomic_decrement): Use correct body macro.
2006-10-17 Jakub Jelinek <jakub@redhat.com>
* include/atomic.h: Add a unique prefix to all local variables
in macros.
* csu/tst-atomic.c (do_test): Test also catomic_* macros.
2006-10-14 Ulrich Drepper <drepper@redhat.com>
* resolv/arpa/nameser.h: Document that ns_t_a6 is deprecated.
[BZ #3313]
* malloc/malloc.c (malloc_consolidate): Don't use get_fast_max to
determine highest fast bin to consolidate, always look into all of
them.
(do_check_malloc_state): Only require for empty bins for large
sizes in main arena.
* libio/stdio.h: Add more __wur attributes.
2006-11-12 Andreas Jaeger <aj@suse.de>
[BZ #2510]
* manual/search.texi (Hash Search Function): Clarify.
(Array Search Function): Clarify.
2006-11-12 Joseph Myers <joseph@codesourcery.com>
[BZ #2830]
* math/atest-exp.c (main): Cast hex value to mp_limb_t before
shifting.
* math/atest-exp2.c (read_mpn_hex): Likewise.
* math/atest-sincos.c (main): Likewise.
* sysdeps/unix/sysv/linux/syscalls.list: Add epoll_pwait.
* sysdeps/unix/sysv/linux/sys/epoll.h: Declare epoll_pwait.
* sysdeps/unix/sysv/linux/Versions (libc): Add epoll_pwait for
version GLIBC_2.6.
* Versions.def: Add GLIBC_2.6 for libc.
* sysdeps/i386/i486/bits/atomic.h: Add catomic_* support.
2006-10-11 Jakub Jelinek <jakub@redhat.com>
* malloc/malloc.c (_int_malloc): Remove unused any_larger variable.
* nis/nis_defaults.c (__nis_default_access): Don't call getenv twice.
* nis/nis_subr.c (nis_getnames): Use __secure_getenv instead of getenv.
* sysdeps/generic/unsecvars.h: Add NIS_PATH.
2006-10-11 Ulrich Drepper <drepper@redhat.com>
* include/atomic.c: Define catomic_* operations.
* sysdeps/x86_64/bits/atomic.h: Likewise. Fix a few minor problems.
* stdlib/cxa_finalize.c: Use catomic_* operations instead of atomic_*.
* malloc/memusage.c: Likewise.
* gmon/mcount.c: Likewise.
* elf/dl-close.c: Likewise.
* elf/dl-open.c: Likewise.
* elf/dl-profile.c: Likewise.
* elf/dl-sym.c: Likewise.
* elf/dl-runtime.c: Likewise.
* elf/dl-fptr.c: Likewise.
* resolv/res_libc.c: Likewise.
2006-10-10 Roland McGrath <roland@frob.com>
* sysdeps/mach/hurd/utimes.c: Use a union to avoid an improper cast.
* sysdeps/mach/hurd/futimes.c: Likewise.
* sysdeps/mach/hurd/lutimes.c: Likewise.
2006-10-09 Ulrich Drepper <drepper@redhat.com>
Jakub Jelinek <jakub@redhat.com>
Implement reference counting of scope records.
* elf/dl-close.c (_dl_close): Remove all scopes from removed objects
from the list in objects which remain. Always allocate new scope
record.
* elf/dl-open.c (dl_open_worker): When growing array for scopes,
don't resize, allocate a new one.
* elf/dl-runtime.c: Update reference counters before using a scope
array.
* elf/dl-sym.c: Likewise.
* elf/dl-libc.c: Adjust for l_scope name change.
* elf/dl-load.c: Likewise.
* elf/dl-object.c: Likewise.
* elf/rtld.c: Likewise.
* include/link.h: Include <rtld-lowlevel.h>. Define struct
r_scoperec. Replace r_scope with pointer to r_scoperec structure.
Add l_scoperec_lock.
* sysdeps/generic/ldsodefs.h: Include <rtld-lowlevel.h>.
* sysdeps/generic/rtld-lowlevel.h: New file.
* include/atomic.h: Rename atomic_and to atomic_and_val and
atomic_or to atomic_or_val. Define new macros atomic_and and
atomic_or which do not return values.
* sysdeps/x86_64/bits/atomic.h: Define atomic_and and atomic_or.
Various cleanups.
* sysdeps/i386/i486/bits/atomic.h: Likewise.
* po/sv.po: Update from translation team.
2006-10-07 Ulrich Drepper <drepper@redhat.com>
* Versions.def: Add GLIBC_2.6 to libpthread.
* include/shlib-compat.h (SHLIB_COMPAT): Expand parameters before use.
(versioned_symbol): Likewise.
(compat_symbol): Likewise.
* po/tr.po: Update from translation team.
* nis/Banner: Removed. It's been integral part forever and the
author info is incomplete anyway.
* libio/Banner: Likewise.
2006-10-06 Ulrich Drepper <drepper@redhat.com>
* version.h (VERSION): Bump to 2.5.90 for new development tree.
2007-01-11 22:51:07 +01:00
|
|
|
trouble with endianness, the internal representation consistently is named
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{INTERNAL} even on big-endian systems where the representations are
|
2001-11-05 09:11:26 +01:00
|
|
|
identical.
|
|
|
|
|
|
|
|
@subsubsection @code{iconv} module data structures
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
So far this section has described how modules are located and considered
|
2001-11-05 09:11:26 +01:00
|
|
|
to be used. What remains to be described is the interface of the modules
|
|
|
|
so that one can write new ones. This section describes the interface as
|
2001-11-07 08:21:22 +01:00
|
|
|
it is in use in January 1999. The interface will change a bit in the
|
2001-11-05 09:11:26 +01:00
|
|
|
future but, with luck, only in an upwardly compatible way.
|
|
|
|
|
|
|
|
The definitions necessary to write new modules are publicly available
|
|
|
|
in the non-standard header @file{gconv.h}. The following text,
|
2001-11-07 08:21:22 +01:00
|
|
|
therefore, describes the definitions from this header file. First,
|
2001-11-05 09:11:26 +01:00
|
|
|
however, it is necessary to get an overview.
|
|
|
|
|
|
|
|
From the perspective of the user of @code{iconv} the interface is quite
|
2001-11-07 08:21:22 +01:00
|
|
|
simple: the @code{iconv_open} function returns a handle that can be used
|
|
|
|
in calls to @code{iconv}, and finally the handle is freed with a call to
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{iconv_close}. The problem is that the handle has to be able to
|
|
|
|
represent the possibly long sequences of conversion steps and also the
|
|
|
|
state of each conversion since the handle is all that is passed to the
|
|
|
|
@code{iconv} function. Therefore, the data structures are really the
|
|
|
|
elements necessary to understanding the implementation.
|
|
|
|
|
|
|
|
We need two different kinds of data structures. The first describes the
|
|
|
|
conversion and the second describes the state etc. There are really two
|
|
|
|
type definitions like this in @file{gconv.h}.
|
|
|
|
@pindex gconv.h
|
|
|
|
|
|
|
|
@comment gconv.h
|
|
|
|
@comment GNU
|
|
|
|
@deftp {Data type} {struct __gconv_step}
|
|
|
|
This data structure describes one conversion a module can perform. For
|
|
|
|
each function in a loaded module with conversion functions there is
|
|
|
|
exactly one object of this type. This object is shared by all users of
|
|
|
|
the conversion (i.e., this object does not contain any information
|
|
|
|
corresponding to an actual conversion; it only describes the conversion
|
|
|
|
itself).
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item struct __gconv_loaded_object *__shlib_handle
|
|
|
|
@itemx const char *__modname
|
|
|
|
@itemx int __counter
|
|
|
|
All these elements of the structure are used internally in the C library
|
|
|
|
to coordinate loading and unloading the shared. One must not expect any
|
|
|
|
of the other elements to be available or initialized.
|
|
|
|
|
|
|
|
@item const char *__from_name
|
|
|
|
@itemx const char *__to_name
|
|
|
|
@code{__from_name} and @code{__to_name} contain the names of the source and
|
|
|
|
destination character sets. They can be used to identify the actual
|
2001-11-07 08:21:22 +01:00
|
|
|
conversion to be carried out since one module might implement conversions
|
2001-11-05 09:11:26 +01:00
|
|
|
for more than one character set and/or direction.
|
|
|
|
|
|
|
|
@item gconv_fct __fct
|
|
|
|
@itemx gconv_init_fct __init_fct
|
|
|
|
@itemx gconv_end_fct __end_fct
|
|
|
|
These elements contain pointers to the functions in the loadable module.
|
|
|
|
The interface will be explained below.
|
|
|
|
|
|
|
|
@item int __min_needed_from
|
|
|
|
@itemx int __max_needed_from
|
|
|
|
@itemx int __min_needed_to
|
|
|
|
@itemx int __max_needed_to;
|
|
|
|
These values have to be supplied in the init function of the module. The
|
|
|
|
@code{__min_needed_from} value specifies how many bytes a character of
|
|
|
|
the source character set at least needs. The @code{__max_needed_from}
|
|
|
|
specifies the maximum value that also includes possible shift sequences.
|
|
|
|
|
|
|
|
The @code{__min_needed_to} and @code{__max_needed_to} values serve the
|
2001-11-07 08:21:22 +01:00
|
|
|
same purpose as @code{__min_needed_from} and @code{__max_needed_from} but
|
2001-11-05 09:11:26 +01:00
|
|
|
this time for the destination character set.
|
|
|
|
|
|
|
|
It is crucial that these values be accurate since otherwise the
|
|
|
|
conversion functions will have problems or not work at all.
|
|
|
|
|
|
|
|
@item int __stateful
|
2001-11-07 08:21:22 +01:00
|
|
|
This element must also be initialized by the init function.
|
|
|
|
@code{int __stateful} is nonzero if the source character set is stateful.
|
2001-11-05 09:11:26 +01:00
|
|
|
Otherwise it is zero.
|
|
|
|
|
|
|
|
@item void *__data
|
|
|
|
This element can be used freely by the conversion functions in the
|
2001-11-07 08:21:22 +01:00
|
|
|
module. @code{void *__data} can be used to communicate extra information
|
|
|
|
from one call to another. @code{void *__data} need not be initialized if
|
|
|
|
not needed at all. If @code{void *__data} element is assigned a pointer
|
|
|
|
to dynamically allocated memory (presumably in the init function) it has
|
|
|
|
to be made sure that the end function deallocates the memory. Otherwise
|
2001-11-05 09:11:26 +01:00
|
|
|
the application will leak memory.
|
|
|
|
|
|
|
|
It is important to be aware that this data structure is shared by all
|
|
|
|
users of this specification conversion and therefore the @code{__data}
|
|
|
|
element must not contain data specific to one specific use of the
|
|
|
|
conversion function.
|
|
|
|
@end table
|
|
|
|
@end deftp
|
|
|
|
|
|
|
|
@comment gconv.h
|
|
|
|
@comment GNU
|
|
|
|
@deftp {Data type} {struct __gconv_step_data}
|
|
|
|
This is the data structure that contains the information specific to
|
|
|
|
each use of the conversion functions.
|
|
|
|
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item char *__outbuf
|
|
|
|
@itemx char *__outbufend
|
|
|
|
These elements specify the output buffer for the conversion step. The
|
|
|
|
@code{__outbuf} element points to the beginning of the buffer, and
|
|
|
|
@code{__outbufend} points to the byte following the last byte in the
|
|
|
|
buffer. The conversion function must not assume anything about the size
|
|
|
|
of the buffer but it can be safely assumed the there is room for at
|
|
|
|
least one complete character in the output buffer.
|
|
|
|
|
|
|
|
Once the conversion is finished, if the conversion is the last step, the
|
|
|
|
@code{__outbuf} element must be modified to point after the last byte
|
|
|
|
written into the buffer to signal how much output is available. If this
|
|
|
|
conversion step is not the last one, the element must not be modified.
|
|
|
|
The @code{__outbufend} element must not be modified.
|
|
|
|
|
|
|
|
@item int __is_last
|
|
|
|
This element is nonzero if this conversion step is the last one. This
|
|
|
|
information is necessary for the recursion. See the description of the
|
|
|
|
conversion function internals below. This element must never be
|
|
|
|
modified.
|
|
|
|
|
|
|
|
@item int __invocation_counter
|
2001-11-07 08:21:22 +01:00
|
|
|
The conversion function can use this element to see how many calls of
|
|
|
|
the conversion function already happened. Some character sets require a
|
2001-11-05 09:11:26 +01:00
|
|
|
certain prolog when generating output, and by comparing this value with
|
2001-11-07 08:21:22 +01:00
|
|
|
zero, one can find out whether it is the first call and whether,
|
|
|
|
therefore, the prolog should be emitted. This element must never be
|
2001-11-05 09:11:26 +01:00
|
|
|
modified.
|
|
|
|
|
|
|
|
@item int __internal_use
|
|
|
|
This element is another one rarely used but needed in certain
|
|
|
|
situations. It is assigned a nonzero value in case the conversion
|
|
|
|
functions are used to implement @code{mbsrtowcs} et.al.@: (i.e., the
|
|
|
|
function is not used directly through the @code{iconv} interface).
|
|
|
|
|
|
|
|
This sometimes makes a difference as it is expected that the
|
|
|
|
@code{iconv} functions are used to translate entire texts while the
|
|
|
|
@code{mbsrtowcs} functions are normally used only to convert single
|
|
|
|
strings and might be used multiple times to convert entire texts.
|
|
|
|
|
|
|
|
But in this situation we would have problem complying with some rules of
|
|
|
|
the character set specification. Some character sets require a prolog,
|
|
|
|
which must appear exactly once for an entire text. If a number of
|
|
|
|
@code{mbsrtowcs} calls are used to convert the text, only the first call
|
|
|
|
must add the prolog. However, because there is no communication between the
|
|
|
|
different calls of @code{mbsrtowcs}, the conversion functions have no
|
|
|
|
possibility to find this out. The situation is different for sequences
|
|
|
|
of @code{iconv} calls since the handle allows access to the needed
|
|
|
|
information.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The @code{int __internal_use} element is mostly used together with
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{__invocation_counter} as follows:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
if (!data->__internal_use
|
|
|
|
&& data->__invocation_counter == 0)
|
|
|
|
/* @r{Emit prolog.} */
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
This element must never be modified.
|
|
|
|
|
|
|
|
@item mbstate_t *__statep
|
|
|
|
The @code{__statep} element points to an object of type @code{mbstate_t}
|
|
|
|
(@pxref{Keeping the state}). The conversion of a stateful character
|
2001-11-07 08:21:22 +01:00
|
|
|
set must use the object pointed to by @code{__statep} to store
|
|
|
|
information about the conversion state. The @code{__statep} element
|
2001-11-05 09:11:26 +01:00
|
|
|
itself must never be modified.
|
|
|
|
|
|
|
|
@item mbstate_t __state
|
|
|
|
This element must @emph{never} be used directly. It is only part of
|
|
|
|
this structure to have the needed space allocated.
|
|
|
|
@end table
|
|
|
|
@end deftp
|
|
|
|
|
|
|
|
@subsubsection @code{iconv} module interfaces
|
|
|
|
|
|
|
|
With the knowledge about the data structures we now can describe the
|
|
|
|
conversion function itself. To understand the interface a bit of
|
2001-11-07 08:21:22 +01:00
|
|
|
knowledge is necessary about the functionality in the C library that
|
2001-11-05 09:11:26 +01:00
|
|
|
loads the objects with the conversions.
|
|
|
|
|
|
|
|
It is often the case that one conversion is used more than once (i.e.,
|
|
|
|
there are several @code{iconv_open} calls for the same set of character
|
|
|
|
sets during one program run). The @code{mbsrtowcs} et.al.@: functions in
|
2001-11-07 08:21:22 +01:00
|
|
|
the GNU C library also use the @code{iconv} functionality, which
|
2001-11-05 09:11:26 +01:00
|
|
|
increases the number of uses of the same functions even more.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Because of this multiple use of conversions, the modules do not get
|
|
|
|
loaded exclusively for one conversion. Instead a module once loaded can
|
|
|
|
be used by an arbitrary number of @code{iconv} or @code{mbsrtowcs} calls
|
2001-11-05 09:11:26 +01:00
|
|
|
at the same time. The splitting of the information between conversion-
|
2001-11-07 08:21:22 +01:00
|
|
|
function-specific information and conversion data makes this possible.
|
2001-11-05 09:11:26 +01:00
|
|
|
The last section showed the two data structures used to do this.
|
|
|
|
|
|
|
|
This is of course also reflected in the interface and semantics of the
|
|
|
|
functions that the modules must provide. There are three functions that
|
|
|
|
must have the following names:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item gconv_init
|
|
|
|
The @code{gconv_init} function initializes the conversion function
|
|
|
|
specific data structure. This very same object is shared by all
|
|
|
|
conversions that use this conversion and, therefore, no state information
|
2001-11-07 08:21:22 +01:00
|
|
|
about the conversion itself must be stored in here. If a module
|
|
|
|
implements more than one conversion, the @code{gconv_init} function will
|
2001-11-05 09:11:26 +01:00
|
|
|
be called multiple times.
|
|
|
|
|
|
|
|
@item gconv_end
|
|
|
|
The @code{gconv_end} function is responsible for freeing all resources
|
|
|
|
allocated by the @code{gconv_init} function. If there is nothing to do,
|
|
|
|
this function can be missing. Special care must be taken if the module
|
|
|
|
implements more than one conversion and the @code{gconv_init} function
|
|
|
|
does not allocate the same resources for all conversions.
|
|
|
|
|
|
|
|
@item gconv
|
|
|
|
This is the actual conversion function. It is called to convert one
|
|
|
|
block of text. It gets passed the conversion step information
|
|
|
|
initialized by @code{gconv_init} and the conversion data, specific to
|
|
|
|
this use of the conversion functions.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
There are three data types defined for the three module interface
|
|
|
|
functions and these define the interface.
|
|
|
|
|
|
|
|
@comment gconv.h
|
|
|
|
@comment GNU
|
|
|
|
@deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *)
|
|
|
|
This specifies the interface of the initialization function of the
|
|
|
|
module. It is called exactly once for each conversion the module
|
|
|
|
implements.
|
|
|
|
|
|
|
|
As explained in the description of the @code{struct __gconv_step} data
|
|
|
|
structure above the initialization function has to initialize parts of
|
|
|
|
it.
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item __min_needed_from
|
|
|
|
@itemx __max_needed_from
|
|
|
|
@itemx __min_needed_to
|
|
|
|
@itemx __max_needed_to
|
|
|
|
These elements must be initialized to the exact numbers of the minimum
|
|
|
|
and maximum number of bytes used by one character in the source and
|
|
|
|
destination character sets, respectively. If the characters all have the
|
|
|
|
same size, the minimum and maximum values are the same.
|
|
|
|
|
|
|
|
@item __stateful
|
|
|
|
This element must be initialized to an nonzero value if the source
|
|
|
|
character set is stateful. Otherwise it must be zero.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
If the initialization function needs to communicate some information
|
2001-11-07 08:21:22 +01:00
|
|
|
to the conversion function, this communication can happen using the
|
|
|
|
@code{__data} element of the @code{__gconv_step} structure. But since
|
|
|
|
this data is shared by all the conversions, it must not be modified by
|
2001-11-05 09:11:26 +01:00
|
|
|
the conversion function. The example below shows how this can be used.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
#define MIN_NEEDED_FROM 1
|
|
|
|
#define MAX_NEEDED_FROM 4
|
|
|
|
#define MIN_NEEDED_TO 4
|
|
|
|
#define MAX_NEEDED_TO 4
|
|
|
|
|
|
|
|
int
|
|
|
|
gconv_init (struct __gconv_step *step)
|
|
|
|
@{
|
|
|
|
/* @r{Determine which direction.} */
|
|
|
|
struct iso2022jp_data *new_data;
|
|
|
|
enum direction dir = illegal_dir;
|
|
|
|
enum variant var = illegal_var;
|
|
|
|
int result;
|
|
|
|
|
|
|
|
if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)
|
|
|
|
@{
|
|
|
|
dir = from_iso2022jp;
|
|
|
|
var = iso2022jp;
|
|
|
|
@}
|
|
|
|
else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)
|
|
|
|
@{
|
|
|
|
dir = to_iso2022jp;
|
|
|
|
var = iso2022jp;
|
|
|
|
@}
|
|
|
|
else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)
|
|
|
|
@{
|
|
|
|
dir = from_iso2022jp;
|
|
|
|
var = iso2022jp2;
|
|
|
|
@}
|
|
|
|
else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)
|
|
|
|
@{
|
|
|
|
dir = to_iso2022jp;
|
|
|
|
var = iso2022jp2;
|
|
|
|
@}
|
|
|
|
|
|
|
|
result = __GCONV_NOCONV;
|
|
|
|
if (dir != illegal_dir)
|
|
|
|
@{
|
|
|
|
new_data = (struct iso2022jp_data *)
|
|
|
|
malloc (sizeof (struct iso2022jp_data));
|
|
|
|
|
|
|
|
result = __GCONV_NOMEM;
|
|
|
|
if (new_data != NULL)
|
|
|
|
@{
|
|
|
|
new_data->dir = dir;
|
|
|
|
new_data->var = var;
|
|
|
|
step->__data = new_data;
|
|
|
|
|
|
|
|
if (dir == from_iso2022jp)
|
|
|
|
@{
|
|
|
|
step->__min_needed_from = MIN_NEEDED_FROM;
|
|
|
|
step->__max_needed_from = MAX_NEEDED_FROM;
|
|
|
|
step->__min_needed_to = MIN_NEEDED_TO;
|
|
|
|
step->__max_needed_to = MAX_NEEDED_TO;
|
|
|
|
@}
|
|
|
|
else
|
|
|
|
@{
|
|
|
|
step->__min_needed_from = MIN_NEEDED_TO;
|
|
|
|
step->__max_needed_from = MAX_NEEDED_TO;
|
|
|
|
step->__min_needed_to = MIN_NEEDED_FROM;
|
|
|
|
step->__max_needed_to = MAX_NEEDED_FROM + 2;
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{Yes, this is a stateful encoding.} */
|
|
|
|
step->__stateful = 1;
|
|
|
|
|
|
|
|
result = __GCONV_OK;
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The function first checks which conversion is wanted. The module from
|
2001-11-07 08:21:22 +01:00
|
|
|
which this function is taken implements four different conversions;
|
2001-11-05 09:11:26 +01:00
|
|
|
which one is selected can be determined by comparing the names. The
|
|
|
|
comparison should always be done without paying attention to the case.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Next, a data structure, which contains the necessary information about
|
2001-11-05 09:11:26 +01:00
|
|
|
which conversion is selected, is allocated. The data structure
|
2001-11-07 08:21:22 +01:00
|
|
|
@code{struct iso2022jp_data} is locally defined since, outside the
|
|
|
|
module, this data is not used at all. Please note that if all four
|
|
|
|
conversions this modules supports are requested there are four data
|
2001-11-05 09:11:26 +01:00
|
|
|
blocks.
|
|
|
|
|
|
|
|
One interesting thing is the initialization of the @code{__min_} and
|
|
|
|
@code{__max_} elements of the step data object. A single ISO-2022-JP
|
|
|
|
character can consist of one to four bytes. Therefore the
|
|
|
|
@code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined
|
|
|
|
this way. The output is always the @code{INTERNAL} character set (aka
|
|
|
|
UCS-4) and therefore each character consists of exactly four bytes. For
|
|
|
|
the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into
|
|
|
|
account that escape sequences might be necessary to switch the character
|
|
|
|
sets. Therefore the @code{__max_needed_to} element for this direction
|
|
|
|
gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the
|
|
|
|
two bytes needed for the escape sequences to single the switching. The
|
|
|
|
asymmetry in the maximum values for the two directions can be explained
|
|
|
|
easily: when reading ISO-2022-JP text, escape sequences can be handled
|
|
|
|
alone (i.e., it is not necessary to process a real character since the
|
|
|
|
effect of the escape sequence can be recorded in the state information).
|
|
|
|
The situation is different for the other direction. Since it is in
|
|
|
|
general not known which character comes next, one cannot emit escape
|
|
|
|
sequences to change the state in advance. This means the escape
|
|
|
|
sequences that have to be emitted together with the next character.
|
|
|
|
Therefore one needs more room than only for the character itself.
|
|
|
|
|
|
|
|
The possible return values of the initialization function are:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item __GCONV_OK
|
|
|
|
The initialization succeeded
|
|
|
|
@item __GCONV_NOCONV
|
|
|
|
The requested conversion is not supported in the module. This can
|
|
|
|
happen if the @file{gconv-modules} file has errors.
|
|
|
|
@item __GCONV_NOMEM
|
|
|
|
Memory required to store additional information could not be allocated.
|
|
|
|
@end table
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
The function called before the module is unloaded is significantly
|
|
|
|
easier. It often has nothing at all to do; in which case it can be left
|
|
|
|
out completely.
|
|
|
|
|
|
|
|
@comment gconv.h
|
|
|
|
@comment GNU
|
|
|
|
@deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *)
|
|
|
|
The task of this function is to free all resources allocated in the
|
|
|
|
initialization function. Therefore only the @code{__data} element of
|
|
|
|
the object pointed to by the argument is of interest. Continuing the
|
|
|
|
example from the initialization function, the finalization function
|
|
|
|
looks like this:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
void
|
|
|
|
gconv_end (struct __gconv_step *data)
|
|
|
|
@{
|
|
|
|
free (data->__data);
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
The most important function is the conversion function itself, which can
|
|
|
|
get quite complicated for complex character sets. But since this is not
|
|
|
|
of interest here, we will only describe a possible skeleton for the
|
|
|
|
conversion function.
|
|
|
|
|
|
|
|
@comment gconv.h
|
|
|
|
@comment GNU
|
|
|
|
@deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int)
|
|
|
|
The conversion function can be called for two basic reason: to convert
|
|
|
|
text or to reset the state. From the description of the @code{iconv}
|
|
|
|
function it can be seen why the flushing mode is necessary. What mode
|
2001-11-07 08:21:22 +01:00
|
|
|
is selected is determined by the sixth argument, an integer. This
|
2001-11-05 09:11:26 +01:00
|
|
|
argument being nonzero means that flushing is selected.
|
|
|
|
|
|
|
|
Common to both modes is where the output buffer can be found. The
|
|
|
|
information about this buffer is stored in the conversion step data. A
|
2001-11-07 08:21:22 +01:00
|
|
|
pointer to this information is passed as the second argument to this
|
|
|
|
function. The description of the @code{struct __gconv_step_data}
|
2001-11-05 09:11:26 +01:00
|
|
|
structure has more information on the conversion step data.
|
|
|
|
|
|
|
|
@cindex stateful
|
|
|
|
What has to be done for flushing depends on the source character set.
|
2001-11-07 08:21:22 +01:00
|
|
|
If the source character set is not stateful, nothing has to be done.
|
|
|
|
Otherwise the function has to emit a byte sequence to bring the state
|
|
|
|
object into the initial state. Once this all happened the other
|
|
|
|
conversion modules in the chain of conversions have to get the same
|
|
|
|
chance. Whether another step follows can be determined from the
|
|
|
|
@code{__is_last} element of the step data structure to which the first
|
2001-11-05 09:11:26 +01:00
|
|
|
parameter points.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
The more interesting mode is when actual text has to be converted. The
|
|
|
|
first step in this case is to convert as much text as possible from the
|
|
|
|
input buffer and store the result in the output buffer. The start of the
|
|
|
|
input buffer is determined by the third argument, which is a pointer to a
|
|
|
|
pointer variable referencing the beginning of the buffer. The fourth
|
2001-11-05 09:11:26 +01:00
|
|
|
argument is a pointer to the byte right after the last byte in the buffer.
|
|
|
|
|
|
|
|
The conversion has to be performed according to the current state if the
|
|
|
|
character set is stateful. The state is stored in an object pointed to
|
|
|
|
by the @code{__statep} element of the step data (second argument). Once
|
|
|
|
either the input buffer is empty or the output buffer is full the
|
|
|
|
conversion stops. At this point, the pointer variable referenced by the
|
|
|
|
third parameter must point to the byte following the last processed
|
|
|
|
byte (i.e., if all of the input is consumed, this pointer and the fourth
|
|
|
|
parameter have the same value).
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
What now happens depends on whether this step is the last one. If it is
|
|
|
|
the last step, the only thing that has to be done is to update the
|
2001-11-05 09:11:26 +01:00
|
|
|
@code{__outbuf} element of the step data structure to point after the
|
2001-11-07 08:21:22 +01:00
|
|
|
last written byte. This update gives the caller the information on how
|
2001-11-05 09:11:26 +01:00
|
|
|
much text is available in the output buffer. In addition, the variable
|
|
|
|
pointed to by the fifth parameter, which is of type @code{size_t}, must
|
|
|
|
be incremented by the number of characters (@emph{not bytes}) that were
|
|
|
|
converted in a non-reversible way. Then, the function can return.
|
|
|
|
|
|
|
|
In case the step is not the last one, the later conversion functions have
|
|
|
|
to get a chance to do their work. Therefore, the appropriate conversion
|
|
|
|
function has to be called. The information about the functions is
|
|
|
|
stored in the conversion data structures, passed as the first parameter.
|
|
|
|
This information and the step data are stored in arrays, so the next
|
|
|
|
element in both cases can be found by simple pointer arithmetic:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
int
|
|
|
|
gconv (struct __gconv_step *step, struct __gconv_step_data *data,
|
|
|
|
const char **inbuf, const char *inbufend, size_t *written,
|
|
|
|
int do_flush)
|
|
|
|
@{
|
|
|
|
struct __gconv_step *next_step = step + 1;
|
|
|
|
struct __gconv_step_data *next_data = data + 1;
|
2002-06-30 06:04:20 +02:00
|
|
|
@dots{}
|
2001-11-05 09:11:26 +01:00
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
The @code{next_step} pointer references the next step information and
|
|
|
|
@code{next_data} the next data record. The call of the next function
|
|
|
|
therefore will look similar to this:
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
next_step->__fct (next_step, next_data, &outerr, outbuf,
|
|
|
|
written, 0)
|
|
|
|
@end smallexample
|
|
|
|
|
|
|
|
But this is not yet all. Once the function call returns the conversion
|
2001-11-07 08:21:22 +01:00
|
|
|
function might have some more to do. If the return value of the function
|
|
|
|
is @code{__GCONV_EMPTY_INPUT}, more room is available in the output
|
|
|
|
buffer. Unless the input buffer is empty the conversion, functions start
|
|
|
|
all over again and process the rest of the input buffer. If the return
|
|
|
|
value is not @code{__GCONV_EMPTY_INPUT}, something went wrong and we have
|
2001-11-05 09:11:26 +01:00
|
|
|
to recover from this.
|
|
|
|
|
|
|
|
A requirement for the conversion function is that the input buffer
|
|
|
|
pointer (the third argument) always point to the last character that
|
|
|
|
was put in converted form into the output buffer. This is trivially
|
|
|
|
true after the conversion performed in the current step, but if the
|
|
|
|
conversion functions deeper downstream stop prematurely, not all
|
|
|
|
characters from the output buffer are consumed and, therefore, the input
|
|
|
|
buffer pointers must be backed off to the right position.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
Correcting the input buffers is easy to do if the input and output
|
|
|
|
character sets have a fixed width for all characters. In this situation
|
|
|
|
we can compute how many characters are left in the output buffer and,
|
|
|
|
therefore, can correct the input buffer pointer appropriately with a
|
|
|
|
similar computation. Things are getting tricky if either character set
|
|
|
|
has characters represented with variable length byte sequences, and it
|
|
|
|
gets even more complicated if the conversion has to take care of the
|
|
|
|
state. In these cases the conversion has to be performed once again, from
|
|
|
|
the known state before the initial conversion (i.e., if necessary the
|
|
|
|
state of the conversion has to be reset and the conversion loop has to be
|
|
|
|
executed again). The difference now is that it is known how much input
|
|
|
|
must be created, and the conversion can stop before converting the first
|
|
|
|
unused character. Once this is done the input buffer pointers must be
|
2001-11-05 09:11:26 +01:00
|
|
|
updated again and the function can return.
|
|
|
|
|
|
|
|
One final thing should be mentioned. If it is necessary for the
|
|
|
|
conversion to know whether it is the first invocation (in case a prolog
|
2001-11-07 08:21:22 +01:00
|
|
|
has to be emitted), the conversion function should increment the
|
|
|
|
@code{__invocation_counter} element of the step data structure just
|
2001-11-05 09:11:26 +01:00
|
|
|
before returning to the caller. See the description of the @code{struct
|
|
|
|
__gconv_step_data} structure above for more information on how this can
|
|
|
|
be used.
|
|
|
|
|
|
|
|
The return value must be one of the following values:
|
|
|
|
|
|
|
|
@table @code
|
|
|
|
@item __GCONV_EMPTY_INPUT
|
|
|
|
All input was consumed and there is room left in the output buffer.
|
|
|
|
@item __GCONV_FULL_OUTPUT
|
|
|
|
No more room in the output buffer. In case this is not the last step
|
|
|
|
this value is propagated down from the call of the next conversion
|
2001-11-07 08:21:22 +01:00
|
|
|
function in the chain.
|
2001-11-05 09:11:26 +01:00
|
|
|
@item __GCONV_INCOMPLETE_INPUT
|
|
|
|
The input buffer is not entirely empty since it contains an incomplete
|
|
|
|
character sequence.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
The following example provides a framework for a conversion function.
|
|
|
|
In case a new conversion has to be written the holes in this
|
|
|
|
implementation have to be filled and that is it.
|
|
|
|
|
|
|
|
@smallexample
|
|
|
|
int
|
|
|
|
gconv (struct __gconv_step *step, struct __gconv_step_data *data,
|
|
|
|
const char **inbuf, const char *inbufend, size_t *written,
|
|
|
|
int do_flush)
|
|
|
|
@{
|
|
|
|
struct __gconv_step *next_step = step + 1;
|
|
|
|
struct __gconv_step_data *next_data = data + 1;
|
|
|
|
gconv_fct fct = next_step->__fct;
|
|
|
|
int status;
|
|
|
|
|
|
|
|
/* @r{If the function is called with no input this means we have}
|
|
|
|
@r{to reset to the initial state. The possibly partly}
|
|
|
|
@r{converted input is dropped.} */
|
|
|
|
if (do_flush)
|
|
|
|
@{
|
|
|
|
status = __GCONV_OK;
|
|
|
|
|
|
|
|
/* @r{Possible emit a byte sequence which put the state object}
|
|
|
|
@r{into the initial state.} */
|
|
|
|
|
|
|
|
/* @r{Call the steps down the chain if there are any but only}
|
|
|
|
@r{if we successfully emitted the escape sequence.} */
|
|
|
|
if (status == __GCONV_OK && ! data->__is_last)
|
|
|
|
status = fct (next_step, next_data, NULL, NULL,
|
|
|
|
written, 1);
|
|
|
|
@}
|
|
|
|
else
|
|
|
|
@{
|
|
|
|
/* @r{We preserve the initial values of the pointer variables.} */
|
|
|
|
const char *inptr = *inbuf;
|
|
|
|
char *outbuf = data->__outbuf;
|
|
|
|
char *outend = data->__outbufend;
|
|
|
|
char *outptr;
|
|
|
|
|
|
|
|
do
|
|
|
|
@{
|
|
|
|
/* @r{Remember the start value for this round.} */
|
|
|
|
inptr = *inbuf;
|
|
|
|
/* @r{The outbuf buffer is empty.} */
|
|
|
|
outptr = outbuf;
|
|
|
|
|
|
|
|
/* @r{For stateful encodings the state must be safe here.} */
|
|
|
|
|
|
|
|
/* @r{Run the conversion loop. @code{status} is set}
|
|
|
|
@r{appropriately afterwards.} */
|
|
|
|
|
|
|
|
/* @r{If this is the last step, leave the loop. There is}
|
|
|
|
@r{nothing we can do.} */
|
|
|
|
if (data->__is_last)
|
|
|
|
@{
|
|
|
|
/* @r{Store information about how many bytes are}
|
|
|
|
@r{available.} */
|
|
|
|
data->__outbuf = outbuf;
|
|
|
|
|
|
|
|
/* @r{If any non-reversible conversions were performed,}
|
|
|
|
@r{add the number to @code{*written}.} */
|
|
|
|
|
|
|
|
break;
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{Write out all output that was produced.} */
|
|
|
|
if (outbuf > outptr)
|
|
|
|
@{
|
|
|
|
const char *outerr = data->__outbuf;
|
|
|
|
int result;
|
|
|
|
|
|
|
|
result = fct (next_step, next_data, &outerr,
|
|
|
|
outbuf, written, 0);
|
|
|
|
|
|
|
|
if (result != __GCONV_EMPTY_INPUT)
|
|
|
|
@{
|
|
|
|
if (outerr != outbuf)
|
|
|
|
@{
|
|
|
|
/* @r{Reset the input buffer pointer. We}
|
|
|
|
@r{document here the complex case.} */
|
|
|
|
size_t nstatus;
|
|
|
|
|
|
|
|
/* @r{Reload the pointers.} */
|
|
|
|
*inbuf = inptr;
|
|
|
|
outbuf = outptr;
|
|
|
|
|
|
|
|
/* @r{Possibly reset the state.} */
|
|
|
|
|
|
|
|
/* @r{Redo the conversion, but this time}
|
|
|
|
@r{the end of the output buffer is at}
|
|
|
|
@r{@code{outerr}.} */
|
|
|
|
@}
|
|
|
|
|
|
|
|
/* @r{Change the status.} */
|
|
|
|
status = result;
|
|
|
|
@}
|
|
|
|
else
|
|
|
|
/* @r{All the output is consumed, we can make}
|
|
|
|
@r{ another run if everything was ok.} */
|
|
|
|
if (status == __GCONV_FULL_OUTPUT)
|
|
|
|
status = __GCONV_OK;
|
|
|
|
@}
|
|
|
|
@}
|
|
|
|
while (status == __GCONV_OK);
|
|
|
|
|
|
|
|
/* @r{We finished one use of this step.} */
|
|
|
|
++data->__invocation_counter;
|
|
|
|
@}
|
|
|
|
|
|
|
|
return status;
|
|
|
|
@}
|
|
|
|
@end smallexample
|
|
|
|
@end deftypevr
|
|
|
|
|
|
|
|
This information should be sufficient to write new modules. Anybody
|
|
|
|
doing so should also take a look at the available source code in the GNU
|
|
|
|
C library sources. It contains many examples of working and optimized
|
|
|
|
modules.
|
|
|
|
|
2001-11-07 08:21:22 +01:00
|
|
|
@c File charset.texi edited October 2001 by Dennis Grace, IBM Corporation
|