codecvt.html: Add more bits, format.
2000-08-28 Benjamin Kosnik <bkoz@purist.soma.redhat.com> * docs/22_locale/codecvt.html: Add more bits, format. From-SVN: r36026
This commit is contained in:
parent
337aa8ed54
commit
a811708d7a
@ -1,3 +1,7 @@
|
||||
2000-08-28 Benjamin Kosnik <bkoz@purist.soma.redhat.com>
|
||||
|
||||
* docs/22_locale/codecvt.html: Add more bits, format.
|
||||
|
||||
2000-08-28 Phil Edwards <pme@sources.redhat.com>
|
||||
|
||||
* docs/configopts.html: Mention new options.
|
||||
|
@ -1,112 +1,512 @@
|
||||
<!-- ================================================================================ -->
|
||||
<!-- This HTML file was created by AbiWord. -->
|
||||
<!-- AbiWord is a free, Open Source word processor. -->
|
||||
<!-- You may obtain more information about AbiWord at www.abisource.com -->
|
||||
<!-- ================================================================================ -->
|
||||
<HTML>
|
||||
<HEAD>
|
||||
<H1>
|
||||
Notes on the codecvt implementation.
|
||||
</H1>
|
||||
</HEAD>
|
||||
<I>
|
||||
prepared by Benjamin Kosnik (bkoz@redhat.com) on August 28, 2000
|
||||
</I>
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
1. Abstract
|
||||
</H2>
|
||||
<P>
|
||||
The standard class codecvt attempts to address conversions between
|
||||
different character encoding schemes. In particular, the standard
|
||||
attempts to detail conversions between the implementation-defined wide
|
||||
characters (hereafter referred to as wchar_t) and the standard type
|
||||
char that is so beloved in classic "C" (which can now be referred to
|
||||
as narrow characters.) This document attempts to describe how the GNU
|
||||
libstdc++-v3 implementation deals with the conversion between wide and
|
||||
narrow characters, and also presents a framework for dealing with the
|
||||
huge number of other encodings that iconv can convert, including
|
||||
Unicode and UTF8. Design issues and requirements are addressed, and
|
||||
examples of correct usage for both the required specializations for
|
||||
wide and narrow characters and the implementation-provided extended
|
||||
functionality are given.
|
||||
</P>
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
2. What the standard says
|
||||
</H2>
|
||||
Around page 425 of the C++ Standard, this charming heading comes into view:
|
||||
|
||||
<BLOCKQUOTE>
|
||||
22.2.1.5 - Template class codecvt [lib.locale.codecvt]
|
||||
</BLOCKQUOTE>
|
||||
|
||||
The text around the codecvt definition gives some clues:
|
||||
|
||||
<BLOCKQUOTE>
|
||||
<I>
|
||||
-1- The class codecvt<internT,externT,stateT> is for use when
|
||||
converting from one codeset to another, such as from wide characters
|
||||
to multibyte characters, between wide character encodings such as
|
||||
Unicode and EUC.
|
||||
</I>
|
||||
</BLOCKQUOTE>
|
||||
|
||||
<P>
|
||||
Hmm. So, in some unspecified way, Unicode encodings and
|
||||
translations between other character sets should be handled by this
|
||||
class.
|
||||
</P>
|
||||
|
||||
<BLOCKQUOTE>
|
||||
<I>
|
||||
-2- The stateT argument selects the pair of codesets being mapped between.
|
||||
</I>
|
||||
</BLOCKQUOTE>
|
||||
|
||||
<P>
|
||||
Ah ha! Another clue...
|
||||
</P>
|
||||
|
||||
<BLOCKQUOTE>
|
||||
<I>
|
||||
-3- The instantiations required in the Table ??
|
||||
(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and
|
||||
codecvt<char,char,mbstate_t>, convert the implementation-defined
|
||||
native character set. codecvt<char,char,mbstate_t> implements a
|
||||
degenerate conversion; it does not convert at
|
||||
all. codecvt<wchar_t,char,mbstate_t> converts between the native
|
||||
character sets for tiny and wide characters. Instantiations on
|
||||
mbstate_t perform conversion between encodings known to the library
|
||||
implementor. Other encodings can be converted by specializing on a
|
||||
user-defined stateT type. The stateT object can contain any state that
|
||||
is useful to communicate to or from the specialized do_convert member.
|
||||
</I>
|
||||
</BLOCKQUOTE>
|
||||
|
||||
<P>
|
||||
At this point, a couple points become clear:
|
||||
|
||||
<P>
|
||||
One: The standard clearly implies that attempts to add non-required
|
||||
(yet useful and widely used) conversions need to do so through the
|
||||
third template parameter, stateT.</P>
|
||||
|
||||
<P>
|
||||
Two: The required conversions, by specifying mbstate_t as the third
|
||||
template parameter, imply an implementation strategy that is mostly
|
||||
(or wholly) based on the underlying C library, and the functions
|
||||
mcsrtombs and wcsrtombs in particular.</P>
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
2. Some thoughts on what would be useful
|
||||
</H2>
|
||||
Probably the most frequently asked question about code conversion is:
|
||||
"So dudes, what's the deal with Unicode strings?" The dude part is
|
||||
optional, but apparently the usefulness of Unicode strings is pretty
|
||||
widely appreciated. Sadly, this specific encoding (And other useful
|
||||
encodings like UTF8, UCS4, ISO 8859-10, etc etc etc) are not mentioned
|
||||
in the C++ standard.
|
||||
|
||||
<P>
|
||||
In particular, the simple implementation detail of wchar_t's size
|
||||
seems to repeatedly confound people. Many systems use a two byte,
|
||||
unsigned integral type to represent wide characters, and use an
|
||||
internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, Java,
|
||||
others.) Other systems, use a four byte, unsigned integral type to
|
||||
represent wide characters, and use an internal encoding of
|
||||
UCS4. (GNU/Linux systems using glibc, in particular.) The C
|
||||
programming language (and thus C++) does not specify a specific size
|
||||
for the type wchar_t.
|
||||
|
||||
<P>
|
||||
Thus, portable C++ code cannot assume a byte size (or endianness) either.
|
||||
|
||||
<P>
|
||||
Getting back to the frequently asked question: What about Unicode strings?
|
||||
|
||||
<P>
|
||||
What magic spell will do this conversion?
|
||||
|
||||
<P>
|
||||
A couple of comments:
|
||||
</P>
|
||||
|
||||
<P>
|
||||
The thought that all one needs to convert between two arbitrary
|
||||
codesets is two types and some kind of state argument is
|
||||
unfortunate. In particular, encodings may be stateless. The naming of
|
||||
the third parameter as stateT is unfortunate, as what is really needed
|
||||
is some kind of generalized type that accounts for the issues that
|
||||
abstract encodings will need. The minimum information that is required
|
||||
includes:
|
||||
</P>
|
||||
|
||||
<UL>
|
||||
<LI>
|
||||
</P>
|
||||
Identifiers for each of the codesets involved in the conversion. For
|
||||
example, using the iconv family of functions from the Single Unix
|
||||
Specification (what used to be called X/Open) hosted on the GNU/Linux
|
||||
operating system allows bi-directional mapping between far more than
|
||||
the following tantalizing possibilities:
|
||||
</P>
|
||||
|
||||
(An edited list taken from <TT>`iconv --list`</TT> on a Red Hat 6.2/Intel system:
|
||||
|
||||
<BLOCKQUOTE>
|
||||
<PRE>
|
||||
8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
|
||||
ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCITT, GREEK, GREEK7-OLD,
|
||||
GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
|
||||
ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
|
||||
ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
|
||||
ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
|
||||
ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
|
||||
UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELITTLE, US-ASCII, US, UTF-8,
|
||||
UTF-16, UTF8, UTF16).
|
||||
</PRE>
|
||||
</BLOCKQUOTE>
|
||||
|
||||
<P>
|
||||
For iconv-based implementations, string literals for each of the
|
||||
encodings (ie. "UCS-2" and "UTF-8") are necessary, although for other,
|
||||
non-iconv implementations a table of enumerated values or some other
|
||||
mechanism may be required.
|
||||
|
||||
<LI>
|
||||
Maximum length of the identifying string literal.
|
||||
|
||||
<LI>
|
||||
Some encodings are require explicit endian-ness. As such, some kind
|
||||
of endian marker or other byte-order marker will be necessary. See
|
||||
"Footnotes for C/C++ developers" in Haible for more information on
|
||||
UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
|
||||
however implementations, most notably Microsoft, vary.)
|
||||
|
||||
<LI>
|
||||
Types representing the conversion state, for conversions involving
|
||||
the machinery in the "C" library, or the conversion descriptor, for
|
||||
conversions using iconv (such as the type iconv_t.) Note that the
|
||||
conversion descriptor encodes more information than a simple encoding
|
||||
state type.
|
||||
|
||||
<LI>
|
||||
Conversion descriptors for both directions of encoding. (ie, both
|
||||
UCS-2 to UTF-8 and UTF-8 to UCS-2.)
|
||||
|
||||
<LI>
|
||||
Something to indicate if the conversion requested if valid.
|
||||
|
||||
<LI>
|
||||
Something to represent if the conversion descriptors are valid.
|
||||
|
||||
<LI>
|
||||
Some way to enforce strict type checking on the internal and
|
||||
external types. As part of this, the size of the internal and
|
||||
external types will need to be known.
|
||||
</UL>
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
3. Problems with "C" code conversions : thread safety, global locales,
|
||||
termination.
|
||||
</H2>
|
||||
|
||||
In addition, multi-threaded and multi-locale environments also impact
|
||||
the design and requirements for code conversions. In particular, they
|
||||
affect the required specialization codecvt<wchar_t, char, mbstate_t>
|
||||
when implemented using standard "C" functions.
|
||||
|
||||
<P>
|
||||
Three problems arise, one big, one of medium importance, and one small.
|
||||
|
||||
<P>
|
||||
First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
|
||||
on all systems required by the GNU tools. For GNU/Linux and glibc,
|
||||
this is not an issue.
|
||||
|
||||
<P>
|
||||
Of medium concern, in the grand scope of things, is that the functions
|
||||
used to implement this specialization work on null-terminated
|
||||
strings. Buffers, especially file buffers, may not be null-terminated,
|
||||
thus giving conversions that end prematurely or are otherwise
|
||||
incorrect. Yikes!
|
||||
|
||||
<P>
|
||||
The last, and fundamental problem, is the assumption of a global
|
||||
locale for all the "C" functions referenced above. For something like
|
||||
C++ iostreams (where codecvt is explicitly used) the notion of
|
||||
multiple locales is fundamental. In practice, most users may not run
|
||||
into this limitation. However, as a quality of implementation issue,
|
||||
the GNU C++ library would like to offer a solution that allows
|
||||
multiple locales and or simultaneous usage with computationally
|
||||
correct results. In short, libstdc++-v3 is trying to offer, as an
|
||||
option, a high-quality implementation, damn the additional complexity!
|
||||
|
||||
<P>
|
||||
For the required specialization codecvt<wchar_t, char, mbstate_t> ,
|
||||
conversions are made between the internal character set (always UCS4
|
||||
on GNU/Linux) and whatever the currently selected locale for the
|
||||
LC_CTYPE category implements.
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
4. Design
|
||||
</H2>
|
||||
The two required specializations are implemented as follows:
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
codecvt<char, char, mbstate_t>
|
||||
</TT>
|
||||
<P>
|
||||
This is a degenerate (ie, does nothing) specialization. Implementing
|
||||
this was a piece of cake.
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
codecvt<char, wchar_t, mbstate_t>
|
||||
</TT>
|
||||
<P>
|
||||
This specialization, by specifying all the template parameters, pretty
|
||||
much ties the hands of implementors. As such, the implementation is
|
||||
straightforward, involving mcsrtombs for the conversions between char
|
||||
to wchar_t and wcsrtombs for conversions between wchar_t and char.
|
||||
|
||||
<P>
|
||||
Neither of these two required specializations deals with Unicode
|
||||
characters. As such, libstdc++-v3 implements a partial specialization
|
||||
of the codecvt class with and iconv wrapper class, __enc_traits as the
|
||||
third template parameter.
|
||||
|
||||
<P>
|
||||
This implementation should be standards conformant. First of all, the
|
||||
standard explicitly points out that instantiations on the third
|
||||
template parameter, stateT, are the proper way to implement
|
||||
non-required conversions. Second of all, the standard says (in Chapter
|
||||
17) that partial specializations of required classes are a-ok. Third
|
||||
of all, the requirements for the stateT type elsewhere in the standard
|
||||
(see 21.1.2 traits typedefs) only indicate that this type be copy
|
||||
constructible.
|
||||
|
||||
<P>
|
||||
As such, the type __enc_traits is defined as a non-templatized, POD
|
||||
type to be used as the third type of a codecvt instantiation. This
|
||||
type is just a wrapper class for iconv, and provides an easy interface
|
||||
to iconv functionality.
|
||||
|
||||
<P>
|
||||
There are two constructors for __enc_traits:
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
__enc_traits() : __in_desc(0), __out_desc(0)
|
||||
</TT>
|
||||
<P>
|
||||
This default constructor sets the internal encoding to some default
|
||||
(currently UCS4) and the external encoding to whatever is returned by
|
||||
nl_langinfo(CODESET).
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
__enc_traits(const char* __int, const char* __ext)
|
||||
</TT>
|
||||
<P>
|
||||
This constructor takes as parameters string literals that indicate the
|
||||
desired internal and external encoding. There are no defaults for
|
||||
either argument.
|
||||
|
||||
<P>
|
||||
One of the issues with iconv is that the string literals identifying
|
||||
conversions are not standardized. Because of this, the thought of
|
||||
mandating and or enforcing some set of pre-determined valid
|
||||
identifiers seems iffy: thus, a more practical (and non-migraine
|
||||
inducing) strategy was implemented: end-users can specify any string
|
||||
(subject to a pre-determined length qualifier, currently 32 bytes) for
|
||||
encodings. It is up to the user to make sure that these strings are
|
||||
valid on the target system.
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
void
|
||||
_M_init()
|
||||
</TT>
|
||||
<P>
|
||||
Strangely enough, this member function attempts to open conversion
|
||||
descriptors for a given __enc_traits object. If the conversion
|
||||
descriptors are not valid, the conversion descriptors returned will
|
||||
not be valid and the resulting calls to the codecvt conversion
|
||||
functions will return error.
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
bool
|
||||
_M_good()
|
||||
</TT>
|
||||
<P>
|
||||
Provides a way to see if the given __enc_traits object has been
|
||||
properly initialized. If the string literals describing the desired
|
||||
internal and external encoding are not valid, initialization will
|
||||
fail, and this will return false. If the internal and external
|
||||
encodings are valid, but iconv_open could not allocate conversion
|
||||
descriptors, this will also return false. Otherwise, the object is
|
||||
ready to convert and will return true.
|
||||
|
||||
<P>
|
||||
<TT>
|
||||
__enc_traits(const __enc_traits&)
|
||||
</TT>
|
||||
<P>
|
||||
As iconv allocates memory and sets up conversion descriptors, the copy
|
||||
constructor can only copy the member data pertaining to the internal
|
||||
and external code conversions, and not the conversion descriptors
|
||||
themselves.
|
||||
|
||||
<P>
|
||||
Definitions for all the required codecvt member functions are provided
|
||||
for this specialization, and usage of codecvt<internal character type,
|
||||
external character type, __enc_traits> is consistent with other
|
||||
codecvt usage.
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
5. Examples
|
||||
</H2>
|
||||
a. conversions involving string literals
|
||||
|
||||
<pre>
|
||||
typedef codecvt_base::result result;
|
||||
typedef unsigned short unicode_t;
|
||||
typedef unicode_t int_type;
|
||||
typedef char ext_type;
|
||||
typedef __enc_traits enc_type;
|
||||
typedef codecvt<int_type, ext_type, enc_type> unicode_codecvt;
|
||||
|
||||
const ext_type* e_lit = "black pearl jasmine tea";
|
||||
int size = strlen(e_lit);
|
||||
int_type i_lit_base[24] =
|
||||
{ 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
|
||||
27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
|
||||
25856, 24832, 2560
|
||||
};
|
||||
const int_type* i_lit = i_lit_base;
|
||||
const ext_type* efrom_next;
|
||||
const int_type* ifrom_next;
|
||||
ext_type* e_arr = new ext_type[size + 1];
|
||||
ext_type* eto_next;
|
||||
int_type* i_arr = new int_type[size + 1];
|
||||
int_type* ito_next;
|
||||
|
||||
// construct a locale object with the specialized facet.
|
||||
locale loc(locale::classic(), new unicode_codecvt);
|
||||
// sanity check the constructed locale has the specialized facet.
|
||||
VERIFY( has_facet<unicode_codecvt>(loc) );
|
||||
const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
|
||||
// convert between const char* and unicode strings
|
||||
unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
|
||||
initialize_state(state01);
|
||||
result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
|
||||
i_arr, i_arr + size, ito_next);
|
||||
VERIFY( r1 == codecvt_base::ok );
|
||||
VERIFY( !int_traits::compare(i_arr, i_lit, size) );
|
||||
VERIFY( efrom_next == e_lit + size );
|
||||
VERIFY( ito_next == i_arr + size );
|
||||
</pre>
|
||||
b. conversions involving std::string
|
||||
c. conversions involving std::filebuf and std::ostream
|
||||
|
||||
More information can be found in the following testcases:
|
||||
codecvt_char_char.cc
|
||||
codecvt_unicode_wchar_t.cc
|
||||
codecvt_unicode_char.cc
|
||||
codecvt_wchar_t_char.cc
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
6. Unresolved Issues
|
||||
</H2>
|
||||
<UL>
|
||||
<LI>
|
||||
a. things that are sketchy, or remain unimplemented
|
||||
do_encoding, max_length and length member functions
|
||||
are only weakly implemented. I have no idea how to do
|
||||
this correctly, and in a generic manner. Nathan?
|
||||
|
||||
<LI>
|
||||
b. conversions involving std::string
|
||||
|
||||
<UL>
|
||||
<LI>
|
||||
how should operators != and == work for string of
|
||||
different/same encoding?
|
||||
|
||||
<LI>
|
||||
what is equal? A byte by byte comparison or an
|
||||
encoding then byte comparison?
|
||||
|
||||
<LI>
|
||||
conversions between narrow, wide, and unicode strings
|
||||
</UL>
|
||||
<LI>
|
||||
c. conversions involving std::filebuf and std::ostream
|
||||
<UL>
|
||||
<LI>
|
||||
how to initialize the state object in a
|
||||
standards-conformant manner?
|
||||
|
||||
<LI>
|
||||
how to synchronize the "C" and "C++" conversion information?
|
||||
|
||||
<LI>
|
||||
wchar_t/char internal buffers and conversions between
|
||||
internal/external buffers?
|
||||
</UL>
|
||||
</UL>
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
7. Acknowledgments
|
||||
</H2>
|
||||
Ulrich Drepper for the iconv suggestions and patient answering of
|
||||
late-night questions, Jason Merrill for the template partial
|
||||
specialization hints, language clarification, and wchar_t fixes.
|
||||
|
||||
<P>
|
||||
<H2>
|
||||
8. Bibliography / Referenced Documents
|
||||
</H2>
|
||||
|
||||
Drepper, Ulrich, GNU libc (glibc) 2.2 manual. In particular, Chapters "6. Character Set Handling" and "7 Locales and Internationalization"
|
||||
|
||||
<P>
|
||||
Drepper, Ulrich, Numerous, late-night email correspondence
|
||||
|
||||
<P>
|
||||
Feather, Clive, "A brief description of Normative Addendum 1," in particular the parts on Extended Character Sets
|
||||
http://www.lysator.liu.se/c/na1.html
|
||||
|
||||
<P>
|
||||
Haible, Bruno, "The Unicode HOWTO" v0.18, 4 August 2000
|
||||
ftp://ftp.ilog.fr/pub/Users/haible/utf8/Unicode-HOWTO.html
|
||||
|
||||
<P>
|
||||
ISO/IEC 14882:1998 Programming languages - C++
|
||||
|
||||
<P>
|
||||
ISO/IEC 9899:1999 Programming languages - C
|
||||
|
||||
<P>
|
||||
Khun, Markus, "UTF-8 and Unicode FAQ for Unix/Linux"
|
||||
http://www.cl.cam.ac.uk/~mgk25/unicode.html
|
||||
|
||||
<P>
|
||||
Langer, Angelika and Klaus Kreft, Standard C++ IOStreams and Locales, Advanced Programmer's Guide and Reference, Addison Wesley Longman, Inc. 2000
|
||||
|
||||
<P>
|
||||
Stroustrup, Bjarne, Appendix D, The C++ Programming Language, Special Edition, Addison Wesley, Inc. 2000
|
||||
|
||||
<P>
|
||||
System Interface Definitions, Issue 6 (IEEE Std. 1003.1-200x)
|
||||
The Open Group/The Institute of Electrical and Electronics Engineers, Inc.
|
||||
http://www.opennc.org/austin/docreg.html
|
||||
|
||||
<!-- Build_Version = 0.7.10 -->
|
||||
<!-- Build_Options = LicensedTrademarks:On Debug:Off Gnome:Off -->
|
||||
<!-- Build_Target = /var/tmp/builds/0961080942/tmp/abi-0.7.10/src/Linux_2.2.14-5.0_i386_OBJ/obj -->
|
||||
<!-- Build_CompileTime = 10:12:56 -->
|
||||
<!-- Build_CompileDate = Jun 15 2000 -->
|
||||
|
||||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml/DTD/xhtml1-strict.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<title>AbiWord Document</title>
|
||||
<style type="text/css">
|
||||
<!--
|
||||
P.norm { margin-top: 0pt; margin-bottom: 0pt }
|
||||
-->
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div>
|
||||
<p class="norm"><span style="font-weight: bold font-size: 16.000000pt;">N</span><span style="font-weight: bold font-size: 16.000000pt;">otes on the</span><span style="font-weight: bold font-size: 16.000000pt;"> codecvt implementation.</span></p>
|
||||
<p class="norm"><span style="font-weight: bold; font-style: italic font-size: 12.000000pt;">prepared by Benjamin Kosnik (bkoz@</span><span style="font-weight: bold; font-style: italic font-size: 12.000000pt;">redhat.com) on August 25, 2000</span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">1. </span><span style="font-weight: bold">Abst</span><span style="font-weight: bold">ract</span></p>
|
||||
<p class="norm">Around page 425 of the C++ Standard, this charming heading comes into view:</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">22.2.1.5 - Template class codecvt [lib.locale.codecvt]</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">The standard class codecvt attempts to address conversions between different character encoding schemes. In particular, the standard attempts to detail conversions between the implementation-defined wide characters (hereafter referred to as wchar_t) and the standard type char that is so beloved in classic "C" (which can now be referred to as narrow characters.) </p>
|
||||
<p class="norm">This document attempts to describe how the GNU libstdc++-v3 implementation deals with the conversion between wide and narrow characters, and also presents a framework for dealing with the huge number of other encodings that iconv can convert, including Unicode and UTF8. Design issues and requirements are addressed, and examples of correct usage for both the required specializations for wide and narrow characters and the implementation-provided extended functionality are given.</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">2. </span><span style="font-weight: bold color: 000000; font-family: Times New Roman; font-size: 12.000000pt;">Intro, ,</span><span style="font-weight: bold color: 000000; font-family: Times New Roman; font-size: 12.000000pt;">standard says</span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">2. </span><span style="font-weight: bold">Som</span><span style="font-weight: bold">e thoughts on what </span><span style="font-weight: bold">would be useful</span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Probably the most frequently asked question about code conversion is: "So dudes, what's the deal with Unicode strings?" The dude part is optional, but apparently the usefulness of Unicode strings is pretty widely appreciated. Sadly, this specific encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10, etc etc etc) are not mentioned in the C++ standard. </p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">In particular, the simple implementation detail of wchar_t's size seems to repeatedly confound people. Many systems use a two byte, unsigned integral type to represent wide characters, and use an internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, Java, others.) Other systems, use a four byte, unsigned integral type to represent wide characters, and use an internal encoding of UCS4. (GNU/Linux systems using glibc, in particular.) The C programming language (and thus C++) does not specify a specific size for the type wchar_t. </p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Thus, portable C++ code cannot assume a byte size (or endianness) either.</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Getting back to the frequently asked question: What about Unicode strings?</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">The text around the codecvt definition gives some clues:</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-style: italic"">-1- The class codecvt<internT,externT,stateT> is for use when converting from one</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">codeset to another, such as from wide characters to multibyte characters, between wide</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">character encodings such as Unicode and EUC. </span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Hmm. So, in some unspecified way, Unicode encodings and translations between other character sets should be handled by this class.</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-style: italic"">-2- The stateT argument selects the pair of codesets being mapped between. </span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Ah ha! Another clue...</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-style: italic"">-3- The instantiations required in the Table ?? (lib.locale.category), namely</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">codecvt<wchar_t,char,mbstate_t> and codecvt<char,char,mbstate_t>, convert the</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">implementation-defined native character set. codecvt<char,char,mbstate_t> implements</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">a degenerate conversion; it does not convert at all. codecvt<wchar_t,char,mbstate_t></span></p>
|
||||
<p class="norm"><span style="font-style: italic"">converts between the native character sets for tiny and wide characters. Instantiations on</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">mbstate_t perform conversion between encodings known to the library implementor.</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">Other encodings can be converted by specializing on a user-defined stateT type. The</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">stateT object can contain any state that is useful to communicate to or from the</span></p>
|
||||
<p class="norm"><span style="font-style: italic"">specialized do_convert member. </span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">At this point, the initial design of the library becomes clear:</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">3. </span><span style="font-weight: bold">How to accomplish </span><span style="font-weight: bold">this: partial specialization with and iconv</span><span style="font-weight: bold"> wrapper class, __enc_traits.</span></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">4. Design</span></p>
|
||||
<p class="norm"> a. goals.</p>
|
||||
<p class="norm"> b. drawbacks</p>
|
||||
<p class="norm"> c. things that are sketchy</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">5. Examples</span></p>
|
||||
<p class="norm"> a. conversions involving string literals</p>
|
||||
<p class="norm"> b. conversions invollving std::string</p>
|
||||
<p class="norm"> c. conversions involving std::filebuf and std::ostream</p>
|
||||
<p class="norm"> </p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">6. Acknowledg</span><span style="font-weight: bold">me</span><span style="font-weight: bold">nts</span></p>
|
||||
<p class="norm">Ulrich Drepper for the iconv suggestions and patient question answering, Jason Merrill for the template partial specialization hints and wchar_t fixes, etc etc etc.</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"><span style="font-weight: bold">7</span><span style="font-weight: bold">. Bibliography</span><span style="font-weight: bold"> / Referenced Documents</span></p>
|
||||
<p class="norm">ISO/IEC 14882:1998 Programming languages - C++</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">ISO/IEC 9899:1999 Programming languages - C</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">glibc-2.2 docs</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">System Interface Definitions, Issue 6 (IEEE Std. 1003.1-200x)</p>
|
||||
<p class="norm">The Open Group/The Institute of Electrical and Electronics Engineers, Inc.</p>
|
||||
<p class="norm">http://www.opennc.org/austin/docreg.html</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Appendix D, The C++ Programming Language, Special Edition, Bjarne Stroustrup, Addison Wesley, Inc. 2000</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Standard C++ IOStreams and Locales, Advanced Programmer's Guide and Reference, Angelika Langer and Klaus Kreft, Addison Wesley Longman, Inc. 2000</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm">Numerous, late-night email correspondence with Ulrich Drepper (drepper@redhat.com).</p>
|
||||
<p class="norm"></p>
|
||||
<p class="norm"></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
Loading…
Reference in New Issue
Block a user