Around page 425 of the C++ Standard, this charming heading comes into view:

The text around the codecvt definition gives some clues:

Hmm. So, in some unspecified way, Unicode encodings and translations between other character sets should be handled by this class.

Ah ha! Another clue...

At this point, a couple points become clear:

One: The standard clearly implies that attempts to add non-required (yet useful and widely used) conversions need to do so through the third template parameter, stateT.

Two: The required conversions, by specifying mbstate_t as the third template parameter, imply an implementation strategy that is mostly (or wholly) based on the underlying C library, and the functions mcsrtombs and wcsrtombs in particular.

The two required specializations are implemented as follows:

codecvt<char, char, mbstate_t>

This is a degenerate (i.e., does nothing) specialization. Implementing this was a piece of cake.

codecvt<char, wchar_t, mbstate_t>

This specialization, by specifying all the template parameters, pretty much ties the hands of implementors. As such, the implementation is straightforward, involving mcsrtombs for the conversions between char to wchar_t and wcsrtombs for conversions between wchar_t and char.

Neither of these two required specializations deals with Unicode characters. As such, libstdc++ implements a partial specialization of the codecvt class with and iconv wrapper class, encoding_state as the third template parameter.

This implementation should be standards conformant. First of all, the standard explicitly points out that instantiations on the third template parameter, stateT, are the proper way to implement non-required conversions. Second of all, the standard says (in Chapter 17) that partial specializations of required classes are a-ok. Third of all, the requirements for the stateT type elsewhere in the standard (see 21.1.2 traits typedefs) only indicate that this type be copy constructible.

As such, the type encoding_state is defined as a non-templatized, POD type to be used as the third type of a codecvt instantiation. This type is just a wrapper class for iconv, and provides an easy interface to iconv functionality.

There are two constructors for encoding_state:

encoding_state() : __in_desc(0), __out_desc(0)

This default constructor sets the internal encoding to some default (currently UCS4) and the external encoding to whatever is returned by nl_langinfo(CODESET).

encoding_state(const char* __int, const char* __ext)

This constructor takes as parameters string literals that indicate the desired internal and external encoding. There are no defaults for either argument.

One of the issues with iconv is that the string literals identifying conversions are not standardized. Because of this, the thought of mandating and or enforcing some set of pre-determined valid identifiers seems iffy: thus, a more practical (and non-migraine inducing) strategy was implemented: end-users can specify any string (subject to a pre-determined length qualifier, currently 32 bytes) for encodings. It is up to the user to make sure that these strings are valid on the target system.

void _M_init()

Strangely enough, this member function attempts to open conversion descriptors for a given encoding_state object. If the conversion descriptors are not valid, the conversion descriptors returned will not be valid and the resulting calls to the codecvt conversion functions will return error.

bool _M_good()

Provides a way to see if the given encoding_state object has been properly initialized. If the string literals describing the desired internal and external encoding are not valid, initialization will fail, and this will return false. If the internal and external encodings are valid, but iconv_open could not allocate conversion descriptors, this will also return false. Otherwise, the object is ready to convert and will return true.

encoding_state(const encoding_state&)

As iconv allocates memory and sets up conversion descriptors, the copy constructor can only copy the member data pertaining to the internal and external code conversions, and not the conversion descriptors themselves.

Definitions for all the required codecvt member functions are provided for this specialization, and usage of codecvt<internal character type, external character type, encoding_state> is consistent with other codecvt usage.

A conversions involving string literal.

  typedef codecvt_base::result                  result;
  typedef unsigned short                        unicode_t;
  typedef unicode_t                             int_type;
  typedef char                                  ext_type;
  typedef encoding_state                          state_type;
  typedef codecvt<int_type, ext_type, state_type> unicode_codecvt;

  const ext_type*       e_lit = "black pearl jasmine tea";
  int                   size = strlen(e_lit);
  int_type              i_lit_base[24] =
  { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
    27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
    25856, 24832, 2560
  };
  const int_type*       i_lit = i_lit_base;
  const ext_type*       efrom_next;
  const int_type*       ifrom_next;
  ext_type*             e_arr = new ext_type[size + 1];
  ext_type*             eto_next;
  int_type*             i_arr = new int_type[size + 1];
  int_type*             ito_next;

  // construct a locale object with the specialized facet.
  locale                loc(locale::classic(), new unicode_codecvt);
  // sanity check the constructed locale has the specialized facet.
  VERIFY( has_facet<unicode_codecvt>(loc) );
  const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
  // convert between const char* and unicode strings
  unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
  initialize_state(state01);
  result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
		     i_arr, i_arr + size, ito_next);
  VERIFY( r1 == codecvt_base::ok );
  VERIFY( !int_traits::compare(i_arr, i_lit, size) );
  VERIFY( efrom_next == e_lit + size );
  VERIFY( ito_next == i_arr + size );

The std::messages facet is probably the most vaguely defined facet in the standard library. It's assumed that this facility was built into the standard library in order to convert string literals from one locale to the other. For instance, converting the "C" locale's const char* c = "please" to a German-localized "bitte" during program execution.

This class has three public member functions, which directly correspond to three protected virtual member functions.

The public member functions are:

catalog open(const string&, const locale&) const

string_type get(catalog, int, int, const string_type&) const

void close(catalog) const

While the virtual functions are:

catalog do_open(const string&, const locale&) const

string_type do_get(catalog, int, int, const string_type&) const

void do_close(catalog) const

A couple of notes on the standard.

First, why is messages_base::catalog specified as a typedef to int? This makes sense for implementations that use catopen and define nl_catd as int, but not for others. Fortunately, it's not heavily used and so only a minor irritant. This has been reported as a possible defect in the standard (LWG 2028).

Second, by making the member functions const, it is impossible to save state in them. Thus, storing away information used in the 'open' member function for use in 'get' is impossible. This is unfortunate.

The 'open' member function in particular seems to be oddly designed. The signature seems quite peculiar. Why specify a const string& argument, for instance, instead of just const char*? Or, why specify a const locale& argument that is to be used in the 'get' member function? How, exactly, is this locale argument useful? What was the intent? It might make sense if a locale argument was associated with a given default message string in the 'open' member function, for instance. Quite murky and unclear, on reflection.

Lastly, it seems odd that messages, which explicitly require code conversion, don't use the codecvt facet. Because the messages facet has only one template parameter, it is assumed that ctype, and not codecvt, is to be used to convert between character sets.

It is implicitly assumed that the locale for the default message string in 'get' is in the "C" locale. Thus, all source code is assumed to be written in English, so translations are always from "en_US" to other, explicitly named locales.

A simple example using the GNU model of message conversion.

#include <iostream>
#include <locale>
using namespace std;

void test01()
{
  typedef messages<char>::catalog catalog;
  const char* dir =
  "/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale";
  const locale loc_de("de_DE");
  const messages<char>& mssg_de = use_facet<messages<char> >(loc_de);

  catalog cat_de = mssg_de.open("libstdc++", loc_de, dir);
  string s01 = mssg_de.get(cat_de, 0, 0, "please");
  string s02 = mssg_de.get(cat_de, 0, 0, "thank you");
  cout << "please in german:" << s01 << '\n';
  cout << "thank you in german:" << s02 << '\n';
  mssg_de.close(cat_de);
}