[1/3] : C N2653 char8_t: Language support

This patch implements the core language and compiler dependent library 
changes proposed in WG14 N2653 [1] for C.  The changes include:
- Use of the existing -fchar8_t and -fno-char8_t options to opt-in to
   (or opt-out of) the following changes when compiling C code.
- Change of type for UTF-8 string literals from array of char to array
   of char8_t (unsigned char).
- A new atomic_char8_t typedef.
- A new ATOMIC_CHAR8_T_LOCK_FREE macro defined in terms of a new
   predefined ATOMIC_CHAR8_T_LOCK_FREE macro.

When -fchar8_t support is enabled for non-C++ modes, the _CHAR8_T_SOURCE 
macro is predefined.  This is the mechanism proposed to glibc to opt-in 
to declarations of the char8_t typedef and c8rtomb and mbrtoc8 functions 
proposed in N2653.  See [2].

Tested on Linux x86_64.

gcc/ChangeLog:

2021-05-31  Tom Honermann  <tom@honermann.net>

          * ginclude/stdatomic.h (atomic_char8_t, ATOMIC_CHAR8_T_LOCK_FREE):
            New typedef and macro.

gcc/c/ChangeLog:

2021-05-31  Tom Honermann  <tom@honermann.net>

          * c-parser.c (c_parser_string_literal): Use char8_t as the type of
            CPP_UTF8STRING when char8_t support is enabled.
          * c-typeck.c (digest_init): Handle initialization of an array
            of character type by a string literal with type array of
            unsigned char.

gcc/c-family/ChangeLog:

2021-05-31  Tom Honermann  <tom@honermann.net>

          * c-cppbuiltin.c (c_cpp_builtins): Define _CHAR8_T_SOURCE if
            char8_t support is enabled in non-C++ language modes.
          * c-lex.c (lex_string): Use char8_t as the type of
            CPP_UTF8STRING when char8_t support is enabled.
          * c-opts.c (c_common_handle_option): Inform the preprocessor if
            char8_t support is enabled.
          * c.opt (fchar8_t): Enable for C language modes.

libcpp/ChangeLog:

2021-05-31  Tom Honermann  <tom@honermann.net>

          * include/cpplib.h (cpp_options): Add char8.

Tom.

[1]: WG14 N2653
      "char8_t: A type for UTF-8 characters and strings (Revision 1)"
      http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2653.htm

[2]: C++20 P0482R6 and C2X N2653: support for char8_t, mbrtoc8(), and 
c8rtomb().
      [Patch 0]: 
https://sourceware.org/pipermail/libc-alpha/2021-June/127230.html
      [Patch 1]: 
https://sourceware.org/pipermail/libc-alpha/2021-June/127231.html
      [Patch 2]: 
https://sourceware.org/pipermail/libc-alpha/2021-June/127232.html
      [Patch 3]: 
https://sourceware.org/pipermail/libc-alpha/2021-June/127233.html

On Sun, 6 Jun 2021, Tom Honermann via Gcc-patches wrote:

> When -fchar8_t support is enabled for non-C++ modes, the _CHAR8_T_SOURCE macro
> is predefined.  This is the mechanism proposed to glibc to opt-in to
> declarations of the char8_t typedef and c8rtomb and mbrtoc8 functions proposed
> in N2653.  See [2].

I don't think glibc should have such a feature test macro, and I don't 
think GCC should define such feature test macros either - _*_SOURCE macros 
are generally for the *user* to define to decide what namespace they want 
visible, not for the compiler to define.  Without proliferating new 
language dialects, __STDC_VERSION__ ought to be sufficient to communicate 
from the compiler to the library (including to GCC's own headers such as 
stdatomic.h).

Also, it seems odd to add a new field to cpp_options without any code in 
libcpp that uses the value of that field.

On 6/7/21 5:11 PM, Joseph Myers wrote:
> On Sun, 6 Jun 2021, Tom Honermann via Gcc-patches wrote:
>
>> When -fchar8_t support is enabled for non-C++ modes, the _CHAR8_T_SOURCE macro
>> is predefined.  This is the mechanism proposed to glibc to opt-in to
>> declarations of the char8_t typedef and c8rtomb and mbrtoc8 functions proposed
>> in N2653.  See [2].
> I don't think glibc should have such a feature test macro, and I don't
> think GCC should define such feature test macros either - _*_SOURCE macros
> are generally for the *user* to define to decide what namespace they want
> visible, not for the compiler to define.  Without proliferating new
> language dialects, __STDC_VERSION__ ought to be sufficient to communicate
> from the compiler to the library (including to GCC's own headers such as
> stdatomic.h).
>
In general I agree, but I think an exception is warranted in this case 
for a few reasons:

 1. The feature includes both core language changes (the change of type
    for u8 string literals) and library changes.  The library changes
    are not actually dependent on the core language change, but they are
    intended to be used together.
 2. Existing use of the char8_t identifier can be found in existing open
    source projects and likely exists in some closed source projects as
    well.  An opt-in approach avoids conflict and the need to
    conditionalize code based on gcc version.
 3. An opt-in approach enables evaluation of the feature prior to any
    WG14 approval.

Tom.

On 6/7/21 5:12 PM, Joseph Myers wrote:
> Also, it seems odd to add a new field to cpp_options without any code in
> libcpp that uses the value of that field.
>
Ah, thank you.  That appears to be leftover code from prior 
experimentation and I failed to identify it as such when preparing the 
patch.  I'll provide a revised patch.

Tom.

On Fri, Jun 11, 2021 at 11:52:41AM -0400, Tom Honermann via Gcc-patches wrote:
> On 6/7/21 5:11 PM, Joseph Myers wrote:
> > On Sun, 6 Jun 2021, Tom Honermann via Gcc-patches wrote:
> > 
> > > When -fchar8_t support is enabled for non-C++ modes, the _CHAR8_T_SOURCE macro
> > > is predefined.  This is the mechanism proposed to glibc to opt-in to
> > > declarations of the char8_t typedef and c8rtomb and mbrtoc8 functions proposed
> > > in N2653.  See [2].
> > I don't think glibc should have such a feature test macro, and I don't
> > think GCC should define such feature test macros either - _*_SOURCE macros
> > are generally for the *user* to define to decide what namespace they want
> > visible, not for the compiler to define.  Without proliferating new
> > language dialects, __STDC_VERSION__ ought to be sufficient to communicate
> > from the compiler to the library (including to GCC's own headers such as
> > stdatomic.h).
> > 
> In general I agree, but I think an exception is warranted in this case for a
> few reasons:
> 
> 1. The feature includes both core language changes (the change of type
>    for u8 string literals) and library changes.  The library changes
>    are not actually dependent on the core language change, but they are
>    intended to be used together.
> 2. Existing use of the char8_t identifier can be found in existing open
>    source projects and likely exists in some closed source projects as
>    well.  An opt-in approach avoids conflict and the need to
>    conditionalize code based on gcc version.
> 3. An opt-in approach enables evaluation of the feature prior to any
>    WG14 approval.

But calling it _CHAR8_T_SOURCE is weird and inconsistent with everything
else.
In C++, there is __cpp_char8_t 201811L predefined macro for char8_t.
Using that in C is not right, sure.
Often we use __SIZEOF_type__ macros not just for sizeof(), but also for
presence check of the types, like
#ifdef __SIZEOF_INT128__
__int128 i;
#else
long long i;
#endif
etc., while char8_t has sizeof (char8_t) == 1, perhaps predefining
__SIZEOF_CHAR8_T__ 1
instead of _CHAR8_T_SOURCE would be better?

	Jakub

On 6/11/21 12:01 PM, Jakub Jelinek wrote:
> On Fri, Jun 11, 2021 at 11:52:41AM -0400, Tom Honermann via Gcc-patches wrote:
>> On 6/7/21 5:11 PM, Joseph Myers wrote:
>>> On Sun, 6 Jun 2021, Tom Honermann via Gcc-patches wrote:
>>>
>>>> When -fchar8_t support is enabled for non-C++ modes, the _CHAR8_T_SOURCE macro
>>>> is predefined.  This is the mechanism proposed to glibc to opt-in to
>>>> declarations of the char8_t typedef and c8rtomb and mbrtoc8 functions proposed
>>>> in N2653.  See [2].
>>> I don't think glibc should have such a feature test macro, and I don't
>>> think GCC should define such feature test macros either - _*_SOURCE macros
>>> are generally for the *user* to define to decide what namespace they want
>>> visible, not for the compiler to define.  Without proliferating new
>>> language dialects, __STDC_VERSION__ ought to be sufficient to communicate
>>> from the compiler to the library (including to GCC's own headers such as
>>> stdatomic.h).
>>>
>> In general I agree, but I think an exception is warranted in this case for a
>> few reasons:
>>
>> 1. The feature includes both core language changes (the change of type
>>     for u8 string literals) and library changes.  The library changes
>>     are not actually dependent on the core language change, but they are
>>     intended to be used together.
>> 2. Existing use of the char8_t identifier can be found in existing open
>>     source projects and likely exists in some closed source projects as
>>     well.  An opt-in approach avoids conflict and the need to
>>     conditionalize code based on gcc version.
>> 3. An opt-in approach enables evaluation of the feature prior to any
>>     WG14 approval.
> But calling it _CHAR8_T_SOURCE is weird and inconsistent with everything
> else.
> In C++, there is __cpp_char8_t 201811L predefined macro for char8_t.
> Using that in C is not right, sure.
> Often we use __SIZEOF_type__ macros not just for sizeof(), but also for
> presence check of the types, like
> #ifdef __SIZEOF_INT128__
> __int128 i;
> #else
> long long i;
> #endif
> etc., while char8_t has sizeof (char8_t) == 1, perhaps predefining
> __SIZEOF_CHAR8_T__ 1
> instead of _CHAR8_T_SOURCE would be better?

I'm open to whatever signaling mechanism would be preferred.  It took me 
a while to settle on _CHAR8_T_SOURCE as the mechanism to propose as I 
didn't find much for other precedents.

I agree that having _CHAR8_T_SOURCE be implied by the -fchar8_t option 
is unusual with respect to other feature test macros.  Is that what you 
find to be weird and inconsistent?

Predefining __SIZEOF_CHAR8_T__ would be consistent with 
__SIZEOF_WCHAR_T__, but kind of strange too since the size is always 1.

Perhaps a better approach would be to follow the __CHAR16_TYPE__ and 
__CHAR32_TYPE__ precedent and define __CHAR8_TYPE__ to unsigned char.  
That is likewise a bit strange since the type would always be unsigned 
char, but it does provide a bit more symmetry.  That could potentially 
have some use as well; for C++, it could be defined as char8_t and 
thereby reflect the difference between the two languages.  Perhaps it 
could be useful in the future as well if WG14 were to add distinct 
char8_t, char16_t, and char32_t types as C++ did (I'm not offering any 
prediction regarding the likelihood of that happening).

Tom.

>
> 	Jakub
>

On Fri, Jun 11, 2021 at 12:20:48PM -0400, Tom Honermann wrote:
> I'm open to whatever signaling mechanism would be preferred.  It took me a
> while to settle on _CHAR8_T_SOURCE as the mechanism to propose as I didn't
> find much for other precedents.
> 
> I agree that having _CHAR8_T_SOURCE be implied by the -fchar8_t option is
> unusual with respect to other feature test macros.  Is that what you find to
> be weird and inconsistent?
> 
> Predefining __SIZEOF_CHAR8_T__ would be consistent with __SIZEOF_WCHAR_T__,
> but kind of strange too since the size is always 1.
> 
> Perhaps a better approach would be to follow the __CHAR16_TYPE__ and
> __CHAR32_TYPE__ precedent and define __CHAR8_TYPE__ to unsigned char.  That
> is likewise a bit strange since the type would always be unsigned char, but
> it does provide a bit more symmetry.  That could potentially have some use
> as well; for C++, it could be defined as char8_t and thereby reflect the
> difference between the two languages.  Perhaps it could be useful in the
> future as well if WG14 were to add distinct char8_t, char16_t, and char32_t
> types as C++ did (I'm not offering any prediction regarding the likelihood
> of that happening).

C++ already predefines
#define __CHAR8_TYPE__ unsigned char
#define __CHAR16_TYPE__ short unsigned int
#define __CHAR32_TYPE__ unsigned int
for -std={c,gnu}++2{0,a,3,b} or -fchar8_t (unless -fno-char8_t), so I agree
just making sure __CHAR8_TYPE__ is defined to unsigned char even for C
is best.
And you probably don't need to do anything in the C patch for it,
void
c_stddef_cpp_builtins(void)
{
  builtin_define_with_value ("__SIZE_TYPE__", SIZE_TYPE, 0);
...
  if (flag_char8_t)
    builtin_define_with_value ("__CHAR8_TYPE__", CHAR8_TYPE, 0);
  builtin_define_with_value ("__CHAR16_TYPE__", CHAR16_TYPE, 0);
  builtin_define_with_value ("__CHAR32_TYPE__", CHAR32_TYPE, 0);
will do that.

	Jakub

On 6/11/21 12:53 PM, Jakub Jelinek wrote:
> On Fri, Jun 11, 2021 at 12:20:48PM -0400, Tom Honermann wrote:
>> I'm open to whatever signaling mechanism would be preferred.  It took me a
>> while to settle on _CHAR8_T_SOURCE as the mechanism to propose as I didn't
>> find much for other precedents.
>>
>> I agree that having _CHAR8_T_SOURCE be implied by the -fchar8_t option is
>> unusual with respect to other feature test macros.  Is that what you find to
>> be weird and inconsistent?
>>
>> Predefining __SIZEOF_CHAR8_T__ would be consistent with __SIZEOF_WCHAR_T__,
>> but kind of strange too since the size is always 1.
>>
>> Perhaps a better approach would be to follow the __CHAR16_TYPE__ and
>> __CHAR32_TYPE__ precedent and define __CHAR8_TYPE__ to unsigned char.  That
>> is likewise a bit strange since the type would always be unsigned char, but
>> it does provide a bit more symmetry.  That could potentially have some use
>> as well; for C++, it could be defined as char8_t and thereby reflect the
>> difference between the two languages.  Perhaps it could be useful in the
>> future as well if WG14 were to add distinct char8_t, char16_t, and char32_t
>> types as C++ did (I'm not offering any prediction regarding the likelihood
>> of that happening).
> C++ already predefines
> #define __CHAR8_TYPE__ unsigned char
> #define __CHAR16_TYPE__ short unsigned int
> #define __CHAR32_TYPE__ unsigned int
> for -std={c,gnu}++2{0,a,3,b} or -fchar8_t (unless -fno-char8_t), so I agree
> just making sure __CHAR8_TYPE__ is defined to unsigned char even for C
> is best.
> And you probably don't need to do anything in the C patch for it,
> void
> c_stddef_cpp_builtins(void)
> {
>    builtin_define_with_value ("__SIZE_TYPE__", SIZE_TYPE, 0);
> ...
>    if (flag_char8_t)
>      builtin_define_with_value ("__CHAR8_TYPE__", CHAR8_TYPE, 0);
>    builtin_define_with_value ("__CHAR16_TYPE__", CHAR16_TYPE, 0);
>    builtin_define_with_value ("__CHAR32_TYPE__", CHAR32_TYPE, 0);
> will do that.

Thank you; I had forgotten that I had already done that work.  I 
confirmed that the proposed changes result in __CHAR8_TYPE__ being 
defined (the tests included with the patch already enforced it).

Tom.

>
> 	Jakub
>