libcpp: Implement C++23 P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31

Hi!

The following patch implements the
P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31
paper.  We already allow UTF-8 characters in the source, so that part
is already implemented, so IMHO all we need to do is pedwarn instead of
just warn for the (default) -Wnormalize=nfc (or for -Wnormalize={id,nkfc})
if the character is not in NFC and to use the unicode XID_Start and
XID_Continue derived code properties to find out what characters are allowed
(the standard actually adds U+005F to XID_Start, but we are handling the
ASCII compatible characters differently already and they aren't allowed
in UCNs in identifiers).  Instead of hardcoding the large tables
in ucnid.tab, this patch makes makeucnid.c read them from the Unicode
tables (13.0.0 version at this point).

The only thing I'm unsure about is the non-pedantic behavior for the
identifier start characters.
The code does:
  /* When -pedantic, we require the character to have been listed by
     the standard for the current language.  Otherwise, we accept the
     union of the acceptable sets for all supported language versions.  */
  valid_flags = C99 | CXX | C11 | CXX23;
  if (CPP_PEDANTIC (pfile))
    {
      if (CPP_OPTION (pfile, cxx23_identifiers))
        valid_flags = CXX23;
      else if (CPP_OPTION (pfile, c11_identifiers))
        valid_flags = C11;
      else if (CPP_OPTION (pfile, c99))
        valid_flags = C99;
      else if (CPP_OPTION (pfile, cplusplus))
        valid_flags = CXX;
    }
  if (! (ucnranges[mn].flags & valid_flags))
      return 0;
  if (CPP_OPTION (pfile, cxx23_identifiers))
    invalid_start_flags = NXX23;
  else if (CPP_OPTION (pfile, c11_identifiers))
    invalid_start_flags = N11;
  else if (CPP_OPTION (pfile, c99))
    invalid_start_flags = N99;
  else
    invalid_start_flags = 0;
...
  /* In C99, UCN digits may not begin identifiers.  In C11 and C++11,
     UCN combining characters may not begin identifiers.  */
  if (ucnranges[mn].flags & invalid_start_flags)
    return 2;
which means when not pedantic, we allow in the middle of identifiers
a union of characters in all the supported language versions, but for
the start character we are much more strict and disallow characters
that the current standard allows for 2nd+ character of id but not for the
first one (but happily accept characters from some other standard even
if they aren't allowed at the start).
Especially with the new -std=c++23 non-pedantic behavior it, that seems quite
restrictive, as -std=c++23 behavior just removes previously accepted
characters from the sets, doesn't add any new ones (all lines with
CXX23 have also C11 sets), but there are 731 ranges with NXX23 but only
81 ranges from that have also N11 set (and 4 ranges with N11 set
but neither CXX23 nor NXX23 set).
If I look for the flags combinations that contain at least one of
N99, N11 and NXX23, one can see:
grep 'N99\|N11\|NXX23' ucnid.h | sed 's/\(|  0,\||CTX,\).*$/\1/' | sort -u
{   0|  0|  0|C11|  0|CXX23|NXX23|  0|  0|  0|  0,
{   0|  0|  0|C11|  0|CXX23|NXX23|  0|NFC|NKC|  0,
{   0|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|  0|  0,
{   0|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|NKC|  0,
{   0|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|NKC|CTX,
{   0|  0|  0|C11|N11|    0|    0|CID|NFC|NKC|  0,
{   0|  0|  0|C11|N11|CXX23|NXX23|  0|  0|  0|  0,
{   0|  0|  0|C11|N11|CXX23|NXX23|CID|NFC|NKC|  0,
{   0|  0|  0|C11|N11|CXX23|NXX23|CID|NFC|NKC|CTX,
{   0|  0|CXX|C11|  0|CXX23|NXX23|CID|NFC|  0|  0,
{ C99|  0|  0|C11|  0|CXX23|NXX23|  0|  0|  0|  0,
{ C99|  0|  0|C11|  0|CXX23|NXX23|  0|NFC|NKC|  0,
{ C99|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|  0|  0,
{ C99|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|NKC|  0,
{ C99|  0|  0|C11|  0|CXX23|NXX23|CID|NFC|NKC|CTX,
{ C99|  0|CXX|C11|  0|CXX23|NXX23|CID|NFC|  0|  0,
{ C99|  0|CXX|C11|  0|CXX23|NXX23|CID|NFC|NKC|  0,
{ C99|N99|  0|C11|  0|    0|    0|CID|NFC|NKC|  0,
{ C99|N99|  0|C11|  0|CXX23|NXX23|CID|NFC|NKC|  0,
{ C99|N99|CXX|C11|  0|CXX23|NXX23|CID|NFC|NKC|  0,
So, I wonder if a better non-pedantic behavior for the
start characters wouldn't be one where character is accepted
as start character whenever it is accepted as start character
by any of the supported standard.  So perhaps before:
  /* In C99, UCN digits may not begin identifiers.  In C11 and C++11,
     UCN combining characters may not begin identifiers.  */
  if (ucnranges[mn].flags & invalid_start_flags)
    return 2;

  return 1;
do
  if (!CPP_PEDANTIC (pfile))
    {
      /* If not -pedantic, accept as character that may
	 begin an identifier a union of characters allowed
	 at that position in each of the character sets.  */
      if ((ucnranges[mn].flags & (C99 | N99)) == C99
	  || (ucnranges[mn].flags & CXX) != 0
	  || (ucnranges[mn].flags & (C11 | N11)) == C11
	  || (ucnranges[mn].flags & (CXX23 | NXX23)) == CXX23)
	return 1;
      return 2;
    }
Or alternatively at least do that for -std=c++23, allow as start
character in that non-pedantic mode also the C11 characters not in N11.

What do you think?

Bootstrapped/regtested on x86_64-linux and i686-linux.

2021-08-06  Jakub Jelinek  <jakub@redhat.com>

	PR c++/100977
libcpp/
	* include/cpplib.h (struct cpp_options): Add cxx23_identifiers.
	* charset.c (CXX23, NXX23): New enumerators.
	(CID, NFC, NKC, CTX): Renumber.
	(ucn_valid_in_identifier): Implement P1949R7 - use CXX23 and
	NXX23 flags for cxx23_identifiers.
	* init.c (struct lang_flags): Add cxx23_identifiers.
	(lang_defaults): Add cxx23_identifiers column.
	(cpp_set_lang): Initialize CPP_OPTION (pfile, cxx23_identifiers).
	* lex.c (warn_about_normalization): If cxx23_identifiers, use
	cpp_pedwarning_with_line instead of cpp_warning_with_line for
	"is not in NFC" diagnostics.
	* makeucnid.c: Adjust usage comment.
	(CXX23, NXX23): New enumerators.
	(all_languages): Add CXX23.
	(not_NFC, not_NFKC, maybe_not_NFC): Renumber.
	(read_derivedcore): New function.
	(write_table): Print also CXX23 and NXX23 columns.
	(main): Require 5 arguments instead of 4, call read_derivedcore.
	* ucnid.h: Regenerated using Unicode 13.0.0 files.
gcc/testsuite/
	* g++.dg/cpp23/normalize1.C: New test.
	* g++.dg/cpp23/normalize2.C: New test.
	* g++.dg/cpp23/normalize3.C: New test.
	* g++.dg/cpp23/normalize4.C: New test.
	* g++.dg/cpp23/normalize5.C: New test.
	* g++.dg/cpp23/normalize6.C: New test.
	* g++.dg/cpp23/normalize7.C: New test.
	* g++.dg/cpp23/ucnid-1-utf8.C: New test.
	* g++.dg/cpp23/ucnid-2-utf8.C: New test.
	* c-c++-common/cpp/normalize-3.c: Expect errors for c++23.

	Jakub

Message ID	20210806080906.GR2380545@tucnak
State	New
Headers	show Return-Path: <gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org> DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B2F1C384C001 Date: Fri, 6 Aug 2021 10:09:06 +0200 To: Jason Merrill <jason@redhat.com>, "Joseph S. Myers" <joseph@codesourcery.com>, Marek Polacek <polacek@redhat.com> Subject: [PATCH] libcpp: Implement C++23 P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31 Message-ID: <20210806080906.GR2380545@tucnak> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Precedence: list From: Jakub Jelinek via Gcc-patches <gcc-patches@gcc.gnu.org> Reply-To: Jakub Jelinek <jakub@redhat.com> Cc: gcc-patches@gcc.gnu.org Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" <gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org>
Series	libcpp: Implement C++23 P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31 \| expand libcpp: Implement C++23 P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31

libcpp: Implement C++23 P1949R7 - C++ Identifier Syntax using Unicode Standard Annex 31

Commit Message

Patch