Implement C11/C++11 set of UCNs allowed in identifiers

This patch updates the set of UCNs supported in identifiers for
C11/C++11, so bringing the general state of C11 support to substantial
parity with that of C99 support (done modulo bugs, extended
identifiers corner cases and floating-point issues / optional features
- in the C11 case there are a couple more optional features without
GCC support, Annex K Bounds-checking interfaces where all that would
be needed would be appropriately conditional rsize_t additions to
stddef.h and stdint.h and Annex L Analyzability where rather more work
would be needed to produce a mode, possibly based on ubsan, where
almost all undefined behavior has bounded effects).

C11 and C++11 do at least have the same set of UCNs permitted in
identifiers, whereas C99 and C++98/C++03 differed (C++03 corrected a
typo where C++98 had 0e0d which should have been 0e8d).  The C11/C++11
set is a superset of the C99 and C++98/C++03 sets, except for the
omission of two Arabic presentation forms U+FD3E ORNATE LEFT
PARENTHESIS and U+FD3F ORNATE RIGHT PARENTHESIS that were previously
allowed by C++.

C99 disallowed UCNs for digits starting identifiers; C++98/C++03 had
no UCNs it allowed in identifiers but not at their start.  C11 and
C++11 instead disallow a specified set of UCNs for combining
characters from starting identifiers.

The C11/C++11 set of characters is generally much bigger than the set
for the previous standard, including large ranges whether or not all
code points in those ranges are currently allocated.  It includes many
more combining characters than before, with those combining characters
also including some with many more precombined NFC forms in Unicode.
The inclusion of more combining characters means that the previous
manual checks for a few context-dependent cases of whether an
identifier is in NFC (for -Wnormalized=) are no longer a feasible
approach.  Instead this patch arranges for those checks to be
generated automatically from the Unicode character data (version 6.3.0
used in this patch) (producing a 2600-line function in place of a
previous 11 lines of code) - I'd welcome comments from any Unicode
experts on whether the logic I'm using to identify non-NFC in these
cases is correct.  As the new combining characters include some that
combine with previous ASCII letters, normalization state also needs to
be updated more precisely for non-UCN parts of identifiers than it did
before.

Bootstrapped with no regressions on x86_64-unknown-linux-gnu.  Applied
to mainline.

Any comments on whether we should consider the Unicode character data
- UnicodeData.txt and DerivedNormalizationProps.txt, a total of about
2MB - as source code for the generated ucnid.h that should be checked
into the repository and included in releases, or as an external build
tool or system library that doesn't need including in the GCC source
code?

gcc/testsuite:
2013-11-15  Joseph Myers  <joseph@codesourcery.com>

	* c-c++-common/cpp/ucnid-2011-1.c: New test.

libcpp:
2013-11-15  Joseph Myers  <joseph@codesourcery.com>

	* ucnid.tab: Add C11 and C11NOSTART data.
	* makeucnid.c (digit): Rename enum value to N99.
	(C11, N11, all_languages): New enum values.
	(NUM_CODE_POINTS, MAX_CODE_POINT): New macros.
	(flags, decomp, combining_value): Use NUM_CODE_POINTS as array
	size.
	(decomp): Use unsigned int as element type.
	(all_decomp): New array.
	(read_ucnid): Handle C11 and C11NOSTART.  Use MAX_CODE_POINT.
	(read_table): Use MAX_CODE_POINT.  Store all decompositions in
	all_decomp.
	(read_derived): Use MAX_CODE_POINT.
	(write_table): Use NUM_CODE_POINTS.  Print N99, C11 and N11
	flags.  Print whole array variable declaration rather than just
	array contents.
	(char_id_valid, write_context_switch): New functions.
	(main): Call write_context_switch.
	* ucnid.h: Regenerate.
	* include/cpplib.h (struct cpp_options): Add c11_identifiers.
	* init.c (struct lang_flags): Add c11_identifiers.
	(cpp_set_lang): Set c11_identifiers option from selected language.
	* internal.h (struct normalize_state): Document "previous" as
	previous starter character.
	(NORMALIZE_STATE_UPDATE_IDNUM): Take character as argument.
	* charset.c (DIG): Rename enum value to N99.
	(C11, N11): New enum values.
	(struct ucnrange): Give name to struct.  Use short for flags and
	unsigned int for end of range.  Include ucnid.h for whole variable
	declaration.
	(ucn_valid_in_identifier): Allow for characters up to 0x10FFFF.
	Allow for C11 in determining valid characters and valid start
	characters.  Use check_nfc for non-Hangul context-dependent
	checks.  Only store starter characters in nst->previous.
	(_cpp_valid_ucn): Pass new argument to
	NORMALIZE_STATE_UPDATE_IDNUM.
	* lex.c (lex_identifier): Pass new argument to
	NORMALIZE_STATE_UPDATE_IDNUM.  Call NORMALIZE_STATE_UPDATE_IDNUM
	after initial non-UCN part of identifier.
	(lex_number): Pass new argument to NORMALIZE_STATE_UPDATE_IDNUM.

Implement C11/C++11 set of UCNs allowed in identifiers

Commit Message

Comments

Patch