diff mbox

RFC: On-demand locations within string-literals

Message ID 1468014566-40305-1-git-send-email-dmalcolm@redhat.com
State New
Headers show

Commit Message

David Malcolm July 8, 2016, 9:49 p.m. UTC
This patch implements precise tracking of source locations for the
individual chars within string literals, so that we can e.g. underline
specific ranges in -Wformat diagnostics.

It should also enable fixing PR inline-asm/57950 ("wrong line numbers
in error messages for inline assembler statements").

I posted a much earlier version of this here:
  "[PATCH 17/22] libcpp: add location tracking within string literals"
    https://gcc.gnu.org/ml/gcc-patches/2015-09/msg00744.html
and:
  "[PATCH 18/22] Track locations within string literals in tree_string"
    https://gcc.gnu.org/ml/gcc-patches/2015-09/msg00743.html
In that old approach, I attempted to capture the location data during
parsing, storing it within a new cpp_string_location class, accessed
by a new TREE_STRING_LOCATION field of STRING_CST.

Doing so would add a pointer to every string literal, and mean storing the
data somewhere (unless we only store it for the "interesting" cases
in a hash somewhere).

Manu implemented an alternative "on-demand" approach in r223470:
in c-format.c which locates the relevant line in the source file and
effectively re-lexes the literal, thus avoiding having to store anything.
That implementation has a simplified lexer that doesn't support
all possible literals ("location_column_from_byte_offset" in c-format.c):

https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=d5a2ddc76a109258297ff345957c35cb50116c94#patch2

In particular, it doesn't support concatenation or macros (amongst other
things).

In the following patch, I've taken the on-demand idea, and reimplemented
it within libcpp's string literal lexer, where the generation of
source-location information is an optional extra aspect of
cpp_interpret_string.
It's disabled during the regular lexer, but it's available through an
interface in input.{c|h} which can rerun the libcpp code and capture
the per-char source_ranges for when we need to issue a diagnostic.

This has the advantage that we share code with the libcpp string
literal lexer, rather than trying to duplicate it, and thus it can handle
everything the "real" lexer can (as it *is* the real lexer).

To handle concatentation the patch adds some extra data storage:
every time a string concatenation happens in c-lex.c: it stores
the locations of the component tokens in a hash_map, keyed by
the spelling location of the start first token
(see class string_concat_db in input.h).

Hence it's only storing extra data for string concatenations,
not for simple string literals.

This approach also handles macros.

I have followup patches in-progess (to c-format.c) that make it use
the new location information to underline bad format strings, and
provide fix-its hints for the format code that should have been
used, for PR c/64955 ("RFE: have -Wformat suggest the correct format
string to use").

Unfortunately this doesn't yet work with the C++ frontend;
the EXPR_LOCATION for the ADDR_EXPR wrapping the literals is
currently UNKNOWN_LOCATION, and this also gets overwritten
by the CALL_EXPR's location due to this in gimplify.c:

2397	  /* FIXME diagnostics: This will mess up gcc.dg/Warray-bounds.c.  */
2398	  /* Make sure arguments have the same location as the function call
2399	     itself.  */
2400	  protected_set_expr_location (*arg_p, call_location);

from 489c40889c8be89bd5bed4b166974f8c1e01e4ee (aka r140917):

+2008-10-06  Aldy Hernandez  <aldyh@redhat.com>
+
+       * gimplify.c (gimplify_arg): Add location argument.  Use it.
+       (gimplify_call_expr): Pass location to gimplify_arg.
+       (gimplify_modify_expr_to_memcpy): Same.
+       (gimplify_modify_expr_to_memset): Same.

which seems to be due to debug information:
  https://gcc.gnu.org/ml/gcc-patches/2008-10/msg00191.html

So this isn't quite ready yet.

Also, this patch currently makes the assumption (in charset.c)
that there's a 1:1 correspondence between bytes in the source
character set and bytes in the execution character set.  This can
be the case if both are, say, UTF-8, but might not hold in
general.

The source char set is UTF-8 or UTF-EBCDIC, and safe-ctype.c has:

# if HOST_CHARSET == HOST_CHARSET_EBCDIC
  #error "FIXME: write tables for EBCDIC"

so presumably we don't actually have any hosts that supports EBCDIC
(do we?); as far as I can tell, we only currently support UTF-8
as the source char set.

Similarly, do we support any targets for which the execution
character set is *not* UTF-8?

Other notes:

- this patch is on top of
  "[PATCH] input.c: add lexing selftests and a test matrix for line_table states"
    https://gcc.gnu.org/ml/gcc-patches/2016-06/msg01340.html
and uses the test matrix idea there to exercise the lexing
under lots of interesting situations.

- string_concat_db has a bit more indirection that I'd like,
but this was necessary in order to get gengtype to work.

- the older approach (storing locations during initial lexing),
had a reasonably compact representation, storing runs of equal
columns-per-char, but it was bit-rotted by the range-packing
optimization of r230331.  FWIW I updated it, and there's a
working version of that idea at:
https://dmalcolm.fedorapeople.org/gcc/2016-07-01/rich-errors-gcc7-v15/0006-FIXME-Location-tracking-within-string-literals.patch

Successfully bootstrapped&regrtested on x86_64-pc-linux-gnu.

Thoughts?

gcc/c-family/ChangeLog:
	* c-common.c (g_string_concat_db): New global.
	* c-common.h (g_string_concat_db): New declaration.
	* c-lex.c (lex_string): When concatenating strings, capture the
	locations of all tokens using a new obstack, and record the
	concatenation locations within g_string_concat_db.
	* c-opts.c (c_common_init_options): Construct g_string_concat_db
	on the ggc-heap.

gcc/ChangeLog:
	* input.c (string_concat::string_concat): New constructor.
	(string_concat_db::string_concat_db): New constructor.
	(string_concat_db::record_string_concatenation): New method.
	(string_concat_db::get_string_concatenation): New method.
	(string_concat_db::get_key_loc): New method.
	(class auto_cpp_string_vec): New class.
	(get_substring_ranges_for_loc): New function.
	(get_source_range_for_substring): New function.
	(get_num_source_ranges_for_substring): New function.
	(test_builtins): Likewise.
	(struct selftest::lexer_test): New struct.
	(selftest::lexer_test::lexer_test): New constructor.
	(selftest::lexer_test::lexer_test): New constructor.
	(selftest::lexer_test::~lexer_test): New destructor.
	(selftest::lexer_test::get_token): New method.
	(selftest::assert_char_at_range): New function.
	(ASSERT_CHAR_AT_RANGE): New macro.
	(selftest::assert_num_substring_ranges): New function.
	(ASSERT_NUM_SUBSTRING_RANGES): New macro.
	(selftest::test_lexer_string_locations_simple): New function.
	(selftest::test_lexer_string_locations_hex): New function.
	(selftest::test_lexer_string_locations_oct): New function.
	(selftest::test_lexer_string_locations_ucn4): New function.
	(selftest::test_lexer_string_locations_ucn8): New function.
	(selftest::test_lexer_string_locations_u8): New function.
	(selftest::test_lexer_string_locations_utf8_source): New function.
	(selftest::test_lexer_string_locations_concatenation_1): New
	function.
	(selftest::test_lexer_string_locations_concatenation_2): New
	function.
	(selftest::test_lexer_string_locations_concatenation_3): New
	function.
	(selftest::test_lexer_string_locations_macro): New function.
	(selftest::input_c_tests): Call the new test functions once per
	case within the line_table test matrix.
	* input.h (struct string_concat): New struct.
	(struct location_hash): New struct.
	(class string_concat_db): New class.
	(get_source_range_for_substring): New prototype.
	* selftest.h (ASSERT_TRUE): Reimplement in terms of...
	(ASSERT_TRUE_AT): New macro.
	(ASSERT_FALSE): Reimplement in terms of...
	(ASSERT_FALSE_AT): New macro.
	(ASSERT_STREQ_AT): Fix typo in comment.

gcc/testsuite/ChangeLog:
	* gcc.dg/plugin/diagnostic-test-string-literals-1.c: New file.
	* gcc.dg/plugin/diagnostic_plugin_test_string_literals.c: New file.
	* gcc.dg/plugin/plugin.exp (plugin_test_list): Add
	diagnostic_plugin_test_string_literals.c and
	diagnostic-test-string-literals-1.c.

libcpp/ChangeLog:
	* charset.c (cpp_substring_ranges::cpp_substring_ranges): New
	constructor.
	(cpp_substring_ranges::~cpp_substring_ranges): New destructor.
	(cpp_substring_ranges::add_range): New method.
	(cpp_substring_ranges::add_n_ranges): New method.
	(_cpp_valid_ucn): Add "char_range" and "loc_reader" params; if
	they are non-NULL, read position information from *loc_reader
	and update char_range->m_finish accordingly.
	(convert_ucn): Add "char_range", "loc_reader", and "ranges"
	params.  If loc_reader is non-NULL, read location information from
	it, and update *ranges accordingly, using char_range.
	Conditionalize the conversion into tbuf on tbuf being non-NULL.
	(convert_hex): Likewise, conditionalizing the call to
	emit_numeric_escape on tbuf.
	(convert_oct): Likewise.
	(convert_escape): Add params "loc_reader" and "ranges".  If
	loc_reader is non-NULL, read location information from it, and
	update *ranges accordingly.  Conditionalize the conversion into
	tbuf on tbuf being non-NULL.
	(cpp_interpret_string): Rename to...
	(cpp_interpret_string_1): ...this, adding params "loc_readers" and
	"out".  Use "to" to conditionalize the initialization and usage of
	"tbuf", such as running the converter.  If "loc_readers" is
	non-NULL, use the instances within it, reading location
	information from them, and passing them to convert_escape; likewise
	write to "out" if loc_readers is non-NULL.
	(cpp_interpret_string): Reimplement in terms to
	cpp_interpret_string_1.
	(cpp_interpret_string_ranges): New function.
	(cpp_string_location_reader::cpp_string_location_reader): New
	constructor.
	(cpp_string_location_reader::get_next): New method.
	* include/cpplib.h (class cpp_string_location_reader): New class.
	(class cpp_substring_ranges): New class.
	(cpp_interpret_string_ranges): New prototype.
	* internal.h (_cpp_valid_ucn): Add params "char_range" and
	"loc_reader".
	* lex.c (forms_identifier_p): Pass NULL for new params to
	_cpp_valid_ucn.
---
 gcc/c-family/c-common.c                            |   5 +
 gcc/c-family/c-common.h                            |   2 +
 gcc/c-family/c-lex.c                               |  24 +-
 gcc/c-family/c-opts.c                              |   3 +
 gcc/input.c                                        | 977 +++++++++++++++++++++
 gcc/input.h                                        |  35 +
 gcc/selftest.h                                     |  30 +-
 .../plugin/diagnostic-test-string-literals-1.c     | 172 ++++
 .../diagnostic_plugin_test_string_literals.c       | 210 +++++
 gcc/testsuite/gcc.dg/plugin/plugin.exp             |   2 +
 libcpp/charset.c                                   | 355 ++++++--
 libcpp/include/cpplib.h                            |  50 ++
 libcpp/internal.h                                  |   4 +-
 libcpp/lex.c                                       |   2 +-
 14 files changed, 1807 insertions(+), 64 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/plugin/diagnostic-test-string-literals-1.c
 create mode 100644 gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_test_string_literals.c

Comments

David Malcolm July 20, 2016, 7:38 p.m. UTC | #1
On Fri, 2016-07-08 at 17:49 -0400, David Malcolm wrote:
[...]

> Also, this patch currently makes the assumption (in charset.c)
> that there's a 1:1 correspondence between bytes in the source
> character set and bytes in the execution character set.  This can
> be the case if both are, say, UTF-8, but might not hold in
> general.
> 
> The source char set is UTF-8 or UTF-EBCDIC, and safe-ctype.c has:
> 
> # if HOST_CHARSET == HOST_CHARSET_EBCDIC
>   #error "FIXME: write tables for EBCDIC"
> 
> so presumably we don't actually have any hosts that supports EBCDIC
> (do we?); as far as I can tell, we only currently support UTF-8
> as the source char set.
> 
> Similarly, do we support any targets for which the execution
> character set is *not* UTF-8?

I brought this up in this thread on the gcc mailing list:
"gcc/libcpp: non-UTF-8 source or execution encodings?"
  https://gcc.gnu.org/ml/gcc/2016-07/msg00091.html
and in particular:
  https://gcc.gnu.org/ml/gcc/2016-07/msg00106.html
it's possible to select the execution char set using at the command
-line for C-family frontends using:
  -fexec-charset=
  -fwide-exec-charset=
e.g. "-fexec-charset=IBM1047" will give one of the variants of EBCDIC.

Given that the internal interface already has a failure mode, I'm
thinking that a reasonable restriction is to only support locations
within string literals for the case where source character set ==
execution character set, and hence we have "convert_no_conversion" as
the converter.  Does that sound sane?  (I can write test coverage for
this).

[...]
Jeff Law July 21, 2016, 4:38 p.m. UTC | #2
On 07/20/2016 01:38 PM, David Malcolm wrote:
> On Fri, 2016-07-08 at 17:49 -0400, David Malcolm wrote:
> [...]
>
>> Also, this patch currently makes the assumption (in charset.c)
>> that there's a 1:1 correspondence between bytes in the source
>> character set and bytes in the execution character set.  This can
>> be the case if both are, say, UTF-8, but might not hold in
>> general.
>>
>> The source char set is UTF-8 or UTF-EBCDIC, and safe-ctype.c has:
>>
>> # if HOST_CHARSET == HOST_CHARSET_EBCDIC
>>   #error "FIXME: write tables for EBCDIC"
>>
>> so presumably we don't actually have any hosts that supports EBCDIC
>> (do we?); as far as I can tell, we only currently support UTF-8
>> as the source char set.
>>
>> Similarly, do we support any targets for which the execution
>> character set is *not* UTF-8?
>
> I brought this up in this thread on the gcc mailing list:
> "gcc/libcpp: non-UTF-8 source or execution encodings?"
>   https://gcc.gnu.org/ml/gcc/2016-07/msg00091.html
> and in particular:
>   https://gcc.gnu.org/ml/gcc/2016-07/msg00106.html
> it's possible to select the execution char set using at the command
> -line for C-family frontends using:
>   -fexec-charset=
>   -fwide-exec-charset=
> e.g. "-fexec-charset=IBM1047" will give one of the variants of EBCDIC.
>
> Given that the internal interface already has a failure mode, I'm
> thinking that a reasonable restriction is to only support locations
> within string literals for the case where source character set ==
> execution character set, and hence we have "convert_no_conversion" as
> the converter.  Does that sound sane?  (I can write test coverage for
> this).
I think this is sane.  We can always revisit later if we change our 
minds, particularly if folks want to do something crazy like self-host 
on an EBCDIC system.

jeff
Martin Sebor July 23, 2016, 9:35 p.m. UTC | #3
On 07/08/2016 03:49 PM, David Malcolm wrote:
> This patch implements precise tracking of source locations for the
> individual chars within string literals, so that we can e.g. underline
> specific ranges in -Wformat diagnostics.
...
> Successfully bootstrapped&regrtested on x86_64-pc-linux-gnu.
>
> Thoughts?

I applied the patch to my -Wformat-length branch and managed to
get it to work in the sprintf-length pass in the middle end, but
I'm not sure I did it right (I suspect not, or at least not the
way it should be done).  I couldn't find other places where the
bits it depends on are used (I copied bits and pieces of the code
I used from the C front end until it all fell into place ).  It
would be great if there was a tutorial on how to plug it in where
it isn't used yet (as in the middle end).  For example, how do
I determine what CLK_XXX constant I should pass to the
cpp_create_reader() function in the middle end?

I ran my tests with the patch and while it handled a good number
of test cases it eventually crashed.  A test case for the ICE is
attached though it needs my -Wformat-length patch to trigger it.
You may be able to tell what's wrong from the debugger context
(included in the test case).  If not, you may be able to
reproduce it by applying my latest patch and hacking the three
argument overload of location_from_offset() the way I did (in
the xyz.i comment).  Or I can send you my latest patch with all
this in place.

Beyond that, the range normally works fine, except when macros
are involved like they are in my tests.  You can see the effect
in the range.out file.  (This works without your patch but it
could very well be because I didn't set it up right.)

That's all I have for now.

Martin
# 1 "/src/gcc-49905/gcc/testsuite/gcc.dg/format/c99-sprintf-length-1.c"

char buffer [8];
extern char *ptr;

void f (__builtin_va_list va)
{
  __builtin_sprintf (buffer + sizeof buffer, "%c", 0);
}

/*
(gdb) r
Starting program: /build/gcc-49905/gcc/cc1 -fpreprocessed xyz.i -quiet -dumpbase xyz.i -mtune=generic -march=x86-64 -auxbase xyz -Wformat=1 -Wformat-length=1 -o xyz.s

Breakpoint 1, location_from_offset (loc=2147483652, offset=1, length=2)
    at /src/gcc-49905/gcc/gimple-ssa-sprintf.c:369
369	    = cpp_create_reader (CLK_GNUC11, ident_hash, line_table);
(gdb) l
364	static location_t
365	location_from_offset (location_t loc, int offset, int length)
366	{
367	#if 1
368	  static struct cpp_reader* parse_in
369	    = cpp_create_reader (CLK_GNUC11, ident_hash, line_table);
370	
371	  static string_concat_db *g_string_concat_db
372	    = new (ggc_alloc <string_concat_db> ()) string_concat_db ();
373	
(gdb) l
374	  source_range range;
375	  if (get_source_range_for_substring (parse_in, g_string_concat_db, loc,
376					      offset, offset + length, &range))
377	    return make_location (range.m_start, range.m_start, range.m_finish);
378	  return loc;
379	#else
(gdb) c
/src/gcc-49905/gcc/testsuite/gcc.dg/format/c99-sprintf-length-1.c: In function ’:
/src/gcc-49905/gcc/testsuite/gcc.dg/format/c99-sprintf-length-1.c:5:6: internal compiler error: in get_substring_ranges_for_loc, at input.c:1310
    and avoid exercising any of the others.  The buffer and objsize macros
      ^
0x1921136 get_substring_ranges_for_loc
	/src/gcc-49905/gcc/input.c:1310
0x19212cc get_source_range_for_substring(cpp_reader*, string_concat_db*, unsigned int, int, int, source_range*)
	/src/gcc-49905/gcc/input.c:1354
0x17eb1ae location_from_offset
	/src/gcc-49905/gcc/gimple-ssa-sprintf.c:375
0x17ed220 compute_format_length
	/src/gcc-49905/gcc/gimple-ssa-sprintf.c:1343
0x17edf4c pass_sprintf_length::compute_format_length(pass_sprintf_length::call_info const&, format_result*)
	/src/gcc-49905/gcc/gimple-ssa-sprintf.c:1880
0x17ee780 pass_sprintf_length::handle_gimple_call(gimple_stmt_iterator)
	/src/gcc-49905/gcc/gimple-ssa-sprintf.c:2122
0x17ee8cf pass_sprintf_length::execute(function*)
	/src/gcc-49905/gcc/gimple-ssa-sprintf.c:2153
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
[Inferior 1 (process 23446) exited with code 04]


 */
$ cat xyz.c && /build/gcc-49905/gcc/xgcc -B/build/gcc-49905/gcc -Wformat -Wformat-length=1 -S xyz.c
char d[2];

#define P(n, f) __builtin_vsprintf (d + sizeof d - n, f, va)

void f (__builtin_va_list va)
{
   __builtin_vsprintf (d + sizeof d - 2, "%#3x", va);
  P (2, "%#3x");
}
xyz.c: In function ‘f’:
xyz.c:7:42: warning: ‘%#3x’ directive writing between 3 and 10 bytes into a region of size 2 [-Wformat-length=]
    __builtin_vsprintf (d + sizeof d - 2, "%#3x", va);
                                          ^~~~~~
xyz.c:7:42: note: using the range [‘1u’, ‘2147483648u’] for directive argument
xyz.c:7:4: note: format output between 4 and 11 bytes into a destination of size 2
    __builtin_vsprintf (d + sizeof d - 2, "%#3x", va);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
xyz.c:8:9: warning: ‘%#3x’ directive writing between 3 and 10 bytes into a region of size 2 [-Wformat-length=]
   P (2, "%#3x");
         ^
xyz.c:3:55: note: in definition of macro ‘P’
 #define P(n, f) __builtin_vsprintf (d + sizeof d - n, f, va)
                                                       ^
xyz.c:8:9: note: using the range [‘1u’, ‘2147483648u’] for directive argument
   P (2, "%#3x");
         ^
xyz.c:3:55: note: in definition of macro ‘P’
 #define P(n, f) __builtin_vsprintf (d + sizeof d - n, f, va)
                                                       ^
xyz.c:3:17: note: format output between 4 and 11 bytes into a destination of size 2
 #define P(n, f) __builtin_vsprintf (d + sizeof d - n, f, va)
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
xyz.c:8:3: note: in expansion of macro ‘P’
   P (2, "%#3x");
   ^
David Malcolm July 24, 2016, 12:37 a.m. UTC | #4
On Sat, 2016-07-23 at 15:35 -0600, Martin Sebor wrote:
> On 07/08/2016 03:49 PM, David Malcolm wrote:
> > This patch implements precise tracking of source locations for the
> > individual chars within string literals, so that we can e.g.
> > underline
> > specific ranges in -Wformat diagnostics.
> ...
> > Successfully bootstrapped&regrtested on x86_64-pc-linux-gnu.
> > 
> > Thoughts?
> 
> I applied the patch to my -Wformat-length branch and managed to
> get it to work in the sprintf-length pass in the middle end, but
> I'm not sure I did it right (I suspect not, or at least not the
> way it should be done).  I couldn't find other places where the
> bits it depends on are used (I copied bits and pieces of the code
> I used from the C front end until it all fell into place ).  It
> would be great if there was a tutorial on how to plug it in where
> it isn't used yet (as in the middle end).  For example, how do
> I determine what CLK_XXX constant I should pass to the
> cpp_create_reader() function in the middle end?

Thanks for trying it out.  I've been reworking the patch, and I have a
much more robust version that I think is close to being ready, with an
easier interface, some bug fixes, and, ahem, EBCDIC support...  and
more usefully to the average user, gracefully handling of other
encodings.

I'd hoped to post the new version on Monday (along with code that wires
it up into c-common/c-format.c).

Though reading your post, I realize now that my interface is in c
-common.h, and that's not going to be usable for you from the middle
end.  Presumably I'm going to need to rework things to be usable from a
gimple-*.c file.  Is it acceptable to somehow wire this up to a
langhook, so that locations for sprintf-style strings are only
available for c-family, with a graceful fallback on other frontends?

Alternatively, I can post the c-common.h interface for review, and we
can get that good enough for trunk, and then we can work on
generalizing it so it's usable from the middle-end for your new pass.

> I ran my tests with the patch and while it handled a good number
> of test cases it eventually crashed.  A test case for the ICE is
> attached though it needs my -Wformat-length patch to trigger it.
> You may be able to tell what's wrong from the debugger context
> (included in the test case).  If not, you may be able to
> reproduce it by applying my latest patch and hacking the three
> argument overload of location_from_offset() the way I did (in
> the xyz.i comment).  Or I can send you my latest patch with all
> this in place.

My patch has changed a lot, so I don't know how useful debugging is
going to be.  I'd be interested in seeing your latest patch.

> Beyond that, the range normally works fine, except when macros
> are involved like they are in my tests.  You can see the effect
> in the range.out file.  (This works without your patch but it
> could very well be because I didn't set it up right.)

Sadly I can't figure out what's going wrong - but the code's changed a
lot at my end since then.  Sorry.

> That's all I have for now.

Martin
> 

Thanks for posting this, it's very helpful.
Dave

>
Martin Sebor Aug. 23, 2016, 3:25 a.m. UTC | #5
>> Beyond that, the range normally works fine, except when macros
>> are involved like they are in my tests.  You can see the effect
>> in the range.out file.  (This works without your patch but it
>> could very well be because I didn't set it up right.)
>
> Sadly I can't figure out what's going wrong - but the code's changed a
> lot at my end since then.  Sorry.

I have integrated the latest (already committed) version of your
patch into my -Wformat-length patch.  Everything (well almost)
works and I get nice ranges for the format string and for (some)
arguments.

I was surprised at how long it took me to switch from the previous
implementation (also copied from c-format.c) to this new API.  As
before, I had to copy bits and pieces of code from other parts of
the compiler to get it to work.  I was also surprised at how complex
making use of it is.  It added 130 lines of code to the pass, which
is 40 more than what I had before.  It seems that the
format_warning_at_substring function from c-format.c (perhaps
generalized and with some reasonable defaults hardcoded) should
be defined where other parts of GCC (including the middle end)
can reuse it.

I ran into a few minor glitches while testing it and I raised
the following bugs for two of them:

   77328 - incorrect caret location in -Wformat calling printf
           via a macro (this was pre-existing)
   77331 - incorrect range location in -Wformat with a concatenated
           format literal (this is new)

The third issue seems like a limitation that I should be able to
overcome but I couldn't figure out how using the new API.  The
problem is that there doesn't seem to be a way to point the caret
at the closing quote of a string, like for example in the following
test.  Even though by default the whole string is underlined (and
the caret points to the opening quote), there doesn't seem to be
a way to specify a range where the caret points to the other quote.
It's no big deal and I only noticed it because one of my tests
started failing, but it seems like it should be possible.

$ cat t.c && /build/gcc-49905/gcc/xgcc -B /build/gcc-49905/gcc -S 
-Wformat t.c
char d [2];

void f (void)
{
   __builtin_sprintf (d, "%sX", "1");
}
t.c: In function ‘f’:
t.c:5:25: warning: writing a terminating nul past the end of the 
destination [-Wformat-length=]
    __builtin_sprintf (d, "%sX", "1");
                          ^~~~~

What I would like to see is similar to what I get when one of
the format string characters is written past the end:

t.c:5:30: warning: writing format character ‘Z’ at offset 4 past the end 
of the destination [-Wformat-length=]
    __builtin_sprintf (d, "%sXYZ", "");
                               ^

Martin
David Malcolm Aug. 23, 2016, 1:59 p.m. UTC | #6
On Mon, 2016-08-22 at 21:25 -0600, Martin Sebor wrote:
> > > Beyond that, the range normally works fine, except when macros
> > > are involved like they are in my tests.  You can see the effect
> > > in the range.out file.  (This works without your patch but it
> > > could very well be because I didn't set it up right.)
> > 
> > Sadly I can't figure out what's going wrong - but the code's
> > changed a
> > lot at my end since then.  Sorry.
> 
> I have integrated the latest (already committed) version of your
> patch into my -Wformat-length patch.  Everything (well almost)
> works and I get nice ranges for the format string and for (some)
> arguments.
> 
> I was surprised at how long it took me to switch from the previous
> implementation (also copied from c-format.c) to this new API.  As
> before, I had to copy bits and pieces of code from other parts of
> the compiler to get it to work.  I was also surprised at how complex
> making use of it is.  It added 130 lines of code to the pass, which
> is 40 more than what I had before.  It seems that the
> format_warning_at_substring function from c-format.c (perhaps
> generalized and with some reasonable defaults hardcoded) should
> be defined where other parts of GCC (including the middle end)
> can reuse it.

I'm guessing that it was difficult because the most useful parts are
currently in c-format.c, whereas your code is in the middle-end.

Is the latest version of your patch posted somewhere where I can see
it?

The substring_loc class should probably be moved from c-family to gcc
also.   We might need a langhook to support that though (not sure yet).

I'd be up for doing these moves (maybe moving
 format_warning_at_substring to diagnostic.h/c), but I'd prefer to see
your patch first.


> I ran into a few minor glitches while testing it and I raised
> the following bugs for two of them:
> 
>    77328 - incorrect caret location in -Wformat calling printf
>            via a macro (this was pre-existing)
>    77331 - incorrect range location in -Wformat with a concatenated
>            format literal (this is new)
> 
> The third issue seems like a limitation that I should be able to
> overcome but I couldn't figure out how using the new API.  The
> problem is that there doesn't seem to be a way to point the caret
> at the closing quote of a string, like for example in the following
> test.  Even though by default the whole string is underlined (and
> the caret points to the opening quote), there doesn't seem to be
> a way to specify a range where the caret points to the other quote.
> It's no big deal and I only noticed it because one of my tests
> started failing, but it seems like it should be possible.
> 
> $ cat t.c && /build/gcc-49905/gcc/xgcc -B /build/gcc-49905/gcc -S 
> -Wformat t.c
> char d [2];
> 
> void f (void)
> {
>    __builtin_sprintf (d, "%sX", "1");
> }
> t.c: In function ‘f’:
> t.c:5:25: warning: writing a terminating nul past the end of the 
> destination [-Wformat-length=]
>     __builtin_sprintf (d, "%sX", "1");
>                           ^~~~~


> What I would like to see is similar to what I get when one of
> the format string characters is written past the end:
> 
> t.c:5:30: warning: writing format character ‘Z’ at offset 4 past the
> end 
> of the destination [-Wformat-length=]
>     __builtin_sprintf (d, "%sXYZ", "");
>                                ^
> 

So would you like the output to look like this:


Option (a): underline whole string, with caret at close-quote

t.c: In function ‘f’:
t.c:5:25: warning: writing a terminating nul past the end of the
destination [-Wformat-length=]
     __builtin_sprintf (d, "%sX", "1");
                           ~~~~^

or like this:

Option (b): just the close-quote

t.c: In function ‘f’:
t.c:5:25: warning: writing a terminating nul past
the end of the
destination [-Wformat-length=]
     __builtin_sprintf (d,
"%sX", "1");
                               ^
?

(do you also emit a note/inform showing the size of d?)

What API are you using to emit the warning?  Given the location of the
string as a whole expressed as a location_t, you can probably do
something like this:

location_t
option_a (location_t string_as_a_whole_loc)
{
  source_range src_range
    = get_range_from_loc (line_table, string_as_a_whole_loc);
  
  return make_location (src_range.m_finish, /* caret */
                        src_range.m_start, src_range.m_finish);
}

location_t
option_b (location_t string_as_a_whole_loc)
{
  source_range
src_range
    = get_range_from_loc (line_table, string_as_a_whole_loc);
 
  return src_range.m_finish;
}

(these could be added to libcpp)

but I get the impression you want something like this integrated into
the format_warning or substring_loc APIs (and it's hard to tell without
seeing your patch).

Hope this is helpful
Dave
Martin Sebor Aug. 23, 2016, 3:17 p.m. UTC | #7
On 08/23/2016 07:59 AM, David Malcolm wrote:
> On Mon, 2016-08-22 at 21:25 -0600, Martin Sebor wrote:
>>>> Beyond that, the range normally works fine, except when macros
>>>> are involved like they are in my tests.  You can see the effect
>>>> in the range.out file.  (This works without your patch but it
>>>> could very well be because I didn't set it up right.)
>>>
>>> Sadly I can't figure out what's going wrong - but the code's
>>> changed a
>>> lot at my end since then.  Sorry.
>>
>> I have integrated the latest (already committed) version of your
>> patch into my -Wformat-length patch.  Everything (well almost)
>> works and I get nice ranges for the format string and for (some)
>> arguments.
>>
>> I was surprised at how long it took me to switch from the previous
>> implementation (also copied from c-format.c) to this new API.  As
>> before, I had to copy bits and pieces of code from other parts of
>> the compiler to get it to work.  I was also surprised at how complex
>> making use of it is.  It added 130 lines of code to the pass, which
>> is 40 more than what I had before.  It seems that the
>> format_warning_at_substring function from c-format.c (perhaps
>> generalized and with some reasonable defaults hardcoded) should
>> be defined where other parts of GCC (including the middle end)
>> can reuse it.
>
> I'm guessing that it was difficult because the most useful parts are
> currently in c-format.c, whereas your code is in the middle-end.
>
> Is the latest version of your patch posted somewhere where I can see
> it?

I'm planning/hoping to post it this week.  It was only difficult
in that I had to gather bits from different parts of the compiler
and figure out how to make them work together.  I.e., it wasn't
a simple matter of replacing one function call with another.  In
the end, it boiled down to replacing the get_location function
with this:

const char *
substring_loc::get_location (location_t *out_loc) const
{
   gcc_assert (out_loc);

   if (!g_string_concat_db)
     g_string_concat_db
       = new (ggc_alloc <string_concat_db> ()) string_concat_db ();

   static struct cpp_reader* parse_in;
   if (!parse_in)
     {
       /* Create and initialize a preprocessing reader.  */
       parse_in = cpp_create_reader (CLK_GNUC99, ident_hash, line_table);
       cpp_init_iconv (parse_in);
     }

   return get_source_location_for_substring (parse_in, g_string_concat_db,
					    m_fmt_string_loc, CPP_STRING,
					    m_caret_idx, m_start_idx, m_end_idx,
					    out_loc);
}

>
> The substring_loc class should probably be moved from c-family to gcc
> also.   We might need a langhook to support that though (not sure yet).
>
> I'd be up for doing these moves (maybe moving
>   format_warning_at_substring to diagnostic.h/c), but I'd prefer to see
> your patch first.
>

Sure.

> So would you like the output to look like this:
>
>
> Option (a): underline whole string, with caret at close-quote
>
> t.c: In function ‘f’:
> t.c:5:25: warning: writing a terminating nul past the end of the
> destination [-Wformat-length=]
>       __builtin_sprintf (d, "%sX", "1");
>                             ~~~~^
>
> or like this:
>
> Option (b): just the close-quote
>
> t.c: In function ‘f’:
> t.c:5:25: warning: writing a terminating nul past
> the end of the
> destination [-Wformat-length=]
>       __builtin_sprintf (d,
> "%sX", "1");
>                                 ^
> ?
>

Either one would work for me.  If this were to become a general
purpose interface then I think it would be nice to let the caller
decide which end/quote (if any) to point the caret at.

> (do you also emit a note/inform showing the size of d?)

Yes.  The full diagnostic looks like this (with the care at
the wrong end as we're discussing):

t.c:5:25: warning: writing a terminating nul past the end of the 
destination [-Wformat-length=]
    __builtin_sprintf (d, "%sX", "A");
                          ^~~~~
t.c:5:3: note: format output 3 bytes into a destination of size 2
    __builtin_sprintf (d, "%sX", "A");
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

>
> What API are you using to emit the warning?

I copied the format_warning_va and format_warning_at subscript
functions from c-format.c and I'm calling the latter.  I'm not
using the hint for anything yet (not sure if there's
an opportunity to make use of it).

> Given the location of the
> string as a whole expressed as a location_t, you can probably do
> something like this:
>
> location_t
> option_a (location_t string_as_a_whole_loc)
> {
>    source_range src_range
>      = get_range_from_loc (line_table, string_as_a_whole_loc);
>
>    return make_location (src_range.m_finish, /* caret */
>                          src_range.m_start, src_range.m_finish);
> }
>
> location_t
> option_b (location_t string_as_a_whole_loc)
> {
>    source_range
> src_range
>      = get_range_from_loc (line_table, string_as_a_whole_loc);
>
>    return src_range.m_finish;
> }
>
> (these could be added to libcpp)
>
> but I get the impression you want something like this integrated into
> the format_warning or substring_loc APIs (and it's hard to tell without
> seeing your patch).

I guess I was hoping for a simple high level interface to warning_at
where I could control the offset of the caret and the underlining
within some bounds.  Completely off the cuff, say if warning_at were
overloaded to take another argument with the offsets, then something
like this (with offsets in characters):

   void my_warning (location_t loc)
   {
     int caret = /* offset of caret from the beginning of loc */;
     int begin = /* optional offset of the start of underlining */;
     int end = /* optional offset of the end of underlining */;

     warning_at (range (loc, caret, begin, end), ...);
   }

Maybe I should prototype it to understand if it can be done and what
the tradeoffs might be.

The most recently posted patch uses the location_from_offset function
that c-format.c used before your changes:

   https://gcc.gnu.org/ml/gcc-patches/2016-08/msg00986.html

In my latest patch I replaced the function and calls to warning_at
with its result with format_warning_at_substring.  I mainly did it
because I thought that was going to be new/recommended API for this
sort of thing.  I wasn't having any problems with previous approach
or looking for enhancements (though the split location for the
format directive and its argument is nice).  Did I misunderstand
what the intent of your changes was?  (I.e., did you not expect
me to make that switch?)

>
> Hope this is helpful
> Dave

Yes, thanks.

Martin
diff mbox

Patch

diff --git a/gcc/c-family/c-common.c b/gcc/c-family/c-common.c
index 936ddfb..90fdc79 100644
--- a/gcc/c-family/c-common.c
+++ b/gcc/c-family/c-common.c
@@ -12901,4 +12901,9 @@  diagnose_mismatched_attributes (tree olddecl, tree newdecl)
   return warned;
 }
 
+/* The global record of string concatentations, for use in
+   extracting locations within string literals.  */
+
+GTY(()) string_concat_db *g_string_concat_db;
+
 #include "gt-c-family-c-common.h"
diff --git a/gcc/c-family/c-common.h b/gcc/c-family/c-common.h
index 3ad5400..7e2e432 100644
--- a/gcc/c-family/c-common.h
+++ b/gcc/c-family/c-common.h
@@ -1116,6 +1116,8 @@  extern tree c_build_bind_expr (location_t, tree, tree);
 extern enum cpp_ttype
 conflict_marker_get_final_tok_kind (enum cpp_ttype tok1_kind);
 
+extern GTY(()) string_concat_db *g_string_concat_db;
+
 /* In c-pch.c  */
 extern void pch_init (void);
 extern void pch_cpp_save_state (void);
diff --git a/gcc/c-family/c-lex.c b/gcc/c-family/c-lex.c
index 8f33d86..4c7e385 100644
--- a/gcc/c-family/c-lex.c
+++ b/gcc/c-family/c-lex.c
@@ -1097,13 +1097,16 @@  lex_string (const cpp_token *tok, tree *valp, bool objc_string, bool translate)
   tree value;
   size_t concats = 0;
   struct obstack str_ob;
+  struct obstack loc_ob;
   cpp_string istr;
   enum cpp_ttype type = tok->type;
 
   /* Try to avoid the overhead of creating and destroying an obstack
      for the common case of just one string.  */
   cpp_string str = tok->val.str;
+  location_t init_loc = tok->src_loc;
   cpp_string *strs = &str;
+  location_t *locs = NULL;
 
   /* objc_at_sign_was_seen is only used when doing Objective-C string
      concatenation.  It is 'true' if we have seen an '@' before the
@@ -1142,16 +1145,21 @@  lex_string (const cpp_token *tok, tree *valp, bool objc_string, bool translate)
 	  else
 	    error ("unsupported non-standard concatenation of string literals");
 	}
+      /* FALLTHROUGH */
 
     case CPP_STRING:
       if (!concats)
 	{
 	  gcc_obstack_init (&str_ob);
+	  gcc_obstack_init (&loc_ob);
 	  obstack_grow (&str_ob, &str, sizeof (cpp_string));
+	  obstack_grow (&loc_ob, &init_loc, sizeof (location_t));
 	}
 
       concats++;
       obstack_grow (&str_ob, &tok->val.str, sizeof (cpp_string));
+      obstack_grow (&loc_ob, &tok->src_loc, sizeof (location_t));
+
       if (objc_string)
 	objc_at_sign_was_seen = false;
       goto retry;
@@ -1164,7 +1172,10 @@  lex_string (const cpp_token *tok, tree *valp, bool objc_string, bool translate)
   /* We have read one more token than we want.  */
   _cpp_backup_tokens (parse_in, 1);
   if (concats)
-    strs = XOBFINISH (&str_ob, cpp_string *);
+    {
+      strs = XOBFINISH (&str_ob, cpp_string *);
+      locs = XOBFINISH (&loc_ob, location_t *);
+    }
 
   if (concats && !objc_string && !in_system_header_at (input_location))
     warning (OPT_Wtraditional,
@@ -1176,6 +1187,12 @@  lex_string (const cpp_token *tok, tree *valp, bool objc_string, bool translate)
     {
       value = build_string (istr.len, (const char *) istr.text);
       free (CONST_CAST (unsigned char *, istr.text));
+      if (concats)
+	{
+	  gcc_assert (locs);
+	  gcc_assert (g_string_concat_db);
+	  g_string_concat_db->record_string_concatenation (concats + 1, locs);
+	}
     }
   else
     {
@@ -1227,7 +1244,10 @@  lex_string (const cpp_token *tok, tree *valp, bool objc_string, bool translate)
   *valp = fix_string_type (value);
 
   if (concats)
-    obstack_free (&str_ob, 0);
+    {
+      obstack_free (&str_ob, 0);
+      obstack_free (&loc_ob, 0);
+    }
 
   return objc_string ? CPP_OBJC_STRING : type;
 }
diff --git a/gcc/c-family/c-opts.c b/gcc/c-family/c-opts.c
index ff6339c..16f525b 100644
--- a/gcc/c-family/c-opts.c
+++ b/gcc/c-family/c-opts.c
@@ -216,6 +216,9 @@  c_common_init_options (unsigned int decoded_options_count,
   unsigned int i;
   struct cpp_callbacks *cb;
 
+  g_string_concat_db
+    = new (ggc_alloc <string_concat_db> ()) string_concat_db ();
+
   parse_in = cpp_create_reader (c_dialect_cxx () ? CLK_GNUCXX: CLK_GNUC89,
 				ident_hash, line_table);
   cb = cpp_get_callbacks (parse_in);
diff --git a/gcc/input.c b/gcc/input.c
index a916597..316b1b5 100644
--- a/gcc/input.c
+++ b/gcc/input.c
@@ -1139,6 +1139,249 @@  dump_location_info (FILE *stream)
 				MAX_SOURCE_LOCATION + 1, UINT_MAX);
 }
 
+/* string_concat's constructor.  */
+
+string_concat::string_concat (int num, location_t *locs)
+  : m_num (num)
+{
+  m_locs = ggc_vec_alloc <location_t> (num);
+  for (int i = 0; i < num; i++)
+    m_locs[i] = locs[i];
+}
+
+/* string_concat_db's constructor.  */
+
+string_concat_db::string_concat_db ()
+{
+  m_table = hash_map <location_hash, string_concat *>::create_ggc (64);
+}
+
+/* Record that a string concatenation occurred, covering NUM
+   string literal tokens.  LOCS is an array of size NUM, containing the
+   locations of the tokens.  A copy of LOCS is taken.  */
+
+void
+string_concat_db::record_string_concatenation (int num, location_t *locs)
+{
+  gcc_assert (num > 1);
+  gcc_assert (locs);
+
+  location_t key_loc = get_key_loc (locs[0]);
+
+  string_concat *concat
+    = new (ggc_alloc <string_concat> ()) string_concat (num, locs);
+  m_table->put (key_loc, concat);
+}
+
+/* Determine if LOC was the location of the the initial token of a
+   concatenation of string literal tokens.
+   If so, *OUT_NUM is written to with the number of tokens, and
+   *OUT_LOCS with the location of an array of locations of the
+   tokens, and return true.  *OUT_LOCS is a borrowed pointer to
+   storage owned by the string_concat_db.
+   Otherwise, return false.  */
+
+bool
+string_concat_db::get_string_concatenation (location_t loc,
+					    int *out_num,
+					    location_t **out_locs)
+{
+  gcc_assert (out_num);
+  gcc_assert (out_locs);
+
+  location_t key_loc = get_key_loc (loc);
+
+  string_concat **concat = m_table->get (key_loc);
+  if (!concat)
+    return false;
+
+  *out_num = (*concat)->m_num;
+  *out_locs =(*concat)->m_locs;
+  return true;
+}
+
+/* Internal function.  Canonicalize LOC into a form suitable for
+   use as a key within the database, stripping away macro expansion,
+   ad-hoc information, and range information, using the location of
+   the start of LOC within an ordinary linemap.  */
+
+location_t
+string_concat_db::get_key_loc (location_t loc)
+{
+  loc = linemap_resolve_location (line_table, loc, LRK_SPELLING_LOCATION,
+				  NULL);
+
+  loc = get_range_from_loc (line_table, loc).m_start;
+
+  return loc;
+}
+
+/* Helper class for use within get_substring_ranges_for_loc.
+   An vec of cpp_string with responsibility for releasing all of the
+   str->text for each str in the vector.  */
+
+class auto_cpp_string_vec :  public auto_vec <cpp_string>
+{
+ public:
+  auto_cpp_string_vec (int alloc)
+    : auto_vec <cpp_string> (alloc) {}
+
+  ~auto_cpp_string_vec ()
+  {
+    /* Clean up the copies within this vec.  */
+    int i;
+    cpp_string *str;
+    FOR_EACH_VEC_ELT (*this, i, str)
+      free (const_cast <unsigned char *> (str->text));
+  }
+};
+
+/* Attempt to populate RANGES with source location information on the
+   individual characters within the string literal found at STRLOC.
+   If CONCATS is non-NULL, then any string literals that the token at
+   STRLOC  was concatenated with are also added to RANGES.
+
+   Return true if successful, or false if any errors occurred (in
+   which case RANGES may be only partially populated and should not
+   be used).
+
+   This is implemented by re-parsing the relevant source line(s).  */
+
+static bool
+get_substring_ranges_for_loc (cpp_reader *pfile,
+			      string_concat_db *concats,
+			      location_t strloc,
+			      cpp_substring_ranges &ranges)
+{
+  gcc_assert (pfile);
+
+  if (strloc == UNKNOWN_LOCATION)
+    return false;
+
+  /* If string concatenation has occurred at STRLOC, get the locations
+     of all of the literal tokens making up the compound string.
+     Otherwise, just use STRLOC.  */
+  int num_locs = 1;
+  location_t *strlocs = &strloc;
+  if (concats)
+    concats->get_string_concatenation (strloc, &num_locs, &strlocs);
+
+  auto_cpp_string_vec strs (num_locs);
+  auto_vec <cpp_string_location_reader> loc_readers (num_locs);
+  for (int i = 0; i < num_locs; i++)
+    {
+      /* Get range of strloc.  We will use it to locate the start and finish
+	 of the literal token within the line.  */
+      source_range src_range = get_range_from_loc (line_table, strlocs[i]);
+
+      if (src_range.m_start >= LINE_MAP_MAX_LOCATION_WITH_COLS)
+	/* If so, we can't reliably determine where the token started within
+	   its line.  */
+	return false;
+
+      if (src_range.m_finish >= LINE_MAP_MAX_LOCATION_WITH_COLS)
+	/* If so, we can't reliably determine where the token finished within
+	   its line.  */
+	return false;
+
+      expanded_location start
+	= expand_location_to_spelling_point (src_range.m_start);
+      expanded_location finish
+	= expand_location_to_spelling_point (src_range.m_finish);
+      if (start.file != finish.file)
+	return false;
+      if (start.line != finish.line)
+	return false;
+      if (start.column > finish.column)
+	return false;
+
+      int line_width;
+      const char *line = location_get_source_line (start.file, start.line,
+						   &line_width);
+      if (line == NULL)
+	return false;
+
+      /* Determine the location of the literal (including quotes
+	 and leading prefix chars, such as the 'u' in a u""
+	 token).  */
+      const char *literal = line + start.column - 1;
+      int literal_length = finish.column - start.column + 1;
+
+      gcc_assert (line_width >= (start.column - 1 + literal_length));
+      cpp_string from;
+      from.len = literal_length;
+      /* Make a copy of the literal, to avoid having to rely on
+	 the lifetime of the copy of the line within the cache.
+	 This will be released by the auto_cpp_string_vec dtor.  */
+      from.text = XDUPVEC (unsigned char, literal, literal_length);
+      strs.safe_push (from);
+      cpp_string_location_reader loc_reader (strlocs[i], line_table);
+      loc_readers.safe_push (loc_reader);
+    }
+
+  /* Rerun cpp_interpret_string, or rather, a modified version of it.  */
+  if (!cpp_interpret_string_ranges (pfile, strs.address (),
+				    loc_readers.address (),
+				    num_locs, &ranges))
+    return false;
+
+  /* Success: "ranges" should now contain information on the string.  */
+  return true;
+}
+
+/* Attempt to populate *OUT with source location information on the
+   range of given characters within the string literal found at STRLOC.
+   START_IDX and END_IDX refer to offsets within the execution character
+   set.
+   If CONCATS is non-NULL, then any string literals that the token at
+   STRLOC was concatenated with are also considered.
+
+   Return true if successful, or false if any errors occurred.
+
+   This is implemented by re-parsing the relevant source line(s).  */
+
+bool
+get_source_range_for_substring (cpp_reader *pfile,
+				string_concat_db *concats,
+				location_t strloc,
+				int start_idx, int end_idx, source_range *out)
+{
+  gcc_checking_assert (start_idx >= 0);
+  gcc_checking_assert (end_idx >= 0);
+  gcc_assert (out);
+
+  cpp_substring_ranges ranges;
+  if (!get_substring_ranges_for_loc (pfile, concats, strloc, ranges))
+    return false;
+
+  if (start_idx >= ranges.get_num_ranges ()
+      || end_idx >= ranges.get_num_ranges ())
+      return false;
+
+  out->m_start = ranges.get_range (start_idx).m_start;
+  out->m_finish = ranges.get_range (end_idx).m_finish;
+  return true;
+}
+
+/* As get_source_range_for_substring, but write to *OUT the number
+   of ranges that are available.  */
+
+bool
+get_num_source_ranges_for_substring (cpp_reader *pfile,
+				     string_concat_db *concats,
+				     location_t strloc,
+				     int *out)
+{
+  gcc_assert (out);
+
+  cpp_substring_ranges ranges;
+  if (!get_substring_ranges_for_loc (pfile, concats, strloc, ranges))
+    return false;
+
+  *out = ranges.get_num_ranges ();
+  return true;
+}
+
 #if CHECKING_P
 
 namespace selftest {
@@ -1481,6 +1724,729 @@  test_lexer (const line_table_case &case_)
   cpp_destroy (parser);
 }
 
+/* A struct for writing lexer tests.  */
+
+struct lexer_test
+{
+  lexer_test (const line_table_case &case_, const char *content);
+  ~lexer_test ();
+
+  const cpp_token *get_token ();
+
+  temp_source_file m_tempfile;
+  temp_line_table m_tmp_lt;
+  cpp_reader *m_parser;
+  string_concat_db m_concats;
+};
+
+/* Constructor.  Override line_table with a new instance based on CASE_,
+   and write CONTENT to a tempfile.  Create a cpp_reader, and use it to
+   start parsing the tempfile.  */
+
+lexer_test::lexer_test (const line_table_case &case_, const char *content) :
+  /* Create a tempfile and write the text to it.  */
+  m_tempfile (SELFTEST_LOCATION, ".c", content),
+  m_tmp_lt (case_),
+  m_parser (cpp_create_reader (CLK_GNUC99, NULL, line_table)),
+  m_concats ()
+{
+  cpp_init_iconv (m_parser);
+
+  /* Parse the file.  */
+  const char *fname = cpp_read_main_file (m_parser,
+					  m_tempfile.get_filename ());
+  ASSERT_NE (fname, NULL);
+}
+
+/* Destructor.  Verify that the next token in m_parser is EOF.  */
+
+lexer_test::~lexer_test ()
+{
+  location_t loc;
+  const cpp_token *tok;
+
+  tok = cpp_get_token_with_location (m_parser, &loc);
+  ASSERT_NE (tok, NULL);
+  ASSERT_EQ (tok->type, CPP_EOF);
+
+  cpp_finish (m_parser, NULL);
+  cpp_destroy (m_parser);
+}
+
+/* Get the next token from m_parser.  */
+
+const cpp_token *
+lexer_test::get_token ()
+{
+  location_t loc;
+  const cpp_token *tok;
+
+  tok = cpp_get_token_with_location (m_parser, &loc);
+  ASSERT_NE (tok, NULL);
+  return tok;
+}
+
+/* Verify that locations within string literals are correctly handled.  */
+
+/* Verify get_source_range_for_substring for token(s) at STRLOC,
+   using the string concatenation database for TEST.
+
+   Assert that the character at index IDX is on EXPECTED_LINE,
+   and that it begins at column EXPECTED_START_COL and ends at
+   EXPECTED_FINISH_COL (unless the locations are beyond
+   LINE_MAP_MAX_LOCATION_WITH_COLS, in which case don't check their
+   columns).  */
+
+static void
+assert_char_at_range (const location &loc,
+		      lexer_test& test,
+		      location_t strloc, int idx, int expected_line,
+		      int expected_start_col, int expected_finish_col)
+{
+  cpp_reader *pfile = test.m_parser;
+  string_concat_db *concats = &test.m_concats;
+
+  source_range actual_range;
+  bool result = get_source_range_for_substring (pfile, concats, strloc,
+						idx, idx, &actual_range);
+  if (should_have_column_data_p (strloc))
+    ASSERT_TRUE_AT (loc, result);
+  else
+    {
+      ASSERT_FALSE_AT (loc, result);
+      return;
+    }
+
+  int actual_start_line = LOCATION_LINE (actual_range.m_start);
+  ASSERT_EQ_AT (loc, expected_line, actual_start_line);
+  int actual_finish_line = LOCATION_LINE (actual_range.m_finish);
+  ASSERT_EQ_AT (loc, expected_line, actual_finish_line);
+
+  if (should_have_column_data_p (actual_range.m_start))
+    {
+      int actual_start_col = LOCATION_COLUMN (actual_range.m_start);
+      ASSERT_EQ_AT (loc, expected_start_col, actual_start_col);
+    }
+  if (should_have_column_data_p (actual_range.m_finish))
+    {
+      int actual_finish_col = LOCATION_COLUMN (actual_range.m_finish);
+      ASSERT_EQ_AT (loc, expected_finish_col, actual_finish_col);
+    }
+}
+
+/* Macro for calling assert_char_at_range, supplying SELFTEST_LOCATION for
+   the effective location of any errors.  */
+
+#define ASSERT_CHAR_AT_RANGE(LEXER_TEST, STRLOC, IDX, EXPECTED_LINE, \
+			     EXPECTED_START_COL,			\
+			     EXPECTED_FINISH_COL)			\
+  assert_char_at_range (SELFTEST_LOCATION, (LEXER_TEST), (STRLOC), (IDX), \
+			(EXPECTED_LINE), \
+			(EXPECTED_START_COL), (EXPECTED_FINISH_COL))
+
+/* Verify get_num_source_ranges_for_substring for token(s) at STRLOC,
+   using the string concatenation database for TEST.
+
+   Assert that the token(s) at STRLOC contain EXPECTED_NUM_RANGES.  */
+
+static void
+assert_num_substring_ranges (const location &loc,
+			     lexer_test& test,
+			     location_t strloc,
+			     int expected_num_ranges)
+{
+  cpp_reader *pfile = test.m_parser;
+  string_concat_db *concats = &test.m_concats;
+
+  int actual_num_ranges;
+  bool result
+    = get_num_source_ranges_for_substring (pfile, concats, strloc,
+					   &actual_num_ranges);
+  if (should_have_column_data_p (strloc))
+    ASSERT_TRUE (result);
+  else
+    {
+      ASSERT_FALSE (result);
+      return;
+    }
+  ASSERT_EQ_AT (loc, expected_num_ranges, actual_num_ranges);
+}
+
+/* Macro for calling assert_num_substring_ranges, supplying
+   SELFTEST_LOCATION for the effective location of any errors.  */
+
+#define ASSERT_NUM_SUBSTRING_RANGES(LEXER_TEST, STRLOC, EXPECTED_NUM_RANGES) \
+    assert_num_substring_ranges (SELFTEST_LOCATION, (LEXER_TEST), (STRLOC), \
+				 (EXPECTED_NUM_RANGES))
+
+/* Lex a simple string literal.  Verify the substring location data, before
+   and after running cpp_interpret_string on it.  */
+
+static void
+test_lexer_string_locations_simple (const line_table_case &case_)
+{
+  /* Digits 0-9 (with 0 at column 10), the simple way.
+     ....................000000000.11111111112.2222222223333333333
+     ....................123456789.01234567890.1234567890123456789
+     We add a trailing comment to ensure that we correctly locate
+     the end of the string literal token.  */
+  const char *content = "        \"0123456789\" /* not a string */\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok, "\"0123456789\"");
+  ASSERT_TOKEN_LOC_EQ (tok, test.m_tempfile.get_filename (), 1, 9, 20);
+
+  /* At this point in lexing, the quote characters are treated as part of
+     the string (they are stripped off by cpp_interpret_string).  */
+
+  ASSERT_EQ (tok->val.str.len, 12);
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("0123456789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.  */
+  for (int i = 0; i < 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1,
+			  10 + i, 10 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 10);
+}
+
+/* Lex a string literal containing a hex-escaped character.
+   Verify the substring location data, before and after running
+   cpp_interpret_string on it.  */
+
+static void
+test_lexer_string_locations_hex (const line_table_case &case_)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as "\x35"
+     and with a space in place of digit 6, to terminate the escaped
+     hex code.
+     ....................000000000.111111.11112222.
+     ....................123456789.012345.67890123.  */
+  const char *content = "        \"01234\\x35 789\"\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok, "\"01234\\x35 789\"");
+  ASSERT_TOKEN_LOC_EQ (tok, test.m_tempfile.get_filename (), 1, 9, 23);
+
+  /* At this point in lexing, the quote characters are treated as part of
+     the string (they are stripped off by cpp_interpret_string).  */
+  ASSERT_EQ (tok->val.str.len, 15);
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("012345 789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.  */
+  for (int i = 0; i < 5; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+  ASSERT_CHAR_AT_RANGE (test, tok->src_loc, 5, 1, 15, 18);
+  for (int i = 6; i <= 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 13 + i, 13 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 10);
+}
+
+/* Lex a string literal containing an octal-escaped character.
+   Verify the substring location data after running cpp_interpret_string
+   on it.  */
+
+static void
+test_lexer_string_locations_oct (const line_table_case &case_)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as "\065"
+     and with a space in place of digit 6, to terminate the escaped
+     octal code.
+     ....................000000000.111111.11112222.2222223333333333444
+     ....................123456789.012345.67890123.4567890123456789012  */
+  const char *content = "        \"01234\\065 789\" /* not a string */\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok, "\"01234\\065 789\"");
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("012345 789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.  */
+  for (int i = 0; i < 5; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+  ASSERT_CHAR_AT_RANGE (test, tok->src_loc, 5, 1, 15, 18);
+  for (int i = 6; i <= 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 13 + i, 13 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 10);
+}
+
+/* Lex a string literal containing UCN 4 characters.
+   Verify the substring location data after running cpp_interpret_string
+   on it.  */
+
+static void
+test_lexer_string_locations_ucn4 (const line_table_case &case_)
+{
+  /* Digits 0-9, expressing digits 5 and 6 as Roman numerals expressed
+     as UCN 4.
+     ....................000000000.111111.111122.222222223.33333333344444
+     ....................123456789.012345.678901.234567890.12345678901234  */
+  const char *content = "        \"01234\\u2174\\u2175789\" /* non-str */\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok, "\"01234\\u2174\\u2175789\"");
+
+  /* Verify that cpp_interpret_string works.
+     The string should be encoded in the execution character
+     set.  Assuming that that is UTF-8, we should have the following:
+     -----------  ----  -----  -------  ----------------
+     Byte offset  Byte  Octal  Unicode  Source Column(s)
+     -----------  ----  -----  -------  ----------------
+     0            0x30         '0'      10
+     1            0x31         '1'      11
+     2            0x32         '2'      12
+     3            0x33         '3'      13
+     4            0x34         '4'      14
+     5            0xE2  \342   U+2174   15-20
+     6            0x85  \205    (cont)  15-20
+     7            0xB4  \264    (cont)  15-20
+     8            0xE2  \342   U+2175   21-26
+     9            0x85  \205    (cont)  21-26
+     10           0xB5  \265    (cont)  21-26
+     11           0x37         '7'      27
+     12           0x38         '8'      28
+     13           0x39         '9'      29
+     -----------  ----  -----  -------  ---------------.  */
+
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("01234\342\205\264\342\205\265789",
+		(const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.
+     '01234'.  */
+  for (int i = 0; i < 4; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+  /* U+2174.  */
+  for (int i = 5; i < 7; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 15, 20);
+  /* U+2175.  */
+  for (int i = 8; i < 10; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 21, 26);
+  /* '789'.  */
+  for (int i = 11; i <= 13; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 16 + i, 16 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 14);
+}
+
+/* Lex a string literal containing UCN 8 characters.
+   Verify the substring location data after running cpp_interpret_string
+   on it.  */
+
+static void
+test_lexer_string_locations_ucn8 (const line_table_case &case_)
+{
+  /* Digits 0-9, expressing digits 5 and 6 as Roman numerals as UCN 8.
+     ....................000000000.111111.1111222222.2222333333333.344444
+     ....................123456789.012345.6789012345.6789012345678.901234  */
+  const char *content = "        \"01234\\U00002174\\U00002175789\" /* */\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok,
+			   "\"01234\\U00002174\\U00002175789\"");
+
+  /* Verify that cpp_interpret_string works.
+     The UTF-8 encoding of the string is identical to that from
+     the ucn4 testcase above; the only difference is the column
+     locations.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("01234\342\205\264\342\205\265789",
+		(const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.
+     '01234'.  */
+  for (int i = 0; i < 4; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+  /* U+2174.  */
+  for (int i = 5; i < 7; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 15, 24);
+  /* U+2175.  */
+  for (int i = 8; i < 10; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 25, 34);
+  /* '789' at columns 35-37  */
+  for (int i = 11; i <= 13; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 24 + i, 24 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 14);
+}
+
+/* Lex a u8-string literal.
+   Verify the substring location data after running cpp_interpret_string
+   on it.  */
+
+static void
+test_lexer_string_locations_u8 (const line_table_case &case_)
+{
+  /* Digits 0-9.
+     ....................000000000.11111111112.22222222233333
+     ....................123456789.01234567890.12345678901234  */
+  const char *content = "      u8\"0123456789\" /* non-str */\n";
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_UTF8STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ (test.m_parser, tok, "u8\"0123456789\"");
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("0123456789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.  */
+  for (int i = 0; i < 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+}
+
+/* Lex a string literal containing UTF-8 source characters.
+   Verify the substring location data after running cpp_interpret_string
+   on it.  */
+
+static void
+test_lexer_string_locations_utf8_source (const line_table_case &case_)
+{
+ /* This string literal is written out to the source file as UTF-8,
+    and is of the form "before mojibake after", where "mojibake"
+    is written as the following four unicode code points:
+       U+6587 CJK UNIFIED IDEOGRAPH-6587
+       U+5B57 CJK UNIFIED IDEOGRAPH-5B57
+       U+5316 CJK UNIFIED IDEOGRAPH-5316
+       U+3051 HIRAGANA LETTER KE.
+     Each of these is 3 bytes wide when encoded in UTF-8, whereas the
+     "before" and "after" are 1 byte per unicode character.
+
+     The numbering shown are "columns", which are *byte* numbers within
+     the line, rather than unicode character numbers.
+
+     .................... 000000000.1111111.
+     .................... 123456789.0123456.  */
+  const char *content = ("        \"before "
+			 /* U+6587 CJK UNIFIED IDEOGRAPH-6587
+			      UTF-8: 0xE6 0x96 0x87
+			      C octal escaped UTF-8: \346\226\207
+			    "column" numbers: 17-19.  */
+			 "\346\226\207"
+
+			 /* U+5B57 CJK UNIFIED IDEOGRAPH-5B57
+			      UTF-8: 0xE5 0xAD 0x97
+			      C octal escaped UTF-8: \345\255\227
+			    "column" numbers: 20-22.  */
+			 "\345\255\227"
+
+			 /* U+5316 CJK UNIFIED IDEOGRAPH-5316
+			      UTF-8: 0xE5 0x8C 0x96
+			      C octal escaped UTF-8: \345\214\226
+			    "column" numbers: 23-25.  */
+			 "\345\214\226"
+
+			 /* U+3051 HIRAGANA LETTER KE
+			      UTF-8: 0xE3 0x81 0x91
+			      C octal escaped UTF-8: \343\201\221
+			    "column" numbers: 26-28.  */
+			 "\343\201\221"
+
+			 /* column numbers 29 onwards
+			  2333333.33334444444444
+			  9012345.67890123456789. */
+			 " after\" /* non-str */\n");
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected token back, with the correct
+     location information.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ
+    (test.m_parser, tok,
+     "\"before \346\226\207\345\255\227\345\214\226\343\201\221 after\"");
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser, &tok->val.str, 1,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ
+    ("before \346\226\207\345\255\227\345\214\226\343\201\221 after",
+     (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Verify ranges of individual characters.  This no longer includes the
+     quotes.
+     Assuming that both source and execution encodings are UTF-8, we have
+     a run of 25 octets in each.  */
+  for (int i = 0; i < 25; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 10 + i, 10 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 25);
+}
+
+/* Test of string literal concatenation.  */
+
+static void
+test_lexer_string_locations_concatenation_1 (const line_table_case &case_)
+{
+  /* Digits 0-9.
+     .....................000000000.111111.11112222222222
+     .....................123456789.012345.67890123456789.  */
+  const char *content = ("        \"01234\" /* non-str */\n"
+			 "        \"56789\" /* non-str */\n");
+  lexer_test test (case_, content);
+
+  location_t input_locs[2];
+
+  /* Verify that we get the expected tokens back.  */
+  auto_vec <cpp_string> input_strings;
+  const cpp_token *tok_a = test.get_token ();
+  ASSERT_EQ (tok_a->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ
+    (test.m_parser, tok_a,
+     "\"01234\"");
+  input_strings.safe_push (tok_a->val.str);
+  input_locs[0] = tok_a->src_loc;
+
+  const cpp_token *tok_b = test.get_token ();
+  ASSERT_EQ (tok_b->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ
+    (test.m_parser, tok_b,
+     "\"56789\"");
+  input_strings.safe_push (tok_b->val.str);
+  input_locs[1] = tok_b->src_loc;
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser,
+				      input_strings.address (), 2,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("0123456789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Simulate c-lex.c's lex_string in order to record concatenation.  */
+  test.m_concats.record_string_concatenation (2, input_locs);
+
+  location_t initial_loc = input_locs[0];
+
+  for (int i = 0; i < 5; i++)
+    ASSERT_CHAR_AT_RANGE (test, initial_loc, i, 1, 10 + i, 10 + i);
+  for (int i = 5; i <= 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, initial_loc, i, 2, 5 + i, 5 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, initial_loc, 10);
+}
+
+/* Another test of string literal concatenation.  */
+
+static void
+test_lexer_string_locations_concatenation_2 (const line_table_case &case_)
+{
+  /* Digits 0-9.
+     .....................000000000.111.11111112222222
+     .....................123456789.012.34567890123456.  */
+  const char *content = ("        \"01\" /* non-str */\n"
+			 "        \"23\" /* non-str */\n"
+			 "        \"45\" /* non-str */\n"
+			 "        \"67\" /* non-str */\n"
+			 "        \"89\" /* non-str */\n");
+  lexer_test test (case_, content);
+
+  auto_vec <cpp_string> input_strings;
+  location_t input_locs[5];
+
+  /* Verify that we get the expected tokens back.  */
+  for (int i = 0; i < 5; i++)
+    {
+      const cpp_token *tok = test.get_token ();
+      ASSERT_EQ (tok->type, CPP_STRING);
+      input_strings.safe_push (tok->val.str);
+      input_locs[i] = tok->src_loc;
+    }
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser,
+				      input_strings.address (), 5,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("0123456789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Simulate c-lex.c's lex_string in order to record concatenation.  */
+  test.m_concats.record_string_concatenation (5, input_locs);
+
+  location_t initial_loc = input_locs[0];
+
+  /* Within ASSERT_CHAR_AT_RANGE (actually assert_char_at_range), we can
+     detect if the initial loc is after LINE_MAP_MAX_LOCATION_WITH_COLS
+     and expect get_source_range_for_substring to fail.
+     However, for a string concatenation test, we can have a case
+     where the initial string is fully before LINE_MAP_MAX_LOCATION_WITH_COLS,
+     but subsequent strings can be after it.
+     Attempting to detect this within assert_char_at_range
+     would overcomplicate the logic for the common test cases, so
+     we detect it here.  */
+  if (should_have_column_data_p (input_locs[0])
+      && !should_have_column_data_p (input_locs[4]))
+    {
+      /* Verify that get_source_range_for_substring gracefully rejects
+	 this case.  */
+      source_range actual_range;
+      bool result
+	= get_source_range_for_substring (test.m_parser, &test.m_concats,
+					  initial_loc, 0, 0, &actual_range);
+      ASSERT_FALSE (result);
+      return;
+    }
+
+  for (int i = 0; i < 5; i++)
+    for (int j = 0; j < 2; j++)
+      ASSERT_CHAR_AT_RANGE (test, initial_loc, (i * 2) + j,
+			    i + 1, 10 + j, 10 + j);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, initial_loc, 10);
+}
+
+/* Another test of string literal concatenation, this time combined with
+   various kinds of escaped characters.  */
+
+static void
+test_lexer_string_locations_concatenation_3 (const line_table_case &case_)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as hex "\x35"
+     digit 6 in ASCII as octal "\066", concatenating multiple strings.  */
+  const char *content
+    /* .000000000.111111.111.1.2222.222.2.2233.333.3333.34444444444555
+       .123456789.012345.678.9.0123.456.7.8901.234.5678.90123456789012. */
+    = ("        \"01234\"  \"\\x35\"  \"\\066\"  \"789\" /* non-str */\n");
+  lexer_test test (case_, content);
+
+  auto_vec <cpp_string> input_strings;
+  location_t input_locs[4];
+
+  /* Verify that we get the expected tokens back.  */
+  for (int i = 0; i < 4; i++)
+    {
+      const cpp_token *tok = test.get_token ();
+      ASSERT_EQ (tok->type, CPP_STRING);
+      input_strings.safe_push (tok->val.str);
+      input_locs[i] = tok->src_loc;
+    }
+
+  /* Verify that cpp_interpret_string works.  */
+  cpp_string dst_string;
+  bool result = cpp_interpret_string (test.m_parser,
+				      input_strings.address (), 4,
+				      &dst_string, CPP_STRING);
+  ASSERT_TRUE (result);
+  ASSERT_STREQ ("0123456789", (const char *)dst_string.text);
+  free (const_cast <unsigned char *> (dst_string.text));
+
+  /* Simulate c-lex.c's lex_string in order to record concatenation.  */
+  test.m_concats.record_string_concatenation (4, input_locs);
+
+  location_t initial_loc = input_locs[0];
+
+  for (int i = 0; i < 4; i++)
+    ASSERT_CHAR_AT_RANGE (test, initial_loc, i, 1, 10 + i, 10 + i);
+  ASSERT_CHAR_AT_RANGE (test, initial_loc, 5, 1, 19, 22);
+  ASSERT_CHAR_AT_RANGE (test, initial_loc, 6, 1, 27, 30);
+  for (int i = 7; i <= 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, initial_loc, i, 1, 28 + i, 28 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, initial_loc, 10);
+}
+
+/* Test of string literal in a macro.  */
+
+static void
+test_lexer_string_locations_macro (const line_table_case &case_)
+{
+  /* Digits 0-9.
+     .....................0000000001111111111.22222222223.
+     .....................1234567890123456789.01234567890.  */
+  const char *content = ("#define MACRO     \"0123456789\" /* non-str */\n"
+			 "  MACRO");
+  lexer_test test (case_, content);
+
+  /* Verify that we get the expected tokens back.  */
+  const cpp_token *tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_PADDING);
+
+  tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_STRING);
+  ASSERT_TOKEN_AS_TEXT_EQ
+    (test.m_parser, tok,
+     "\"0123456789\"");
+
+  /* Verify ranges of individual characters.  We ought to
+     see columns within the macro definition.  */
+  for (int i = 0; i < 9; i++)
+    ASSERT_CHAR_AT_RANGE (test, tok->src_loc, i, 1, 20 + i, 20 + i);
+
+  ASSERT_NUM_SUBSTRING_RANGES (test, tok->src_loc, 10);
+
+  tok = test.get_token ();
+  ASSERT_EQ (tok->type, CPP_PADDING);
+}
+
 /* A table of interesting location_t values, giving one axis of our test
    matrix.  */
 
@@ -1539,6 +2505,17 @@  input_c_tests ()
 	  /* Run all tests for the given case within the test matrix.  */
 	  test_accessing_ordinary_linemaps (c);
 	  test_lexer (c);
+	  test_lexer_string_locations_simple (c);
+	  test_lexer_string_locations_hex (c);
+	  test_lexer_string_locations_oct (c);
+	  test_lexer_string_locations_ucn4 (c);
+	  test_lexer_string_locations_ucn8 (c);
+	  test_lexer_string_locations_u8 (c);
+	  test_lexer_string_locations_utf8_source (c);
+	  test_lexer_string_locations_concatenation_1 (c);
+	  test_lexer_string_locations_concatenation_2 (c);
+	  test_lexer_string_locations_concatenation_3 (c);
+	  test_lexer_string_locations_macro (c);
 
 	  num_cases_tested++;
 	}
diff --git a/gcc/input.h b/gcc/input.h
index ae4fecf..2f77afe 100644
--- a/gcc/input.h
+++ b/gcc/input.h
@@ -82,4 +82,39 @@  void dump_location_info (FILE *stream);
 
 void diagnostics_file_cache_fini (void);
 
+struct GTY(()) string_concat
+{
+  string_concat (int num, location_t *locs);
+
+  int m_num;
+  location_t * GTY ((atomic)) m_locs;
+};
+
+struct location_hash : int_hash <location_t, UNKNOWN_LOCATION> { };
+
+class GTY(()) string_concat_db
+{
+ public:
+  string_concat_db ();
+  void record_string_concatenation (int num, location_t *locs);
+
+  bool get_string_concatenation (location_t loc,
+				 int *out_num,
+				 location_t **out_locs);
+
+ private:
+  static location_t get_key_loc (location_t loc);
+
+ public:
+  /* This would be private, but must be public for use by
+     gtype-desc.c.  */
+  hash_map <location_hash, string_concat *> *m_table;
+};
+
+extern bool get_source_range_for_substring (cpp_reader *pfile,
+					    string_concat_db *concats,
+					    location_t strloc,
+					    int start_idx, int end_idx,
+					    source_range *out);
+
 #endif
diff --git a/gcc/selftest.h b/gcc/selftest.h
index 967e76b..02e4694 100644
--- a/gcc/selftest.h
+++ b/gcc/selftest.h
@@ -102,13 +102,19 @@  extern int num_passes;
    ::selftest::fail if it false.  */
 
 #define ASSERT_TRUE(EXPR)				\
+  ASSERT_TRUE_AT (SELFTEST_LOCATION, (EXPR))
+
+/* Like ASSERT_TRUE, but treat LOC as the effective location of the
+   selftest.  */
+
+#define ASSERT_TRUE_AT(LOC, EXPR)			\
   SELFTEST_BEGIN_STMT					\
   const char *desc = "ASSERT_TRUE (" #EXPR ")";		\
   bool actual = (EXPR);					\
   if (actual)						\
-    ::selftest::pass (SELFTEST_LOCATION, desc);	\
+    ::selftest::pass ((LOC), desc);			\
   else							\
-    ::selftest::fail (SELFTEST_LOCATION, desc);		\
+    ::selftest::fail ((LOC), desc);			\
   SELFTEST_END_STMT
 
 /* Evaluate EXPR and coerce to bool, calling
@@ -116,13 +122,19 @@  extern int num_passes;
    ::selftest::fail if it true.  */
 
 #define ASSERT_FALSE(EXPR)					\
+  ASSERT_FALSE_AT (SELFTEST_LOCATION, (EXPR))
+
+/* Like ASSERT_FALSE, but treat LOC as the effective location of the
+   selftest.  */
+
+#define ASSERT_FALSE_AT(LOC, EXPR)				\
   SELFTEST_BEGIN_STMT						\
-  const char *desc = "ASSERT_FALSE (" #EXPR ")";		\
-  bool actual = (EXPR);					\
-  if (actual)							\
-    ::selftest::fail (SELFTEST_LOCATION, desc);				\
-  else								\
-    ::selftest::pass (SELFTEST_LOCATION, desc);				\
+  const char *desc = "ASSERT_FALSE (" #EXPR ")";			\
+  bool actual = (EXPR);							\
+  if (actual)								\
+    ::selftest::fail ((LOC), desc);			\
+  else									\
+    ::selftest::pass ((LOC), desc);					\
   SELFTEST_END_STMT
 
 /* Evaluate EXPECTED and ACTUAL and compare them with ==, calling
@@ -167,7 +179,7 @@  extern int num_passes;
 			    (EXPECTED), (ACTUAL));		    \
   SELFTEST_END_STMT
 
-/* Like ASSERT_STREQ_AT, but treat LOC as the effective location of the
+/* Like ASSERT_STREQ, but treat LOC as the effective location of the
    selftest.  */
 
 #define ASSERT_STREQ_AT(LOC, EXPECTED, ACTUAL)			    \
diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic-test-string-literals-1.c b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-string-literals-1.c
new file mode 100644
index 0000000..73c5b2b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic-test-string-literals-1.c
@@ -0,0 +1,172 @@ 
+/* { dg-do compile } */
+/* { dg-options "-O -fdiagnostics-show-caret" } */
+
+/* This is a collection of unittests for ranges within string literals,
+   using diagnostic_plugin_test_string_literals, which handles
+   "__emit_string_literal_range" by generating a warning at the given
+   subset of a string literal.
+
+   The indices are 0-based.  It's easiest to verify things using string
+   literals that are runs of 0-based digits (to avoid having to count
+   characters).  */
+
+extern void __emit_string_literal_range (const char *literal,
+					 int start_idx, int end_idx);
+
+void
+test_simple_string_literal (void)
+{
+  __emit_string_literal_range ("0123456789", /* { dg-warning "range" } */
+			       6, 7);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("0123456789",
+                                       ^~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_concatenated_string_literal (void)
+{
+  __emit_string_literal_range ("01234" "56789", /* { dg-warning "range" } */
+			       3, 6);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234" "56789",
+                                    ^~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_multiline_string_literal (void)
+{
+  __emit_string_literal_range ("01234" /* { dg-warning "range" } */
+                               "56789",
+                               3, 6);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234"
+                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+                                "56789",
+                                ~~~  
+   { dg-end-multiline-output "" } */
+  /* FIXME: why does the above need two trailing spaces?  */
+}
+
+/* Tests of various unicode encodings.
+
+   Digits 0 through 9 are unicode code points:
+      U+0030 DIGIT ZERO
+      ...
+      U+0039 DIGIT NINE
+   However, these are not always valid as UCN (see the comment in
+   libcpp/charset.c:_cpp_valid_ucn).
+
+   Hence we need to test UCN using an alternative unicode
+   representation of numbers; let's use Roman numerals,
+   (though these start at one, not zero):
+      U+2170 SMALL ROMAN NUMERAL ONE
+      ...
+      U+2174 SMALL ROMAN NUMERAL FIVE  ("v")
+      U+2175 SMALL ROMAN NUMERAL SIX   ("vi")
+      ...
+      U+2178 SMALL ROMAN NUMERAL NINE.  */
+
+void
+test_hex (void)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as "\x35"
+     and with a space in place of digit 6, to terminate the escaped
+     hex code.  */
+  __emit_string_literal_range ("01234\x35 789", /* { dg-warning "range" } */
+			       3, 7);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234\x35 789"
+                                    ^~~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_oct (void)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as "\065"
+     and with a space in place of digit 6, to terminate the escaped
+     octal code.  */
+  __emit_string_literal_range ("01234\065 789", /* { dg-warning "range" } */
+			       3, 7);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234\065 789"
+                                    ^~~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_multiple (void)
+{
+  /* Digits 0-9, expressing digit 5 in ASCII as hex "\x35"
+     digit 6 in ASCII as octal "\066", concatenating multiple strings.  */
+  __emit_string_literal_range ("01234"  "\x35"  "\066"  "789", /* { dg-warning "range" } */
+			       3, 8);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234"  "\x35"  "\066"  "789",
+                                    ^~~~~~~~~~~~~~~~~~~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_ucn4 (void)
+{
+  /* Digits 0-9, expressing digits 5 and 6 as Roman numerals expressed
+     as UCN 4.
+     The resulting string is encoded as UTF-8.  Most of the digits are 1 byte
+     each, but digits 5 and 6 are encoded with 3 bytes each.
+     Hence to underline digits 4-7 we need to underling using bytes 4-11 in
+     the UTF-8 encoding.  */
+  __emit_string_literal_range ("01234\u2174\u2175789", /* { dg-warning "range" } */
+			       4, 11);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234\u2174\u2175789",
+                                     ^~~~~~~~~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_ucn8 (void)
+{
+  /* Digits 0-9, expressing digits 5 and 6 as Roman numerals as UCN 8.
+     The resulting string is the same as as in test_ucn4 above, and hence
+     has the same UTF-8 encoding, and so we again need to underline bytes
+     4-11 in the UTF-8 encoding in order to underline digits 4-7.  */
+  __emit_string_literal_range ("01234\U00002174\U00002175789", /* { dg-warning "range" } */
+			       4, 11);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range ("01234\U00002174\U00002175789",
+                                     ^~~~~~~~~~~~~~~~~~~~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_u8 (void)
+{
+  /* Digits 0-9.  */
+  __emit_string_literal_range (u8"0123456789", /* { dg-warning "range" } */
+			       4, 7);
+/* { dg-begin-multiline-output "" }
+   __emit_string_literal_range (u8"0123456789",
+                                       ^~~~
+   { dg-end-multiline-output "" } */
+}
+
+void
+test_macro (void)
+{
+#define START "01234"  /* { dg-warning "range" } */
+  __emit_string_literal_range (START
+                               "56789",
+                               3, 6);
+/* { dg-begin-multiline-output "" }
+ #define START "01234"
+                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+   __emit_string_literal_range (START
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+                                "56789",
+                                ~~~
+   { dg-end-multiline-output "" } */
+}
diff --git a/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_test_string_literals.c b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_test_string_literals.c
new file mode 100644
index 0000000..d92c2b5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/plugin/diagnostic_plugin_test_string_literals.c
@@ -0,0 +1,210 @@ 
+/* This plugin uses the diagnostics code to verify tracking of source code
+   locations within string literals.  */
+/* { dg-options "-O" } */
+
+#include "gcc-plugin.h"
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "tm.h"
+#include "tree.h"
+#include "stringpool.h"
+#include "toplev.h"
+#include "basic-block.h"
+#include "hash-table.h"
+#include "vec.h"
+#include "ggc.h"
+#include "basic-block.h"
+#include "tree-ssa-alias.h"
+#include "internal-fn.h"
+#include "gimple-fold.h"
+#include "tree-eh.h"
+#include "gimple-expr.h"
+#include "is-a.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "tree.h"
+#include "tree-pass.h"
+#include "intl.h"
+#include "plugin-version.h"
+#include "c-family/c-common.h"
+#include "diagnostic.h"
+#include "context.h"
+#include "print-tree.h"
+#include "cpplib.h"
+#include "c-family/c-pragma.h"
+
+int plugin_is_GPL_compatible;
+
+/* A custom pass for printing string literal location information.  */
+
+const pass_data pass_data_test_string_literals =
+{
+  GIMPLE_PASS, /* type */
+  "test_string_literals", /* name */
+  OPTGROUP_NONE, /* optinfo_flags */
+  TV_NONE, /* tv_id */
+  PROP_ssa, /* properties_required */
+  0, /* properties_provided */
+  0, /* properties_destroyed */
+  0, /* todo_flags_start */
+  0, /* todo_flags_finish */
+};
+
+class pass_test_string_literals : public gimple_opt_pass
+{
+public:
+  pass_test_string_literals(gcc::context *ctxt)
+    : gimple_opt_pass(pass_data_test_string_literals, ctxt)
+  {}
+
+  /* opt_pass methods: */
+  bool gate (function *) { return true; }
+  virtual unsigned int execute (function *);
+
+}; // class pass_test_string_literals
+
+/* Determine if STMT is a call with NUM_ARGS arguments to a function
+   named FUNCNAME.
+   If so, return STMT as a gcall *.  Otherwise return NULL.  */
+
+static gcall *
+check_for_named_call (gimple *stmt,
+		      const char *funcname, unsigned int num_args)
+{
+  gcc_assert (funcname);
+
+  gcall *call = dyn_cast <gcall *> (stmt);
+  if (!call)
+    return NULL;
+
+  tree fndecl = gimple_call_fndecl (call);
+  if (!fndecl)
+    return NULL;
+
+  if (strcmp (IDENTIFIER_POINTER (DECL_NAME (fndecl)), funcname))
+    return NULL;
+
+  if (gimple_call_num_args (call) != num_args)
+    {
+      error_at (stmt->location, "expected number of args: %i (got %i)",
+		num_args, gimple_call_num_args (call));
+      return NULL;
+    }
+
+  return call;
+}
+
+/* Emit a warning covering SRC_RANGE, with the caret at the start of
+   SRC_RANGE.  */
+
+static void
+emit_warning (source_range src_range)
+{
+  location_t loc
+    = make_location (src_range.m_start, src_range.m_start, src_range.m_finish);
+  warning_at (loc, 0, "range %i:%i-%i:%i",
+	      LOCATION_LINE (src_range.m_start),
+	      LOCATION_COLUMN (src_range.m_start),
+	      LOCATION_LINE (src_range.m_finish),
+	      LOCATION_COLUMN (src_range.m_finish));
+}
+
+/* Support code for verifying that we are correctly tracking ranges
+   within string literals, for use by diagnostic-test-string-literals-*.c.
+   Emit a warning showing the range of a string literal, for each call to
+   a function named "__emit_string_literal_range".
+   The initial argument should be a string literal; arguments 2 and 3
+   should be integer constants, giving the range within the string
+   to be printed.  */
+
+static void
+test_string_literals (gimple *stmt)
+{
+  gcall *call = check_for_named_call (stmt, "__emit_string_literal_range", 3);
+  if (!call)
+    return;
+
+  /* We expect an ADDR_EXPR with a STRING_CST inside it for the
+     initial arg.  */
+  tree t_addr_string = gimple_call_arg (call, 0);
+  if (TREE_CODE (t_addr_string) != ADDR_EXPR)
+    {
+      error_at (call->location, "string literal required for arg 1");
+      return;
+    }
+
+  tree t_string = TREE_OPERAND (t_addr_string, 0);
+  if (TREE_CODE (t_string) != STRING_CST)
+    {
+      error_at (call->location, "string literal required for arg 1");
+      return;
+    }
+
+  tree t_start_idx = gimple_call_arg (call, 1);
+  if (TREE_CODE (t_start_idx) != INTEGER_CST)
+    {
+      error_at (call->location, "integer constant required for arg 2");
+      return;
+    }
+  int start_idx = TREE_INT_CST_LOW (t_start_idx);
+
+  tree t_end_idx = gimple_call_arg (call, 2);
+  if (TREE_CODE (t_end_idx) != INTEGER_CST)
+    {
+      error_at (call->location, "integer constant required for arg 3");
+      return;
+    }
+  int end_idx = TREE_INT_CST_LOW (t_end_idx);
+
+  /* A STRING_CST doesn't have a location, but the ADDR_EXPR does.  */
+  location_t strloc = EXPR_LOCATION (t_addr_string);
+  source_range src_range;
+  if (get_source_range_for_substring (parse_in, g_string_concat_db, strloc,
+				      start_idx, end_idx, &src_range))
+    emit_warning (src_range);
+  else
+    error_at (strloc, "unable to read substring range");
+}
+
+/* Call test_string_literals on every statement within FUN.  */
+
+unsigned int
+pass_test_string_literals::execute (function *fun)
+{
+  gimple_stmt_iterator gsi;
+  basic_block bb;
+
+  FOR_EACH_BB_FN (bb, fun)
+    for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+      {
+	gimple *stmt = gsi_stmt (gsi);
+	test_string_literals (stmt);
+      }
+
+  return 0;
+}
+
+/* Entrypoint for the plugin.  Create and register the custom pass.  */
+
+int
+plugin_init (struct plugin_name_args *plugin_info,
+	     struct plugin_gcc_version *version)
+{
+  struct register_pass_info pass_info;
+  const char *plugin_name = plugin_info->base_name;
+  int argc = plugin_info->argc;
+  struct plugin_argument *argv = plugin_info->argv;
+
+  if (!plugin_default_version_check (version, &gcc_version))
+    return 1;
+
+  pass_info.pass = new pass_test_string_literals (g);
+  pass_info.reference_pass_name = "ssa";
+  pass_info.ref_pass_instance_number = 1;
+  pass_info.pos_op = PASS_POS_INSERT_AFTER;
+  register_callback (plugin_name, PLUGIN_PASS_MANAGER_SETUP, NULL,
+		     &pass_info);
+
+  return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/plugin/plugin.exp b/gcc/testsuite/gcc.dg/plugin/plugin.exp
index f039c8d..3c2383a 100644
--- a/gcc/testsuite/gcc.dg/plugin/plugin.exp
+++ b/gcc/testsuite/gcc.dg/plugin/plugin.exp
@@ -71,6 +71,8 @@  set plugin_test_list [list \
 	  diagnostic-test-expressions-1.c } \
     { diagnostic_plugin_show_trees.c \
 	  diagnostic-test-show-trees-1.c } \
+    { diagnostic_plugin_test_string_literals.c \
+	  diagnostic-test-string-literals-1.c } \
     { location_overflow_plugin.c \
 	  location-overflow-test-1.c \
 	  location-overflow-test-2.c } \
diff --git a/libcpp/charset.c b/libcpp/charset.c
index 2d07942..a4bddc8 100644
--- a/libcpp/charset.c
+++ b/libcpp/charset.c
@@ -812,6 +812,51 @@  cpp_host_to_exec_charset (cpp_reader *pfile, cppchar_t c)
 
 
 
+/* cpp_substring_ranges's constructor. */
+
+cpp_substring_ranges::cpp_substring_ranges () :
+  m_ranges (NULL),
+  m_num_ranges (0),
+  m_alloc_ranges (8)
+{
+  m_ranges = XNEWVEC (source_range, m_alloc_ranges);
+}
+
+/* cpp_substring_ranges's destructor. */
+
+cpp_substring_ranges::~cpp_substring_ranges ()
+{
+  free (m_ranges);
+}
+
+/* Add RANGE to the vector of source_range information.  */
+
+void
+cpp_substring_ranges::add_range (source_range range)
+{
+  if (m_num_ranges >= m_alloc_ranges)
+    {
+      m_alloc_ranges *= 2;
+      m_ranges
+	= (source_range *)xrealloc (m_ranges,
+				    sizeof (source_range) * m_alloc_ranges);
+    }
+  m_ranges[m_num_ranges++] = range;
+}
+
+/* Read NUM ranges from LOC_READER, adding them to the vector of source_range
+   information.  */
+
+void
+cpp_substring_ranges::add_n_ranges (int num,
+				    cpp_string_location_reader &loc_reader)
+{
+  for (int i = 0; i < num; i++)
+    add_range (loc_reader.get_next ());
+}
+
+
+
 /* Utility routine that computes a mask of the form 0000...111... with
    WIDTH 1-bits.  */
 static inline size_t
@@ -980,18 +1025,27 @@  ucn_valid_in_identifier (cpp_reader *pfile, cppchar_t c,
    one beyond the UCN, or to the syntactically invalid character.
 
    IDENTIFIER_POS is 0 when not in an identifier, 1 for the start of
-   an identifier, or 2 otherwise.  */
+   an identifier, or 2 otherwise.
+
+   If CHAR_RANGE and LOC_READER are non-NULL, then position information is
+   read from *LOC_READER and CHAR_RANGE->m_finish is updated accordingly.  */
 
 bool
 _cpp_valid_ucn (cpp_reader *pfile, const uchar **pstr,
 		const uchar *limit, int identifier_pos,
-		struct normalize_state *nst, cppchar_t *cp)
+		struct normalize_state *nst, cppchar_t *cp,
+		source_range *char_range,
+		cpp_string_location_reader *loc_reader)
 {
   cppchar_t result, c;
   unsigned int length;
   const uchar *str = *pstr;
   const uchar *base = str - 2;
 
+  /* char_range and loc_reader must either be both NULL, or both be
+     non-NULL.  */
+  gcc_assert ((char_range != NULL) == (loc_reader != NULL));
+
   if (!CPP_OPTION (pfile, cplusplus) && !CPP_OPTION (pfile, c99))
     cpp_error (pfile, CPP_DL_WARNING,
 	       "universal character names are only valid in C++ and C99");
@@ -1021,6 +1075,8 @@  _cpp_valid_ucn (cpp_reader *pfile, const uchar **pstr,
       if (!ISXDIGIT (c))
 	break;
       str++;
+      if (loc_reader)
+	char_range->m_finish = loc_reader->get_next ().m_finish;
       result = (result << 4) + hex_value (c);
     }
   while (--length && str < limit);
@@ -1086,11 +1142,18 @@  _cpp_valid_ucn (cpp_reader *pfile, const uchar **pstr,
 }
 
 /* Convert an UCN, pointed to by FROM, to UTF-8 encoding, then translate
-   it to the execution character set and write the result into TBUF.
-   An advanced pointer is returned.  Issues all relevant diagnostics.  */
+   it to the execution character set and write the result into TBUF,
+   if TBUF is non-NULL.
+   An advanced pointer is returned.  Issues all relevant diagnostics.
+   If LOC_READER is non-NULL, then RANGES must be non-NULL and CHAR_RANGE
+   contains the location of the character so far: location information
+   is read from *LOC_READER, and *RANGES is updated accordingly.  */
 static const uchar *
 convert_ucn (cpp_reader *pfile, const uchar *from, const uchar *limit,
-	     struct _cpp_strbuf *tbuf, struct cset_converter cvt)
+	     struct _cpp_strbuf *tbuf, struct cset_converter cvt,
+	     source_range char_range,
+	     cpp_string_location_reader *loc_reader,
+	     cpp_substring_ranges *ranges)
 {
   cppchar_t ucn;
   uchar buf[6];
@@ -1099,8 +1162,17 @@  convert_ucn (cpp_reader *pfile, const uchar *from, const uchar *limit,
   int rval;
   struct normalize_state nst = INITIAL_NORMALIZE_STATE;
 
+  /* loc_reader and ranges must either be both NULL, or both be non-NULL.  */
+  gcc_assert ((loc_reader != NULL) == (ranges != NULL));
+
   from++;  /* Skip u/U.  */
-  _cpp_valid_ucn (pfile, &from, limit, 0, &nst, &ucn);
+
+  if (loc_reader)
+    /* The u/U is part of the spelling of this character.  */
+    char_range.m_finish = loc_reader->get_next ().m_finish;
+
+  _cpp_valid_ucn (pfile, &from, limit, 0, &nst,
+		  &ucn, &char_range, loc_reader);
 
   rval = one_cppchar_to_utf8 (ucn, &bufp, &bytesleft);
   if (rval)
@@ -1109,9 +1181,20 @@  convert_ucn (cpp_reader *pfile, const uchar *from, const uchar *limit,
       cpp_errno (pfile, CPP_DL_ERROR,
 		 "converting UCN to source character set");
     }
-  else if (!APPLY_CONVERSION (cvt, buf, 6 - bytesleft, tbuf))
-    cpp_errno (pfile, CPP_DL_ERROR,
-	       "converting UCN to execution character set");
+  else
+    {
+      if (tbuf)
+	if (!APPLY_CONVERSION (cvt, buf, 6 - bytesleft, tbuf))
+	  cpp_errno (pfile, CPP_DL_ERROR,
+		     "converting UCN to execution character set");
+
+      if (loc_reader)
+	{
+	  int num_encoded_bytes = 6 - bytesleft;
+	  for (int i = 0; i < num_encoded_bytes; i++)
+	    ranges->add_range (char_range);
+	}
+    }
 
   return from;
 }
@@ -1167,31 +1250,48 @@  emit_numeric_escape (cpp_reader *pfile, cppchar_t n,
 }
 
 /* Convert a hexadecimal escape, pointed to by FROM, to the execution
-   character set and write it into the string buffer TBUF.  Returns an
-   advanced pointer, and issues diagnostics as necessary.
+   character set and write it into the string buffer TBUF (if non-NULL).
+   Returns an advanced pointer, and issues diagnostics as necessary.
    No character set translation occurs; this routine always produces the
    execution-set character with numeric value equal to the given hex
-   number.  You can, e.g. generate surrogate pairs this way.  */
+   number.  You can, e.g. generate surrogate pairs this way.
+   If LOC_READER is non-NULL, then RANGES must be non-NULL and CHAR_RANGE
+   contains the location of the character so far: location information
+   is read from *LOC_READER, and *RANGES is updated accordingly.  */
 static const uchar *
 convert_hex (cpp_reader *pfile, const uchar *from, const uchar *limit,
-	     struct _cpp_strbuf *tbuf, struct cset_converter cvt)
+	     struct _cpp_strbuf *tbuf, struct cset_converter cvt,
+	     source_range char_range,
+	     cpp_string_location_reader *loc_reader,
+	     cpp_substring_ranges *ranges)
 {
   cppchar_t c, n = 0, overflow = 0;
   int digits_found = 0;
   size_t width = cvt.width;
   size_t mask = width_to_mask (width);
 
+  /* loc_reader and ranges must either be both NULL, or both be non-NULL.  */
+  gcc_assert ((loc_reader != NULL) == (ranges != NULL));
+
   if (CPP_WTRADITIONAL (pfile))
     cpp_warning (pfile, CPP_W_TRADITIONAL,
 	         "the meaning of '\\x' is different in traditional C");
 
-  from++;  /* Skip 'x'.  */
+  /* Skip 'x'.  */
+  from++;
+
+  /* The 'x' is part of the spelling of this character.  */
+  if (loc_reader)
+    char_range.m_finish = loc_reader->get_next ().m_finish;
+
   while (from < limit)
     {
       c = *from;
       if (! hex_p (c))
 	break;
       from++;
+      if (loc_reader)
+	char_range.m_finish = loc_reader->get_next ().m_finish;
       overflow |= n ^ (n << 4 >> 4);
       n = (n << 4) + hex_value (c);
       digits_found = 1;
@@ -1211,7 +1311,10 @@  convert_hex (cpp_reader *pfile, const uchar *from, const uchar *limit,
       n &= mask;
     }
 
-  emit_numeric_escape (pfile, n, tbuf, cvt);
+  if (tbuf)
+    emit_numeric_escape (pfile, n, tbuf, cvt);
+  if (ranges)
+    ranges->add_range (char_range);
 
   return from;
 }
@@ -1221,10 +1324,16 @@  convert_hex (cpp_reader *pfile, const uchar *from, const uchar *limit,
    advanced pointer, and issues diagnostics as necessary.
    No character set translation occurs; this routine always produces the
    execution-set character with numeric value equal to the given octal
-   number.  */
+   number.
+   If LOC_READER is non-NULL, then RANGES must be non-NULL and CHAR_RANGE
+   contains the location of the character so far: location information
+   is read from *LOC_READER, and *RANGES is updated accordingly.  */
 static const uchar *
 convert_oct (cpp_reader *pfile, const uchar *from, const uchar *limit,
-	     struct _cpp_strbuf *tbuf, struct cset_converter cvt)
+	     struct _cpp_strbuf *tbuf, struct cset_converter cvt,
+	     source_range char_range,
+	     cpp_string_location_reader *loc_reader,
+	     cpp_substring_ranges *ranges)
 {
   size_t count = 0;
   cppchar_t c, n = 0;
@@ -1232,12 +1341,17 @@  convert_oct (cpp_reader *pfile, const uchar *from, const uchar *limit,
   size_t mask = width_to_mask (width);
   bool overflow = false;
 
+  /* loc_reader and ranges must either be both NULL, or both be non-NULL.  */
+  gcc_assert ((loc_reader != NULL) == (ranges != NULL));
+
   while (from < limit && count++ < 3)
     {
       c = *from;
       if (c < '0' || c > '7')
 	break;
       from++;
+      if (loc_reader)
+	char_range.m_finish = loc_reader->get_next ().m_finish;
       overflow |= n ^ (n << 3 >> 3);
       n = (n << 3) + c - '0';
     }
@@ -1249,18 +1363,26 @@  convert_oct (cpp_reader *pfile, const uchar *from, const uchar *limit,
       n &= mask;
     }
 
-  emit_numeric_escape (pfile, n, tbuf, cvt);
+  if (tbuf)
+    emit_numeric_escape (pfile, n, tbuf, cvt);
+  if (ranges)
+    ranges->add_range (char_range);
 
   return from;
 }
 
 /* Convert an escape sequence (pointed to by FROM) to its value on
    the target, and to the execution character set.  Do not scan past
-   LIMIT.  Write the converted value into TBUF.  Returns an advanced
-   pointer.  Handles all relevant diagnostics.  */
+   LIMIT.  Write the converted value into TBUF, if TBUF is non-NULL.
+   Returns an advanced pointer.  Handles all relevant diagnostics.
+   If LOC_READER is non-NULL, then RANGES must be non-NULL: location
+   information is read from *LOC_READER, and *RANGES is updated
+   accordingly.  */
 static const uchar *
 convert_escape (cpp_reader *pfile, const uchar *from, const uchar *limit,
-		struct _cpp_strbuf *tbuf, struct cset_converter cvt)
+		struct _cpp_strbuf *tbuf, struct cset_converter cvt,
+		cpp_string_location_reader *loc_reader,
+		cpp_substring_ranges *ranges)
 {
   /* Values of \a \b \e \f \n \r \t \v respectively.  */
 #if HOST_CHARSET == HOST_CHARSET_ASCII
@@ -1273,20 +1395,28 @@  convert_escape (cpp_reader *pfile, const uchar *from, const uchar *limit,
 
   uchar c;
 
+  /* Record the location of the backslash.  */
+  source_range char_range;
+  if (loc_reader)
+    char_range = loc_reader->get_next ();
+
   c = *from;
   switch (c)
     {
       /* UCNs, hex escapes, and octal escapes are processed separately.  */
     case 'u': case 'U':
-      return convert_ucn (pfile, from, limit, tbuf, cvt);
+      return convert_ucn (pfile, from, limit, tbuf, cvt,
+			  char_range, loc_reader, ranges);
 
     case 'x':
-      return convert_hex (pfile, from, limit, tbuf, cvt);
+      return convert_hex (pfile, from, limit, tbuf, cvt,
+			  char_range, loc_reader, ranges);
       break;
 
     case '0':  case '1':  case '2':  case '3':
     case '4':  case '5':  case '6':  case '7':
-      return convert_oct (pfile, from, limit, tbuf, cvt);
+      return convert_oct (pfile, from, limit, tbuf, cvt,
+			  char_range, loc_reader, ranges);
 
       /* Various letter escapes.  Get the appropriate host-charset
 	 value into C.  */
@@ -1338,10 +1468,11 @@  convert_escape (cpp_reader *pfile, const uchar *from, const uchar *limit,
 	}
     }
 
-  /* Now convert what we have to the execution character set.  */
-  if (!APPLY_CONVERSION (cvt, &c, 1, tbuf))
-    cpp_errno (pfile, CPP_DL_ERROR,
-	       "converting escape sequence to execution character set");
+  if (tbuf)
+    /* Now convert what we have to the execution character set.  */
+    if (!APPLY_CONVERSION (cvt, &c, 1, tbuf))
+      cpp_errno (pfile, CPP_DL_ERROR,
+		 "converting escape sequence to execution character set");
 
   return from + 1;
 }
@@ -1374,28 +1505,50 @@  converter_for_type (cpp_reader *pfile, enum cpp_ttype type)
    are to be converted from the source to the execution character set,
    escape sequences translated, and finally all are to be
    concatenated.  WIDE indicates whether or not to produce a wide
-   string.  The result is written into TO.  Returns true for success,
-   false for failure.  */
-bool
-cpp_interpret_string (cpp_reader *pfile, const cpp_string *from, size_t count,
-		      cpp_string *to,  enum cpp_ttype type)
+   string.  If TO is non-NULL, the result is written into TO.
+   If LOC_READERS and OUT are non-NULL, then location information
+   is read from LOC_READERS (which must be an array of length COUNT),
+   and location information is written to *RANGES.
+   Returns true for success, false for failure.  */
+static bool
+cpp_interpret_string_1 (cpp_reader *pfile, const cpp_string *from, size_t count,
+			cpp_string *to,  enum cpp_ttype type,
+			cpp_string_location_reader *loc_readers,
+			cpp_substring_ranges *out)
 {
   struct _cpp_strbuf tbuf;
   const uchar *p, *base, *limit;
   size_t i;
   struct cset_converter cvt = converter_for_type (pfile, type);
 
-  tbuf.asize = MAX (OUTBUF_BLOCK_SIZE, from->len);
-  tbuf.text = XNEWVEC (uchar, tbuf.asize);
-  tbuf.len = 0;
+  /* loc_readers and out must either be both NULL, or both be non-NULL.  */
+  gcc_assert ((loc_readers != NULL) == (out != NULL));
+
+  if (to)
+    {
+      tbuf.asize = MAX (OUTBUF_BLOCK_SIZE, from->len);
+      tbuf.text = XNEWVEC (uchar, tbuf.asize);
+      tbuf.len = 0;
+    }
 
   for (i = 0; i < count; i++)
     {
+      cpp_string_location_reader *loc_reader = NULL;
+      if (loc_readers)
+	loc_reader = &loc_readers[i];
+
       p = from[i].text;
       if (*p == 'u')
 	{
-	  if (*++p == '8')
-	    p++;
+	  p++;
+	  if (loc_reader)
+	    loc_reader->get_next ();
+	  if (*p == '8')
+	    {
+	      p++;
+	      if (loc_reader)
+		loc_reader->get_next ();
+	    }
 	}
       else if (*p == 'L' || *p == 'U') p++;
       if (*p == 'R')
@@ -1414,13 +1567,26 @@  cpp_interpret_string (cpp_reader *pfile, const cpp_string *from, size_t count,
 
 	  /* Raw strings are all normal characters; these can be fed
 	     directly to convert_cset.  */
-	  if (!APPLY_CONVERSION (cvt, p, limit - p, &tbuf))
-	    goto fail;
+	  if (to)
+	    if (!APPLY_CONVERSION (cvt, p, limit - p, &tbuf))
+	      goto fail;
+
+	  if (loc_reader)
+	    /* FIXME: If generating source ranges, assume we have a 1:1
+	       correspondence between bytes in the source encoding and bytes
+	       in the execution encoding (e.g. if we have a UTF-8 to UTF-8
+	       conversion), so that the run of byte in the source file
+	       corresponds to a run of bytes in the execution string.  */
+	    out->add_n_ranges (limit - p, *loc_reader);
 
 	  continue;
 	}
 
-      p++; /* Skip leading quote.  */
+      /* Skip leading quote.  */
+      p++;
+      if (loc_reader)
+	loc_reader->get_next ();
+
       limit = from[i].text + from[i].len - 1; /* Skip trailing quote.  */
 
       for (;;)
@@ -1432,29 +1598,80 @@  cpp_interpret_string (cpp_reader *pfile, const cpp_string *from, size_t count,
 	    {
 	      /* We have a run of normal characters; these can be fed
 		 directly to convert_cset.  */
-	      if (!APPLY_CONVERSION (cvt, base, p - base, &tbuf))
-		goto fail;
+	      if (to)
+		if (!APPLY_CONVERSION (cvt, base, p - base, &tbuf))
+		  goto fail;
+	    /* FIXME: similar to above: this assumes we have a 1:1
+	       correspondence between bytes in the source encoding and bytes
+	       in the execution encoding.  */
+	      if (loc_reader)
+		out->add_n_ranges (p - base, *loc_reader);
 	    }
 	  if (p == limit)
 	    break;
 
-	  p = convert_escape (pfile, p + 1, limit, &tbuf, cvt);
+	  struct _cpp_strbuf *tbuf_ptr = to ? &tbuf : NULL;
+	  p = convert_escape (pfile, p + 1, limit, tbuf_ptr, cvt,
+			      loc_reader, out);
 	}
     }
-  /* NUL-terminate the 'to' buffer and translate it to a cpp_string
-     structure.  */
-  emit_numeric_escape (pfile, 0, &tbuf, cvt);
-  tbuf.text = XRESIZEVEC (uchar, tbuf.text, tbuf.len);
-  to->text = tbuf.text;
-  to->len = tbuf.len;
+
+  if (to)
+    {
+      /* NUL-terminate the 'to' buffer and translate it to a cpp_string
+	 structure.  */
+      emit_numeric_escape (pfile, 0, &tbuf, cvt);
+      tbuf.text = XRESIZEVEC (uchar, tbuf.text, tbuf.len);
+      to->text = tbuf.text;
+      to->len = tbuf.len;
+    }
+
   return true;
 
  fail:
   cpp_errno (pfile, CPP_DL_ERROR, "converting to execution character set");
-  free (tbuf.text);
+  if (to)
+    free (tbuf.text);
   return false;
 }
 
+/* FROM is an array of cpp_string structures of length COUNT.  These
+   are to be converted from the source to the execution character set,
+   escape sequences translated, and finally all are to be
+   concatenated.  WIDE indicates whether or not to produce a wide
+   string.  The result is written into TO.  Returns true for success,
+   false for failure.  */
+bool
+cpp_interpret_string (cpp_reader *pfile, const cpp_string *from, size_t count,
+		      cpp_string *to,  enum cpp_ttype type)
+{
+  return cpp_interpret_string_1 (pfile, from, count, to, type, NULL, NULL);
+}
+
+/* This function mimics the behavior of cpp_interpret_string, but
+   rather than generating a string in the execution character set,
+   *OUT is written to with the source code ranges of the characters
+   in such a string.
+   FROM and LOC_READERS should both be arrays of length COUNT.
+   Returns true for success, false for failure.  */
+
+bool
+cpp_interpret_string_ranges (cpp_reader *pfile, const cpp_string *from,
+			     cpp_string_location_reader *loc_readers,
+			     size_t count,
+			     cpp_substring_ranges *out)
+{
+  /* SOURCE_CHARSET "UTF-8" */
+#if HOST_CHARSET == HOST_CHARSET_ASCII
+
+  /* We assume UTF-8 to UTF-8 conversion.  */
+  return cpp_interpret_string_1 (pfile, from, count, NULL, CPP_STRING,
+				 loc_readers, out);
+#else
+  return false;
+#endif
+}
+
 /* Subroutine of do_line and do_linemarker.  Convert escape sequences
    in a string, but do not perform character set conversion.  */
 bool
@@ -1818,3 +2035,39 @@  _cpp_default_encoding (void)
 
   return current_encoding;
 }
+
+/* Implementation of class cpp_string_location_reader.  */
+
+/* Constructor for cpp_string_location_reader.  */
+
+cpp_string_location_reader::
+cpp_string_location_reader (source_location src_loc,
+			    line_maps *line_table)
+: m_line_table (line_table)
+{
+  src_loc = get_range_from_loc (line_table, src_loc).m_start;
+
+  /* SRC_LOC might be a macro location.  It only makes sense to do
+     column-by-column calculations on ordinary maps, so get the
+     corresponding location in an ordinary map.  */
+  m_loc
+    = linemap_resolve_location (line_table, src_loc,
+				LRK_SPELLING_LOCATION, NULL);
+
+  const line_map_ordinary *map
+    = linemap_check_ordinary (linemap_lookup (line_table, m_loc));
+  m_offset_per_column = (1 << map->m_range_bits);
+}
+
+/* Get the range of the next source byte.  */
+
+source_range
+cpp_string_location_reader::get_next ()
+{
+  source_range result;
+  result.m_start = m_loc;
+  result.m_finish = m_loc;
+  if (m_loc <= LINE_MAP_MAX_LOCATION_WITH_COLS)
+    m_loc += m_offset_per_column;
+  return result;
+}
diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
index 543f3b9..b6dd39c 100644
--- a/libcpp/include/cpplib.h
+++ b/libcpp/include/cpplib.h
@@ -743,6 +743,51 @@  struct GTY(()) cpp_hashnode {
   union _cpp_hashnode_value GTY ((desc ("CPP_HASHNODE_VALUE_IDX (%1)"))) value;
 };
 
+/* A class for iterating through the source locations within a
+   string token (before escapes are interpreted, and before
+   concatenation).  */
+
+class cpp_string_location_reader {
+ public:
+  cpp_string_location_reader (source_location src_loc,
+			      line_maps *line_table);
+
+  source_range get_next ();
+
+ private:
+  source_location m_loc;
+  int m_offset_per_column;
+  line_maps *m_line_table;
+};
+
+/* A class for storing the source ranges of all of the characters within
+   a string literal, after escapes are interpreted, and after
+   concatenation.
+
+   This is not GTY-marked, as instances are intended to be temporary.  */
+
+class cpp_substring_ranges
+{
+ public:
+  cpp_substring_ranges ();
+  ~cpp_substring_ranges ();
+
+  int get_num_ranges () const { return m_num_ranges; }
+  source_range get_range (int idx) const
+  {
+    linemap_assert (idx < m_num_ranges);
+    return m_ranges[idx];
+  }
+
+  void add_range (source_range range);
+  void add_n_ranges (int num, cpp_string_location_reader &loc_reader);
+
+ private:
+  source_range *m_ranges;
+  int m_num_ranges;
+  int m_alloc_ranges;
+};
+
 /* Call this first to get a handle to pass to other functions.
 
    If you want cpplib to manage its own hashtable, pass in a NULL
@@ -829,6 +874,11 @@  extern cppchar_t cpp_interpret_charconst (cpp_reader *, const cpp_token *,
 extern bool cpp_interpret_string (cpp_reader *,
 				  const cpp_string *, size_t,
 				  cpp_string *, enum cpp_ttype);
+extern bool cpp_interpret_string_ranges (cpp_reader *pfile,
+					 const cpp_string *from,
+					 cpp_string_location_reader *readers,
+					 size_t count,
+					 cpp_substring_ranges *out);
 extern bool cpp_interpret_string_notranslate (cpp_reader *,
 					      const cpp_string *, size_t,
 					      cpp_string *, enum cpp_ttype);
diff --git a/libcpp/internal.h b/libcpp/internal.h
index ca2b498..4a5cd3c 100644
--- a/libcpp/internal.h
+++ b/libcpp/internal.h
@@ -754,7 +754,9 @@  struct normalize_state
 extern bool _cpp_valid_ucn (cpp_reader *, const unsigned char **,
 			    const unsigned char *, int,
 			    struct normalize_state *state,
-			    cppchar_t *);
+			    cppchar_t *,
+			    source_range *char_range,
+			    cpp_string_location_reader *loc_reader);
 extern void _cpp_destroy_iconv (cpp_reader *);
 extern unsigned char *_cpp_convert_input (cpp_reader *, const char *,
 					  unsigned char *, size_t, size_t,
diff --git a/libcpp/lex.c b/libcpp/lex.c
index 236418d..4e71965 100644
--- a/libcpp/lex.c
+++ b/libcpp/lex.c
@@ -1247,7 +1247,7 @@  forms_identifier_p (cpp_reader *pfile, int first,
       cppchar_t s;
       buffer->cur += 2;
       if (_cpp_valid_ucn (pfile, &buffer->cur, buffer->rlimit, 1 + !first,
-			  state, &s))
+			  state, &s, NULL, NULL))
 	return true;
       buffer->cur -= 2;
     }