[2/6] Generate virtual locations for tokens

This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it. If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to
speak. First it modifies the macro expander to make it create a macro
map for each macro expansion. It then allocates a virtual location for
each resulting token. As a result, a new kind of token (struct
cpp_expanded_token) is created. That token is just a normal token plus
a virtual location. That expanded token is then pushed onto a new kind
of context, suprisingly named an expanded token context. cpp_get_token
is modified to recognize expanded tokens from extended token contexts
and return the normal tokens from there. A similar modification was
done to cpp_get_token_with_location but this one now returns the
virtual location of the expanded token. Note that expanded tokens
don't leak out from libcpp. cpp_get_token and
cpp_get_token_with_location shield the rest of the world from that
implementation detail.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

Our beloved expand_location has been modified to resolve virtual
locations to spelling locations -- when macro expansion tracking is
turned on. Otherwise the behaviour is unchanged compared to before.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

The combination of this patch and the previous one bootstraps with
--enable-languages=all,ada and passes regression tests on
x86_64-unknown-linux-gnu.

gcc/
	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
	preprocessor related options.

gcc/c-family/

	* c.opt (ftrack-macro-expansion): New option. Handle it with and
	without argument.
	* c-opts.c (c_common_handle_option)<case
	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
	cases. Handle -ftrack-macro-expansion with and without argument.

libcpp/

	* include/cpplib.h (struct cpp_options)<track_macro_expansion>:
	New option.
	* internal.h (struct cpp_extended_token): New struct.
	(enum context_tokens_kind): New enum.
	(struct cpp_context)<tokens_kind>: New member of type enum
	context_tokens_kind.
	(_cpp_remaining_tokens_num_in_context): Declare new function.
	* init.c (cpp_create_reader): Initialize the base context to zero.
	(_cpp_remaining_tokens_num_in_context): Define new function.
	(_cpp_token_from_context_at): Define new static function.
	(cpp_peek_token): Use new _cpp_remaining_tokens_num_in_context and
	_cpp_token_from_context_at.
	* macro.c (enum macro_arg_token_kind): New enum.
	(struct macro_arg_token_iter): New struct.
	(maybe_adjust_loc_for_trad_cpp, push_extended_tokens_context)
	(alloc_expanded_args_mem, ensure_expanded_args_room)
	(alloc_args_buff, extend_arg_tokens_room, set_arg_token)
	(get_arg_token_location, arg_token_ptr_at, arg_token_at)
	(next_macro_token_ptr, prev_macro_token_ptr, macro_token_ptr_at)
	(next_arg_location_ptr, macro_arg_token_iter_init)
	(macro_arg_token_iter_get_token)
	(macro_arg_token_iter_get_location, macro_arg_token_iter_forward)
	(expanded_token_index, tokens_buff_new, tokens_buff_count)
	(tokens_buff_last_token_ptr, tokens_buff_put_token_to)
	(tokens_buff_append_token, tokens_buff_remove_last_token)
	(reached_end_of_context, consume_next_token_from_context): New
	static functions.
	(cpp_get_token_1): New static function. Splitted and extended from ...
	(cpp_get_token): ... here.
	(stringify_arg): Use the new arg_token_at.
	(paste_tokens): Support the new cpp_extended_token struct kind of
	tokens in the macro token stream. Use the new next_macro_token_ptr
	function.
	(collect_args): Use the new alloc_args_buff to allocate the buffer
	holding the tokens of the arguments. Use arg_token_ptr_at and
	arg_token_at to access the tokens of an argument. Use
	extend_arg_tokens_room to grow the argument tokens buffer. Use
	cpp_get_token_1 to get virtual locations of the tokens of the
	arguments. Use set_arg_token to set a token into the tokens buffer
	of the argument.
	(expand_arg): Use the new alloc_expanded_args_mem,
	push_extended_tokens_context and set_arg_token.
	(_cpp_pop_context): Really free the memory hold by the context.
	(replace_args): Use the new tokens_buff_new, linemap_enter_macro,
	macro_arg_token_iter_init, macro_arg_token_iter_get_token,
	macro_arg_token_iter_forward, tokens_buff_remove_last_token,
	tokens_buff_count, tokens_buff_last_token_ptr,
	tokens_buff_append_token, expanded_token_index,
	push_extended_tokens_context.
	(next_context, push_ptoken_context, _cpp_push_token_context)
	(_cpp_push_text_context): Propertly initialize the context.
	(enter_macro_context): Add a parameter for macro expansion point.
	Pass it to replace_args and to the "used" cpp callback.  When
	-ftrack-macro-expansion is in effect, create a macro map for the
	macro expansion and use it to allocate proper virtual locations
	for tokens resulting from the expansion.
	(cpp_get_token_with_location): Use cpp_get_token_1 and
	maybe_adjust_loc_for_trad_cpp.
	(_cpp_backup_tokens): Support the new kinds of token contexts.
---
 gcc/c-family/c-opts.c   |   13 +
 gcc/c-family/c.opt      |    8 +
 gcc/doc/cppopts.texi    |   17 +
 gcc/doc/invoke.texi     |    6 +-
 libcpp/include/cpplib.h |    8 +
 libcpp/init.c           |    1 +
 libcpp/internal.h       |   43 ++-
 libcpp/lex.c            |   44 ++-
 libcpp/macro.c          | 1169 ++++++++++++++++++++++++++++++++++++++++++-----
 9 files changed, 1187 insertions(+), 122 deletions(-)

[2/6] Generate virtual locations for tokens

Commit Message

Patch