[2/6] Generate virtual locations for tokens

This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it.  If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to speak.
First it modifies the macro expander to make it create a macro map for
each macro expansion. It then allocates a virtual location for each
resulting token.  Virtual locations of tokens resulting from macro
expansions are then stored on a special kind of context called an
"expanded tokens context".  In other words, in an expanded tokens
context, there are tokens resulting from macro expansion and their
associated virtual locations.  cpp_get_token_with_location is modified
to return the virtual location of tokens resulting from macro
expansion.  Note that once all tokens from an expanded token context have
been consumed and the context and is freed, the memory used to store the
virtual locations of the tokens held in that context is freed as well.
This helps reducing the overall peak memory consumption.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

gcc/ChangeLog
2011-10-15  Tom Tromey  <tromey@redhat.com>
	    Dodji Seketeli  <dodji@redhat.com>

	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
	preprocessor related options.

gcc/c-family/ChangeLog
2011-10-15  Tom Tromey  <tromey@redhat.com>
 	    Dodji Seketeli  <dodji@redhat.com>

	* c.opt (ftrack-macro-expansion): New option. Handle it with and
	without argument.
	* c-opts.c (c_common_handle_option)<case
	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
	cases. Handle -ftrack-macro-expansion with and without argument.

libcpp/ChangeLog
2011-10-15  Tom Tromey  <tromey@redhat.com>
	    Dodji Seketeli  <dodji@redhat.com>

	* include/cpplib.h (struct cpp_options)<track_macro_expansion>:
	New option.
	* internal.h (struct macro_context): New struct.
	(enum context_tokens_kind): New enum.
	(struct cpp_context)<tokens_kind>: New member of type enum
	context_tokens_kind.
	(struct cpp_context)<macro>: Remove this.  Replace it with an enum
	of macro and  macro_context.
	(struct cpp_context)<direct_p>: Remove.
	(_cpp_remaining_tokens_num_in_context): Declare new function.
	* directives.c (destringize_and_run): Adjust.
	* lex.c (_cpp_remaining_tokens_num_in_context)
	(_cpp_token_from_context_at): Define new functions
	(cpp_peek_token): Use them.
	* init.c (cpp_create_reader): Initialize the base context to zero.
	(_cpp_token_from_context_at): Define new static function.
	(cpp_peek_token): Use new _cpp_remaining_tokens_num_in_context and
	_cpp_token_from_context_at.
	* macro.c (struct macro_arg)<virt_locs, expanded_virt_locs>: New
	members.
	(enum macro_arg_token_kind): New enum.
	(struct macro_arg_token_iter): New struct.
	(maybe_adjust_loc_for_trad_cpp, push_extended_tokens_context)
	(alloc_expanded_arg_mem, ensure_expanded_arg_room)
	(delete_macro_args, set_arg_token, get_arg_token_location)
	(arg_token_ptr_at, macro_arg_token_iter_init)
	(macro_arg_token_iter_get_token)
	(macro_arg_token_iter_get_location, macro_arg_token_iter_forward)
	(expanded_token_index, tokens_buff_new, tokens_buff_count)
	(tokens_buff_last_token_ptr, tokens_buff_put_token_to)
	(tokens_buff_add_token, tokens_buff_remove_last_token)
	(reached_end_of_context, consume_next_token_from_context): New
	static functions.
	(cpp_get_token_1): New static function. Split and extended from
	cpp_get_token.  Use reached_end_of_context and
	consume_next_token_from_context.  Unify its return point.  Move
	the location tweaking from cpp_get_token_with_location in here.
	(cpp_get_token): Use cpp_get_token_1
	(stringify_arg): Use the new arg_token_at.
	(paste_all_tokens): Support tokens coming from extended tokens
	contexts.
	(collect_args): Return the number of collected arguments, by
	parameter.  Store virtual locations of tokens that constitute the
	collected args.
	(funlike_invocation_p): Return the number of collected arguments,
	by parameter.
	(enter_macro_context): Add a parameter for macro expansion point.
	Pass it to replace_args and to the "used" cpp callback.  Get the
	number of function-like macro arguments from funlike_invocation_p,
	pass it to the new delete_macro_args to free the memory used by
	macro args.  When -ftrack-macro-expansion is in effect, for macros
	that have no arguments, create a macro map for the macro expansion
	and use it to allocate proper virtual locations for tokens
	resulting from the expansion.  Push an extended tokens context
	containing the tokens resulting from macro expansion and their
	virtual locations.
	(replace_args): Rename the different variables named 'count' into
	variables with more meaningful names.  Create a macro map;
	allocate virtual locations of tokens resulting from this
	expansion.  Use macro_arg_token_iter to iterate over tokens of a
	given macro.  Handle the case of the argument of
	-ftrack-macro-expansion being < 2.  Don't free macro arguments
	memory resulting from expand_arg here, as these are freed by the
	caller of replace_arg using delete_macro_args now.  Push extended
	token context.
	(next_context, push_ptoken_context, _cpp_push_token_context)
	(_cpp_push_text_context): Properly initialize the context.
	(expand_arg): Use the new alloc_expanded_arg_mem,
	push_extended_tokens_context, cpp_get_token_1, and set_arg_token.
	(_cpp_pop_context): Really free the memory held by the context.
	Handle freeing memory used by extended tokens contexts.
	(cpp_get_token_with_location): Use cpp_get_token_1.
	(cpp_sys_macro_p): Adjust.
	(_cpp_backup_tokens): Support the new kinds of token contexts.
	* traditional.c (recursive_macro): Adjust.

---
 gcc/ChangeLog             |    7 +
 gcc/c-family/ChangeLog    |    9 +
 gcc/c-family/c-opts.c     |   12 +
 gcc/c-family/c.opt        |    8 +
 gcc/doc/cppopts.texi      |   18 +
 gcc/doc/invoke.texi       |    6 +-
 gcc/input.c               |    2 +-
 libcpp/ChangeLog          |   79 +++
 libcpp/directives.c       |    4 +-
 libcpp/include/cpplib.h   |    8 +
 libcpp/include/line-map.h |    2 +-
 libcpp/init.c             |    3 +-
 libcpp/internal.h         |   58 ++-
 libcpp/lex.c              |   41 ++-
 libcpp/line-map.c         |    2 +-
 libcpp/macro.c            | 1326 ++++++++++++++++++++++++++++++++++++++++-----
 libcpp/traditional.c      |    2 +-
 17 files changed, 1434 insertions(+), 153 deletions(-)

[2/6] Generate virtual locations for tokens

Commit Message

Patch