[2/7] Generate virtual locations for tokens

From: Dodji Seketeli <dodji@redhat.com>

Jason Merrill <jason@redhat.com> writes:

> On 08/09/2011 10:54 AM, Dodji Seketeli wrote:
> > +      goto ftrack_macro_expansion_with_arg;
> > +
> > +    case OPT_ftrack_macro_expansion_:
> > +    ftrack_macro_expansion_with_arg:
> 
> Instead of the goto, just write /* Fall through.  */

Fixed in the patch below.

> 
> > +consumption if necessary. Value @samp{0} of @var{level} de-activates
> > +this option just as if no @option{-ftrack-macro-expansion} was present
> > +on the command line. Value @samp{1} tracks tokens locations in a
> > +degraded mode for the sake of minimal memory overhead. In this mode
> > +all tokens resulting from the expansion of an argument of a
> > +function-like macro have the same location. Value @samp{2} tracks
> > +tokens locations completely. This value is the most memory hungry. It
> > +is the default value.
> 
> "It is the default value" sounds to me like no -ftrack-macro-expansion
> option is equivalent to -ftrack-macro-expansion=2, rather than =0.

I tried to propose a better wording in the attached patch.

> 
> > +     expansion of arguments of function-like macro. all macro
> > +     expansion. 2 Means we do track all macro expansions. This last
> 
> Seems like the "all macro expansion" is left over from a previous
> version of this sentence.

Oops, good catch.  Fixed.

> 
> > +/* This describes some additional data that is added to the macro
> > +   token context of type cpp_context, when -ftrack-macro-expansion is
> > +   on.  */
> > +typedef struct
> > +{
> > +  /* The node of the macro we are referring to.  */
> > +  cpp_hashnode *macro_node;
> > +  /* This buffer contains an array of virtual locations.  The virtual
> > +     location at index 0 is the virtual location of the token at index
> > +     0 in the current instance of cpp_context; similarly for all the
> > +     other virtual locations.  */
> > +  source_location *virt_locs;
> > +  /* This is a pointer to the current virtual location.  This is used
> > +     to iterate over the virtual locations while we iterate over the
> > +     tokens they belong to.  */
> > +  source_location *cur_virt_loc;
> > +} macro_context;
> 
> Why track virtual locations separately rather than use them directly
> as the src_loc of the tokens?

My understanding is that a cpp_token (yielded by the public entry points
cpp_get_token and cpp_get_token_with_location) is supposed to be
immutable.  E.g:

#define MACRO FOO //#0

MACRO  //#1

MACRO  //#2

The virtual locations of FOO in #1 and #2 must be different and be
able to coexist at a given point in time.  E.g, let T be the token
representing FOO.  Then suppose that a client code would get a hold on
T as a result of calling cpp_get_token (and assume that T->src_loc is
its virtual location) at #1.  Then once we reach #2, the T->src_loc
would magically change and be set to the location #2.  The client code
would mistakenly think that T->src_loc still points to #1.  Oops.

A given virtual location of a token, on the other hand, depends on the
particular macro expansion context of that token, at a point in time.
I guess that's why there is a dedicated function
(cpp_get_token_with_location as opposed to cpp_get_token) to get the
virtual location of that token in the current macro expansion context.

To comply with this invariant, the initial patch of Tom was cloning T
into T', and was using T'->src_loc to track the virtual location of T.
Which is close to what you are proposing, while respecting the
invariant.  But it turned out that was using too much memory :-(.  So
we devised this scheme instead.

Thanks.

From: Dodji Seketeli <dodji@redhat.com>
Date: Sat, 4 Dec 2010 14:04:29 +0100
Subject: [PATCH 2/7] Generate virtual locations for tokens

This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it.  If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to speak.
First it modifies the macro expander to make it create a macro map for
each macro expansion. It then allocates a virtual location for each
resulting token.  Virtual locations of tokens resulting from macro
expansions are then stored on a special kind of context called an
"expanded tokens context".  In other words, in an expanded tokens
context, there are tokens resulting from macro expansion and their
associated virtual locations.  cpp_get_token_with_location is modified
to return the virtual location of tokens resulting from macro
expansion.  Note that once all tokens from an expanded token context have
been consumed and the context and is freed, the memory used to store the
virtual locations of the tokens held in that context is freed as well.
This helps reducing the overall peak memory consumption.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

The combination of this patch and the previous one bootstraps with
	--enable-languages=all,ada and passes regression tests on
x86_64-unknown-linux-gnu.

gcc/
	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
	preprocessor related options.

gcc/c-family/

	* c.opt (ftrack-macro-expansion): New option. Handle it with and
	without argument.
	* c-opts.c (c_common_handle_option)<case
	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
	cases. Handle -ftrack-macro-expansion with and without argument.

libcpp/

	* include/cpplib.h (struct cpp_options)<track_macro_expansion>:
	New option.
	* internal.h (struct macro_context): New struct.
	(enum context_tokens_kind): New enum.
	(struct cpp_context)<tokens_kind>: New member of type enum
	context_tokens_kind.
	(struct cpp_context)<macro>: Change the type of this to void.
	(struct cpp_context)<direct_p>: Remove.
	(_cpp_remaining_tokens_num_in_context): Declare new function.
	* lex.c (_cpp_remaining_tokens_num_in_context)
	(_cpp_token_from_context_at): Define new functions
	(cpp_peek_token): Use them.
	* init.c (cpp_create_reader): Initialize the base context to zero.
	(_cpp_token_from_context_at): Define new static function.
	(cpp_peek_token): Use new _cpp_remaining_tokens_num_in_context and
	_cpp_token_from_context_at.
	* macro.c (struct macro_arg)<expanded_capacity, virt_locs>:
	(struct macro_arg)<virt_locs_capacity, expanded_virt_locs>: New
	members.
	(enum macro_arg_token_kind): New enum.
	(struct macro_arg_token_iter): New struct.
	(maybe_adjust_loc_for_trad_cpp, push_extended_tokens_context)
	(alloc_expanded_args_mem, ensure_expanded_args_room)
	(delete_macro_args, set_arg_token, get_arg_token_location)
	(arg_token_ptr_at, macro_arg_token_iter_init)
	(macro_arg_token_iter_get_token)
	(macro_arg_token_iter_get_location, macro_arg_token_iter_forward)
	(expanded_token_index, tokens_buff_new, tokens_buff_count)
	(tokens_buff_last_token_ptr, tokens_buff_put_token_to)
	(tokens_buff_append_token, tokens_buff_remove_last_token)
	(reached_end_of_context, consume_next_token_from_context): New
	static functions.
	(cpp_get_token_1): New static function. Split and extended from
	cpp_get_token.  Use reached_end_of_context and
	consume_next_token_from_context.
	(cpp_get_token): Use cpp_get_token_1
	(stringify_arg): Use the new arg_token_at.
	(paste_all_tokens): Support tokens coming from extended tokens
	contexts.
	(collect_args): Return the number of collected arguments, by
	parameter.  Store virtual locations of tokens that constitute the
	collected args.
	(funlike_invocation_p): Return the number of collected arguments,
	by parameter.
	(enter_macro_context): Add a parameter for macro expansion point.
	Pass it to replace_args and to the "used" cpp callback.  Get the
	number of function-like macro arguments from funlike_invocation_p,
	pass it to the new delete_macro_args to free the memory used by
	macro args.  When -ftrack-macro-expansion is in effect, for macros
	that have no arguments, create a macro map for the macro expansion
	and use it to allocate proper virtual locations for tokens
	resulting from the expansion.  Push an extended tokens context
	containing the tokens resulting from macro expansion and their
	virtual locations.
	(replace_args): Rename the different variables named 'count' into
	variables with more meaningful names.  Create a macro map;
	allocate virtual locations of tokens resulting from this
	expansion.  Use macro_arg_token_iter to iterate over tokens of a
	given macro.  Handle the case of the argument of
	-ftrack-macro-expansion being < 2.  Don't free macro arguments
	memory resulting from expand_arg here, as these are freed by the
	caller of replace_arg using delete_macro_args now.  Push extended
	token context.
	(next_context, push_ptoken_context, _cpp_push_token_context)
	(_cpp_push_text_context): Properly initialize the context.
	(expand_arg): Use the new alloc_expanded_args_mem,
	push_extended_tokens_context, cpp_get_token_1, and set_arg_token.
	(_cpp_pop_context): Really free the memory held by the context.
	Handle freeing memory used by extended tokens contexts.
	(cpp_get_token_with_location): Use cpp_get_token_1 and
	maybe_adjust_loc_for_trad_cpp.
	(_cpp_backup_tokens): Support the new kinds of token contexts.
---
 gcc/c-family/c-opts.c   |   12 +
 gcc/c-family/c.opt      |    8 +
 gcc/doc/cppopts.texi    |   18 +
 gcc/doc/invoke.texi     |    6 +-
 libcpp/include/cpplib.h |    8 +
 libcpp/init.c           |    1 +
 libcpp/internal.h       |   54 ++-
 libcpp/lex.c            |   41 ++-
 libcpp/macro.c          | 1308 ++++++++++++++++++++++++++++++++++++++++++-----
 9 files changed, 1321 insertions(+), 135 deletions(-)

[2/7] Generate virtual locations for tokens

Commit Message

Comments

Patch