Patchwork [2/6] Generate virtual locations for tokens

login
register
mail settings
Submitter Dodji Seketeli
Date Dec. 10, 2010, 11:11 a.m.
Message ID <1291979498-1604-4-git-send-email-dodji@redhat.com>
Download mbox | patch
Permalink /patch/75080/
State New
Headers show

Comments

Dodji Seketeli - Dec. 10, 2010, 11:11 a.m.
This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it. If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to
speak. First it modifies the macro expander to make it create a macro
map for each macro expansion. It then allocates a virtual location for
each resulting token. As a result, a new kind of token (struct
cpp_expanded_token) is created. That token is just a normal token plus
a virtual location. That expanded token is then pushed onto a new kind
of context, suprisingly named an expanded token context. cpp_get_token
is modified to recognize expanded tokens from extended token contexts
and return the normal tokens from there. A similar modification was
done to cpp_get_token_with_location but this one now returns the
virtual location of the expanded token. Note that expanded tokens
don't leak out from libcpp. cpp_get_token and
cpp_get_token_with_location shield the rest of the world from that
implementation detail.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

Our beloved expand_location has been modified to resolve virtual
locations to spelling locations -- when macro expansion tracking is
turned on. Otherwise the behaviour is unchanged compared to before.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

The combination of this patch and the previous one bootstraps with
--enable-languages=all,ada and passes regression tests on
x86_64-unknown-linux-gnu.

gcc/
	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
	preprocessor related options.

gcc/c-family/

	* c.opt (ftrack-macro-expansion): New option. Handle it with and
	without argument.
	* c-opts.c (c_common_handle_option)<case
	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
	cases. Handle -ftrack-macro-expansion with and without argument.

libcpp/

	* include/cpplib.h (struct cpp_options)<track_macro_expansion>:
	New option.
	* internal.h (struct cpp_extended_token): New struct.
	(enum context_tokens_kind): New enum.
	(struct cpp_context)<tokens_kind>: New member of type enum
	context_tokens_kind.
	(_cpp_remaining_tokens_num_in_context): Declare new function.
	* init.c (cpp_create_reader): Initialize the base context to zero.
	(_cpp_remaining_tokens_num_in_context): Define new function.
	(_cpp_token_from_context_at): Define new static function.
	(cpp_peek_token): Use new _cpp_remaining_tokens_num_in_context and
	_cpp_token_from_context_at.
	* macro.c (enum macro_arg_token_kind): New enum.
	(struct macro_arg_token_iter): New struct.
	(maybe_adjust_loc_for_trad_cpp, push_extended_tokens_context)
	(alloc_expanded_args_mem, ensure_expanded_args_room)
	(alloc_args_buff, extend_arg_tokens_room, set_arg_token)
	(get_arg_token_location, arg_token_ptr_at, arg_token_at)
	(next_macro_token_ptr, prev_macro_token_ptr, macro_token_ptr_at)
	(next_arg_location_ptr, macro_arg_token_iter_init)
	(macro_arg_token_iter_get_token)
	(macro_arg_token_iter_get_location, macro_arg_token_iter_forward)
	(expanded_token_index, tokens_buff_new, tokens_buff_count)
	(tokens_buff_last_token_ptr, tokens_buff_put_token_to)
	(tokens_buff_append_token, tokens_buff_remove_last_token)
	(reached_end_of_context, consume_next_token_from_context): New
	static functions.
	(cpp_get_token_1): New static function. Splitted and extended from ...
	(cpp_get_token): ... here.
	(stringify_arg): Use the new arg_token_at.
	(paste_tokens): Support the new cpp_extended_token struct kind of
	tokens in the macro token stream. Use the new next_macro_token_ptr
	function.
	(collect_args): Use the new alloc_args_buff to allocate the buffer
	holding the tokens of the arguments. Use arg_token_ptr_at and
	arg_token_at to access the tokens of an argument. Use
	extend_arg_tokens_room to grow the argument tokens buffer. Use
	cpp_get_token_1 to get virtual locations of the tokens of the
	arguments. Use set_arg_token to set a token into the tokens buffer
	of the argument.
	(expand_arg): Use the new alloc_expanded_args_mem,
	push_extended_tokens_context and set_arg_token.
	(_cpp_pop_context): Really free the memory hold by the context.
	(replace_args): Use the new tokens_buff_new, linemap_enter_macro,
	macro_arg_token_iter_init, macro_arg_token_iter_get_token,
	macro_arg_token_iter_forward, tokens_buff_remove_last_token,
	tokens_buff_count, tokens_buff_last_token_ptr,
	tokens_buff_append_token, expanded_token_index,
	push_extended_tokens_context.
	(next_context, push_ptoken_context, _cpp_push_token_context)
	(_cpp_push_text_context): Propertly initialize the context.
	(enter_macro_context): Add a parameter for macro expansion point.
	Pass it to replace_args and to the "used" cpp callback.  When
	-ftrack-macro-expansion is in effect, create a macro map for the
	macro expansion and use it to allocate proper virtual locations
	for tokens resulting from the expansion.
	(cpp_get_token_with_location): Use cpp_get_token_1 and
	maybe_adjust_loc_for_trad_cpp.
	(_cpp_backup_tokens): Support the new kinds of token contexts.
---
 gcc/c-family/c-opts.c   |   13 +
 gcc/c-family/c.opt      |    8 +
 gcc/doc/cppopts.texi    |   17 +
 gcc/doc/invoke.texi     |    6 +-
 libcpp/include/cpplib.h |    8 +
 libcpp/init.c           |    1 +
 libcpp/internal.h       |   43 ++-
 libcpp/lex.c            |   44 ++-
 libcpp/macro.c          | 1169 ++++++++++++++++++++++++++++++++++++++++++-----
 9 files changed, 1187 insertions(+), 122 deletions(-)

Patch

diff --git a/gcc/c-family/c-opts.c b/gcc/c-family/c-opts.c
index befd644..4c2e37e 100644
--- a/gcc/c-family/c-opts.c
+++ b/gcc/c-family/c-opts.c
@@ -621,6 +621,19 @@  c_common_handle_option (size_t scode, const char *arg, int value,
       cpp_opts->preprocessed = value;
       break;
 
+    case OPT_ftrack_macro_expansion:
+      if (value)
+	value = 2;
+      goto ftrack_macro_expansion_with_arg;
+
+    case OPT_ftrack_macro_expansion_:
+    ftrack_macro_expansion_with_arg:
+      if (arg && *arg != '\0')
+	cpp_opts->track_macro_expansion = value;
+      else
+	cpp_opts->track_macro_expansion = 2;
+      break;
+
     case OPT_frepo:
       flag_use_repository = value;
       if (value)
diff --git a/gcc/c-family/c.opt b/gcc/c-family/c.opt
index 8682471..25d0cb6 100644
--- a/gcc/c-family/c.opt
+++ b/gcc/c-family/c.opt
@@ -909,6 +909,14 @@  fpreprocessed
 C ObjC C++ ObjC++
 Treat the input file as already preprocessed
 
+ftrack-macro-expansion
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+; converted into ftrack-macro-expansion=
+
+ftrack-macro-expansion=
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+-ftrack-macro-expansion=<0|1|2>  Track locations of tokens coming from macro expansion and display them in error messages
+
 fpretty-templates
 C++ ObjC++ Var(flag_pretty_templates) Init(1)
 -fno-pretty-templates Do not pretty-print template specializations as the template signature followed by the arguments
diff --git a/gcc/doc/cppopts.texi b/gcc/doc/cppopts.texi
index 57624ed..a4f8b32 100644
--- a/gcc/doc/cppopts.texi
+++ b/gcc/doc/cppopts.texi
@@ -583,6 +583,23 @@  correct column numbers in warnings or errors, even if tabs appear on the
 line.  If the value is less than 1 or greater than 100, the option is
 ignored.  The default is 8.
 
+@item -ftrack-macro-expansion@r{[}=@var{level}@r{]}
+@opindex ftrack-macro-expansion
+Track locations of tokens across macro expansions. This allows the
+compiler to emit diagnostic about the current macro expansion stack
+when a compilation error occurs in a macro expansion. Using this
+option makes the preprocessor and the compiler consume more
+memory. The @var{level} parameter can be used to choose the level of
+precision of token location tracking thus decreasing the memory
+consumption if necessary. Value @samp{0} of @var{level} de-activates
+this option just as if no @option{-ftrack-macro-expansion} was present
+on the command line. Value @samp{1} tracks tokens locations in a
+degraded mode for the sake of minimal memory overhead. In this mode
+all tokens resulting from the expansion of an argument of a
+function-like macro have the same location. Value @samp{2} tracks
+tokens locations completely. This value is the most memory hungry. It
+is the default value.
+
 @item -fexec-charset=@var{charset}
 @opindex fexec-charset
 @cindex character set, execution
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 7c80415..07f2b20 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -415,9 +415,9 @@  Objective-C and Objective-C++ Dialects}.
 -iwithprefixbefore @var{dir}  -isystem @var{dir} @gol
 -imultilib @var{dir} -isysroot @var{dir} @gol
 -M  -MM  -MF  -MG  -MP  -MQ  -MT  -nostdinc  @gol
--P  -fworking-directory  -remap @gol
--trigraphs  -undef  -U@var{macro}  -Wp,@var{option} @gol
--Xpreprocessor @var{option}}
+-P -ftrack-macro-expansion -fworking-directory @gol
+-remap -trigraphs  -undef  -U@var{macro}  @gol
+-Wp,@var{option} -Xpreprocessor @var{option}}
 
 @item Assembler Option
 @xref{Assembler Options,,Passing Options to the Assembler}.
diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
index 8fa2881..783576b 100644
--- a/libcpp/include/cpplib.h
+++ b/libcpp/include/cpplib.h
@@ -389,6 +389,14 @@  struct cpp_options
      bother trying to do macro expansion and whatnot.  */
   unsigned char preprocessed;
 
+  /* Nonzero means we are tracking locations of tokens involved in
+     macro expansion. 1 Means we track the location in degraded mode
+     where we do not track locations of tokens resulting from the
+     expansion of arguments of function-like macro. all macro
+     expansion. 2 Means we do track all macro expansions. This last
+     option is the one that consumes the highest amount of memory.  */
+  unsigned char track_macro_expansion;
+
   /* Nonzero means handle C++ alternate operator names.  */
   unsigned char operator_names;
 
diff --git a/libcpp/init.c b/libcpp/init.c
index 8836143..02023e8 100644
--- a/libcpp/init.c
+++ b/libcpp/init.c
@@ -151,6 +151,7 @@  cpp_create_reader (enum c_lang lang, hash_table *table,
   init_library ();
 
   pfile = XCNEW (cpp_reader);
+  memset (&pfile->base_context, 0, sizeof (pfile->base_context));
 
   cpp_set_lang (pfile, lang);
   CPP_OPTION (pfile, warn_multichar) = 1;
diff --git a/libcpp/internal.h b/libcpp/internal.h
index f02c878..9124276 100644
--- a/libcpp/internal.h
+++ b/libcpp/internal.h
@@ -139,6 +139,44 @@  struct tokenrun
 #define CUR(c) ((c)->u.trad.cur)
 #define RLIMIT(c) ((c)->u.trad.rlimit)
 
+
+/* This is a "special" token representation that reuses a "normal"
+   token and adds a location. The location is useful when the tokens
+   results from macro expansion. In that case LOCATION encodes the
+   locus of the token in the macro definition as well as the locus of
+   the token at the macro expansion point. This representation is
+   used when -ftrack-macro-expansion is on, and for tokens resulting
+   from macro expansion only.  */
+struct cpp_extended_token
+{
+  const cpp_token *token;
+  /* The location of the token in the context of the macro expansion.
+     This is different from the location of the token member above
+     which would be the location of the token in the context of the
+     definition of the macro.  */
+  source_location location;
+};
+typedef struct cpp_extended_token cpp_ext_token;
+
+/* The kind of tokens carried by a cpp_context.  */
+enum context_tokens_kind {
+  /* This is the value of cpp_context::tokens_kind if u.iso.first
+     contains an instance of cpp_token **.  */
+  TOKENS_KIND_INDIRECT,
+  /* This is the value of cpp_context::tokens_kind if u.iso.first
+     contains an instance of cpp_token *.  */
+  TOKENS_KIND_DIRECT,
+  /* This is the value of cpp_context::tokens_kind if u.iso.first
+     contains an instance of cpp_extended_token. This is used only
+     when the -ftrack-macro-expansion flag is on. Note that in this
+     case the static type of u.iso.first is still considered to be
+     cpp_token ** even though each element of the array pointed to by
+     u.iso.first are effectively instances of cpp_exted_token. This is
+     a hack done so that the size of u.iso.first remains "small" when
+     the flag -ftrack-macro-expansion is not used.  */
+  TOKENS_KIND_EXTENDED
+};
+
 typedef struct cpp_context cpp_context;
 struct cpp_context
 {
@@ -171,8 +209,8 @@  struct cpp_context
   /* For a macro context, the macro node, otherwise NULL.  */
   cpp_hashnode *macro;
 
-  /* True if utoken element is token, else ptoken.  */
-  bool direct_p;
+  /* This determines the type of tokens hold by this context.  */
+  enum context_tokens_kind tokens_kind;
 };
 
 struct lexer_state
@@ -601,6 +639,7 @@  extern cpp_token *_cpp_lex_direct (cpp_reader *);
 extern int _cpp_equiv_tokens (const cpp_token *, const cpp_token *);
 extern void _cpp_init_tokenrun (tokenrun *, unsigned int);
 extern cpp_hashnode *_cpp_lex_identifier (cpp_reader *, const char *);
+extern int _cpp_remaining_tokens_num_in_context (cpp_reader *);
 
 /* In init.c.  */
 extern void _cpp_maybe_push_include_file (cpp_reader *);
diff --git a/libcpp/lex.c b/libcpp/lex.c
index bcf292c..3071952 100644
--- a/libcpp/lex.c
+++ b/libcpp/lex.c
@@ -1719,6 +1719,41 @@  next_tokenrun (tokenrun *run)
   return run->next;
 }
 
+/* Return the number of not yet processed token in the the current
+   context.  */
+int
+_cpp_remaining_tokens_num_in_context (cpp_reader *pfile)
+{
+  cpp_context *context = pfile->context;
+  if (context->tokens_kind == TOKENS_KIND_DIRECT)
+    return (LAST (context).token - FIRST (context).token) / sizeof (cpp_token);
+  else if (context->tokens_kind == TOKENS_KIND_INDIRECT)
+    return (LAST (context).ptoken - FIRST (context).ptoken) / sizeof (cpp_token *);
+  else if (context->tokens_kind == TOKENS_KIND_EXTENDED)
+    return (LAST (context).ptoken - FIRST (context).ptoken) / sizeof (cpp_ext_token);
+  else
+      abort ();
+}
+
+/* Returns the token present at index INDEX in the current context.
+   If INDEX is zero, the next token to be processed is returned.  */
+static const cpp_token*
+_cpp_token_from_context_at (cpp_reader *pfile, int index)
+{
+  cpp_context *context = pfile->context;
+  if (context->tokens_kind == TOKENS_KIND_DIRECT)
+    return &(FIRST (context).token[index]);
+  else if (context->tokens_kind == TOKENS_KIND_INDIRECT)
+    return FIRST (context).ptoken[index];
+  else if (context->tokens_kind == TOKENS_KIND_EXTENDED)
+    {
+      cpp_ext_token *ptr = (cpp_ext_token *) FIRST (context).ptoken;
+      return ptr[index].token;
+    }
+ else
+   abort ();
+}
+
 /* Look ahead in the input stream.  */
 const cpp_token *
 cpp_peek_token (cpp_reader *pfile, int index)
@@ -1730,15 +1765,10 @@  cpp_peek_token (cpp_reader *pfile, int index)
   /* First, scan through any pending cpp_context objects.  */
   while (context->prev)
     {
-      ptrdiff_t sz = (context->direct_p
-                      ? LAST (context).token - FIRST (context).token
-                      : LAST (context).ptoken - FIRST (context).ptoken);
+      ptrdiff_t sz = _cpp_remaining_tokens_num_in_context (pfile);
 
       if (index < (int) sz)
-        return (context->direct_p
-                ? FIRST (context).token + index
-                : *(FIRST (context).ptoken + index));
-
+        return _cpp_token_from_context_at (pfile, index);
       index -= (int) sz;
       context = context->prev;
     }
diff --git a/libcpp/macro.c b/libcpp/macro.c
index 419114c..edb59e9 100644
--- a/libcpp/macro.c
+++ b/libcpp/macro.c
@@ -37,15 +37,55 @@  struct macro_arg
   const cpp_token *stringified;	/* Stringified argument.  */
   unsigned int count;		/* # of tokens in argument.  */
   unsigned int expanded_count;	/* # of tokens in expanded argument.  */
+  size_t expanded_capacity;       /* total size of expanded array.  */
+};
+
+/* The kind of macro tokens which the instance of
+   macro_arg_token_iter is supposed to iterate over.  */
+enum macro_arg_token_kind {
+  MACRO_ARG_TOKEN_NORMAL,
+  /* This is a macro argument token that got transformed into a string
+     litteral, e.g. #foo.  */
+  MACRO_ARG_TOKEN_STRINGIFIED,
+  /* This is a token resulting from the expansion of a macro
+     argument that was itself a macro.  */
+  MACRO_ARG_TOKEN_EXPANDED
+};
+
+/* An iterator over token coming from a function line macro
+   argument.  */
+typedef struct macro_arg_token_iter macro_arg_token_iter;
+struct macro_arg_token_iter
+{
+  /* The cpp_reader the macro comes from.  */
+  cpp_reader *pfile;
+  /* The kind of token which we are supposed to iterator over.  */
+  enum macro_arg_token_kind kind;
+  /* The function-like macro the tokens come from.  */
+  const macro_arg *arg;
+  /* A pointer to the current token pointed to by the iterator.  */
+  const cpp_token **token_ptr;
+  /* A pointer to the "full" location of the current token. If
+     -ftrack-macro-expansion is used this location tracks locuses
+     accross macro expansion.  */
+  const source_location *location_ptr;
+#ifdef ENABLE_CHECKING
+  /* The number of times the iterator went forward. This useful only
+     when checking is enabled.  */
+  size_t num_forwards;
+#endif
 };
 
 /* Macro expansion.  */
 
 static int enter_macro_context (cpp_reader *, cpp_hashnode *,
-				const cpp_token *);
+				const cpp_token *, source_location);
 static int builtin_macro (cpp_reader *, cpp_hashnode *);
 static void push_ptoken_context (cpp_reader *, cpp_hashnode *, _cpp_buff *,
 				 const cpp_token **, unsigned int);
+static void push_extended_tokens_context (cpp_reader *, cpp_hashnode *,
+					  _cpp_buff *, const cpp_token **,
+					  unsigned int);
 static _cpp_buff *collect_args (cpp_reader *, const cpp_hashnode *,
 				_cpp_buff **);
 static cpp_context *next_context (cpp_reader *);
@@ -55,8 +95,50 @@  static const cpp_token *new_string_token (cpp_reader *, uchar *, unsigned int);
 static const cpp_token *stringify_arg (cpp_reader *, macro_arg *);
 static void paste_all_tokens (cpp_reader *, const cpp_token *);
 static bool paste_tokens (cpp_reader *, const cpp_token **, const cpp_token *);
+static void alloc_expanded_args_mem (cpp_reader *, macro_arg *, size_t);
+static void ensure_expanded_args_room (cpp_reader *, macro_arg *, size_t);
+static _cpp_buff *alloc_args_buff (cpp_reader *, size_t);
+static _cpp_buff *extend_arg_tokens_room (cpp_reader *, _cpp_buff *, size_t);
+static void set_arg_token (cpp_reader *, macro_arg *, const cpp_token *,
+			   source_location, size_t, enum macro_arg_token_kind);
+static const source_location *get_arg_token_location (cpp_reader *,
+						      const macro_arg *,
+						      enum macro_arg_token_kind);
+static const cpp_token **arg_token_ptr_at (cpp_reader *, const macro_arg *,
+					   size_t, enum macro_arg_token_kind);
+static const cpp_token *arg_token_at (cpp_reader *, macro_arg *, size_t,
+				      enum macro_arg_token_kind);
+static const cpp_token **next_macro_token_ptr (cpp_reader *, const cpp_token **);
+static const cpp_token **prev_macro_token_ptr (cpp_reader *, const cpp_token **);
+static const cpp_token **macro_token_ptr_at (cpp_reader *, const cpp_token **, int);
+static source_location *next_arg_location_ptr (cpp_reader *,
+					       const source_location *);
+static void macro_arg_token_iter_init (macro_arg_token_iter *, cpp_reader*,
+				       enum macro_arg_token_kind,
+				       const macro_arg *, const cpp_token **);
+static const cpp_token *macro_arg_token_iter_get_token
+(const macro_arg_token_iter *it);
+static source_location macro_arg_token_iter_get_location
+(const macro_arg_token_iter *);
+static void macro_arg_token_iter_forward (macro_arg_token_iter *);
+static _cpp_buff *tokens_buff_new (cpp_reader *, size_t);
+static size_t tokens_buff_count (cpp_reader *, _cpp_buff *);
+static const cpp_token **tokens_buff_last_token_ptr (cpp_reader *,
+						      _cpp_buff *);
+static const cpp_token **tokens_buff_put_token_to
+(cpp_reader *, const cpp_token **, const cpp_token *, source_location,
+ source_location, const struct line_map *, unsigned int *);
+
+static const cpp_token **tokens_buff_append_token (cpp_reader *,
+						   _cpp_buff *,
+						   const cpp_token *,
+						   source_location,
+						   source_location,
+						   const struct line_map *,
+						   unsigned int *);
+static void tokens_buff_remove_last_token (cpp_reader *, _cpp_buff *);
 static void replace_args (cpp_reader *, cpp_hashnode *, cpp_macro *,
-			  macro_arg *);
+			  macro_arg *, source_location);
 static _cpp_buff *funlike_invocation_p (cpp_reader *, cpp_hashnode *,
 					_cpp_buff **);
 static bool create_iso_definition (cpp_reader *, cpp_macro *);
@@ -70,6 +152,11 @@  static bool warn_of_redefinition (cpp_reader *, cpp_hashnode *,
 static bool parse_params (cpp_reader *, cpp_macro *);
 static void check_trad_stringification (cpp_reader *, const cpp_macro *,
 					const cpp_string *);
+static bool reached_end_of_context (cpp_context *);
+static void consume_next_token_from_context (cpp_reader *pfile,
+					     const cpp_token **,
+					     source_location *);
+static const cpp_token* cpp_get_token_1 (cpp_reader *, source_location *);
 
 /* Emits a warning if NODE is a macro defined in the main file that
    has not been used.  */
@@ -369,7 +456,8 @@  stringify_arg (cpp_reader *pfile, macro_arg *arg)
   /* Loop, reading in the argument's tokens.  */
   for (i = 0; i < arg->count; i++)
     {
-      const cpp_token *token = arg->first[i];
+      const cpp_token *token = arg_token_at (pfile, arg, i,
+					     MACRO_ARG_TOKEN_NORMAL);
 
       if (token->type == CPP_PADDING)
 	{
@@ -511,7 +599,7 @@  paste_tokens (cpp_reader *pfile, const cpp_token **plhs, const cpp_token *rhs)
 static void
 paste_all_tokens (cpp_reader *pfile, const cpp_token *lhs)
 {
-  const cpp_token *rhs;
+  const cpp_token *rhs = NULL;
   cpp_context *context = pfile->context;
 
   do
@@ -521,10 +609,16 @@  paste_all_tokens (cpp_reader *pfile, const cpp_token *lhs)
 	 object-like macro, or a function-like macro with arguments
 	 inserted.  In either case, the constraints to #define
 	 guarantee we have at least one more token.  */
-      if (context->direct_p)
+      if (context->tokens_kind == TOKENS_KIND_DIRECT)
 	rhs = FIRST (context).token++;
-      else
+      else if (context->tokens_kind == TOKENS_KIND_INDIRECT)
 	rhs = *FIRST (context).ptoken++;
+      else if (context->tokens_kind == TOKENS_KIND_EXTENDED)
+	{
+	  rhs = ((cpp_ext_token *) FIRST (context).ptoken)->token;
+	  FIRST (context).ptoken =
+	    next_macro_token_ptr (pfile, FIRST (context).ptoken);
+	}
 
       if (rhs->type == CPP_PADDING)
 	{
@@ -598,14 +692,16 @@  collect_args (cpp_reader *pfile, const cpp_hashnode *node,
   macro_arg *args, *arg;
   const cpp_token *token;
   unsigned int argc;
+  source_location virt_loc;
 
   macro = node->value.macro;
   if (macro->paramc)
     argc = macro->paramc;
   else
     argc = 1;
-  buff = _cpp_get_buff (pfile, argc * (50 * sizeof (cpp_token *)
-				       + sizeof (macro_arg)));
+
+  buff = alloc_args_buff (pfile, argc);
+
   base_buff = buff;
   args = (macro_arg *) buff->base;
   memset (args, 0, argc * sizeof (macro_arg));
@@ -626,15 +722,16 @@  collect_args (cpp_reader *pfile, const cpp_hashnode *node,
       for (;;)
 	{
 	  /* Require space for 2 new tokens (including a CPP_EOF).  */
-	  if ((unsigned char *) &arg->first[ntokens + 2] > buff->limit)
+	  if ((unsigned char *) arg_token_ptr_at (pfile, arg, ntokens + 2,
+						  MACRO_ARG_TOKEN_NORMAL)
+	      > buff->limit)
 	    {
-	      buff = _cpp_append_extend_buff (pfile, buff,
-					      1000 * sizeof (cpp_token *));
+	      buff = extend_arg_tokens_room (pfile, buff, 1000);
 	      arg->first = (const cpp_token **) buff->cur;
 	    }
 
-	  token = cpp_get_token (pfile);
-
+	  token = cpp_get_token_1 (pfile, &virt_loc);
+	  
 	  if (token->type == CPP_PADDING)
 	    {
 	      /* Drop leading padding.  */
@@ -690,7 +787,7 @@  collect_args (cpp_reader *pfile, const cpp_hashnode *node,
 		  BUFF_FRONT (*pragma_buff) += sizeof (cpp_token *);
 		  if (token->type == CPP_PRAGMA_EOL)
 		    break;
-		  token = cpp_get_token (pfile);
+		  token = cpp_get_token_1 (pfile, &virt_loc);
 		}
 	      while (token->type != CPP_EOF);
 
@@ -704,22 +801,28 @@  collect_args (cpp_reader *pfile, const cpp_hashnode *node,
 	      else
 		continue;
 	    }
-
-	  arg->first[ntokens++] = token;
+	  set_arg_token (pfile, arg, token, virt_loc,
+			 ntokens, MACRO_ARG_TOKEN_NORMAL);
+	  ntokens++;
 	}
 
       /* Drop trailing padding.  */
-      while (ntokens > 0 && arg->first[ntokens - 1]->type == CPP_PADDING)
+      while (ntokens > 0
+	     && arg_token_at (pfile, arg, ntokens - 1,
+			      MACRO_ARG_TOKEN_NORMAL)->type == CPP_PADDING)
 	ntokens--;
 
       arg->count = ntokens;
-      arg->first[ntokens] = &pfile->eof;
+      set_arg_token (pfile, arg, &pfile->eof, pfile->eof.src_loc,
+		     ntokens, MACRO_ARG_TOKEN_NORMAL);
 
       /* Terminate the argument.  Excess arguments loop back and
 	 overwrite the final legitimate argument, before failing.  */
       if (argc <= macro->paramc)
 	{
-	  buff->cur = (unsigned char *) &arg->first[ntokens + 1];
+	  buff->cur =
+	    (unsigned char *) arg_token_ptr_at (pfile, arg, ntokens + 1,
+						MACRO_ARG_TOKEN_NORMAL);
 	  if (argc != macro->paramc)
 	    arg++;
 	}
@@ -823,13 +926,15 @@  macro_real_token_count (const cpp_macro *macro)
 /* Push the context of a macro with hash entry NODE onto the context
    stack.  If we can successfully expand the macro, we push a context
    containing its yet-to-be-rescanned replacement list and return one.
-   If there were additionally any unexpanded deferred #pragma directives
-   among macro arguments, push another context containing the
-   pragma tokens before the yet-to-be-rescanned replacement list
-   and return two.  Otherwise, we don't push a context and return zero.  */
+   If there were additionally any unexpanded deferred #pragma
+   directives among macro arguments, push another context containing
+   the pragma tokens before the yet-to-be-rescanned replacement list
+   and return two.  Otherwise, we don't push a context and return
+   zero. LOCATION is the location of the expansion point of the
+   macro.  */
 static int
 enter_macro_context (cpp_reader *pfile, cpp_hashnode *node,
-		     const cpp_token *result)
+		     const cpp_token *token, source_location location)
 {
   /* The presence of a macro invalidates a file's controlling macro.  */
   pfile->mi_valid = false;
@@ -877,8 +982,13 @@  enter_macro_context (cpp_reader *pfile, cpp_hashnode *node,
 	    }
 
 	  if (macro->paramc > 0)
-	    replace_args (pfile, node, macro, (macro_arg *) buff->base);
-	  _cpp_release_buff (pfile, buff);
+	    replace_args (pfile, node, macro,
+			  (macro_arg *) buff->base,
+			  location);
+	  /* Unlike _cpp_release_buff, calling _cpp_free_buff
+	     actually frees the memory for real. This reduces peak
+	     memory usage on source code with a lot of macros.  */
+	  _cpp_free_buff (buff);
 	}
 
       /* Disable the macro within its expansion.  */
@@ -892,19 +1002,47 @@  enter_macro_context (cpp_reader *pfile, cpp_hashnode *node,
 	}
 
       if (pfile->cb.used)
-	pfile->cb.used (pfile, result->src_loc, node);
+	pfile->cb.used (pfile, location, node);
 
       macro->used = 1;
 
       if (macro->paramc == 0)
-	_cpp_push_token_context (pfile, node, macro->exp.tokens,
-				 macro_real_token_count (macro));
+	{
+	  if (CPP_OPTION (pfile, track_macro_expansion))
+	    {
+	      unsigned int i, count = macro->count;
+	      const cpp_token *src = macro->exp.tokens;
+	      const struct line_map *map;
+	      _cpp_buff *macro_tokens =
+		tokens_buff_new (pfile, count);
+		
+	      /* Create a macro map to record the locations of the
+		 tokens that are involved in the expansion. LOCATION
+		 is the location of the macro expansion point.  */
+	      map  = linemap_enter_macro (pfile->line_table,
+					  macro, location, count);
+	      for (i = 0; i < count; ++i)
+		{
+		  tokens_buff_append_token (pfile, macro_tokens, src,
+					    src->src_loc, src->src_loc,
+					    map, &i);
+		  ++src;
+		}
+	      push_extended_tokens_context (pfile, node, macro_tokens,
+					    (const cpp_token **)
+					    macro_tokens->base,
+					    count);
+	    }
+	  else
+	    _cpp_push_token_context (pfile, node, macro->exp.tokens,
+				     macro_real_token_count (macro));
+	}
 
       if (pragma_buff)
 	{
 	  if (!pfile->state.in_directive)
 	    _cpp_push_token_context (pfile, NULL,
-				     padding_token (pfile, result), 1);
+				     padding_token (pfile, token), 1);
 	  do
 	    {
 	      _cpp_buff *tail = pragma_buff->next;
@@ -926,33 +1064,336 @@  enter_macro_context (cpp_reader *pfile, cpp_hashnode *node,
   return builtin_macro (pfile, node);
 }
 
+/* Allocates a buffer to hold tokens coming from macro arguments. If
+   -ftrack-macro-expansion is used the buffer holds instances of
+   cpp_ext_token, otherwise it holds instance of cpp_token *.  The
+   buffer is big enough to contain NUM_ARGS instances of macro_arg
+   data, and for each instance of macro_arg can contain 50 tokens that
+   can be accessed from the macro_arg::first pointer.  */
+static _cpp_buff*
+alloc_args_buff (cpp_reader *pfile, size_t num_args)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  size_t arg_size;
+
+  if (track_macro_exp_p)
+    arg_size = 50 * sizeof (cpp_ext_token) + sizeof (macro_arg);
+  else
+    arg_size = 50 * sizeof (cpp_token *) + sizeof (macro_arg);
+
+  return _cpp_get_buff (pfile, num_args * arg_size);
+}
+
+/* Enlarges the memory room pointed to by the macro_arg::first pointer
+   to make it capable of containing NUM_ELEMS more worth of tokens.
+
+   Creates a new buffer big enough to hold NUM_ELEMS tokens as well as
+   a the uncommitted tokens remaining in BUFF, chain that new buffer
+   to the end of BUFF, and return it. BUFF->CUR must be the memory room to
+   store tokens belonging to a function-like macro argument (the
+   macro_arg::first pointer).  */
+static _cpp_buff *
+extend_arg_tokens_room (cpp_reader *pfile, _cpp_buff *buff, size_t num_elems)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  size_t token_size =
+    (track_macro_exp_p) ? sizeof (cpp_ext_token) : sizeof (cpp_token *);
+
+  return _cpp_append_extend_buff (pfile, buff, num_elems * token_size);
+}
+
+/* Set the INDEXth token of the macro argument ARG. TOKEN is the
+   token to set, LOCATION is its location. If TOKEN comes from a
+   macro expansion and if -ftrack-macro-location is in effect,
+   LOCATION must be the location that encodes locuses accross macro
+   expansion. Otherwise it has to be TOKEN->SRC_LOC. KIND is the kind
+   of tokens the argument ARG is supposed to contain.  */
+static void
+set_arg_token (cpp_reader *pfile, macro_arg *arg, const cpp_token *token,
+	       source_location location, size_t index,
+	       enum macro_arg_token_kind kind)
+{  
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  const cpp_token **token_ptr;
+
+  token_ptr = arg_token_ptr_at (pfile, arg, index, kind);
+  if (track_macro_exp_p)
+    {
+      ((cpp_ext_token *) token_ptr)->token = (cpp_token *) token;
+      ((cpp_ext_token *) token_ptr)->location = location;
+    }
+  else
+    *token_ptr = token;
+}
+
+/* Get the pointer to the location of the argument token of the
+   function-like macro argument ARG.  */
+static const source_location *
+get_arg_token_location (cpp_reader *pfile,
+			const macro_arg *arg,
+			enum macro_arg_token_kind kind)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  const cpp_token **token_ptr = arg_token_ptr_at (pfile, arg, 0, kind);
+
+  if (token_ptr == NULL)
+    return NULL;
+
+  if (track_macro_exp_p)
+      return &((cpp_ext_token *) token_ptr)->location;
+  return (source_location *) &((*token_ptr)->src_loc);
+
+}
+
+/* Return the pointer to the INDEXth token of the macro argument ARG.
+   KIND specifies the kind of token the macro argument ARG
+   contains.  */
+static const cpp_token **
+arg_token_ptr_at (cpp_reader *pfile, const macro_arg *arg,
+		  size_t index, enum macro_arg_token_kind kind)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  const cpp_token **ptr = NULL;
+
+  switch (kind)
+    {
+    case MACRO_ARG_TOKEN_NORMAL:
+      ptr = arg->first;
+      break;
+    case MACRO_ARG_TOKEN_STRINGIFIED:
+      ptr = (const cpp_token **) &arg->stringified;
+      break;
+    case MACRO_ARG_TOKEN_EXPANDED:
+      ptr = arg->expanded;
+      break;
+    }
+
+  if (track_macro_exp_p)
+    return (const cpp_token **) &((cpp_ext_token *) ptr)[index].token;
+  else
+    return &ptr[index];
+}
+
+/* Return the INDEXth token of the macro argument ARG.  KIND specifies
+   the kind of token the macro argument ARG contains.  */
+static const cpp_token *
+arg_token_at (cpp_reader *pfile, macro_arg *arg, size_t index,
+	      enum macro_arg_token_kind kind)
+{
+  return *arg_token_ptr_at (pfile, arg, index, kind);
+}
+
+/* Return the address of the token coming right after *TOKEN_PTR. The
+   function handles the different types of tokens. If
+   -ftrack-macro-expansion is used, the token pointed to by TOKEN_PTR
+   is considered to be an instance of cpp_ext_token otherwise it's a
+   cpp_token*.  */
+static const cpp_token **
+next_macro_token_ptr (cpp_reader *pfile, const cpp_token **token_ptr)
+{
+  return macro_token_ptr_at (pfile, token_ptr, 1);
+}
+
+/* Return the address of the token coming right before *TOKEN_PTR. The
+   function handles the different types of tokens. If
+   -ftrack-macro-expansion is used, the token pointed to by TOKEN_PTR
+   is considered to be an instance of cpp_ext_token otherwise it's a
+   cpp_token*.  */
+static const cpp_token **
+prev_macro_token_ptr (cpp_reader *pfile, const cpp_token **token_ptr)
+{
+  return macro_token_ptr_at (pfile, token_ptr, -1);
+}
+
+/* Return the address of the token at index INDEX, starting from
+   TOKEN_PTR. The function handles the different types of tokens. If
+   -ftrack-macro-expansion is used, the token pointed to by TOKEN_PTR
+   is considered to be an instance of cpp_ext_token otherwise it's a
+   cpp_token *.  */
+static const cpp_token **
+macro_token_ptr_at (cpp_reader *pfile,
+		    const cpp_token **token_ptr,
+		    int index)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (track_macro_exp_p)
+    return &((cpp_ext_token *) (token_ptr))[index].token;
+  return &token_ptr[index];
+}
+
+/* Return the address of the location of the token coming right after
+   the token which location is pointed to by LOCATION. If
+   -ftrack-macro-expansion is used, the token is considered to be an
+   instance of cpp_ext_token otherwise it's a cpp_token *.  */
+static source_location *
+next_arg_location_ptr (cpp_reader *pfile,
+		       const source_location *location)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (track_macro_exp_p)
+    {
+      cpp_ext_token *base;
+      base = (cpp_ext_token *)
+	((unsigned char*) location - offsetof (cpp_ext_token, location));
+      return &base[1].location;
+    }
+  else
+    abort ();
+}
+
+/* Initialize an iterator so that it iterates over the tokens of a
+   function-like macro argument.  KIND is the kind of tokens we want
+   ITER to iterate over. TOKEN_PTR points the first token ITER will
+   iterate over.  */
+static void
+macro_arg_token_iter_init (macro_arg_token_iter *iter,
+			   cpp_reader *pfile,
+			   enum macro_arg_token_kind kind,
+			   const macro_arg *arg,
+			   const cpp_token **token_ptr)
+{
+  iter->pfile = pfile;
+  iter->kind = kind;
+  iter->arg = arg;
+  iter->token_ptr = token_ptr;
+  iter->location_ptr = get_arg_token_location (pfile, arg, kind);
+#ifdef ENABLE_CHECKING
+  iter->num_forwards = 0;
+#endif
+}
+
+/* Move the iterator one token forward. Note that if IT was
+   initialized on an argument that has a stringified token, moving it
+   foward doesn't make sense as a stringified token is essentially one
+   string.  */
+static void
+macro_arg_token_iter_forward (macro_arg_token_iter *it)
+{
+  bool track_macro_exp_p = CPP_OPTION (it->pfile,
+				       track_macro_expansion);
+
+  switch (it->kind)
+    {
+    case MACRO_ARG_TOKEN_NORMAL:
+    case MACRO_ARG_TOKEN_EXPANDED:
+      it->token_ptr = next_macro_token_ptr (it->pfile,
+					    it->token_ptr);
+      if (track_macro_exp_p)
+	it->location_ptr =
+	  next_arg_location_ptr (it->pfile,
+				 it->location_ptr);
+      break;
+    case MACRO_ARG_TOKEN_STRINGIFIED:
+#ifdef ENABLE_CHECKING
+      if (it->num_forwards > 0)
+	abort ();
+      it->num_forwards++;
+#endif
+      break;
+    }
+}
+
+/* Return the token pointed to by the iterator.  */
+static const cpp_token *
+macro_arg_token_iter_get_token (const macro_arg_token_iter *it)
+{
+#ifdef ENABLE_CHECKING
+  if (it->kind == MACRO_ARG_TOKEN_STRINGIFIED
+      && it->num_forwards > 0)
+    abort ();
+#endif
+  if (it->token_ptr == NULL)
+    return NULL;
+  return *it->token_ptr;
+}
+
+/* Return the location of the token pointed to by the iterator.*/
+static source_location
+macro_arg_token_iter_get_location (const macro_arg_token_iter *it)
+{
+#ifdef ENABLE_CHECKING
+  if (it->kind == MACRO_ARG_TOKEN_STRINGIFIED
+      && it->num_forwards > 0)
+    abort ();
+#endif
+  return *it->location_ptr;
+}
+
+/* Return the index of a token [resulting from macro expansion] inside
+   the total list of tokens resulting from a given macro
+   expansion. The index can be different depending on whether if we
+   want each tokens resulting from function-like macro arguments
+   expansion to have a different location or not.
+
+   E.g, consider this function like macro: 
+
+        #define M(x) x - 3
+
+   Then consider us "calling" it (and thus expanding it) like:
+   
+       M(1+2)
+
+   It will be expanded into:
+
+       1+2-3
+
+   Let's consider the case of the token '2'.
+
+   Its index can be 1 (it's the third token of the set of tokens
+   resulting from the expansion) or it can be 0 if we consider that
+   all tokens resulting from the expansion of the argument "1+2" have
+   the same index, which is 0. In this later case, the index of token
+   '-' would then be 1 and the index of token '3' would be 2.
+   
+   The later case is useful to use less memory e.g, for the case of
+   the user using the option -ftrack-macro-expansion=1.
+
+   This is a subroutine of replace_args.  */
+inline static unsigned
+expanded_token_index (cpp_reader *pfile, cpp_macro *macro,
+		      const cpp_token *cur_replacement_token,
+		      unsigned absolute_token_index)
+{
+  if (CPP_OPTION (pfile, track_macro_expansion) > 1)
+    return absolute_token_index;  
+  return cur_replacement_token - macro->exp.tokens;
+}
+
 /* Replace the parameters in a function-like macro of NODE with the
    actual ARGS, and place the result in a newly pushed token context.
    Expand each argument before replacing, unless it is operated upon
-   by the # or ## operators.  */
+   by the # or ## operators. EXPANSION_POINT_LOC is the location of
+   the expansion point of the macro. E.g, the location of the
+   function-like macro invocation.  */
 static void
-replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg *args)
+replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro,
+	      macro_arg *args, source_location expansion_point_loc)
 {
   unsigned int i, total;
   const cpp_token *src, *limit;
-  const cpp_token **dest, **first;
+  const cpp_token **first = NULL;
   macro_arg *arg;
-  _cpp_buff *buff;
-  unsigned int count;
+  _cpp_buff *buff = NULL;
+  unsigned int exp_count;
+  const struct line_map *map = NULL;
+  bool track_macro_exp_p;
 
   /* First, fully macro-expand arguments, calculating the number of
      tokens in the final expansion as we go.  The ordering of the if
      statements below is subtle; we must handle stringification before
      pasting.  */
-  count = macro_real_token_count (macro);
-  total = count;
-  limit = macro->exp.tokens + count;
+  exp_count = macro_real_token_count (macro);
+  total = exp_count;
+  limit = macro->exp.tokens + exp_count;
 
   for (src = macro->exp.tokens; src < limit; src++)
     if (src->type == CPP_MACRO_ARG)
       {
 	/* Leading and trailing padding tokens.  */
 	total += 2;
+	exp_count += 2;
 
 	/* We have an argument.  If it is not being stringified or
 	   pasted it is macro-replaced before insertion.  */
@@ -974,67 +1415,189 @@  replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg
 	  }
       }
 
+  /* When the compiler is called with the -ftrack-macro-expansion
+     flag, we need to keep track of the location of each token that
+     results from macro expansion.
+
+     A token resulting from macro expansion is not a new token. It is
+     simply the same token as the token coming from the macro
+     definition. The new things that are allocated are the buffer that
+     holds the tokens resulting from macro expansion and a new
+     location that records many things like the locus of the expansion
+     point as well as the original locus inside the definition of the
+     macro. This is basically what the buffer buff holds: a set of
+     cpp_token* and a location.
+
+     The memory allocated to store the tokens and their locations is
+     going to be freed once the context of macro expansion is popped.
+     
+     As far as tokens are concerned, the memory overhead of
+     -ftrack-macro-expansion is proportional to the number of
+     macros that get expanded multiplied by sizeof (source_location).
+     The good news is that extra memory gets freed when the macro
+     context is freed, i.e shortly after the macro got expanded.  */
+
+  /* Is the -ftrack-macro-expansion flag in effect?  */
+  track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
   /* Now allocate space for the expansion, copy the tokens and replace
-     the arguments.  */
-  buff = _cpp_get_buff (pfile, total * sizeof (cpp_token *));
+     the arguments.  This memory must be freed when the context of
+     the macro MACRO is popped.  */
+  buff = tokens_buff_new (pfile, total);
+
   first = (const cpp_token **) buff->base;
-  dest = first;
 
+  /* Create a macro map to record the locations of the tokens that are
+     involved in the expansion. Note that the expansion point is set
+     to the location of the closing parenthesis. Otherwise, the
+     subsequent map created for the first token that comes after the
+     macro map might have a wrong line number. What would lead to
+     tokens with wrong line numbers after the macro expansion. This
+     adds up to the memory overhead of the -ftrack-macro-expansion
+     flag; for every macro that is expanded, a "macro map" is
+     created.  */
+  if (track_macro_exp_p)
+    {
+      int num_macro_tokens = total;
+      if (track_macro_exp_p < 2)
+	/* Then the number of macro tokens won't take in account the
+	   fact that function-like macro arguments can expand to
+	   multiple tokens. This is to save memory at the expense of
+	   accuracy.
+
+	   Suppose we have #define SQARE(A) A * A
+
+	   And then we do SQARE(2+3)
+
+	   Then the tokens 2, +, 3, will have the same location, saying they come
+	   from the expansion of the argument A.  */
+	num_macro_tokens = exp_count;
+      map = linemap_enter_macro (pfile->line_table, macro,
+				 expansion_point_loc,
+				 num_macro_tokens);
+    }
+  i = 0;
   for (src = macro->exp.tokens; src < limit; src++)
     {
-      unsigned int count;
-      const cpp_token **from, **paste_flag;
+      unsigned int arg_tokens_count;
+      macro_arg_token_iter from;
+      const cpp_token **paste_flag = NULL;
+      const cpp_token **tmp_token_ptr;
 
       if (src->type != CPP_MACRO_ARG)
 	{
-	  *dest++ = src;
+	  unsigned index = expanded_token_index (pfile, macro, src, i);
+	  tokens_buff_append_token (pfile, buff, src,
+				    src->src_loc, src->src_loc,
+				    map, &index);
+	  i += 1;
 	  continue;
 	}
 
       paste_flag = 0;
       arg = &args[src->val.macro_arg.arg_no - 1];
       if (src->flags & STRINGIFY_ARG)
-	count = 1, from = &arg->stringified;
+	{
+	  arg_tokens_count = 1;
+	  macro_arg_token_iter_init (&from, pfile,
+				     MACRO_ARG_TOKEN_STRINGIFIED,
+				     arg, &arg->stringified);
+	}
       else if (src->flags & PASTE_LEFT)
-	count = arg->count, from = arg->first;
+	{
+	  arg_tokens_count = arg->count;
+	  macro_arg_token_iter_init (&from, pfile,
+				     MACRO_ARG_TOKEN_NORMAL,
+				     arg, arg->first);
+	}
       else if (src != macro->exp.tokens && (src[-1].flags & PASTE_LEFT))
 	{
-	  count = arg->count, from = arg->first;
-	  if (dest != first)
+	  int num_toks;
+	  arg_tokens_count = arg->count;
+	  macro_arg_token_iter_init (&from, pfile,
+				     MACRO_ARG_TOKEN_NORMAL,
+				     arg, arg->first);
+
+	  num_toks = tokens_buff_count (pfile, buff);
+
+	  if (num_toks != 0)
 	    {
-	      if (dest[-1]->type == CPP_COMMA
+	      tmp_token_ptr = tokens_buff_last_token_ptr (pfile, buff);
+
+	      if ((*tmp_token_ptr)->type == CPP_COMMA
 		  && macro->variadic
 		  && src->val.macro_arg.arg_no == macro->paramc)
 		{
 		  /* Swallow a pasted comma if from == NULL, otherwise
 		     drop the paste flag.  */
-		  if (from == NULL)
-		    dest--;
+		  if (macro_arg_token_iter_get_token (&from) == NULL)
+		    tokens_buff_remove_last_token (pfile, buff);
 		  else
-		    paste_flag = dest - 1;
+		    paste_flag = tmp_token_ptr;
 		}
-	      /* Remove the paste flag if the RHS is a placemarker.  */
-	      else if (count == 0)
-		paste_flag = dest - 1;
+	      else if (arg_tokens_count == 0)
+		paste_flag = tmp_token_ptr;
 	    }
 	}
       else
-	count = arg->expanded_count, from = arg->expanded;
+	{
+	  arg_tokens_count = arg->expanded_count;
+	  macro_arg_token_iter_init (&from, pfile,
+				     MACRO_ARG_TOKEN_EXPANDED,
+				     arg, arg->expanded);
+	}
 
       /* Padding on the left of an argument (unless RHS of ##).  */
       if ((!pfile->state.in_directive || pfile->state.directive_wants_padding)
 	  && src != macro->exp.tokens && !(src[-1].flags & PASTE_LEFT))
-	*dest++ = padding_token (pfile, src);
+	{
+	  const cpp_token *t = padding_token (pfile, src);
+	  unsigned index = expanded_token_index (pfile, macro, src, i);
+	  tokens_buff_append_token (pfile, buff, t,
+				    t->src_loc, t->src_loc,
+				    map, &index);
+	}
 
-      if (count)
+      if (arg_tokens_count)
 	{
-	  memcpy (dest, from, count * sizeof (cpp_token *));
-	  dest += count;
+	  unsigned int j;
+	  for (j = 0; j < arg_tokens_count; ++j)
+	    {
+	      /* So if track_macro_exp_p is < 2, the user wants to
+		 save extra memory while tracking macro expansion locations.
+	         So in that case here is what we do:
+
+		 Suppose we have #define SQARE(A) A * A
+
+		 And then we do SQARE(2+3)
+
+		 Then the tokens 2, +, 3, will have the same location, saying they come
+		 from the expansion of the argument A.
+
+	      So that means we are going to ignore the COUNT tokens
+	      resulting from the expansion of the current macro
+	      arugment. In other words all the COUNT tokens resulting
+	      from the expansion of the macro argument will have the
+	      index I. Normally, each of those token should have
+	      index I+J.  */
+	      unsigned token_index = i;
+	      unsigned index;
+	      if (track_macro_exp_p > 1)
+		token_index += j;
+
+	      index = expanded_token_index (pfile, macro, src, token_index);
+	      tokens_buff_append_token (pfile, buff,
+					macro_arg_token_iter_get_token (&from),
+					macro_arg_token_iter_get_location (&from),
+					src->src_loc, map, &index);
+	      macro_arg_token_iter_forward (&from);
+	    }
 
 	  /* With a non-empty argument on the LHS of ##, the last
 	     token should be flagged PASTE_LEFT.  */
 	  if (src->flags & PASTE_LEFT)
-	    paste_flag = dest - 1;
+	    paste_flag =
+	      (const cpp_token **) tokens_buff_last_token_ptr (pfile, buff);
 	}
       else if (CPP_PEDANTIC (pfile) && ! macro->syshdr
 	       && ! CPP_OPTION (pfile, c99)
@@ -1050,7 +1613,11 @@  replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg
 
       /* Avoid paste on RHS (even case count == 0).  */
       if (!pfile->state.in_directive && !(src->flags & PASTE_LEFT))
-	*dest++ = &pfile->avoid_paste;
+	{
+	  const cpp_token *t = &pfile->avoid_paste;
+	  tokens_buff_append_token (pfile, buff, t, t->src_loc,
+				    t->src_loc, NULL, NULL);
+	}
 
       /* Add a new paste flag, or remove an unwanted one.  */
       if (paste_flag)
@@ -1064,6 +1631,8 @@  replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg
 	    token->flags = (*paste_flag)->flags & ~PASTE_LEFT;
 	  *paste_flag = token;
 	}
+
+      i += arg_tokens_count;
     }
 
   /* Free the expanded arguments.  */
@@ -1071,7 +1640,12 @@  replace_args (cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg
     if (args[i].expanded)
       free (args[i].expanded);
 
-  push_ptoken_context (pfile, node, buff, first, dest - first);
+  if (track_macro_exp_p)
+    push_extended_tokens_context (pfile, node, buff, first,
+				  tokens_buff_count (pfile, buff));
+  else
+    push_ptoken_context (pfile, node, buff, first,
+			 tokens_buff_count (pfile, buff));
 }
 
 /* Return a special padding token, with padding inherited from SOURCE.  */
@@ -1099,6 +1673,7 @@  next_context (cpp_reader *pfile)
   if (result == 0)
     {
       result = XNEW (cpp_context);
+      memset (result, 0, sizeof (cpp_context));
       result->prev = pfile->context;
       result->next = 0;
       pfile->context->next = result;
@@ -1115,7 +1690,7 @@  push_ptoken_context (cpp_reader *pfile, cpp_hashnode *macro, _cpp_buff *buff,
 {
   cpp_context *context = next_context (pfile);
 
-  context->direct_p = false;
+  context->tokens_kind = TOKENS_KIND_INDIRECT;
   context->macro = macro;
   context->buff = buff;
   FIRST (context).ptoken = first;
@@ -1127,13 +1702,30 @@  void
 _cpp_push_token_context (cpp_reader *pfile, cpp_hashnode *macro,
 			 const cpp_token *first, unsigned int count)
 {
+   cpp_context *context = next_context (pfile);
+ 
+   context->tokens_kind = TOKENS_KIND_DIRECT;
+   context->macro = macro;
+   context->buff = NULL;
+  FIRST (context).token = first;
+  LAST (context).token = first + count;
+}
+
+/* Push a list of tokens.  */
+static void
+push_extended_tokens_context (cpp_reader *pfile, cpp_hashnode *macro,
+			      _cpp_buff *token_buff,
+			      const cpp_token **first,
+			      unsigned int count)
+{
   cpp_context *context = next_context (pfile);
 
-  context->direct_p = true;
+  context->tokens_kind = TOKENS_KIND_EXTENDED;
+  context->buff = token_buff;
   context->macro = macro;
-  context->buff = NULL;
-  FIRST (context).token = first;
-  LAST (context).token = first + count;
+  FIRST (context).ptoken = first;
+  LAST (context).ptoken =
+    macro_token_ptr_at (pfile, first, count);
 }
 
 /* Push a traditional macro's replacement text.  */
@@ -1143,7 +1735,7 @@  _cpp_push_text_context (cpp_reader *pfile, cpp_hashnode *macro,
 {
   cpp_context *context = next_context (pfile);
 
-  context->direct_p = true;
+  context->tokens_kind = TOKENS_KIND_DIRECT;
   context->macro = macro;
   context->buff = NULL;
   CUR (context) = start;
@@ -1151,6 +1743,196 @@  _cpp_push_text_context (cpp_reader *pfile, cpp_hashnode *macro,
   macro->flags |= NODE_DISABLED;
 }
 
+/* Creates a buffer that holds tokens a.k.a "token buffer", usually
+   for the purpose of storing them on a cpp_context. If the
+   -ftrack-macro-expansion flag is in effect the buffer holds a set of
+   cpp_ext_token. Otherwise it holds a set of cpp_token*.  */
+static _cpp_buff*
+tokens_buff_new (cpp_reader *pfile, size_t len)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  size_t size =
+    track_macro_exp_p
+    ? len * sizeof (cpp_ext_token)
+    : len * sizeof (cpp_token *);
+
+  return _cpp_get_buff (pfile, size);
+}
+
+/* Returns the number of tokens contained in a token buffer. If the
+   -ftrack-macro-expansion flag is in effect the buffer holds a set of
+   cpp_ext_token. Otherwise it holds a set of cpp_token*.  */
+static size_t
+tokens_buff_count (cpp_reader *pfile, _cpp_buff *buff)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  size_t size =
+    track_macro_exp_p
+    ? sizeof (cpp_ext_token)
+    : sizeof (cpp_token *);
+
+  return (BUFF_FRONT (buff) - buff->base) / size;
+}
+
+/* Return a pointer to the last token contained in the token buffer
+   BUFF. If the -ftrack-macro-expansion flag is in effect the buffer
+   holds a set of cpp_ext_token. Otherwise it holds a set of
+   cpp_token*.*/
+static const cpp_token **
+tokens_buff_last_token_ptr (cpp_reader *pfile, _cpp_buff *buff)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (track_macro_exp_p)
+    return &((cpp_ext_token *) BUFF_FRONT (buff))[-1].token;
+  return &((const cpp_token **) BUFF_FRONT (buff))[-1];
+}
+
+/* Remove the last token contained in the token buffer BUFF. If the
+   -ftrack-macro-expansion flag is in effect the buffer holds a set of
+   cpp_ext_token. Otherwise it holds a set of cpp_token*.*/
+static void
+tokens_buff_remove_last_token (cpp_reader *pfile, _cpp_buff *buff)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (BUFF_FRONT (buff) > buff->base)
+    {
+      if (track_macro_exp_p)
+	BUFF_FRONT (buff) =
+	  (unsigned char *) &((cpp_ext_token *) BUFF_FRONT (buff))[-1];
+      else
+	BUFF_FRONT (buff) =
+	  (unsigned char *) &((const cpp_token **) BUFF_FRONT (buff))[-1];
+    }
+}
+
+/* Insert a token into the token buffer at the position pointed to by
+   DEST. Note that the buffer is not enlarged so the previous token
+   that was at *DEST is overwritten. TOKEN is the token to
+   insert. DEF_LOC is the full location of the token, i.e, the
+   location possibly encoding its locus accross macro expansion. If
+   TOKEN is an argument of a function like macro (inside a macro
+   replacement list), PARM_DEF_LOC is the location of the macro
+   parameter that TOKEN is replacing.  If TOKEN doesn't come from a
+   macro expansion, then PARM_DEF_LOC can just be set to the same
+   value as DEF_LOC. If MAP is non null, it means TOKEN comes from a
+   macro expansion and MAP is the macro map associated to the
+   macro. MACRO_TOKEN_INDEX points to the index of the token in the
+   macro map; It is not considered if MAP is NULL. Upon successful
+   completion this function returns the a pointer to the position of
+   the token coming right after the insertion point.
+
+   If the -ftrack-macro-expansion flag is in effect the buffer holds a
+   set of cpp_ext_token. Otherwise it holds a set of cpp_token*.  */
+static inline const cpp_token **
+tokens_buff_put_token_to (cpp_reader *pfile,
+			  const cpp_token **dest,
+			  const cpp_token *token,
+			  source_location def_loc,
+			  source_location parm_def_loc,			  
+			  const struct line_map *map,
+			  unsigned int *macro_token_index)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+  source_location macro_loc = def_loc;
+  const cpp_token **result;
+
+  if (track_macro_exp_p)
+    {
+      cpp_ext_token *token_dest = (cpp_ext_token *) dest;
+      if (map)
+	macro_loc = linemap_add_macro_token (map, *macro_token_index,
+					     def_loc, parm_def_loc);
+      token_dest->token = token;
+      token_dest->location = macro_loc;
+      result =  &token_dest[1].token;
+    }
+  else
+    {
+      *dest = token;
+      result = &dest[1];
+    }
+  return result;
+}
+
+/* Appends a token to the end of the token buffer BUFFER.  TOKEN is
+   the token to append. DEF_LOC is the full location of the token,
+   i.e, the location possibly encoding its locus accross macro
+   expansion. If TOKEN is an argument of a function like macro (inside
+   a macro replacement list), PARM_DEF_LOC is the location of the
+   macro parameter that TOKEN is replacing.  If TOKEN doesn't come
+   from a macro expansion, then PARM_DEF_LOC can just be set to the
+   same value as DEF_LOC. If MAP is non null, it means TOKEN comes
+   from a macro expansion and MAP is the macro map associated to the
+   macro. MACRO_TOKEN_INDEX points to the index of the token in the
+   macro map; It is not considered if MAP is NULL. Upon successful
+   completion this function returns the a pointer to the position of
+   the token coming right after the insertion point.
+
+   If the -ftrack-macro-expansion flag is in effect the buffer holds a
+   set of cpp_ext_token. Otherwise it holds a set of cpp_token*.
+ */
+static const cpp_token **
+tokens_buff_append_token (cpp_reader *pfile,
+			  _cpp_buff *buffer,
+			  const cpp_token *token,
+			  source_location def_loc,
+			  source_location parm_def_loc,
+			  const struct line_map *map,
+			  unsigned int *macro_token_index)
+{
+  const cpp_token **result;
+
+  result =
+    tokens_buff_put_token_to (pfile,
+			      (const cpp_token **) BUFF_FRONT (buffer),
+			      token, def_loc, parm_def_loc,
+			      map, macro_token_index);
+
+  BUFF_FRONT (buffer) = (unsigned char *) result;
+  return result;
+}
+
+/* Allocate space for the function-like macro argument ARG to store
+   the tokens resulting from the macro-expansion of the tokens that
+   make up ARG itself. That space is allocated in ARG->expanded and
+   needs to be freed using free.  */
+static void
+alloc_expanded_args_mem (cpp_reader *pfile, macro_arg *arg, size_t capacity)
+{
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (track_macro_exp_p)
+    arg->expanded = (const cpp_token **) XNEWVEC (cpp_ext_token, capacity);
+  else
+    arg->expanded = XNEWVEC (const cpp_token *, capacity);
+
+  arg->expanded_capacity = capacity;
+}
+
+/* If necessary, enlarge ARG->expanded to so that it can contain SIZE
+   more tokens.  */
+static void
+ensure_expanded_args_room (cpp_reader *pfile, macro_arg *arg, size_t size)
+{
+  bool track_macro_exp_p;
+
+  if (size <= arg->expanded_capacity)
+    return;
+
+  size *= 2;
+  track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
+
+  if (track_macro_exp_p)
+    arg->expanded = (const cpp_token **)
+	XRESIZEVEC (cpp_ext_token, arg->expanded, size);
+  else
+    arg->expanded =
+      XRESIZEVEC (const cpp_token *, arg->expanded, size);
+  arg->expanded_capacity = size;
+}
+
 /* Expand an argument ARG before replacing parameters in a
    function-like macro.  This works by pushing a context with the
    argument's tokens, and then expanding that into a temporary buffer
@@ -1162,6 +1944,7 @@  expand_arg (cpp_reader *pfile, macro_arg *arg)
 {
   unsigned int capacity;
   bool saved_warn_trad;
+  bool track_macro_exp_p = CPP_OPTION (pfile, track_macro_expansion);
 
   if (arg->count == 0)
     return;
@@ -1172,26 +1955,30 @@  expand_arg (cpp_reader *pfile, macro_arg *arg)
 
   /* Loop, reading in the arguments.  */
   capacity = 256;
-  arg->expanded = XNEWVEC (const cpp_token *, capacity);
+  alloc_expanded_args_mem (pfile, arg, capacity);
+
+  if (track_macro_exp_p)
+    push_extended_tokens_context (pfile, NULL, NULL,
+				  arg->first, arg->count + 1);
+  else
+    push_ptoken_context (pfile, NULL, NULL,
+			 arg->first, arg->count + 1);
 
-  push_ptoken_context (pfile, NULL, NULL, arg->first, arg->count + 1);
   for (;;)
     {
       const cpp_token *token;
+      source_location location;
 
-      if (arg->expanded_count + 1 >= capacity)
-	{
-	  capacity *= 2;
-	  arg->expanded = XRESIZEVEC (const cpp_token *, arg->expanded,
-                                      capacity);
-	}
+      ensure_expanded_args_room (pfile, arg, arg->expanded_count + 1);
 
-      token = cpp_get_token (pfile);
+      token = cpp_get_token_1 (pfile, &location);
 
       if (token->type == CPP_EOF)
 	break;
 
-      arg->expanded[arg->expanded_count++] = token;
+      set_arg_token (pfile, arg, token, location,
+		     arg->expanded_count, MACRO_ARG_TOKEN_EXPANDED);
+      arg->expanded_count++;
     }
 
   _cpp_pop_context (pfile);
@@ -1200,8 +1987,9 @@  expand_arg (cpp_reader *pfile, macro_arg *arg)
 }
 
 /* Pop the current context off the stack, re-enabling the macro if the
-   context represented a macro's replacement list.  The context
-   structure is not freed so that we can re-use it later.  */
+   context represented a macro's replacement list.  Initially the
+   context structure was not freed so that we can re-use it later, but
+   now we do free it to reduce peak memory consumption.  */
 void
 _cpp_pop_context (cpp_reader *pfile)
 {
@@ -1211,14 +1999,85 @@  _cpp_pop_context (cpp_reader *pfile)
     context->macro->flags &= ~NODE_DISABLED;
 
   if (context->buff)
-    _cpp_release_buff (pfile, context->buff);
+    {
+      /* Decrease memory peak consumption by freeing the memory used
+	 by the context.  */
+      _cpp_free_buff (context->buff);
+    }
 
   pfile->context = context->prev;
+  /* decrease peak memory consumption by feeing the context.  */
+  pfile->context->next = NULL;
+  free (context);
 }
 
-/* External routine to get a token.  Also used nearly everywhere
-   internally, except for places where we know we can safely call
-   _cpp_lex_token directly, such as lexing a directive name.
+/* Return TRUE if we reached the end of the set of tokens stored in
+   CONTEXT, FALSE otherwise.  */
+static bool
+reached_end_of_context (cpp_context *context)
+{
+  if (context->tokens_kind == TOKENS_KIND_DIRECT)
+      return FIRST (context).token == LAST (context).token;
+  else if (context->tokens_kind == TOKENS_KIND_INDIRECT
+	   || context->tokens_kind == TOKENS_KIND_EXTENDED)
+    return FIRST (context).ptoken == LAST (context).ptoken;
+  else
+    abort ();
+}
+
+/* Consume the next token contained in the current context of PFILE,
+   and return it in *TOKEN. It's "full location" is returned in
+   *LOCATION. If -ftrack-macro-location is in effeect, fFull location"
+   means the location encoding the locus of the token accross macro
+   expansion; otherwise it's just is the "normal" location of the
+   token which (*TOKEN)->src_loc.  */
+static void
+consume_next_token_from_context (cpp_reader *pfile,
+				 const cpp_token ** token,
+				 source_location *location)
+{
+  cpp_context *c = pfile->context;
+
+  if ((c)->tokens_kind == TOKENS_KIND_DIRECT)
+    {
+      *token = FIRST (c).token;
+      *location = (*token)->src_loc;
+      FIRST (c).token++;
+    }
+  else if ((c)->tokens_kind == TOKENS_KIND_INDIRECT)		
+    {
+      *token = *FIRST (c).ptoken;
+      *location = (*token)->src_loc;
+      FIRST (c).ptoken++;
+    }
+  else if ((c)->tokens_kind == TOKENS_KIND_EXTENDED)
+    {
+      *token = ((cpp_ext_token *) FIRST (c).ptoken)->token;
+      *location = ((cpp_ext_token *) FIRST (c).ptoken)->location;
+      FIRST (c).ptoken = next_macro_token_ptr (pfile, FIRST (c).ptoken);
+    }
+  else
+    abort ();
+}
+
+/* In the traditionnal mode of the preprocessor, if we are currently
+   in a directive, the location of a token must be the location of the
+   start of the directive line. This function returns the proper
+   location if we are in the traditionnal mode, and just returns
+   LOCATION otherwise.   */
+
+static inline source_location
+maybe_adjust_loc_for_trad_cpp (cpp_reader *pfile, source_location location)
+{
+  if (CPP_OPTION (pfile, traditional))
+    {
+      if (pfile->state.in_directive)
+	return pfile->directive_line;
+    }
+  return location;
+}
+
+/* Routine to get a token as well as its location.
 
    Macro expansions and directives are transparently handled,
    including entering included files.  Thus tokens are post-macro
@@ -1226,12 +2085,40 @@  _cpp_pop_context (cpp_reader *pfile)
    see CPP_EOF only at EOF.  Internal callers also see it when meeting
    a directive inside a macro call, when at the end of a directive and
    state.in_directive is still 1, and at the end of argument
-   pre-expansion.  */
-const cpp_token *
-cpp_get_token (cpp_reader *pfile)
+   pre-expansion.
+
+   LOC is an out parameter; *LOC is set to the location "as expected
+   by the user".  This matters when a token results from macro
+   expansion -- the token's location will indicate where the macro is
+   defined (the spelling location of the token) but *LOC will be a
+   virtual location of the token. Virtual location means a location
+   that possibly encodes many types of locus at once. A virtual
+   location can encode the location of a token resulting from macro
+   expansion or not. If the token results from macro expansion its
+   virtual location encodes (at the same time):
+     - the spelling location of the token
+     - the locus of the macro expansion point
+     - the locus the point where the token got instantiated as part of
+       the macro expansion process.
+     (YES, IT ENCODES ALL THESE THREE AT THE SAME TIME! and maybe more.)
+
+   You can learn more about the different locuses encoded in a map by
+   reading the extensive comments of the line_map_macro and line_map
+   structs in line-map.h.  A virtual location, indeed.
+
+   The linemap API can then be used to retrieve the particular locus
+   we are interested in.
+
+   Otherwise *LOC is set to the same location as the location carried
+   by the returned token.  */
+static const cpp_token*
+cpp_get_token_1 (cpp_reader *pfile, source_location *location)
 {
   const cpp_token *result;
   bool can_set = pfile->set_invocation_location;
+  /* This token is a virtual token that either encodes a location
+     related to macro expansion or a spelling location.  */
+  source_location virt_loc = 0;
   pfile->set_invocation_location = false;
 
   for (;;)
@@ -1241,20 +2128,23 @@  cpp_get_token (cpp_reader *pfile)
 
       /* Context->prev == 0 <=> base context.  */
       if (!context->prev)
-	result = _cpp_lex_token (pfile);
-      else if (FIRST (context).token != LAST (context).token)
 	{
-	  if (context->direct_p)
-	    result = FIRST (context).token++;
-	  else
-	    result = *FIRST (context).ptoken++;
-
+	  result = _cpp_lex_token (pfile);
+	  virt_loc = result->src_loc;
+	}
+      else if (!reached_end_of_context (context))
+	{
+	  consume_next_token_from_context (pfile, &result,
+					   &virt_loc);
 	  if (result->flags & PASTE_LEFT)
 	    {
 	      paste_all_tokens (pfile, result);
 	      if (pfile->state.in_directive)
 		continue;
-	      return padding_token (pfile, result);
+	      result = padding_token (pfile, result);
+	      if (location)
+		*location = result->src_loc;
+	      return result;
 	    }
 	}
       else
@@ -1262,6 +2152,8 @@  cpp_get_token (cpp_reader *pfile)
 	  _cpp_pop_context (pfile);
 	  if (pfile->state.in_directive)
 	    continue;
+	  if (location)
+	    *location = pfile->avoid_paste.src_loc;
 	  return &pfile->avoid_paste;
 	}
 
@@ -1299,7 +2191,8 @@  cpp_get_token (cpp_reader *pfile)
 				      || (peek_tok->flags & PREV_WHITE));
 		  node = pfile->cb.macro_to_expand (pfile, result);
 		  if (node)
-		    ret = enter_macro_context (pfile, node, result);
+		    ret = enter_macro_context (pfile, node, result,
+					       virt_loc);
 		  else if (whitespace_after)
 		    {
 		      /* If macro_to_expand hook returned NULL and it
@@ -1316,12 +2209,16 @@  cpp_get_token (cpp_reader *pfile)
 		}
 	    }
 	  else
-	    ret = enter_macro_context (pfile, node, result);
+	    ret = enter_macro_context (pfile, node, result, 
+				       virt_loc);
 	  if (ret)
  	    {
 	      if (pfile->state.in_directive || ret == 2)
 		continue;
-	      return padding_token (pfile, result);
+	      result = padding_token (pfile, result);
+	      if (location)
+		*location = result->src_loc;
+	      return result;
 	    }
 	}
       else
@@ -1338,27 +2235,74 @@  cpp_get_token (cpp_reader *pfile)
       break;
     }
 
-  return result;
+  if (location)
+    *location = virt_loc;
+  return result;  
 }
 
-/* Like cpp_get_token, but also returns a location separate from the
-   one provided by the returned token.  LOC is an out parameter; *LOC
-   is set to the location "as expected by the user".  This matters
-   when a token results from macro expansion -- the token's location
-   will indicate where the macro is defined, but *LOC will be the
-   location of the start of the expansion.  */
+/* External routine to get a token.  Also used nearly everywhere
+   internally, except for places where we know we can safely call
+   _cpp_lex_token directly, such as lexing a directive name.
+
+   Macro expansions and directives are transparently handled,
+   including entering included files.  Thus tokens are post-macro
+   expansion, and after any intervening directives.  External callers
+   see CPP_EOF only at EOF.  Internal callers also see it when meeting
+   a directive inside a macro call, when at the end of a directive and
+   state.in_directive is still 1, and at the end of argument
+   pre-expansion.  */
+const cpp_token *
+cpp_get_token (cpp_reader *pfile)
+{
+  return cpp_get_token_1 (pfile, NULL);
+}
+
+/* Like cpp_get_token, but also returns a virtual token location
+   separate from the spelling location carried by the returned token.
+
+   LOC is an out parameter; *LOC is set to the location "as expected
+   by the user".  This matters when a token results from macro
+   expansion; in that case the token's spelling location indicates the
+   locus of the token in the definition of the macro but *LOC
+   virtually encodes all the other meaningful locuses associated to
+   the token.
+
+   What? virtual location? Yes, virtual location.
+
+   If the token results from macro expansion and if macro expansion
+   location tracking is enbled its virtual location encodes (at the
+   same time):
+
+   - the spelling location of the token the locus of the macro
+   - expansion point the locus the point where the token got
+   - instantiated as part of the macro expansion process.
+
+   You have to use the linemap API to get the locus you are interested
+   in from a given virtual location.
+
+   If macro expansion tracking is off and if the token results from
+   macro expansion the virtual location is the expansion point of the
+   macro that got expanded.
+
+   When the token doesn't result from macro expansion, the virtual
+   location is just the same thing as its spelling location.  */
+
 const cpp_token *
 cpp_get_token_with_location (cpp_reader *pfile, source_location *loc)
 {
   const cpp_token *result;
 
   pfile->set_invocation_location = true;
-  result = cpp_get_token (pfile);
+  result = cpp_get_token_1 (pfile, loc);
   if (pfile->context->macro)
-    *loc = pfile->invocation_location;
+    {
+      if (!CPP_OPTION (pfile, track_macro_expansion))
+	*loc = pfile->invocation_location;
+    }
   else
     *loc = result->src_loc;
 
+  *loc = maybe_adjust_loc_for_trad_cpp (pfile, *loc);
   return result;
 }
 
@@ -1425,10 +2369,15 @@  _cpp_backup_tokens (cpp_reader *pfile, unsigned int count)
     {
       if (count != 1)
 	abort ();
-      if (pfile->context->direct_p)
+      if (pfile->context->tokens_kind == TOKENS_KIND_DIRECT)
 	FIRST (pfile->context).token--;
-      else
+      else if (pfile->context->tokens_kind == TOKENS_KIND_INDIRECT)
 	FIRST (pfile->context).ptoken--;
+      else if (pfile->context->tokens_kind == TOKENS_KIND_EXTENDED)
+	FIRST (pfile->context).ptoken =
+	  prev_macro_token_ptr (pfile, FIRST (pfile->context).ptoken);
+      else
+	abort ();
     }
 }