Generate virtual locations for tokens

This second instalment uses the infrastructure of the previous patch
to allocate a macro map for each macro expansion and assign a virtual
location to each token resulting from the expansion.

To date when cpp_get_token comes across a token that happens to be a
macro, the macro expander kicks in, expands the macro, pushes the
resulting tokens onto a "token context" and returns a dummy padding
token. The next call to cpp_get_token goes look into the token context
for the next token [which is going to result from the previous macro
expansion] and returns it.  If the token is a macro, the macro expander
kicks in and you know the story.

This patch piggy-backs on that macro expansion process, so to speak.
First it modifies the macro expander to make it create a macro map for
each macro expansion. It then allocates a virtual location for each
resulting token.  Virtual locations of tokens resulting from macro
expansions are then stored on a special kind of context called an
"expanded tokens context".  In other words, in an expanded tokens
context, there are tokens resulting from macro expansion and their
associated virtual locations.  cpp_get_token_with_location is modified
to return the virtual location of tokens resulting from macro
expansion.  Note that once all tokens from an expanded token context have
been consumed and the context and is freed, the memory used to store the
virtual locations of the tokens held in that context is freed as well.
This helps reducing the overall peak memory consumption.

The client code that was getting macro expansion point location from
cpp_get_token_with_location now gets virtual location from it. Those
virtual locations can in turn be resolved into the different
interesting physical locations thanks to the linemap API exposed by
the previous patch.

Expensive progress. Possibly. So this whole virtual location
allocation business is switched off by default. So by default no
extended token is created. No extended token context is created
either. One has to use -ftrack-macro-expansion to switch this on. This
complicates the code but I believe it can be useful as some of our
friends found out at http://llvm.org/bugs/show_bug.cgi?id=5610

The patch tries to reduce the memory consumption by freeing some token
context memory that was being reused before. I didn't notice any
compilation slow down due to this immediate freeing on my GNU/Linux
system.

As no client code tries to resolve virtual locations to anything but
what was being done before, no new test case has been added.

Co-Authored-By: Dodji Seketeli <dodji@redhat.com>

From-SVN: r180082

gcc/ChangeLog

@@ -1,3 +1,10 @@
+2011-10-15  Tom Tromey  <tromey@redhat.com>
+	    Dodji Seketeli  <dodji@redhat.com>
+
+	* doc/cppopts.texi (-ftrack-macro-expansion): Document new option.
+	* doc/invoke.texi (-ftrack-macro-expansion): Add this to the list of
+	preprocessor related options.
+
 2011-10-15  Tom Tromey  <tromey@redhat>
 	    Dodji Seketeli  <dodji@redhat.com>
 

gcc/c-family/ChangeLog

@@ -1,3 +1,12 @@
+2011-10-15  Tom Tromey  <tromey@redhat.com>
+	    Dodji Seketeli  <dodji@redhat.com>
+
+	* c.opt (ftrack-macro-expansion): New option. Handle it with and
+	without argument.
+	* c-opts.c (c_common_handle_option)<case
+	OPT_ftrack_macro_expansion_, case OPT_ftrack_macro_expansion>: New
+	cases. Handle -ftrack-macro-expansion with and without argument.
+
 2011-10-15  Tom Tromey  <tromey@redhat.com>
 	    Dodji Seketeli  <dodji@redhat.com>
 

gcc/c-family/c-opts.c

@@ -628,6 +628,18 @@ c_common_handle_option (size_t scode, const char *arg, int value,
       cpp_opts->preprocessed = value;
       break;
 
+    case OPT_ftrack_macro_expansion:
+      if (value)
+	value = 2;
+      /* Fall Through.  */
+
+    case OPT_ftrack_macro_expansion_:
+      if (arg && *arg != '\0')
+	cpp_opts->track_macro_expansion = value;
+      else
+	cpp_opts->track_macro_expansion = 2;
+      break;
+
     case OPT_frepo:
       flag_use_repository = value;
       if (value)

gcc/c-family/c.opt

@@ -945,6 +945,14 @@ fpreprocessed
 C ObjC C++ ObjC++
 Treat the input file as already preprocessed
 
+ftrack-macro-expansion
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+; converted into ftrack-macro-expansion=
+
+ftrack-macro-expansion=
+C ObjC C++ ObjC++ JoinedOrMissing RejectNegative UInteger
+-ftrack-macro-expansion=<0|1|2>  Track locations of tokens coming from macro expansion and display them in error messages
+
 fpretty-templates
 C++ ObjC++ Var(flag_pretty_templates) Init(1)
 -fno-pretty-templates Do not pretty-print template specializations as the template signature followed by the arguments

gcc/doc/cppopts.texi

@@ -583,6 +583,24 @@ correct column numbers in warnings or errors, even if tabs appear on the
 line.  If the value is less than 1 or greater than 100, the option is
 ignored.  The default is 8.
 
+@item -ftrack-macro-expansion@r{[}=@var{level}@r{]}
+@opindex ftrack-macro-expansion
+Track locations of tokens across macro expansions. This allows the
+compiler to emit diagnostic about the current macro expansion stack
+when a compilation error occurs in a macro expansion. Using this
+option makes the preprocessor and the compiler consume more
+memory. The @var{level} parameter can be used to choose the level of
+precision of token location tracking thus decreasing the memory
+consumption if necessary. Value @samp{0} of @var{level} de-activates
+this option just as if no @option{-ftrack-macro-expansion} was present
+on the command line. Value @samp{1} tracks tokens locations in a
+degraded mode for the sake of minimal memory overhead. In this mode
+all tokens resulting from the expansion of an argument of a
+function-like macro have the same location. Value @samp{2} tracks
+tokens locations completely. This value is the most memory hungry.
+When this option is given no argument, the default parameter value is
+@samp{2}.
+
 @item -fexec-charset=@var{charset}
 @opindex fexec-charset
 @cindex character set, execution

gcc/doc/invoke.texi

@@ -429,9 +429,9 @@ Objective-C and Objective-C++ Dialects}.
 -iwithprefixbefore @var{dir}  -isystem @var{dir} @gol
 -imultilib @var{dir} -isysroot @var{dir} @gol
 -M  -MM  -MF  -MG  -MP  -MQ  -MT  -nostdinc  @gol
--P  -fworking-directory  -remap @gol
--trigraphs  -undef  -U@var{macro}  -Wp,@var{option} @gol
--Xpreprocessor @var{option}}
+-P -ftrack-macro-expansion -fworking-directory @gol
+-remap -trigraphs  -undef  -U@var{macro}  @gol
+-Wp,@var{option} -Xpreprocessor @var{option}}
 
 @item Assembler Option
 @xref{Assembler Options,,Passing Options to the Assembler}.

gcc/input.c

@@ -1,5 +1,5 @@
 /* Data and functions related to line maps and input files.
-   Copyright (C) 2004, 2007, 2008, 2009, 2010
+   Copyright (C) 2004, 2007, 2008, 2009, 2010, 2011
    Free Software Foundation, Inc.
 
 This file is part of GCC.

libcpp/ChangeLog

@@ -1,3 +1,82 @@
+2011-10-15  Tom Tromey  <tromey@redhat.com>
+	    Dodji Seketeli  <dodji@redhat.com>
+
+	* include/cpplib.h (struct cpp_options)<track_macro_expansion>:
+	New option.
+	* internal.h (struct macro_context): New struct.
+	(enum context_tokens_kind): New enum.
+	(struct cpp_context)<tokens_kind>: New member of type enum
+	context_tokens_kind.
+	(struct cpp_context)<macro>: Remove this.  Replace it with an enum
+	of macro and  macro_context.
+	(struct cpp_context)<direct_p>: Remove.
+	(_cpp_remaining_tokens_num_in_context): Declare new function.
+	* directives.c (destringize_and_run): Adjust.
+	* lex.c (_cpp_remaining_tokens_num_in_context)
+	(_cpp_token_from_context_at): Define new functions
+	(cpp_peek_token): Use them.
+	* init.c (cpp_create_reader): Initialize the base context to zero.
+	(_cpp_token_from_context_at): Define new static function.
+	(cpp_peek_token): Use new _cpp_remaining_tokens_num_in_context and
+	_cpp_token_from_context_at.
+	* macro.c (struct macro_arg)<virt_locs, expanded_virt_locs>: New
+	members.
+	(enum macro_arg_token_kind): New enum.
+	(struct macro_arg_token_iter): New struct.
+	(maybe_adjust_loc_for_trad_cpp, push_extended_tokens_context)
+	(alloc_expanded_arg_mem, ensure_expanded_arg_room)
+	(delete_macro_args, set_arg_token, get_arg_token_location)
+	(arg_token_ptr_at, macro_arg_token_iter_init)
+	(macro_arg_token_iter_get_token)
+	(macro_arg_token_iter_get_location, macro_arg_token_iter_forward)
+	(expanded_token_index, tokens_buff_new, tokens_buff_count)
+	(tokens_buff_last_token_ptr, tokens_buff_put_token_to)
+	(tokens_buff_add_token, tokens_buff_remove_last_token)
+	(reached_end_of_context, consume_next_token_from_context): New
+	static functions.
+	(cpp_get_token_1): New static function. Split and extended from
+	cpp_get_token.  Use reached_end_of_context and
+	consume_next_token_from_context.  Unify its return point.  Move
+	the location tweaking from cpp_get_token_with_location in here.
+	(cpp_get_token): Use cpp_get_token_1
+	(stringify_arg): Use the new arg_token_at.
+	(paste_all_tokens): Support tokens coming from extended tokens
+	contexts.
+	(collect_args): Return the number of collected arguments, by
+	parameter.  Store virtual locations of tokens that constitute the
+	collected args.
+	(funlike_invocation_p): Return the number of collected arguments,
+	by parameter.
+	(enter_macro_context): Add a parameter for macro expansion point.
+	Pass it to replace_args and to the "used" cpp callback.  Get the
+	number of function-like macro arguments from funlike_invocation_p,
+	pass it to the new delete_macro_args to free the memory used by
+	macro args.  When -ftrack-macro-expansion is in effect, for macros
+	that have no arguments, create a macro map for the macro expansion
+	and use it to allocate proper virtual locations for tokens
+	resulting from the expansion.  Push an extended tokens context
+	containing the tokens resulting from macro expansion and their
+	virtual locations.
+	(replace_args): Rename the different variables named 'count' into
+	variables with more meaningful names.  Create a macro map;
+	allocate virtual locations of tokens resulting from this
+	expansion.  Use macro_arg_token_iter to iterate over tokens of a
+	given macro.  Handle the case of the argument of
+	-ftrack-macro-expansion being < 2.  Don't free macro arguments
+	memory resulting from expand_arg here, as these are freed by the
+	caller of replace_arg using delete_macro_args now.  Push extended
+	token context.
+	(next_context, push_ptoken_context, _cpp_push_token_context)
+	(_cpp_push_text_context): Properly initialize the context.
+	(expand_arg): Use the new alloc_expanded_arg_mem,
+	push_extended_tokens_context, cpp_get_token_1, and set_arg_token.
+	(_cpp_pop_context): Really free the memory held by the context.
+	Handle freeing memory used by extended tokens contexts.
+	(cpp_get_token_with_location): Use cpp_get_token_1.
+	(cpp_sys_macro_p): Adjust.
+	(_cpp_backup_tokens): Support the new kinds of token contexts.
+	* traditional.c (recursive_macro): Adjust.
+
 2011-10-15  Tom Tromey  <tromey@redhat>
 	    Dodji Seketeli  <dodji@redhat.com>
 

libcpp/directives.c

@@ -1,7 +1,7 @@
 /* CPP Library. (Directive handling.)
    Copyright (C) 1986, 1987, 1989, 1992, 1993, 1994, 1995, 1996, 1997, 1998,
    1999, 2000, 2001, 2002, 2003, 2004, 2005,
-   2007, 2008, 2009, 2010 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
    Contributed by Per Bothner, 1994-95.
    Based on CCCP program by Paul Rubin, June 1986
    Adapted to ANSI C, Richard Stallman, Jan 1987
@@ -1742,7 +1742,7 @@ destringize_and_run (cpp_reader *pfile, const cpp_string *in)
   saved_cur_run = pfile->cur_run;
 
   pfile->context = XNEW (cpp_context);
-  pfile->context->macro = 0;
+  pfile->context->c.macro = 0;
   pfile->context->prev = 0;
   pfile->context->next = 0;
 

libcpp/include/cpplib.h

@@ -393,6 +393,14 @@ struct cpp_options
      bother trying to do macro expansion and whatnot.  */
   unsigned char preprocessed;
 
+  /* Nonzero means we are tracking locations of tokens involved in
+     macro expansion. 1 Means we track the location in degraded mode
+     where we do not track locations of tokens resulting from the
+     expansion of arguments of function-like macro.  2 Means we do
+     track all macro expansions. This last option is the one that
+     consumes the highest amount of memory.  */
+  unsigned char track_macro_expansion;
+
   /* Nonzero means handle C++ alternate operator names.  */
   unsigned char operator_names;
 

libcpp/include/line-map.h

@@ -1,5 +1,5 @@
 /* Map logical line numbers to (source file, line number) pairs.
-   Copyright (C) 2001, 2003, 2004, 2007, 2008, 2009, 2010
+   Copyright (C) 2001, 2003, 2004, 2007, 2008, 2009, 2010, 2011
    Free Software Foundation, Inc.
 
 This program is free software; you can redistribute it and/or modify it

libcpp/init.c

@@ -154,6 +154,7 @@ cpp_create_reader (enum c_lang lang, hash_table *table,
   init_library ();
 
   pfile = XCNEW (cpp_reader);
+  memset (&pfile->base_context, 0, sizeof (pfile->base_context));
 
   cpp_set_lang (pfile, lang);
   CPP_OPTION (pfile, warn_multichar) = 1;
@@ -213,7 +214,7 @@ cpp_create_reader (enum c_lang lang, hash_table *table,
 
   /* Initialize the base context.  */
   pfile->context = &pfile->base_context;
-  pfile->base_context.macro = 0;
+  pfile->base_context.c.macro = 0;
   pfile->base_context.prev = pfile->base_context.next = 0;
 
   /* Aligned and unaligned storage.  */

libcpp/internal.h

@@ -1,6 +1,6 @@
 /* Part of CPP library.
    Copyright (C) 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007,
-   2008, 2009, 2010 Free Software Foundation, Inc.
+   2008, 2009, 2010, 2011 Free Software Foundation, Inc.
 
 This program is free software; you can redistribute it and/or modify it
 under the terms of the GNU General Public License as published by the
@@ -139,6 +139,40 @@ struct tokenrun
 #define CUR(c) ((c)->u.trad.cur)
 #define RLIMIT(c) ((c)->u.trad.rlimit)
 
+/* This describes some additional data that is added to the macro
+   token context of type cpp_context, when -ftrack-macro-expansion is
+   on.  */
+typedef struct
+{
+  /* The node of the macro we are referring to.  */
+  cpp_hashnode *macro_node;
+  /* This buffer contains an array of virtual locations.  The virtual
+     location at index 0 is the virtual location of the token at index
+     0 in the current instance of cpp_context; similarly for all the
+     other virtual locations.  */
+  source_location *virt_locs;
+  /* This is a pointer to the current virtual location.  This is used
+     to iterate over the virtual locations while we iterate over the
+     tokens they belong to.  */
+  source_location *cur_virt_loc;
+} macro_context;
+
+/* The kind of tokens carried by a cpp_context.  */
+enum context_tokens_kind {
+  /* This is the value of cpp_context::tokens_kind if u.iso.first
+     contains an instance of cpp_token **.  */
+  TOKENS_KIND_INDIRECT,
+  /* This is the value of cpp_context::tokens_kind if u.iso.first
+     contains an instance of cpp_token *.  */
+  TOKENS_KIND_DIRECT,
+  /* This is the value of cpp_context::tokens_kind when the token
+     context contains tokens resulting from macro expansion.  In that
+     case struct cpp_context::macro points to an instance of struct
+     macro_context.  This is used only when the
+     -ftrack-macro-expansion flag is on.  */
+  TOKENS_KIND_EXTENDED
+};
+
 typedef struct cpp_context cpp_context;
 struct cpp_context
 {
@@ -168,11 +202,24 @@ struct cpp_context
      When the context is popped, the buffer is released.  */
   _cpp_buff *buff;
 
-  /* For a macro context, the macro node, otherwise NULL.  */
-  cpp_hashnode *macro;
+  /* If tokens_kind is TOKEN_KIND_EXTENDED, then (as we thus are in a
+     macro context) this is a pointer to an instance of macro_context.
+     Otherwise if tokens_kind is *not* TOKEN_KIND_EXTENDED, then, if
+     we are in a macro context, this is a pointer to an instance of
+     cpp_hashnode, representing the name of the macro this context is
+     for.  If we are not in a macro context, then this is just NULL.
+     Note that when tokens_kind is TOKEN_KIND_EXTENDED, the memory
+     used by the instance of macro_context pointed to by this member
+     is de-allocated upon de-allocation of the instance of struct
+     cpp_context.  */
+  union
+  {
+    macro_context *mc;
+    cpp_hashnode *macro;
+  } c;
 
-  /* True if utoken element is token, else ptoken.  */
-  bool direct_p;
+  /* This determines the type of tokens held by this context.  */
+  enum context_tokens_kind tokens_kind;
 };
 
 struct lexer_state
@@ -605,6 +652,7 @@ extern cpp_token *_cpp_lex_direct (cpp_reader *);
 extern int _cpp_equiv_tokens (const cpp_token *, const cpp_token *);
 extern void _cpp_init_tokenrun (tokenrun *, unsigned int);
 extern cpp_hashnode *_cpp_lex_identifier (cpp_reader *, const char *);
+extern int _cpp_remaining_tokens_num_in_context (cpp_reader *);
 
 /* In init.c.  */
 extern void _cpp_maybe_push_include_file (cpp_reader *);

libcpp/lex.c

@@ -1703,6 +1703,38 @@ next_tokenrun (tokenrun *run)
   return run->next;
 }
 
+/* Return the number of not yet processed token in the the current
+   context.  */
+int
+_cpp_remaining_tokens_num_in_context (cpp_reader *pfile)
+{
+  cpp_context *context = pfile->context;
+  if (context->tokens_kind == TOKENS_KIND_DIRECT)
+    return ((LAST (context).token - FIRST (context).token)
+	    / sizeof (cpp_token));
+  else if (context->tokens_kind == TOKENS_KIND_INDIRECT
+	   || context->tokens_kind == TOKENS_KIND_EXTENDED)
+    return ((LAST (context).ptoken - FIRST (context).ptoken)
+	    / sizeof (cpp_token *));
+  else
+      abort ();
+}
+
+/* Returns the token present at index INDEX in the current context.
+   If INDEX is zero, the next token to be processed is returned.  */
+static const cpp_token*
+_cpp_token_from_context_at (cpp_reader *pfile, int index)
+{
+  cpp_context *context = pfile->context;
+  if (context->tokens_kind == TOKENS_KIND_DIRECT)
+    return &(FIRST (context).token[index]);
+  else if (context->tokens_kind == TOKENS_KIND_INDIRECT
+	   || context->tokens_kind == TOKENS_KIND_EXTENDED)
+    return FIRST (context).ptoken[index];
+ else
+   abort ();
+}
+
 /* Look ahead in the input stream.  */
 const cpp_token *
 cpp_peek_token (cpp_reader *pfile, int index)
@@ -1714,15 +1746,10 @@ cpp_peek_token (cpp_reader *pfile, int index)
   /* First, scan through any pending cpp_context objects.  */
   while (context->prev)
     {
-      ptrdiff_t sz = (context->direct_p
-                      ? LAST (context).token - FIRST (context).token
-                      : LAST (context).ptoken - FIRST (context).ptoken);
+      ptrdiff_t sz = _cpp_remaining_tokens_num_in_context (pfile);
 
       if (index < (int) sz)
-        return (context->direct_p
-                ? FIRST (context).token + index
-                : *(FIRST (context).ptoken + index));
-
+        return _cpp_token_from_context_at (pfile, index);
       index -= (int) sz;
       context = context->prev;
     }

libcpp/line-map.c

@@ -1,5 +1,5 @@
 /* Map logical line numbers to (source file, line number) pairs.
-   Copyright (C) 2001, 2003, 2004, 2007, 2008, 2009
+   Copyright (C) 2001, 2003, 2004, 2007, 2008, 2009, 2010, 2011
    Free Software Foundation, Inc.
 
 This program is free software; you can redistribute it and/or modify it

libcpp/traditional.c

@@ -738,7 +738,7 @@ recursive_macro (cpp_reader *pfile, cpp_hashnode *node)
       do
 	{
 	  depth++;
-	  if (context->macro == node && depth > 20)
+	  if (context->c.macro == node && depth > 20)
 	    break;
 	  context = context->prev;
 	}