Patchwork [gimplefe] Re-factor parser/lexer

login
register
mail settings
Submitter Diego Novillo
Date Nov. 22, 2010, 5:44 p.m.
Message ID <20101122174416.GA26860@google.com>
Download mbox | patch
Permalink /patch/72567/
State New
Headers show

Comments

Diego Novillo - Nov. 22, 2010, 5:44 p.m.
This patch starts re-factoring the lexer and parser.  We now read
input files from the command line, but parsing is still largely
non-functional.  I'll address that in the next patch I'm working
on.


Diego.


gimple/ChangeLog

2010-11-22  Diego Novillo  <dnovillo@google.com>

	* Make-lang.in (GIMPLE_PARSER_H): Add CPPLIB_H and VEC_H
	(gimple-lang.o, parser.o): Fix dependencies.
	* parser.h: Include cpplib.h and vec.h.
	(struct gimple_lexer): Define.
	(struct gimple_parser): Define.
	(gimple_token): Define.
	* config-lang.in (gtfiles): Add parser.h and parser.c
	* gimple-lang.c: Do not include flags.h nor tm.h.
	(handle_leaf_attribute): New.
	(gimple_attributes): Add attribute 'leaf'.
	* parser.c: Do not include cpplib.h nor input.h.
	Include toplev.h, timevar.h.
	Move include of ggc.h earlier in the file.
	Change all references to cpp_token with gimple_token.
	(parser_gc_root__): New.
	(gl_init): New.
	(gp_init): New.
	(gl_lex): New.
	(gl_consume_token): New.
	(gp_parse): New.
	(gp_finish): New.
	(gimple_main): Call gp_init, gl_lex, gp_parse and
	gp_finish.
Sandeep Soni - Dec. 5, 2010, 10:35 a.m.
On Mon, Nov 22, 2010 at 11:14 PM, Diego Novillo <dnovillo@google.com> wrote:
> This patch starts re-factoring the lexer and parser.  We now read
> input files from the command line, but parsing is still largely
> non-functional.  I'll address that in the next patch I'm working
> on.
>

I am unable to clear build stage 2 after adding this patch to my local
repository. make gives an error as

make error: No rule for gt-gimple-gimple-lang.h. Needed for gimple-lang.o

Am I missing something?
Diego Novillo - Dec. 7, 2010, 8:57 p.m.
On Sun, Dec 5, 2010 at 05:35, Sandeep Soni <soni.sandeepb@gmail.com> wrote:

> I am unable to clear build stage 2 after adding this patch to my local
> repository. make gives an error as
>
> make error: No rule for gt-gimple-gimple-lang.h. Needed for gimple-lang.o

I made changes that require reconfiguration.  Did you try to build
from an existing build tree?  Can you try out of a clean slate?  I
just built it without any problems.


Diego.
Sandeep Soni - Dec. 8, 2010, 6 a.m.
On Wed, Dec 8, 2010 at 2:27 AM, Diego Novillo <dnovillo@google.com> wrote:
> On Sun, Dec 5, 2010 at 05:35, Sandeep Soni <soni.sandeepb@gmail.com> wrote:
>
>> I am unable to clear build stage 2 after adding this patch to my local
>> repository. make gives an error as
>>
>> make error: No rule for gt-gimple-gimple-lang.h. Needed for gimple-lang.o
>
> I made changes that require reconfiguration.  Did you try to build
> from an existing build tree?  Can you try out of a clean slate?  I
> just built it without any problems.
>
>
> Diego.
>

I made a clean build actually. The error seems weird. The build system
is trying to find a rule for gt-gimple-gimple-lang.h. I don't know if
that is correct behaviour. I seem to have applied the patch wrong so
something is incorrect in gimple/make-lang.in

Is it possible for me to copy only that file from the index?
Diego Novillo - Dec. 9, 2010, 8:33 p.m.
On Wed, Dec 8, 2010 at 01:00, Sandeep Soni <soni.sandeepb@gmail.com> wrote:

> I made a clean build actually. The error seems weird. The build system
> is trying to find a rule for gt-gimple-gimple-lang.h. I don't know if
> that is correct behaviour. I seem to have applied the patch wrong so
> something is incorrect in gimple/make-lang.in
>
> Is it possible for me to copy only that file from the index?

You seem to have a mangled up tree.  Can you try again with a fresh
checkout of the branch?


Diego.
Sandeep Soni - Dec. 12, 2010, 5:35 p.m.
On Fri, Dec 10, 2010 at 2:03 AM, Diego Novillo <dnovillo@google.com> wrote:
> You seem to have a mangled up tree.  Can you try again with a fresh
> checkout of the branch?

Got it up and running by doing a fresh checkout. I seem to have got
something wrong earlier (I have been making way too many mistakes)
I will probably try and write some code to have the entire type system
in place. Its anyways long overdue.
Thanks for the help!

Patch

Index: gimple/Make-lang.in
===================================================================
--- gimple/Make-lang.in	(revision 166537)
+++ gimple/Make-lang.in	(working copy)
@@ -24,7 +24,7 @@ 
 GIMPLE_EXE = gimple1$(exeext)
 # The GIMPLE-specific object files inclued in $(GIMPLE_EXE).
 GIMPLE_OBJS = gimple/gimple-lang.o gimple/parser.o attribs.o
-GIMPLE_PARSER_H = gimple/parser.h
+GIMPLE_PARSER_H = gimple/parser.h $(CPPLIB_H) $(VEC_H)
 GIMPLE_TREE_H = gimple/gimple-tree.h
 
 # Rules
@@ -75,10 +75,10 @@  $(GIMPLE_EXE): $(GIMPLE_OBJS) $(BACKEND)
 
 # Dependencies
 gimple/gimple-lang.o: gimple/gimple-lang.c $(CONFIG_H) $(SYSTEM_H) \
-	coretypes.h flags.h $(TM_H) $(TREE_H) $(TARGET_H) \
-	langhooks.h $(LANGHOOKS_DEF_H) debug.h $(GIMPLE_PARSER_H) \
-	$(DIAGNOSTIC_CORE_H) $(TOPLEV_H) $(GIMPLE_TREE_H) \
-	$(GGC_H) gtype-gimple.h gt-gimple-gimple-lang.h
+	coretypes.h $(TREE_H) $(TARGET_H) langhooks.h $(LANGHOOKS_DEF_H) \
+	debug.h $(GIMPLE_PARSER_H) $(GIMPLE_H) $(TOPLEV_H) $(GIMPLE_TREE_H) \
+	diagnostic-core.h $(TOPLEV_H) $(GGC_H) gtype-gimple.h \
+	gt-gimple-gimple-lang.h
 gimple/parser.o: gimple/parser.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
-	$(CPPLIB_H) $(INPUT_H) $(DIAGNOSTIC_H) $(TREE_H) $(GIMPLE_H) \
-	$(TOPLEV_H) $(GIMPLE_PARSER_H)
+	$(DIAGNOSTIC_H) $(TOPLEV_H) $(TIMEVAR_H) $(TREE_H) $(GIMPLE_H) \
+	$(GIMPLE_PARSER_H) $(GGC_H) gt-gimple-parser.h
Index: gimple/parser.h
===================================================================
--- gimple/parser.h	(revision 166537)
+++ gimple/parser.h	(working copy)
@@ -22,6 +22,53 @@  along with GCC; see the file COPYING3.  
 #ifndef GIMPLE_PARSER_H
 #define GIMPLE_PARSER_H
 
+#include "cpplib.h"
+#include "vec.h"
+
+struct gimple_parser;
+typedef cpp_token gimple_token;
+
+DEF_VEC_O (gimple_token);
+DEF_VEC_ALLOC_O (gimple_token, gc);
+
+/* The GIMPLE lexer.  */
+
+typedef struct GTY(()) gimple_lexer {
+  /* Associated parser.  */
+  struct gimple_parser *parser;
+
+  /* Path to the main input file name.  */
+  const char *filename;
+
+  /* The cpp reader to get pre-processed tokens.  */
+  struct GTY((skip)) cpp_reader *reader;
+
+  /* The array of tokens read by the lexer.  */
+  VEC(gimple_token, gc) *tokens;
+
+  /* Token to be consumed by the parser.  */
+  size_t cur_token_ix;
+} gimple_lexer;
+
+
+/* The GIMPLE parser.  */
+
+typedef struct GTY(()) gimple_parser {
+  /* Reader we use for lexing.  */
+  gimple_lexer *lexer;
+
+  /* Non-zero if '-dy' is enabled (dump debugging information during
+     parsing).  */
+  unsigned int debug_p : 1;
+
+  /* Line table.  */
+  struct line_maps *line_table;
+
+  /* Identifier table.  */
+  struct GTY((skip)) ht *ident_hash;
+} gimple_parser;
+
+
 /* In parser.c  */
 extern void gimple_main (int);
  
Index: gimple/config-lang.in
===================================================================
--- gimple/config-lang.in	(revision 166537)
+++ gimple/config-lang.in	(working copy)
@@ -22,6 +22,6 @@  language="gimple"
 compilers="gimple1\$(exeext)"
 stagestuff="gimple1\$(exeext)"
 
-gtfiles="\$(srcdir)/gimple/gimple-tree.h \$(srcdir)/gimple/gimple-lang.c"
+gtfiles="\$(srcdir)/gimple/gimple-tree.h \$(srcdir)/gimple/gimple-lang.c \$(srcdir)/gimple/parser.h \$(srcdir)/gimple/parser.c"
 
 build_by_default=yes
Index: gimple/gimple-lang.c
===================================================================
--- gimple/gimple-lang.c	(revision 166537)
+++ gimple/gimple-lang.c	(working copy)
@@ -22,8 +22,6 @@  along with GCC; see the file COPYING3.  
 #include "config.h"
 #include "system.h"
 #include "coretypes.h"
-#include "flags.h"
-#include "tm.h"
 #include "tree.h"
 #include "target.h"
 #include "langhooks.h"
@@ -37,6 +35,7 @@  along with GCC; see the file COPYING3.  
 
 static tree handle_noreturn_attribute (tree *, tree, tree, int, bool *);
 static tree handle_const_attribute (tree *, tree, tree, int, bool *);
+static tree handle_leaf_attribute (tree *, tree, tree, int, bool *);
 static tree handle_malloc_attribute (tree *, tree, tree, int, bool *);
 static tree handle_pure_attribute (tree *, tree, tree, int, bool *);
 static tree handle_novops_attribute (tree *, tree, tree, int, bool *);
@@ -56,6 +55,8 @@  const struct attribute_spec gimple_attri
   /* The same comments as for noreturn attributes apply to const ones.  */
   { "const",                  0, 0, true,  false, false,
 			      handle_const_attribute },
+  { "leaf",                   0, 0, true,  false, false,
+			      handle_leaf_attribute },
   { "malloc",                 0, 0, true,  false, false,
 			      handle_malloc_attribute },
   { "pure",                   0, 0, true,  false, false,
@@ -211,6 +212,29 @@  handle_const_attribute (tree *node, tree
 }
 
 
+/* Handle a "leaf" attribute; arguments as in
+   struct attribute_spec.handler.  */
+
+static tree
+handle_leaf_attribute (tree *node, tree name,
+		       tree ARG_UNUSED (args),
+		       int ARG_UNUSED (flags), bool *no_add_attrs)
+{
+  if (TREE_CODE (*node) != FUNCTION_DECL)
+    {
+      warning (OPT_Wattributes, "%qE attribute ignored", name);
+      *no_add_attrs = true;
+    }
+  if (!TREE_PUBLIC (*node))
+    {
+      warning (OPT_Wattributes, "%qE attribute has no effect on unit local functions", name);
+      *no_add_attrs = true;
+    }
+
+  return NULL_TREE;
+}
+
+
 /* Handle a "malloc" attribute; arguments as in
    struct attribute_spec.handler.  */
 
Index: gimple/parser.c
===================================================================
--- gimple/parser.c	(revision 166541)
+++ gimple/parser.c	(working copy)
@@ -22,21 +22,27 @@  along with GCC; see the file COPYING3.  
 #include "config.h"
 #include "system.h"
 #include "coretypes.h"
-#include "cpplib.h"
-#include "input.h"
 #include "diagnostic.h"
+#include "toplev.h"
+#include "timevar.h"
 #include "tree.h"
 #include "gimple.h"
-#include "toplev.h"
 #include "parser.h"
+#include "ggc.h"
+
+/* The GIMPLE parser.  Note: do not use this variable directly.  It is
+   declared here only to serve as a root for the GC machinery.  The
+   parser pointer should be passed as a parameter to every function
+   that needs to access it.  */
+static GTY(()) gimple_parser *parser_gc_root__;
 
 /* Consumes a token if the EXPECTED_TOKEN_TYPE is exactly the one we 
    are looking for. The token is obtained by reading it from the reader P.  */
  
-static const cpp_token * 
+static const gimple_token * 
 gimple_parse_expect_token (cpp_reader *p, enum cpp_ttype expected_token_type)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   next_token = cpp_peek_token (p, 0);
 
@@ -65,7 +71,7 @@  gimple_parse_expect_token (cpp_reader *p
 static void
 gimple_parse_expect_subcode (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
   const char *text;
   int i;
 
@@ -98,7 +104,7 @@  gimple_parse_expect_subcode (cpp_reader 
 static void 
 gimple_parse_expect_lhs (cpp_reader *p)
 {  
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   /* Just before the name of the identifier we might get the symbol 
      of dereference too. If we do get it then consume that token, else
@@ -117,7 +123,7 @@  gimple_parse_expect_lhs (cpp_reader *p)
 static void 
 gimple_parse_expect_rhs1 (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
   next_token = cpp_peek_token (p, 0);
 
   /* Currently there is duplication in the following blocks but there
@@ -152,7 +158,7 @@  gimple_parse_expect_rhs1 (cpp_reader *p)
 static void 
 gimple_parse_expect_rhs2 (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
   next_token = cpp_peek_token (p, 0);
 
   /* ??? Can there be more possibilities than these ?  */
@@ -190,7 +196,7 @@  gimple_parse_assign_stmt (cpp_reader *p)
 static void
 gimple_parse_expect_op1 (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
   next_token = cpp_peek_token (p, 0);
 
   switch (next_token->type)
@@ -213,7 +219,7 @@  gimple_parse_expect_op1 (cpp_reader *p)
 static void
 gimple_parse_expect_op2 (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
   next_token = cpp_peek_token (p, 0);
 
   switch (next_token->type)
@@ -302,7 +308,7 @@  gimple_parse_label_stmt (cpp_reader *p)
 static void
 gimple_parse_switch_stmt (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   gimple_parse_expect_token (p, CPP_LESS);
   gimple_parse_expect_token (p, CPP_NAME);
@@ -353,7 +359,7 @@  gimple_parse_expect_function_name (cpp_r
 static void
 gimple_parse_expect_return_var (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   next_token = cpp_peek_token (p, 0);
 
@@ -371,7 +377,7 @@  gimple_parse_expect_return_var (cpp_read
 static void
 gimple_parse_expect_argument (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   next_token = cpp_peek_token (p, 0);
 
@@ -399,7 +405,7 @@  gimple_parse_expect_argument (cpp_reader
 static void
 gimple_parse_call_stmt (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   gimple_parse_expect_function_name (p);
   gimple_parse_expect_return_var (p);
@@ -436,7 +442,7 @@  gimple_parse_return_stmt (cpp_reader *p)
    for recognizing the statements in a function body.  */
 
 static void 
-gimple_parse_stmt (cpp_reader *p, const cpp_token *tok)
+gimple_parse_stmt (cpp_reader *p, const gimple_token *tok)
 {
   const char *text;
   int i;
@@ -523,7 +529,7 @@  gimple_parse_expect_field_decl (cpp_read
 static void
 gimple_parse_record_type (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   gimple_parse_expect_token (p, CPP_LESS);
   gimple_parse_expect_token (p, CPP_NAME);
@@ -568,7 +574,7 @@  gimple_parse_record_type (cpp_reader *p)
 static void
 gimple_parse_union_type (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   gimple_parse_expect_token (p, CPP_LESS);
   gimple_parse_expect_token (p, CPP_NAME);
@@ -624,7 +630,7 @@  gimple_parse_expect_const_decl (cpp_read
 static void
 gimple_parse_enum_type (cpp_reader *p)
 {
-  const cpp_token *next_token;
+  const gimple_token *next_token;
 
   gimple_parse_expect_token (p, CPP_LESS);
   gimple_parse_expect_token (p, CPP_NAME);
@@ -653,7 +659,7 @@  gimple_parse_enum_type (cpp_reader *p)
    for recognizing the type and variable declarations. */
 
 static void
-gimple_parse_type (cpp_reader *p, const cpp_token *tok)
+gimple_parse_type (cpp_reader *p, const gimple_token *tok)
 {
   const char *text;
   int i;
@@ -689,36 +695,113 @@  gimple_parse_type (cpp_reader *p, const 
 }
 
 
-/* Main entry point for the GIMPLE front end.  */
+/* Initialize the lexer.  */
 
-void
-gimple_main (int debug_p ATTRIBUTE_UNUSED)
+static gimple_lexer *
+gl_init (gimple_parser *p)
 {
-  /* We invoke the parser here.  */
-  cpp_reader *p;
-  const cpp_token *tok;
-  const char *input_file = "/tmp/gimple.txt";
-  const char *output_file;
-  bool inside_type_section_p = true;
-  struct line_maps *line_tab;
+  gimple_lexer *l;
 
-  line_tab = ggc_alloc_cleared_line_maps ();
-  linemap_init (line_tab);
-  p = cpp_create_reader (CLK_GNUC99, ident_hash, line_tab);
-  output_file = cpp_read_main_file (p, input_file);
-  if (output_file)
+  l = ggc_alloc_cleared_gimple_lexer ();
+  l->parser = p;
+  l->filename = main_input_filename;
+  l->reader = cpp_create_reader (CLK_GNUC99, p->ident_hash, p->line_table);
+  l->filename = cpp_read_main_file (l->reader, l->filename);
+  l->cur_token_ix = 0;
+
+  return l;
+}
+
+
+/* Initialize the parser data structures.  */
+
+static gimple_parser *
+gp_init (int debug_p)
+{
+  gimple_parser *p = ggc_alloc_cleared_gimple_parser ();
+  p->debug_p = debug_p;
+  line_table = p->line_table = ggc_alloc_cleared_line_maps ();
+  p->ident_hash = ident_hash;
+  linemap_init (p->line_table);
+  p->lexer = gl_init (p);
+
+  return p;
+}
+
+
+/* Get all the tokens from the file in LEXER.  */
+
+static void
+gl_lex (gimple_lexer *lexer)
+{
+  const gimple_token *gimple_tok;
+
+  timevar_push (TV_CPP);
+
+  do
     {
-      tok = cpp_get_token (p);
-      while (tok->type != CPP_EOF)
-	{
-	  if (inside_type_section_p)
-	    gimple_parse_type (p, tok);
-	  else
-	    gimple_parse_stmt (p, tok);
-	  tok = cpp_get_token (p);
-	}
+      location_t loc;
+
+      gimple_tok = cpp_get_token_with_location (lexer->reader, &loc);
+      if (gimple_tok->type != CPP_EOF)
+	VEC_safe_push (gimple_token, gc, lexer->tokens, gimple_tok);
     }
+  while (gimple_tok->type != CPP_EOF);
 
-  cpp_finish (p, NULL);
-  cpp_destroy (p);   
+  timevar_pop (TV_CPP);
+}
+
+
+/* Consume the next token from PARSER.  */
+
+static gimple_token *
+gl_consume_token (gimple_lexer *lexer)
+{
+  return VEC_index (gimple_token, lexer->tokens, lexer->cur_token_ix++);
 }
+
+/* Parse the translation unit in PARSER.  */
+
+static void
+gp_parse (gimple_parser *parser)
+{
+  while (!VEC_empty (gimple_token, parser->lexer->tokens))
+    {
+      gimple_token *tok = gl_consume_token (parser->lexer);
+      if (1)
+	gimple_parse_type (parser->lexer->reader, tok);
+      else
+	gimple_parse_stmt (parser->lexer->reader, tok);
+    }
+}
+
+
+/* Finalize parsing and release allocated memory in PARSER.  */
+
+static void
+gp_finish (gimple_parser *parser)
+{
+  cpp_finish (parser->lexer->reader, NULL);
+  cpp_destroy (parser->lexer->reader);
+  parser_gc_root__ = NULL;
+}
+
+
+/* Main entry point for the GIMPLE front end.  */
+
+void
+gimple_main (int debug_p)
+{
+  gimple_parser *parser;
+
+  parser_gc_root__ = parser = gp_init (debug_p);
+
+  if (parser->lexer->filename == NULL)
+    return;
+
+  gl_lex (parser->lexer);
+  gp_parse (parser);
+  gp_finish (parser);
+}
+
+#include "gt-gimple-parser.h"