diff mbox

[3/5] qapi: add qapi2texi script

Message ID 1435917064-17827-4-git-send-email-marcandre.lureau@redhat.com
State New
Headers show

Commit Message

Marc-André Lureau July 3, 2015, 9:51 a.m. UTC
From: Marc-André Lureau <marcandre.lureau@gmail.com>

As the name suggests, the qapi2texi script converts JSON QAPI
description into a standalone texi file suitable for different target
formats.

As the documentation format doesn't seem to be specified, it parses the
following blocks before each declaration with some variations:

  ##
  # @symbol
  #
  # body
  #
  # @arg: foo
  # @arg: #optional foo
  #
  # Returns: returns
  # Since: version
  # Notes: notes
  ##

Using the json declaration, it's able to give extra information about
the type of arguments and return value expected.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi.py      |  78 ++++++++++++++++++-
 scripts/qapi2texi.py | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 286 insertions(+), 4 deletions(-)
 create mode 100644 scripts/qapi2texi.py

Comments

Eric Blake July 27, 2015, 10:31 p.m. UTC | #1
On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> From: Marc-André Lureau <marcandre.lureau@gmail.com>
> 
> As the name suggests, the qapi2texi script converts JSON QAPI
> description into a standalone texi file suitable for different target
> formats.
> 
> As the documentation format doesn't seem to be specified, it parses the
> following blocks before each declaration with some variations:

docs/qapi-code-gen.txt tried to give a sample documentation.  Feel free
to formalize that, and to fix non-conforming uses, if you desire.  It'll
be a big one-time audit of the .json files, but getting things
consistent, _and keeping them that way by automatic conformance checks
in your conversion tool_, is fine by me.

> 
>   ##
>   # @symbol
>   #
>   # body
>   #
>   # @arg: foo
>   # @arg: #optional foo
>   #
>   # Returns: returns
>   # Since: version
>   # Notes: notes
>   ##
> 
> Using the json declaration, it's able to give extra information about
> the type of arguments and return value expected.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi.py      |  78 ++++++++++++++++++-
>  scripts/qapi2texi.py | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 286 insertions(+), 4 deletions(-)
>  create mode 100644 scripts/qapi2texi.py

Are you intending to apply this after Markus' big work on QMP Introspection?

> 
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 06d7fc2..70208e8 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -103,6 +103,53 @@ class QAPIExprError(Exception):
>          return error_path(self.info['parent']) + \
>              "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
>  
> +class QAPIDoc:

In particular, this should probably be QAPIDoc(object) (new style class
declaration).

> @@ -762,6 +823,15 @@ def parse_schema(fname):
>          print >>sys.stderr, e
>          exit(1)
>  
> +def parse_schema_full(fname):
> +    try:
> +        schema = QAPISchema(open(fname, "r"))
> +        check_exprs(schema.exprs)
> +        return schema.exprs
> +    except (QAPISchemaError, QAPIExprError), e:
> +        print >>sys.stderr, e
> +        exit(1)

and this may need to be reworked on top of Markus' new parser class.

> +++ b/scripts/qapi2texi.py
> @@ -0,0 +1,212 @@
> +# QAPI texi generator
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.
> +# See the COPYING file in the top-level directory.

Although I'm not a lawyer, my understanding is that for GPL to apply,
there has to be an explicit mention of Copyright.

> +exprs = parse_schema_full(sys.argv[4])

And you'll probably want to rewrite this in terms of the visitor interface.

> +for cmd in exprs:
> +    expr = cmd['expr']
> +    docs = cmd['info']['doc']
> +    (kind, name) = expr.items()[0]
> +
> +    for d in docs[0:-1]:
> +        print d.comment
> +
> +    texi = {"command": texi_command,
> +            "struct": texi_struct,
> +            "enum": texi_enum,
> +            "union": texi_union,
> +            "alternate": texi_alternate,
> +            "event": texi_event}
> +    try:
> +        texi[kind](expr, docs[-1])
> +    except KeyError:
> +        raise ValueError("Unknown expression kind '%s'" % kind)

although this is a rather cute way to polymorphically invoke the correct
helper function.
Marc-Andre Lureau July 27, 2015, 11:39 p.m. UTC | #2
Hi

----- Original Message -----
> On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> > From: Marc-André Lureau <marcandre.lureau@gmail.com>
> > 
> > As the name suggests, the qapi2texi script converts JSON QAPI
> > description into a standalone texi file suitable for different target
> > formats.
> > 
> > As the documentation format doesn't seem to be specified, it parses the
> > following blocks before each declaration with some variations:
> 
> docs/qapi-code-gen.txt tried to give a sample documentation.  Feel free
> to formalize that, and to fix non-conforming uses, if you desire.  It'll
> be a big one-time audit of the .json files, but getting things
> consistent, _and keeping them that way by automatic conformance checks
> in your conversion tool_, is fine by me.

Thanks, the patch series evolved quite a bit since this first series.

The current branch is: https://github.com/elmarco/qemu/commits/qapi

And I found about that doc, so I removed that comment in commit message.
 
> > 
> >   ##
> >   # @symbol
> >   #
> >   # body
> >   #
> >   # @arg: foo
> >   # @arg: #optional foo
> >   #
> >   # Returns: returns
> >   # Since: version
> >   # Notes: notes
> >   ##
> > 
> > Using the json declaration, it's able to give extra information about
> > the type of arguments and return value expected.
> > 
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  scripts/qapi.py      |  78 ++++++++++++++++++-
> >  scripts/qapi2texi.py | 212
> >  +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 286 insertions(+), 4 deletions(-)
> >  create mode 100644 scripts/qapi2texi.py
> 
> Are you intending to apply this after Markus' big work on QMP Introspection?

Yes, I am in contact with Markus, I propose we revisit this series after his patches are merged.

Thanks for your comments so far, 

> > 
> > diff --git a/scripts/qapi.py b/scripts/qapi.py
> > index 06d7fc2..70208e8 100644
> > --- a/scripts/qapi.py
> > +++ b/scripts/qapi.py
> > @@ -103,6 +103,53 @@ class QAPIExprError(Exception):
> >          return error_path(self.info['parent']) + \
> >              "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
> >  
> > +class QAPIDoc:
> 
> In particular, this should probably be QAPIDoc(object) (new style class
> declaration).

I just run pep8 fine on my file, I don't know about that new syntax.

> 
> > @@ -762,6 +823,15 @@ def parse_schema(fname):
> >          print >>sys.stderr, e
> >          exit(1)
> >  
> > +def parse_schema_full(fname):
> > +    try:
> > +        schema = QAPISchema(open(fname, "r"))
> > +        check_exprs(schema.exprs)
> > +        return schema.exprs
> > +    except (QAPISchemaError, QAPIExprError), e:
> > +        print >>sys.stderr, e
> > +        exit(1)
> 
> and this may need to be reworked on top of Markus' new parser class.
> 
> > +++ b/scripts/qapi2texi.py
> > @@ -0,0 +1,212 @@
> > +# QAPI texi generator
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.
> > +# See the COPYING file in the top-level directory.
> 
> Although I'm not a lawyer, my understanding is that for GPL to apply,
> there has to be an explicit mention of Copyright.
> 

ah ok, I'll add it then


> > +exprs = parse_schema_full(sys.argv[4])
> 
> And you'll probably want to rewrite this in terms of the visitor interface.
> 
> > +for cmd in exprs:
> > +    expr = cmd['expr']
> > +    docs = cmd['info']['doc']
> > +    (kind, name) = expr.items()[0]
> > +
> > +    for d in docs[0:-1]:
> > +        print d.comment
> > +
> > +    texi = {"command": texi_command,
> > +            "struct": texi_struct,
> > +            "enum": texi_enum,
> > +            "union": texi_union,
> > +            "alternate": texi_alternate,
> > +            "event": texi_event}
> > +    try:
> > +        texi[kind](expr, docs[-1])
> > +    except KeyError:
> > +        raise ValueError("Unknown expression kind '%s'" % kind)
> 
> although this is a rather cute way to polymorphically invoke the correct
> helper function.
> 
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
> 
>
diff mbox

Patch

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 06d7fc2..70208e8 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -103,6 +103,53 @@  class QAPIExprError(Exception):
         return error_path(self.info['parent']) + \
             "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
 
+class QAPIDoc:
+    def __init__(self, comment):
+        self.symbol = None
+        self.comment = ""
+        self.args = OrderedDict()
+        self.meta = OrderedDict()
+        self.section = None
+
+        for line in comment.split('\n'):
+            line = ' '.join(line.split())
+            split = line.split(' ', 1)
+            key = split[0].rstrip(':')
+
+            if key.startswith("@"):
+                key = key[1:]
+                line = split[1] if len(split) > 1 else ""
+                if self.symbol == None:
+                    self.symbol = key
+                else:
+                    self.start_section(self.args, key)
+            elif key in ("Since", "Returns", "Notes"):
+                line = split[1] if len(split) > 1 else ""
+                self.start_section(self.meta, key)
+            self.append_comment(line)
+
+        self.end_section()
+
+    def append_comment(self, line):
+        if len(line) == 0:
+            return
+        if self.section != None:
+           self.section.append(line)
+        elif self.comment == "":
+            self.comment = line
+        else:
+            self.comment += line if line[0] == ' ' else " " + line
+
+    def end_section(self):
+        if self.section != None:
+            dic = self.section[0]
+            key = self.section[1]
+            dic[key] = " ".join(self.section[2:]).strip()
+
+    def start_section(self, dic, key):
+        self.end_section()
+        self.section = [dic, key, ""]
+
 class QAPISchema:
 
     def __init__(self, fp, previously_included = [], incl_info = None):
@@ -118,11 +165,14 @@  class QAPISchema:
         self.line = 1
         self.line_pos = 0
         self.exprs = []
+        self.comment = ""
+        self.apidoc = []
         self.accept()
 
         while self.tok != None:
             expr_info = {'file': fname, 'line': self.line,
-                         'parent': self.incl_info}
+                         'parent': self.incl_info, 'doc': self.apidoc}
+            self.apidoc = []
             expr = self.get_expr(False)
             if isinstance(expr, dict) and "include" in expr:
                 if len(expr) != 1:
@@ -154,7 +204,7 @@  class QAPISchema:
                 self.exprs.extend(exprs_include.exprs)
             else:
                 expr_elem = {'expr': expr,
-                             'info': expr_info}
+                             'info': expr_info }
                 self.exprs.append(expr_elem)
 
     def accept(self):
@@ -165,8 +215,19 @@  class QAPISchema:
             self.val = None
 
             if self.tok == '#':
-                self.cursor = self.src.find('\n', self.cursor)
-            elif self.tok in ['{', '}', ':', ',', '[', ']']:
+                end = self.src.find('\n', self.cursor)
+                if self.src[self.cursor] != "#":
+                    self.comment += self.src[self.cursor:end+1]
+                self.cursor = end+1
+                continue
+            else:
+                apidoc = QAPIDoc(self.comment)
+                if apidoc.symbol != None or \
+                   not apidoc.comment.find("*-*"):
+                    self.apidoc.append(apidoc)
+                self.comment = ""
+
+            if self.tok in ['{', '}', ':', ',', '[', ']']:
                 return
             elif self.tok == "'":
                 string = ''
@@ -762,6 +823,15 @@  def parse_schema(fname):
         print >>sys.stderr, e
         exit(1)
 
+def parse_schema_full(fname):
+    try:
+        schema = QAPISchema(open(fname, "r"))
+        check_exprs(schema.exprs)
+        return schema.exprs
+    except (QAPISchemaError, QAPIExprError), e:
+        print >>sys.stderr, e
+        exit(1)
+
 #
 # Code generation helpers
 #
diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
new file mode 100644
index 0000000..a3879a4
--- /dev/null
+++ b/scripts/qapi2texi.py
@@ -0,0 +1,212 @@ 
+# QAPI texi generator
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+
+command_fmt = """
+@deftypefn {type} {{{ret}}} {name} @
+{{{args}}}
+
+{body}
+
+@end deftypefn
+
+""".format
+
+enum_fmt = """
+@deftp Enum {name}
+
+{body}
+
+@end deftp
+
+""".format
+
+struct_fmt = """
+@deftp {type} {name} @
+{{{attrs}}}
+
+{body}
+
+@end deftp
+
+""".format
+
+
+def subst_vars(doc):
+    return re.sub(r'@(\w*)', r'@var{\1}', doc)
+
+
+def texi_args(expr):
+    data = expr["data"] if "data" in expr else {}
+    if type(data) == str:
+        args = data
+    else:
+        args = []
+        for v, t in data.iteritems():
+            if type(t) == str and t.startswith("**"):
+                args.append("%s@dots{}" % v)
+            elif v.startswith("*"):
+                v = v[1:]
+                args.append("['%s': @var{%s}]" % (v, t))
+            else:
+                args.append("'%s': @var{%s}" % (v, t))
+        args = ", ".join(args)
+    return args
+
+
+def texi_body(doc, arg="@var"):
+    body = "@table %s\n" % arg
+    for arg, c in doc.args.iteritems():
+        if c.startswith("#optional"):
+            c = c[10:]
+            arg += "*"
+        body += "@item %s\n%s\n" % (arg, subst_vars(c))
+    body += "@end table\n"
+    body += subst_vars(doc.comment)
+
+    for k in ("Returns", "Notes", "Since"):
+        if k not in doc.meta:
+            continue
+        body += "\n@quotation %s\n%s\n@end quotation" % \
+                (k, subst_vars(doc.meta[k]))
+    return body
+
+
+def texi_alternate(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Alternate",
+                     name=doc.symbol,
+                     attrs="[ " + args + " ]",
+                     body=body)
+
+
+def texi_union(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Union",
+                     name=doc.symbol,
+                     attrs="[ " + args + " ]",
+                     body=body)
+
+
+def texi_enum(expr, doc):
+    body = texi_body(doc, "@samp")
+    print enum_fmt(name=doc.symbol,
+                   body=body)
+
+
+def texi_struct(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Struct",
+                     name=doc.symbol,
+                     attrs="@{ " + args + " @}",
+                     body=body)
+
+
+def texi_command(expr, doc):
+    args = texi_args(expr)
+    ret = expr["returns"] if "returns" in expr else ""
+    body = texi_body(doc)
+    print command_fmt(type="Command",
+                      name=doc.symbol,
+                      ret=ret,
+                      args="(" + args + ")",
+                      body=body)
+
+
+def texi_event(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print command_fmt(type="Event",
+                      name=doc.symbol,
+                      ret="",
+                      args="(" + args + ")",
+                      body=body)
+
+
+if len(sys.argv) != 5:
+    print >>sys.stderr, "%s: need exactly 4 arguments" % sys.argv[0]
+    sys.exit(1)
+
+exprs = parse_schema_full(sys.argv[4])
+
+print """
+\input texinfo
+@setfilename {filename}
+@documentlanguage en
+@exampleindent 0
+@paragraphindent 0
+
+@settitle {title}
+
+@ifinfo
+@direntry
+* QEMU: (qemu-doc).    {title}
+@end direntry
+@end ifinfo
+
+@titlepage
+@title {title} {version}
+@end titlepage
+
+@ifnottex
+@node Top
+@top
+
+This is the API reference for QEMU {version}.
+
+@menu
+* API Reference::
+* Commands and Events Index::
+* Data Types Index::
+@end menu
+
+@end ifnottex
+
+@contents
+
+@node API Reference
+@chapter API Reference
+
+@c man begin DESCRIPTION
+""".format(title=sys.argv[1], version=sys.argv[2], filename=sys.argv[3])
+
+for cmd in exprs:
+    expr = cmd['expr']
+    docs = cmd['info']['doc']
+    (kind, name) = expr.items()[0]
+
+    for d in docs[0:-1]:
+        print d.comment
+
+    texi = {"command": texi_command,
+            "struct": texi_struct,
+            "enum": texi_enum,
+            "union": texi_union,
+            "alternate": texi_alternate,
+            "event": texi_event}
+    try:
+        texi[kind](expr, docs[-1])
+    except KeyError:
+        raise ValueError("Unknown expression kind '%s'" % kind)
+
+print """
+@c man end
+
+@c man begin SEEALSO
+The HTML documentation of QEMU for more precise information.
+@c man end
+
+@node Commands and Events Index
+@unnumbered Commands and Events Index
+@printindex fn
+@node Data Types Index
+@unnumbered Data Types Index
+@printindex tp
+@bye
+"""