[v9,07/17] qapi: Start converting to new qapi union layout

We have two issues with our qapi union layout:
1) Even though the QMP wire format spells the tag 'type', the
C code spells it 'kind', requiring some hacks in the generator.
2) The C struct uses an anonymous union, which places all tag
values in the same namespace as all non-variant members. This
leads to spurious collisions if a tag value matches a QMP name.

This patch is the front end for a series that converts to a
saner qapi union layout.  By the end of the series, we will no
longer have the type/kind mismatch, and all tag values will be
under a named union, which requires clients to access
'obj->u.value' instead of 'obj->value'.  But since the
conversion touches a number of files, it is easiest if we
temporarily support BOTH layouts simultaneously.

Given a simple union qapi type:

{ 'union':'Foo', 'data': { 'a':'int', 'b':'bool' } }

we make the following changes in generated qapi-types.h:

| struct Foo {
|-    FooKind kind;
|-    union { /* union tag is @kind */
|+    union {
|+        FooKind kind;
|+        FooKind type;
|+    };
|+    union { /* union tag is @type */
|         void *data;
|         int64_t a;
|         bool b;
|+        union { /* union tag is @type */
|+            void *data;
|+            int64_t a;
|+            bool b;
|+        } u;
|     };
| };

Flat unions do not need the anonymous union for the tag member,
as we already fixed that to use the member name instead of 'kind'
back in commit 0f61af3e.  On the other hand, the duplication
means that we temporarily cannot support 'u' as a branch name.

Later, when the conversions are complete, we will remove the
duplication hacks and restore support for 'u' as a branch name.

Note, however, that we do not rename the generated enum, which
is still 'FooKind'.  A further patch could generate implicit
enums as 'FooType', but that causes more churn to C code, and
gets harder since the generator already reserved the '*Kind'
namespace, but there are already QMP constructs with '*Type'
naming which means we cannot easily reserve it for qapi.

Signed-off-by: Eric Blake <eblake@redhat.com>

---
v9: new patch, but incorporates parts of v5 31/46 and Markus' RFC:
http://lists.gnu.org/archive/html/qemu-devel/2015-10/msg02236.html
---
 scripts/qapi-types.py                   | 26 +++++++++++++++++++-------
 scripts/qapi-visit.py                   | 24 +++++++++---------------
 tests/qapi-schema/qapi-schema-test.json |  4 +++-
 tests/qapi-schema/qapi-schema-test.out  |  4 ++--
 4 files changed, 33 insertions(+), 25 deletions(-)

Eric Blake <eblake@redhat.com> writes:

> We have two issues with our qapi union layout:
> 1) Even though the QMP wire format spells the tag 'type', the
> C code spells it 'kind', requiring some hacks in the generator.
> 2) The C struct uses an anonymous union, which places all tag
> values in the same namespace as all non-variant members. This
> leads to spurious collisions if a tag value matches a QMP name.
>
> This patch is the front end for a series that converts to a
> saner qapi union layout.  By the end of the series, we will no
> longer have the type/kind mismatch, and all tag values will be
> under a named union, which requires clients to access
> 'obj->u.value' instead of 'obj->value'.  But since the
> conversion touches a number of files, it is easiest if we
> temporarily support BOTH layouts simultaneously.
>
> Given a simple union qapi type:
>
> { 'union':'Foo', 'data': { 'a':'int', 'b':'bool' } }
>
> we make the following changes in generated qapi-types.h:
>
> | struct Foo {
> |-    FooKind kind;
> |-    union { /* union tag is @kind */
> |+    union {
> |+        FooKind kind;
> |+        FooKind type;
> |+    };
> |+    union { /* union tag is @type */
> |         void *data;
> |         int64_t a;
> |         bool b;
> |+        union { /* union tag is @type */
> |+            void *data;
> |+            int64_t a;
> |+            bool b;
> |+        } u;
> |     };
> | };

This is clever and ugly in equal measure.  I respect that.  Fortunately,
it's also temporary.

> Flat unions do not need the anonymous union for the tag member,
> as we already fixed that to use the member name instead of 'kind'
> back in commit 0f61af3e.

Unlike then, we need multiple commits for simple unions, because they're
more widely used?

>                           On the other hand, the duplication
> means that we temporarily cannot support 'u' as a branch name.

Separate paragraph, because now you're talking about the *other*
anonymous union.

> Later, when the conversions are complete, we will remove the
> duplication hacks and restore support for 'u' as a branch name.
>
> Note, however, that we do not rename the generated enum, which
> is still 'FooKind'.  A further patch could generate implicit
> enums as 'FooType', but that causes more churn to C code, and
> gets harder since the generator already reserved the '*Kind'
> namespace, but there are already QMP constructs with '*Type'
> naming which means we cannot easily reserve it for qapi.

Oh, we can reserve whatever we want in QAPI, it's just a lot of churn to
adapt the QAPI-using code.

I'd simply say "but that would cause substantial churn to C code, as
there are already QAPI definitions with '*Type' naming".

> Signed-off-by: Eric Blake <eblake@redhat.com>
>
> ---
> v9: new patch, but incorporates parts of v5 31/46 and Markus' RFC:
> http://lists.gnu.org/archive/html/qemu-devel/2015-10/msg02236.html
> ---
>  scripts/qapi-types.py                   | 26 +++++++++++++++++++-------
>  scripts/qapi-visit.py                   | 24 +++++++++---------------
>  tests/qapi-schema/qapi-schema-test.json |  4 +++-
>  tests/qapi-schema/qapi-schema-test.out  |  4 ++--
>  4 files changed, 33 insertions(+), 25 deletions(-)
>

First part: generate C structs as described in the commit message.

> diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
> index bcef39d..0a14451 100644
> --- a/scripts/qapi-types.py
> +++ b/scripts/qapi-types.py
> @@ -136,11 +136,23 @@ struct %(c_name)s {
>      if base:
>          ret += gen_struct_fields([], base)
>      else:
> +        # TODO As a hack, we emit both 'kind' and 'type'. Ultimately, we
> +        # want to use only 'type', but the conversion is large enough to
> +        # require staging over several commits.
>          ret += mcgen('''
> -    %(c_type)s kind;
> +    union {
> +        %(c_type)s kind;
> +        %(c_type)s type;
> +    };
>  ''',
>                       c_type=c_name(variants.tag_member.type.name))
>
> +    # TODO As a hack, we emit the union twice, once as an anonymous union
> +    # and once as a named union.  Ultimately, we want to use only the
> +    # named union version (as it avoids conflicts between tag values as
> +    # branch names competing with non-variant QMP names), but the conversion
> +    # is large enough to require staging over several commits.
> +    tmp = ''
>      # FIXME: What purpose does data serve, besides preventing a union that
>      # has a branch named 'data'? We use it in qapi-visit.py to decide
>      # whether to bypass the switch statement if visiting the discriminator
> @@ -149,25 +161,25 @@ struct %(c_name)s {
>      # should not be any data leaks even without a data pointer.  Or, if
>      # 'data' is merely added to guarantee we don't have an empty union,
>      # shouldn't we enforce that at .json parse time?
> -    ret += mcgen('''
> +    tmp += mcgen('''
>      union { /* union tag is @%(c_name)s */
>          void *data;
>  ''',
> -                 # TODO ugly special case for simple union
> -                 # Use same tag name in C as on the wire to get rid of
> -                 # it, then: c_name=c_name(variants.tag_member.name)
> -                 c_name=c_name(variants.tag_name or 'kind'))
> +                 c_name=c_name(variants.tag_member.name))
>
>      for var in variants.variants:
>          # Ugly special case for simple union TODO get rid of it
>          typ = var.simple_union_type() or var.type
> -        ret += mcgen('''
> +        tmp += mcgen('''
>          %(c_type)s %(c_name)s;
>  ''',
>                       c_type=typ.c_type(),
>                       c_name=c_name(var.name))
>
> +    ret += tmp
> +    ret += '    ' + '\n    '.join(tmp.split('\n'))
>      ret += mcgen('''
> +    } u;
>      };
>  };
>  ''')

It took me some head-scratching to understand why this generates
correctly indented output.  If it wasn't temporary code, I'd ask for
cleanup.

Second part: convert qapi-visit.py.  Not mentioned in commit message.
Separate patch, perhaps?

> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index 91bf350..2afe811 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -182,18 +182,18 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
>      if (err) {
>          goto out;
>      }
> -    visit_get_next_type(v, (int*) &(*obj)->kind, %(c_name)s_qtypes, name, &err);
> +    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
>      if (err) {
>          goto out_obj;
>      }
> -    switch ((*obj)->kind) {
> +    switch ((*obj)->type) {
>  ''',
>                  c_name=c_name(name))
>
>      for var in variants.variants:
>          ret += mcgen('''
>      case %(case)s:
> -        visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, name, &err);
> +        visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
>          break;
>  ''',
>                       case=c_enum_const(variants.tag_member.type.name,
> @@ -255,22 +255,16 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
>      visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, "%(name)s", &err);
>  ''',
>                       c_type=variants.tag_member.type.c_name(),
> -                     # TODO ugly special case for simple union
> -                     # Use same tag name in C as on the wire to get rid of
> -                     # it, then: c_name=c_name(variants.tag_member.name)
> -                     c_name='kind',
> +                     c_name=c_name(variants.tag_member.name),
>                       name=variants.tag_member.name)
>      ret += gen_err_check(label='out_obj')
>      ret += mcgen('''
> -    if (!visit_start_union(v, !!(*obj)->data, &err) || err) {
> +    if (!visit_start_union(v, !!(*obj)->u.data, &err) || err) {
>          goto out_obj;
>      }
>      switch ((*obj)->%(c_name)s) {
>  ''',
> -                 # TODO ugly special case for simple union
> -                 # Use same tag name in C as on the wire to get rid of
> -                 # it, then: c_name=c_name(variants.tag_member.name)
> -                 c_name=c_name(variants.tag_name or 'kind'))
> +                 c_name=c_name(variants.tag_member.name))
>
>      for var in variants.variants:
>          # TODO ugly special case for simple union
> @@ -282,13 +276,13 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
>                                         var.name))
>          if simple_union_type:
>              ret += mcgen('''
> -        visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, "data", &err);
> +        visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, "data", &err);
>  ''',
>                           c_type=simple_union_type.c_name(),
>                           c_name=c_name(var.name))
>          else:
>              ret += mcgen('''
> -        visit_type_implicit_%(c_type)s(v, &(*obj)->%(c_name)s, &err);
> +        visit_type_implicit_%(c_type)s(v, &(*obj)->u.%(c_name)s, &err);
>  ''',
>                           c_type=var.type.c_name(),
>                           c_name=c_name(var.name))
> @@ -304,7 +298,7 @@ out_obj:
>      error_propagate(errp, err);
>      err = NULL;
>      if (*obj) {
> -        visit_end_union(v, !!(*obj)->data, &err);
> +        visit_end_union(v, !!(*obj)->u.data, &err);
>      }
>      error_propagate(errp, err);
>      err = NULL;

Third part: work around temporary clash with 'u'.  Needs to remain in
this patch, obviously.  Suggest to amend the commit message to say

    On the other hand, the duplication means that we temporarily cannot
    support 'u' as a branch name.  Adapt a few tests that do.

> diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
> index 22e15eb..876ce18 100644
> --- a/tests/qapi-schema/qapi-schema-test.json
> +++ b/tests/qapi-schema/qapi-schema-test.json
> @@ -113,8 +113,10 @@
>  # should still be valid as a type or union branch name. And although
>  # '*Kind' and '*List' are forbidden as type names, they should not be
>  # forbidden as a member or branch name.
> +# TODO - we temporarily do not support 'u' as branch name, while converting
> +# code to use the new union layout
>  { 'struct': 'has_a', 'data': { 'MyKind': 'int', 'MyList': ['int'] } }
> -{ 'union': 'u', 'data': { 'u': 'uint8', 'myKind': 'has_a',
> +{ 'union': 'u', 'data': { 'u8': 'uint8', 'myKind': 'has_a',
>                            'myList': 'has_a' } }
>
>  # testing commands
> diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
> index feaf20d..cb12435 100644
> --- a/tests/qapi-schema/qapi-schema-test.out
> +++ b/tests/qapi-schema/qapi-schema-test.out
> @@ -202,10 +202,10 @@ object has_a
>      member MyKind: int optional=False
>      member MyList: intList optional=False
>  object u
> -    case u: :obj-uint8-wrapper
> +    case u8: :obj-uint8-wrapper
>      case myKind: :obj-has_a-wrapper
>      case myList: :obj-has_a-wrapper
> -enum uKind ['u', 'myKind', 'myList']
> +enum uKind ['u8', 'myKind', 'myList']
>  command user_def_cmd None -> None
>     gen=True success_response=True
>  command user_def_cmd1 :obj-user_def_cmd1-arg -> None

On 10/22/2015 07:54 AM, Markus Armbruster wrote:

> 
> This is clever and ugly in equal measure.  I respect that.  Fortunately,
> it's also temporary.
> 
>> Flat unions do not need the anonymous union for the tag member,
>> as we already fixed that to use the member name instead of 'kind'
>> back in commit 0f61af3e.
> 
> Unlike then, we need multiple commits for simple unions, because they're
> more widely used?

Yes. In fact, both you and I expressed surprise back then that the main
body of qemu didn't need adjusting - our only use of flat unions was
hidden behind QDict manipulations rather than direct generated qapi
struct, explaining why nothing was affected when we converted flat
unions.  But a useful note for the commit message at any rate.

> 
>>                           On the other hand, the duplication
>> means that we temporarily cannot support 'u' as a branch name.
> 
> Separate paragraph, because now you're talking about the *other*
> anonymous union.
> 
>> Later, when the conversions are complete, we will remove the
>> duplication hacks and restore support for 'u' as a branch name.

And based on comments on 3/17, I'm deferring any testsuite changes
related to 'u' collisions until after this conversion to inline base is
complete, so this part of the commit message actually disappears in v10
because I'm no longer touching qapi-schema-test this early.

>>
>> Note, however, that we do not rename the generated enum, which
>> is still 'FooKind'.  A further patch could generate implicit
>> enums as 'FooType', but that causes more churn to C code, and
>> gets harder since the generator already reserved the '*Kind'
>> namespace, but there are already QMP constructs with '*Type'
>> naming which means we cannot easily reserve it for qapi.
> 
> Oh, we can reserve whatever we want in QAPI, it's just a lot of churn to
> adapt the QAPI-using code.
> 
> I'd simply say "but that would cause substantial churn to C code, as
> there are already QAPI definitions with '*Type' naming".

Okay.


>>      for var in variants.variants:
>>          # Ugly special case for simple union TODO get rid of it
>>          typ = var.simple_union_type() or var.type
>> -        ret += mcgen('''
>> +        tmp += mcgen('''
>>          %(c_type)s %(c_name)s;
>>  ''',
>>                       c_type=typ.c_type(),
>>                       c_name=c_name(var.name))
>>
>> +    ret += tmp
>> +    ret += '    ' + '\n    '.join(tmp.split('\n'))
>>      ret += mcgen('''
>> +    } u;
>>      };
>>  };
>>  ''')
> 
> It took me some head-scratching to understand why this generates
> correctly indented output.  If it wasn't temporary code, I'd ask for
> cleanup.

Would a comment help?  It's because we add 4 spaces after each newline,
but need an indent prior to the first line of tmp, and the '} u;' line
picks up four spaces after the last line of tmp.

> 
> Second part: convert qapi-visit.py.  Not mentioned in commit message.
> Separate patch, perhaps?

Sure, I could split.


> 
> Third part: work around temporary clash with 'u'.  Needs to remain in
> this patch, obviously.  Suggest to amend the commit message to say
> 
>     On the other hand, the duplication means that we temporarily cannot
>     support 'u' as a branch name.  Adapt a few tests that do.

Or, rather, dropped entirely, because the tests for collisions with 'u'
will be deferred until after the conversion is complete.

Eric Blake <eblake@redhat.com> writes:

> On 10/22/2015 07:54 AM, Markus Armbruster wrote:
>
>> 
>> This is clever and ugly in equal measure.  I respect that.  Fortunately,
>> it's also temporary.
>> 
>>> Flat unions do not need the anonymous union for the tag member,
>>> as we already fixed that to use the member name instead of 'kind'
>>> back in commit 0f61af3e.
>> 
>> Unlike then, we need multiple commits for simple unions, because they're
>> more widely used?
>
> Yes. In fact, both you and I expressed surprise back then that the main
> body of qemu didn't need adjusting - our only use of flat unions was
> hidden behind QDict manipulations rather than direct generated qapi
> struct, explaining why nothing was affected when we converted flat
> unions.  But a useful note for the commit message at any rate.
>
>> 
>>>                           On the other hand, the duplication
>>> means that we temporarily cannot support 'u' as a branch name.
>> 
>> Separate paragraph, because now you're talking about the *other*
>> anonymous union.
>> 
>>> Later, when the conversions are complete, we will remove the
>>> duplication hacks and restore support for 'u' as a branch name.
>
> And based on comments on 3/17, I'm deferring any testsuite changes
> related to 'u' collisions until after this conversion to inline base is
> complete, so this part of the commit message actually disappears in v10
> because I'm no longer touching qapi-schema-test this early.
>
>>>
>>> Note, however, that we do not rename the generated enum, which
>>> is still 'FooKind'.  A further patch could generate implicit
>>> enums as 'FooType', but that causes more churn to C code, and
>>> gets harder since the generator already reserved the '*Kind'
>>> namespace, but there are already QMP constructs with '*Type'
>>> naming which means we cannot easily reserve it for qapi.
>> 
>> Oh, we can reserve whatever we want in QAPI, it's just a lot of churn to
>> adapt the QAPI-using code.
>> 
>> I'd simply say "but that would cause substantial churn to C code, as
>> there are already QAPI definitions with '*Type' naming".
>
> Okay.
>
>
>>>      for var in variants.variants:
>>>          # Ugly special case for simple union TODO get rid of it
>>>          typ = var.simple_union_type() or var.type
>>> -        ret += mcgen('''
>>> +        tmp += mcgen('''
>>>          %(c_type)s %(c_name)s;
>>>  ''',
>>>                       c_type=typ.c_type(),
>>>                       c_name=c_name(var.name))
>>>
>>> +    ret += tmp
>>> +    ret += '    ' + '\n    '.join(tmp.split('\n'))
>>>      ret += mcgen('''
>>> +    } u;
>>>      };
>>>  };
>>>  ''')
>> 
>> It took me some head-scratching to understand why this generates
>> correctly indented output.  If it wasn't temporary code, I'd ask for
>> cleanup.
>
> Would a comment help?  It's because we add 4 spaces after each newline,
> but need an indent prior to the first line of tmp, and the '} u;' line
> picks up four spaces after the last line of tmp.

Yes.  Let's not worry about it, it's just temporary scaffolding.

>> Second part: convert qapi-visit.py.  Not mentioned in commit message.
>> Separate patch, perhaps?
>
> Sure, I could split.
>
>
>> 
>> Third part: work around temporary clash with 'u'.  Needs to remain in
>> this patch, obviously.  Suggest to amend the commit message to say
>> 
>>     On the other hand, the duplication means that we temporarily cannot
>>     support 'u' as a branch name.  Adapt a few tests that do.
>
> Or, rather, dropped entirely, because the tests for collisions with 'u'
> will be deferred until after the conversion is complete.

If deferring is easy, go for it.