iotest 233 failing

I'm now getting failures on iotest 233:

233   fail       [15:26:01] [15:26:03]   2.1s   (last: 1.3s)  output mismatch (see 233.out.bad)
Failures: 233
Failed 1 of 1 iotests

Looks like I recently updated to gnutls-3.7.2-1.fc34 on June 1, could
that be the culprit for the error message being reordered?

On Thu, Jun 10, 2021 at 03:34:46PM -0500, Eric Blake wrote:
> I'm now getting failures on iotest 233:
> 
> 233   fail       [15:26:01] [15:26:03]   2.1s   (last: 1.3s)  output mismatch (see 233.out.bad)
> --- /home/eblake/qemu/tests/qemu-iotests/233.out
> +++ 233.out.bad
> @@ -65,6 +65,6 @@
>  == final server log ==
>  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
>  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
> -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client1,L=R'lyeh,C=South Pacific is denied
> -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client3,L=R'lyeh,C=South Pacific is denied
> +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client1,CN=localhost is denied
> +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client3,CN=localhost is denied
>  *** done
> Failures: 233
> Failed 1 of 1 iotests
> 
> Looks like I recently updated to gnutls-3.7.2-1.fc34 on June 1, could
> that be the culprit for the error message being reordered?

It is possible I guess. They have indeed made such a change in the past
and reverted it when I pointed out that this is effectively an ABI for
apps, because access control lists are based on matching the distinguish
name string, as an opaque string. The cause certainly needs investigating
as a matter of urgency because this is ABI for QEMU's authz access control
lists.


Regards,
Daniel

On Thu, Jun 10, 2021 at 10:31:14PM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 10, 2021 at 03:34:46PM -0500, Eric Blake wrote:
> > I'm now getting failures on iotest 233:
> > 
> > 233   fail       [15:26:01] [15:26:03]   2.1s   (last: 1.3s)  output mismatch (see 233.out.bad)
> > --- /home/eblake/qemu/tests/qemu-iotests/233.out
> > +++ 233.out.bad
> > @@ -65,6 +65,6 @@
> >  == final server log ==
> >  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
> >  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
> > -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client1,L=R'lyeh,C=South Pacific is denied
> > -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client3,L=R'lyeh,C=South Pacific is denied
> > +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client1,CN=localhost is denied
> > +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client3,CN=localhost is denied
> >  *** done
> > Failures: 233
> > Failed 1 of 1 iotests
> > 
> > Looks like I recently updated to gnutls-3.7.2-1.fc34 on June 1, could
> > that be the culprit for the error message being reordered?
> 
> It is possible I guess. They have indeed made such a change in the past
> and reverted it when I pointed out that this is effectively an ABI for
> apps, because access control lists are based on matching the distinguish
> name string, as an opaque string. The cause certainly needs investigating
> as a matter of urgency because this is ABI for QEMU's authz access control
> lists.

There is an ominous sounding NEWS item in 3.7.2

** certtool: When producing certificates and certificate requests, subject DN
   components that are provided individually will now be ordered by
   assumed scale (e.g. Country before State, Organization before
   OrganizationalUnit).  This change also affects the order in which
   certtool prompts interactively.  Please rely on the template
   mechanism for automated use of certtool! (#1243)

This ordering change in certtool seems to correspond with the new order
you see above in the distinguished name, so I wonder if the certtool
change had accidental side effects.


Regards,
Daniel

On Thu, Jun 10, 2021 at 10:33:40PM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 10, 2021 at 10:31:14PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Jun 10, 2021 at 03:34:46PM -0500, Eric Blake wrote:
> > > I'm now getting failures on iotest 233:
> > > 
> > > 233   fail       [15:26:01] [15:26:03]   2.1s   (last: 1.3s)  output mismatch (see 233.out.bad)
> > > --- /home/eblake/qemu/tests/qemu-iotests/233.out
> > > +++ 233.out.bad
> > > @@ -65,6 +65,6 @@
> > >  == final server log ==
> > >  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
> > >  qemu-nbd: option negotiation failed: Verify failed: No certificate was found.
> > > -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client1,L=R'lyeh,C=South Pacific is denied
> > > -qemu-nbd: option negotiation failed: TLS x509 authz check for CN=localhost,O=Cthulhu Dark Lord Enterprises client3,L=R'lyeh,C=South Pacific is denied
> > > +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client1,CN=localhost is denied
> > > +qemu-nbd: option negotiation failed: TLS x509 authz check for C=South Pacific,L=R'lyeh,O=Cthulhu Dark Lord Enterprises client3,CN=localhost is denied
> > >  *** done
> > > Failures: 233
> > > Failed 1 of 1 iotests
> > > 
> > > Looks like I recently updated to gnutls-3.7.2-1.fc34 on June 1, could
> > > that be the culprit for the error message being reordered?
> > 
> > It is possible I guess. They have indeed made such a change in the past
> > and reverted it when I pointed out that this is effectively an ABI for
> > apps, because access control lists are based on matching the distinguish
> > name string, as an opaque string. The cause certainly needs investigating
> > as a matter of urgency because this is ABI for QEMU's authz access control
> > lists.
> 
> There is an ominous sounding NEWS item in 3.7.2
> 
> ** certtool: When producing certificates and certificate requests, subject DN
>    components that are provided individually will now be ordered by
>    assumed scale (e.g. Country before State, Organization before
>    OrganizationalUnit).  This change also affects the order in which
>    certtool prompts interactively.  Please rely on the template
>    mechanism for automated use of certtool! (#1243)
> 
> This ordering change in certtool seems to correspond with the new order
> you see above in the distinguished name, so I wonder if the certtool
> change had accidental side effects.

Right so iotest 233 of course creates a new certificate, and so it picks
up the new certtool behaviour, which means the certificate it generates
has the distinguished name in the new order. This is good because it
means the gnutls API for querying the distinguished name is still using
the same ordering as before. This is bad because the iotest is obviously
susceptible to changes it the way the certificate is created.

I think we might just need to apply a filter to strip the distinguished
name from the output.

Regards,
Daniel