Patchwork virtio: limit avail bytes lookahead

login
register
mail settings
Submitter Michael S. Tsirkin
Date Nov. 1, 2012, 4:07 p.m.
Message ID <20121101160721.GA9176@redhat.com>
Download mbox | patch
Permalink /patch/196268/
State New
Headers show

Comments

Michael S. Tsirkin - Nov. 1, 2012, 4:07 p.m.
Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
a regression in virtio-net performance because it looks
into the ring aggressively while we really only care
about a single packet worth of buffers.
To fix, add parameters limiting lookahead, and
use in virtqueue_avail_bytes.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>

---

Edivaldo could you please confirm this fixes regression?
Amit Shah - Nov. 2, 2012, 9:56 a.m.
On (Thu) 01 Nov 2012 [18:07:21], Michael S. Tsirkin wrote:
> Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
> a regression in virtio-net performance because it looks
> into the ring aggressively while we really only care
> about a single packet worth of buffers.
> To fix, add parameters limiting lookahead, and
> use in virtqueue_avail_bytes.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>

Acked-by: Amit Shah <amit.shah@redhat.com>

		Amit
Stefan Hajnoczi - Nov. 2, 2012, 10:18 a.m.
On Thu, Nov 1, 2012 at 5:07 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
> a regression in virtio-net performance because it looks
> into the ring aggressively while we really only care
> about a single packet worth of buffers.
> To fix, add parameters limiting lookahead, and
> use in virtqueue_avail_bytes.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>

Nice, much simpler than the ideas I had.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Michael S. Tsirkin - Nov. 2, 2012, 2:48 p.m.
On Fri, Nov 02, 2012 at 11:18:18AM +0100, Stefan Hajnoczi wrote:
> On Thu, Nov 1, 2012 at 5:07 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
> > a regression in virtio-net performance because it looks
> > into the ring aggressively while we really only care
> > about a single packet worth of buffers.
> > To fix, add parameters limiting lookahead, and
> > use in virtqueue_avail_bytes.
> >
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>
> 
> Nice, much simpler than the ideas I had.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Anthony could you apply this out of band please so this stops
biting people?

Thanks,
MST
Stefan Hajnoczi - Nov. 2, 2012, 7:44 p.m.
On Fri, Nov 2, 2012 at 3:48 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Fri, Nov 02, 2012 at 11:18:18AM +0100, Stefan Hajnoczi wrote:
>> On Thu, Nov 1, 2012 at 5:07 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
>> > a regression in virtio-net performance because it looks
>> > into the ring aggressively while we really only care
>> > about a single packet worth of buffers.
>> > To fix, add parameters limiting lookahead, and
>> > use in virtqueue_avail_bytes.
>> >
>> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> > Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>
>>
>> Nice, much simpler than the ideas I had.
>>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>
> Anthony could you apply this out of band please so this stops
> biting people?

Especially for the 1.3 release so that we don't have a virtio
performance regression.

Stefan
Michael S. Tsirkin - Nov. 27, 2012, 4:25 p.m.
On Thu, Nov 01, 2012 at 06:07:21PM +0200, Michael S. Tsirkin wrote:
> Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
> a regression in virtio-net performance because it looks
> into the ring aggressively while we really only care
> about a single packet worth of buffers.
> To fix, add parameters limiting lookahead, and
> use in virtqueue_avail_bytes.
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>

Ping.
Anthony - going to apply this?


> ---
> 
> Edivaldo could you please confirm this fixes regression?
> 
> diff --git a/hw/virtio-serial-bus.c b/hw/virtio-serial-bus.c
> index d20bd8b..a761cdc 100644
> --- a/hw/virtio-serial-bus.c
> +++ b/hw/virtio-serial-bus.c
> @@ -297,7 +297,7 @@ size_t virtio_serial_guest_ready(VirtIOSerialPort *port)
>      if (use_multiport(port->vser) && !port->guest_connected) {
>          return 0;
>      }
> -    virtqueue_get_avail_bytes(vq, &bytes, NULL);
> +    virtqueue_get_avail_bytes(vq, &bytes, NULL, UINT_MAX, 0);
>      return bytes;
>  }
>  
> diff --git a/hw/virtio.c b/hw/virtio.c
> index ec8b7d8..f40a8c5 100644
> --- a/hw/virtio.c
> +++ b/hw/virtio.c
> @@ -336,7 +336,8 @@ static unsigned virtqueue_next_desc(hwaddr desc_pa,
>  }
>  
>  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
> -                               unsigned int *out_bytes)
> +                               unsigned int *out_bytes,
> +                               unsigned max_in_bytes, unsigned max_out_bytes)
>  {
>      unsigned int idx;
>      unsigned int total_bufs, in_total, out_total;
> @@ -385,6 +386,9 @@ void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>              } else {
>                  out_total += vring_desc_len(desc_pa, i);
>              }
> +            if (in_total >= max_in_bytes && out_total >= max_out_bytes) {
> +                goto done;
> +            }
>          } while ((i = virtqueue_next_desc(desc_pa, i, max)) != max);
>  
>          if (!indirect)
> @@ -392,6 +396,7 @@ void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>          else
>              total_bufs++;
>      }
> +done:
>      if (in_bytes) {
>          *in_bytes = in_total;
>      }
> @@ -405,12 +410,8 @@ int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
>  {
>      unsigned int in_total, out_total;
>  
> -    virtqueue_get_avail_bytes(vq, &in_total, &out_total);
> -    if ((in_bytes && in_bytes < in_total)
> -        || (out_bytes && out_bytes < out_total)) {
> -        return 1;
> -    }
> -    return 0;
> +    virtqueue_get_avail_bytes(vq, &in_total, &out_total, in_bytes, out_bytes);
> +    return in_bytes <= in_total && out_bytes <= out_total;
>  }
>  
>  void virtqueue_map_sg(struct iovec *sg, hwaddr *addr,
> diff --git a/hw/virtio.h b/hw/virtio.h
> index ac482be..1278328 100644
> --- a/hw/virtio.h
> +++ b/hw/virtio.h
> @@ -150,7 +150,8 @@ int virtqueue_pop(VirtQueue *vq, VirtQueueElement *elem);
>  int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
>                            unsigned int out_bytes);
>  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
> -                               unsigned int *out_bytes);
> +                               unsigned int *out_bytes,
> +                               unsigned max_in_bytes, unsigned max_out_bytes);
>  
>  void virtio_notify(VirtIODevice *vdev, VirtQueue *vq);
>
Edivaldo de Araujo Pereira - Nov. 27, 2012, 4:54 p.m.
Dear friends, 

Please excuse-me for not reporting earlier... I confirm that the patch by Michael really fixes the problem I've reported. The regression has gone away when I used it, so I think it is good to be applied.

Thanks,

Edivaldo de Araújo Pereira


--- Em ter, 27/11/12, Michael S. Tsirkin <mst@redhat.com> escreveu:

> De: Michael S. Tsirkin <mst@redhat.com>
> Assunto: Re: [PATCH] virtio: limit avail bytes lookahead
> Para: "Amit Shah" <amit.shah@redhat.com>
> Cc: "Stefan Hajnoczi" <stefanha@gmail.com>, "Edivaldo de Araujo Pereira" <edivaldoapereira@yahoo.com.br>, qemu-devel@nongnu.org, "Anthony Liguori" <anthony@codemonkey.ws>, "Bug 1066055" <1066055@bugs.launchpad.net>
> Data: Terça-feira, 27 de Novembro de 2012, 8:25
> On Thu, Nov 01, 2012 at 06:07:21PM
> +0200, Michael S. Tsirkin wrote:
> > Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f
> introduced
> > a regression in virtio-net performance because it
> looks
> > into the ring aggressively while we really only care
> > about a single packet worth of buffers.
> > To fix, add parameters limiting lookahead, and
> > use in virtqueue_avail_bytes.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>
> 
> Ping.
> Anthony - going to apply this?
> 
> 
> > ---
> > 
> > Edivaldo could you please confirm this fixes
> regression?
> > 
> > diff --git a/hw/virtio-serial-bus.c
> b/hw/virtio-serial-bus.c
> > index d20bd8b..a761cdc 100644
> > --- a/hw/virtio-serial-bus.c
> > +++ b/hw/virtio-serial-bus.c
> > @@ -297,7 +297,7 @@ size_t
> virtio_serial_guest_ready(VirtIOSerialPort *port)
> >      if (use_multiport(port->vser)
> && !port->guest_connected) {
> >          return 0;
> >      }
> > -    virtqueue_get_avail_bytes(vq,
> &bytes, NULL);
> > +    virtqueue_get_avail_bytes(vq,
> &bytes, NULL, UINT_MAX, 0);
> >      return bytes;
> >  }
> >  
> > diff --git a/hw/virtio.c b/hw/virtio.c
> > index ec8b7d8..f40a8c5 100644
> > --- a/hw/virtio.c
> > +++ b/hw/virtio.c
> > @@ -336,7 +336,8 @@ static unsigned
> virtqueue_next_desc(hwaddr desc_pa,
> >  }
> >  
> >  void virtqueue_get_avail_bytes(VirtQueue *vq,
> unsigned int *in_bytes,
> > -             
>              
>    unsigned int *out_bytes)
> > +             
>              
>    unsigned int *out_bytes,
> > +             
>              
>    unsigned max_in_bytes, unsigned
> max_out_bytes)
> >  {
> >      unsigned int idx;
> >      unsigned int total_bufs, in_total,
> out_total;
> > @@ -385,6 +386,9 @@ void
> virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int
> *in_bytes,
> >              } else
> {
> >               
>   out_total += vring_desc_len(desc_pa, i);
> >              }
> > +            if (in_total
> >= max_in_bytes && out_total >= max_out_bytes)
> {
> > +             
>   goto done;
> > +            }
> >          } while ((i =
> virtqueue_next_desc(desc_pa, i, max)) != max);
> >  
> >          if (!indirect)
> > @@ -392,6 +396,7 @@ void
> virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int
> *in_bytes,
> >          else
> >             
> total_bufs++;
> >      }
> > +done:
> >      if (in_bytes) {
> >          *in_bytes =
> in_total;
> >      }
> > @@ -405,12 +410,8 @@ int
> virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
> >  {
> >      unsigned int in_total, out_total;
> >  
> > -    virtqueue_get_avail_bytes(vq,
> &in_total, &out_total);
> > -    if ((in_bytes && in_bytes <
> in_total)
> > -        || (out_bytes &&
> out_bytes < out_total)) {
> > -        return 1;
> > -    }
> > -    return 0;
> > +    virtqueue_get_avail_bytes(vq,
> &in_total, &out_total, in_bytes, out_bytes);
> > +    return in_bytes <= in_total
> && out_bytes <= out_total;
> >  }
> >  
> >  void virtqueue_map_sg(struct iovec *sg, hwaddr
> *addr,
> > diff --git a/hw/virtio.h b/hw/virtio.h
> > index ac482be..1278328 100644
> > --- a/hw/virtio.h
> > +++ b/hw/virtio.h
> > @@ -150,7 +150,8 @@ int virtqueue_pop(VirtQueue *vq,
> VirtQueueElement *elem);
> >  int virtqueue_avail_bytes(VirtQueue *vq, unsigned
> int in_bytes,
> >               
>             unsigned int
> out_bytes);
> >  void virtqueue_get_avail_bytes(VirtQueue *vq,
> unsigned int *in_bytes,
> > -             
>              
>    unsigned int *out_bytes);
> > +             
>              
>    unsigned int *out_bytes,
> > +             
>              
>    unsigned max_in_bytes, unsigned
> max_out_bytes);
> >  
> >  void virtio_notify(VirtIODevice *vdev, VirtQueue
> *vq);
> >  
>
Anthony Liguori - Nov. 27, 2012, 7:47 p.m.
"Michael S. Tsirkin" <mst@redhat.com> writes:

> On Thu, Nov 01, 2012 at 06:07:21PM +0200, Michael S. Tsirkin wrote:
>> Commit 0d8d7690850eb0cf2b2b60933cf47669a6b6f18f introduced
>> a regression in virtio-net performance because it looks
>> into the ring aggressively while we really only care
>> about a single packet worth of buffers.
>> To fix, add parameters limiting lookahead, and
>> use in virtqueue_avail_bytes.
>> 
>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>> Reported-by: Edivaldo de Araujo Pereira <edivaldoapereira@yahoo.com.br>
>
> Ping.
> Anthony - going to apply this?

Yes, I've got it queued now.  In the future, please top post patches.

Regards,

Anthony Liguori

>
>
>> ---
>> 
>> Edivaldo could you please confirm this fixes regression?
>> 
>> diff --git a/hw/virtio-serial-bus.c b/hw/virtio-serial-bus.c
>> index d20bd8b..a761cdc 100644
>> --- a/hw/virtio-serial-bus.c
>> +++ b/hw/virtio-serial-bus.c
>> @@ -297,7 +297,7 @@ size_t virtio_serial_guest_ready(VirtIOSerialPort *port)
>>      if (use_multiport(port->vser) && !port->guest_connected) {
>>          return 0;
>>      }
>> -    virtqueue_get_avail_bytes(vq, &bytes, NULL);
>> +    virtqueue_get_avail_bytes(vq, &bytes, NULL, UINT_MAX, 0);
>>      return bytes;
>>  }
>>  
>> diff --git a/hw/virtio.c b/hw/virtio.c
>> index ec8b7d8..f40a8c5 100644
>> --- a/hw/virtio.c
>> +++ b/hw/virtio.c
>> @@ -336,7 +336,8 @@ static unsigned virtqueue_next_desc(hwaddr desc_pa,
>>  }
>>  
>>  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>> -                               unsigned int *out_bytes)
>> +                               unsigned int *out_bytes,
>> +                               unsigned max_in_bytes, unsigned max_out_bytes)
>>  {
>>      unsigned int idx;
>>      unsigned int total_bufs, in_total, out_total;
>> @@ -385,6 +386,9 @@ void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>>              } else {
>>                  out_total += vring_desc_len(desc_pa, i);
>>              }
>> +            if (in_total >= max_in_bytes && out_total >= max_out_bytes) {
>> +                goto done;
>> +            }
>>          } while ((i = virtqueue_next_desc(desc_pa, i, max)) != max);
>>  
>>          if (!indirect)
>> @@ -392,6 +396,7 @@ void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>>          else
>>              total_bufs++;
>>      }
>> +done:
>>      if (in_bytes) {
>>          *in_bytes = in_total;
>>      }
>> @@ -405,12 +410,8 @@ int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
>>  {
>>      unsigned int in_total, out_total;
>>  
>> -    virtqueue_get_avail_bytes(vq, &in_total, &out_total);
>> -    if ((in_bytes && in_bytes < in_total)
>> -        || (out_bytes && out_bytes < out_total)) {
>> -        return 1;
>> -    }
>> -    return 0;
>> +    virtqueue_get_avail_bytes(vq, &in_total, &out_total, in_bytes, out_bytes);
>> +    return in_bytes <= in_total && out_bytes <= out_total;
>>  }
>>  
>>  void virtqueue_map_sg(struct iovec *sg, hwaddr *addr,
>> diff --git a/hw/virtio.h b/hw/virtio.h
>> index ac482be..1278328 100644
>> --- a/hw/virtio.h
>> +++ b/hw/virtio.h
>> @@ -150,7 +150,8 @@ int virtqueue_pop(VirtQueue *vq, VirtQueueElement *elem);
>>  int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
>>                            unsigned int out_bytes);
>>  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
>> -                               unsigned int *out_bytes);
>> +                               unsigned int *out_bytes,
>> +                               unsigned max_in_bytes, unsigned max_out_bytes);
>>  
>>  void virtio_notify(VirtIODevice *vdev, VirtQueue *vq);
>>

Patch

diff --git a/hw/virtio-serial-bus.c b/hw/virtio-serial-bus.c
index d20bd8b..a761cdc 100644
--- a/hw/virtio-serial-bus.c
+++ b/hw/virtio-serial-bus.c
@@ -297,7 +297,7 @@  size_t virtio_serial_guest_ready(VirtIOSerialPort *port)
     if (use_multiport(port->vser) && !port->guest_connected) {
         return 0;
     }
-    virtqueue_get_avail_bytes(vq, &bytes, NULL);
+    virtqueue_get_avail_bytes(vq, &bytes, NULL, UINT_MAX, 0);
     return bytes;
 }
 
diff --git a/hw/virtio.c b/hw/virtio.c
index ec8b7d8..f40a8c5 100644
--- a/hw/virtio.c
+++ b/hw/virtio.c
@@ -336,7 +336,8 @@  static unsigned virtqueue_next_desc(hwaddr desc_pa,
 }
 
 void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
-                               unsigned int *out_bytes)
+                               unsigned int *out_bytes,
+                               unsigned max_in_bytes, unsigned max_out_bytes)
 {
     unsigned int idx;
     unsigned int total_bufs, in_total, out_total;
@@ -385,6 +386,9 @@  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
             } else {
                 out_total += vring_desc_len(desc_pa, i);
             }
+            if (in_total >= max_in_bytes && out_total >= max_out_bytes) {
+                goto done;
+            }
         } while ((i = virtqueue_next_desc(desc_pa, i, max)) != max);
 
         if (!indirect)
@@ -392,6 +396,7 @@  void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
         else
             total_bufs++;
     }
+done:
     if (in_bytes) {
         *in_bytes = in_total;
     }
@@ -405,12 +410,8 @@  int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
 {
     unsigned int in_total, out_total;
 
-    virtqueue_get_avail_bytes(vq, &in_total, &out_total);
-    if ((in_bytes && in_bytes < in_total)
-        || (out_bytes && out_bytes < out_total)) {
-        return 1;
-    }
-    return 0;
+    virtqueue_get_avail_bytes(vq, &in_total, &out_total, in_bytes, out_bytes);
+    return in_bytes <= in_total && out_bytes <= out_total;
 }
 
 void virtqueue_map_sg(struct iovec *sg, hwaddr *addr,
diff --git a/hw/virtio.h b/hw/virtio.h
index ac482be..1278328 100644
--- a/hw/virtio.h
+++ b/hw/virtio.h
@@ -150,7 +150,8 @@  int virtqueue_pop(VirtQueue *vq, VirtQueueElement *elem);
 int virtqueue_avail_bytes(VirtQueue *vq, unsigned int in_bytes,
                           unsigned int out_bytes);
 void virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
-                               unsigned int *out_bytes);
+                               unsigned int *out_bytes,
+                               unsigned max_in_bytes, unsigned max_out_bytes);
 
 void virtio_notify(VirtIODevice *vdev, VirtQueue *vq);