| Message ID | 87B246BB5ED53A4C98E4F9A35839EDE128F6F49D@nkgeml506-mbs.china.huawei.com |
|---|---|
| State | New |
| Headers | show |
On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > When used a tap as net driver for vm, if too many packets was delivered to the > guest os via tap interface, the guest os will be blocked on io events for a long > time, while tap driver was busying process packets. > > kvm vcpu thread block on io lock call trace: > __lll_lock_wait > _L_lock_1004 > __pthread_mutex_lock > qemu_mutex_lock > kvm_cpu_exec > qemu_kvm_cpu_thread_fn > start_thread > > qemu io thread call trace: > ... > qemu_net_queue_send > tap_send > qemu_iohandler_poll > main_loop_wait > main_loop > > > I think the qemu io lock time should be as small as possible, and the io work > slice should be limited at a particular ration or time. > > --- > Signed-off-by: Wangkai <wangkai86@huawei.com> How many packets are you seeing in a single tap_send() call? Have you profiled the tap_send() code path? Maybe it is performing some operation that is very slow. By the way, if you want good performance you should use vhost_net instead of userspace vhost_net. Userspace virtio-net is not very optimized. Stefan
> -----Original Message----- > From: Stefan Hajnoczi [mailto:stefanha@redhat.com] > Sent: Friday, July 11, 2014 9:04 PM > To: Wangkai (Kevin,C) > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > > When used a tap as net driver for vm, if too many packets was > > delivered to the guest os via tap interface, the guest os will be > > blocked on io events for a long time, while tap driver was busying > process packets. > > > > kvm vcpu thread block on io lock call trace: > > __lll_lock_wait > > _L_lock_1004 > > __pthread_mutex_lock > > qemu_mutex_lock > > kvm_cpu_exec > > qemu_kvm_cpu_thread_fn > > start_thread > > > > qemu io thread call trace: > > ... > > qemu_net_queue_send > > tap_send > > qemu_iohandler_poll > > main_loop_wait > > main_loop > > > > > > I think the qemu io lock time should be as small as possible, and the > > io work slice should be limited at a particular ration or time. > > > > --- > > Signed-off-by: Wangkai <wangkai86@huawei.com> > > How many packets are you seeing in a single tap_send() call? > > Have you profiled the tap_send() code path? Maybe it is performing > some operation that is very slow. > > By the way, if you want good performance you should use vhost_net > instead of userspace vhost_net. Userspace virtio-net is not very > optimized. > > Stefan Hi Stefan, I am not use profile, just debug with gdb and code review. When packets delivered, I found the VM was hung, and I check qemu run State by gdb, I see the call trace for IO thread and vcpu thread, and I add debug info to check how many packets within tap_send, the info below: total recv 393520 time 1539821 us total recv 1270 time 4931 us total recv 257872 time 995828 us total recv 10745 time 41438 us total recv 505387 time 2000925 us total recv 535 time 1891 us total recv 168934 time 646008 us regards, Wangkai
On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote: > > > > -----Original Message----- > > From: Stefan Hajnoczi [mailto:stefanha@redhat.com] > > Sent: Friday, July 11, 2014 9:04 PM > > To: Wangkai (Kevin,C) > > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang > > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap > > > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > > > When used a tap as net driver for vm, if too many packets was > > > delivered to the guest os via tap interface, the guest os will be > > > blocked on io events for a long time, while tap driver was busying > > process packets. > > > > > > kvm vcpu thread block on io lock call trace: > > > __lll_lock_wait > > > _L_lock_1004 > > > __pthread_mutex_lock > > > qemu_mutex_lock > > > kvm_cpu_exec > > > qemu_kvm_cpu_thread_fn > > > start_thread > > > > > > qemu io thread call trace: > > > ... > > > qemu_net_queue_send > > > tap_send > > > qemu_iohandler_poll > > > main_loop_wait > > > main_loop > > > > > > > > > I think the qemu io lock time should be as small as possible, and the > > > io work slice should be limited at a particular ration or time. > > > > > > --- > > > Signed-off-by: Wangkai <wangkai86@huawei.com> > > > > How many packets are you seeing in a single tap_send() call? > > > > Have you profiled the tap_send() code path? Maybe it is performing > > some operation that is very slow. > > > > By the way, if you want good performance you should use vhost_net > > instead of userspace vhost_net. Userspace virtio-net is not very > > optimized. > > > > Stefan > > > Hi Stefan, > > I am not use profile, just debug with gdb and code review. It's worth understanding the root cause for this behavior because something is probably wrong. > When packets delivered, I found the VM was hung, and I check qemu run > > State by gdb, I see the call trace for IO thread and vcpu thread, and > > I add debug info to check how many packets within tap_send, the info below: > > total recv 393520 time 1539821 us > total recv 1270 time 4931 us > total recv 257872 time 995828 us > total recv 10745 time 41438 us > total recv 505387 time 2000925 us 505387 packets or 505387 bytes? If that's packets, then even with small 64-byte packets that would mean 32 MB of pending data! Are you running a networking benchmark where you'd expect lots of packets? Have you checked how lot the time between tap_send() calls is? Perhaps something else in QEMU is blocking the event loop so packets accumulate in the host kernel. Stefan
> -----Original Message----- > From: Stefan Hajnoczi [mailto:stefanha@redhat.com] > Sent: Monday, July 14, 2014 4:43 PM > To: Wangkai (Kevin,C) > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap > > On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote: > > > > > > > -----Original Message----- > > > From: Stefan Hajnoczi [mailto:stefanha@redhat.com] > > > Sent: Friday, July 11, 2014 9:04 PM > > > To: Wangkai (Kevin,C) > > > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang > > > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap > > > > > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > > > > When used a tap as net driver for vm, if too many packets was > > > > delivered to the guest os via tap interface, the guest os will be > > > > blocked on io events for a long time, while tap driver was > busying > > > process packets. > > > > > > > > kvm vcpu thread block on io lock call trace: > > > > __lll_lock_wait > > > > _L_lock_1004 > > > > __pthread_mutex_lock > > > > qemu_mutex_lock > > > > kvm_cpu_exec > > > > qemu_kvm_cpu_thread_fn > > > > start_thread > > > > > > > > qemu io thread call trace: > > > > ... > > > > qemu_net_queue_send > > > > tap_send > > > > qemu_iohandler_poll > > > > main_loop_wait > > > > main_loop > > > > > > > > > > > > I think the qemu io lock time should be as small as possible, and > > > > the io work slice should be limited at a particular ration or > time. > > > > > > > > --- > > > > Signed-off-by: Wangkai <wangkai86@huawei.com> > > > > > > How many packets are you seeing in a single tap_send() call? > > > > > > Have you profiled the tap_send() code path? Maybe it is performing > > > some operation that is very slow. > > > > > > By the way, if you want good performance you should use vhost_net > > > instead of userspace vhost_net. Userspace virtio-net is not very > > > optimized. > > > > > > Stefan > > > > > > Hi Stefan, > > > > I am not use profile, just debug with gdb and code review. > > It's worth understanding the root cause for this behavior because > something is probably wrong. > > > When packets delivered, I found the VM was hung, and I check qemu run > > > > State by gdb, I see the call trace for IO thread and vcpu thread, and > > > > I add debug info to check how many packets within tap_send, the info > below: > > > > total recv 393520 time 1539821 us > > total recv 1270 time 4931 us > > total recv 257872 time 995828 us > > total recv 10745 time 41438 us > > total recv 505387 time 2000925 us > > 505387 packets or 505387 bytes? > > If that's packets, then even with small 64-byte packets that would mean > 32 MB of pending data! > > Are you running a networking benchmark where you'd expect lots of > packets? > > Have you checked how lot the time between tap_send() calls is? Perhaps > something else in QEMU is blocking the event loop so packets accumulate > in the host kernel. > > Stefan Hi Stefan, Here the detail network: +--------------------------------------------+ | The host add tap1 and eth10 to bridge 'br1'| +--------+ | +------------+ | | send | | | VM eth1-+-tap1 --- bridge --- eth10 --+---------------------+ packets| | +------------+ | | | +--------------------------------------------+ +--------+ Qemu start vm by virtio, use tap interface, option is: -net nic,vlan=101, model=virtio -net tap,vlan=101,ifname=tap1,script=no,downscript=no And add tap1 and eth10 to bridge br1 in the host: Brctl addif br1 tap1 Brctl addif br1 eth10 total recv 505387 time 2000925 us: means call tap_send once dealing 505387 packets, the packet payload was 300 bytes, and time use for tap_send() was 2,000,925 micro-seconds, time was measured by record time stamp at function tap_send() start and end. We just test the performance of VM. Regards Wangkai
On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote: > Here the detail network: > > +--------------------------------------------+ > | The host add tap1 and eth10 to bridge 'br1'| +--------+ > | +------------+ | | send | > | | VM eth1-+-tap1 --- bridge --- eth10 --+---------------------+ packets| > | +------------+ | | | > +--------------------------------------------+ +--------+ > > Qemu start vm by virtio, use tap interface, option is: > -net nic,vlan=101, model=virtio -net tap,vlan=101,ifname=tap1,script=no,downscript=no Use the newer -netdev/-device syntax to get offload support and slightly better performance: -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device virtio-net-pci,netdev=tap0 > And add tap1 and eth10 to bridge br1 in the host: > Brctl addif br1 tap1 > Brctl addif br1 eth10 > > total recv 505387 time 2000925 us: > means call tap_send once dealing 505387 packets, the packet payload was 300 bytes, and > time use for tap_send() was 2,000,925 micro-seconds, time was measured by record time stamp > at function tap_send() start and end. > > We just test the performance of VM. That is 150 MB of incoming packets in a single tap_send(). Network rx queues are maybe a few 1000 packets so I wonder what is going on here. Maybe more packets are arriving while QEMU is reading them and we keep looping. That's strange though because the virtio-net rx virtqueue should fill up (it only has 256 entries). Can you investigate more and find out exactly what is going on? It's not clear yet that adding a budget is the solution or just hiding a deeper problem. Stefan
> -----Original Message----- > From: Stefan Hajnoczi [mailto:stefanha@gmail.com] > Sent: Tuesday, July 15, 2014 11:00 PM > To: Wangkai (Kevin,C) > Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org; > aliguori@amazon.com > Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking > on tap > > On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote: > > Here the detail network: > > > > +--------------------------------------------+ > > | The host add tap1 and eth10 to bridge 'br1'| +- > -------+ > > | +------------+ | | > send | > > | | VM eth1-+-tap1 --- bridge --- eth10 --+---------------------+ > > | | packets| > > | +------------+ | | > | > > +--------------------------------------------+ > > +--------------------------------------------+ +--------+ > > > > Qemu start vm by virtio, use tap interface, option is: > > -net nic,vlan=101, model=virtio -net > > tap,vlan=101,ifname=tap1,script=no,downscript=no > > Use the newer -netdev/-device syntax to get offload support and > slightly better performance: > > -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device > virtio-net-pci,netdev=tap0 > > > And add tap1 and eth10 to bridge br1 in the host: > > Brctl addif br1 tap1 > > Brctl addif br1 eth10 > > > > total recv 505387 time 2000925 us: > > means call tap_send once dealing 505387 packets, the packet payload > > was 300 bytes, and time use for tap_send() was 2,000,925 > > micro-seconds, time was measured by record time stamp at function > tap_send() start and end. > > > > We just test the performance of VM. > > That is 150 MB of incoming packets in a single tap_send(). Network rx > queues are maybe a few 1000 packets so I wonder what is going on here. > > Maybe more packets are arriving while QEMU is reading them and we keep > looping. That's strange though because the virtio-net rx virtqueue > should fill up (it only has 256 entries). > > Can you investigate more and find out exactly what is going on? It's > not clear yet that adding a budget is the solution or just hiding a > deeper problem. > > Stefan [Wangkai (Kevin,C)] Hi Stefan, Yes, I have one machine keep sending packets 300M/second, about 1M packets per second, I am checking the code, you mean virtqueue has only 256 entries, is this member keep the value? vq->vring.num I will check and add some debug info to check this problem. I have got the most time taking by the virtio_net_receive() virtio_net_receive qemu_deliver_packet qemu_net_queue_send net_hub_port_receive qemu_deliver_packet qemu_net_queue_send tap_send qemu_iohandler_poll main_loop_wait main_loop main regards Wangkai
> -----Original Message----- > From: Stefan Hajnoczi [mailto:stefanha@gmail.com] > Sent: Tuesday, July 15, 2014 11:00 PM > To: Wangkai (Kevin,C) > Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org; > aliguori@amazon.com > Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking > on tap > > On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote: > > Here the detail network: > > > > +--------------------------------------------+ > > | The host add tap1 and eth10 to bridge 'br1'| +- > -------+ > > | +------------+ | | > send | > > | | VM eth1-+-tap1 --- bridge --- eth10 --+---------------------+ > > | | packets| > > | +------------+ | | > | > > +--------------------------------------------+ > > +--------------------------------------------+ +--------+ > > > > Qemu start vm by virtio, use tap interface, option is: > > -net nic,vlan=101, model=virtio -net > > tap,vlan=101,ifname=tap1,script=no,downscript=no > > Use the newer -netdev/-device syntax to get offload support and > slightly better performance: > > -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device > virtio-net-pci,netdev=tap0 > > > And add tap1 and eth10 to bridge br1 in the host: > > Brctl addif br1 tap1 > > Brctl addif br1 eth10 > > > > total recv 505387 time 2000925 us: > > means call tap_send once dealing 505387 packets, the packet payload > > was 300 bytes, and time use for tap_send() was 2,000,925 > > micro-seconds, time was measured by record time stamp at function > tap_send() start and end. > > > > We just test the performance of VM. > > That is 150 MB of incoming packets in a single tap_send(). Network rx > queues are maybe a few 1000 packets so I wonder what is going on here. > > Maybe more packets are arriving while QEMU is reading them and we keep > looping. That's strange though because the virtio-net rx virtqueue > should fill up (it only has 256 entries). > > Can you investigate more and find out exactly what is going on? It's > not clear yet that adding a budget is the solution or just hiding a > deeper problem. > > Stefan [Wangkai (Kevin,C)] Hi Stefan, I think I have found the problem, why 256 entries virtqueue cannot prevent packets receiving. I have start one SMP guest, which have 2 cores, one core was pending on Io, and the other core was receiving the packets, and QEMU filling the virtqueue while the guest kernel was moving the packets out from the queue and process. They were racing, only if the guest got enough packets and receive slower than QEMU sending, the virtqueue full, then finish once receive. And I have tried -netdev/-device syntax start guest again, got very few Improvement. Regards Wangkai
On 07/17/2014 11:43 AM, Wangkai (Kevin,C) wrote: > >> -----Original Message----- >> From: Stefan Hajnoczi [mailto:stefanha@gmail.com] >> Sent: Tuesday, July 15, 2014 11:00 PM >> To: Wangkai (Kevin,C) >> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org; >> aliguori@amazon.com >> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking >> on tap >> >> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote: >>> Here the detail network: >>> >>> +--------------------------------------------+ >>> | The host add tap1 and eth10 to bridge 'br1'| +- >> -------+ >>> | +------------+ | | >> send | >>> | | VM eth1-+-tap1 --- bridge --- eth10 --+---------------------+ >>> | | packets| >>> | +------------+ | | >> | >>> +--------------------------------------------+ >>> +--------------------------------------------+ +--------+ >>> >>> Qemu start vm by virtio, use tap interface, option is: >>> -net nic,vlan=101, model=virtio -net >>> tap,vlan=101,ifname=tap1,script=no,downscript=no >> Use the newer -netdev/-device syntax to get offload support and >> slightly better performance: >> >> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device >> virtio-net-pci,netdev=tap0 >> >>> And add tap1 and eth10 to bridge br1 in the host: >>> Brctl addif br1 tap1 >>> Brctl addif br1 eth10 >>> >>> total recv 505387 time 2000925 us: >>> means call tap_send once dealing 505387 packets, the packet payload >>> was 300 bytes, and time use for tap_send() was 2,000,925 >>> micro-seconds, time was measured by record time stamp at function >> tap_send() start and end. >>> We just test the performance of VM. >> That is 150 MB of incoming packets in a single tap_send(). Network rx >> queues are maybe a few 1000 packets so I wonder what is going on here. >> >> Maybe more packets are arriving while QEMU is reading them and we keep >> looping. That's strange though because the virtio-net rx virtqueue >> should fill up (it only has 256 entries). >> >> Can you investigate more and find out exactly what is going on? It's >> not clear yet that adding a budget is the solution or just hiding a >> deeper problem. >> >> Stefan > [Wangkai (Kevin,C)] > > Hi Stefan, > > I think I have found the problem, why 256 entries virtqueue cannot prevent > packets receiving. > > I have start one SMP guest, which have 2 cores, one core was pending on > Io, and the other core was receiving the packets, and QEMU filling the > virtqueue while the guest kernel was moving the packets out from the > queue and process. > > They were racing, only if the guest got enough packets and receive slower > than QEMU sending, the virtqueue full, then finish once receive. I hit the similar issue in the past. Using pktgen to inject packets from tap to guest directly, then guest was slow to respond any other io event. This is similar to the tx and we have tx_burst or tx_timer to solve it. But for rx, probably we could do some limitation in tap_send() like this patch since e1000 may have the same issue. Generically, we need some mechanism to guarantee the fairness between devices to prevent one from starving the others. > > And I have tried -netdev/-device syntax start guest again, got very few > Improvement. > > Regards > Wangkai >
> -----Original Message----- > From: Jason Wang [mailto:jasowang@redhat.com] > Sent: Thursday, July 17, 2014 1:37 PM > To: Wangkai (Kevin,C); Stefan Hajnoczi > Cc: Lee yang; qemu-devel@nongnu.org; Stefan Hajnoczi; > aliguori@amazon.com; Michael S. Tsirkin > Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking > on tap > > On 07/17/2014 11:43 AM, Wangkai (Kevin,C) wrote: > > > >> -----Original Message----- > >> From: Stefan Hajnoczi [mailto:stefanha@gmail.com] > >> Sent: Tuesday, July 15, 2014 11:00 PM > >> To: Wangkai (Kevin,C) > >> Cc: Stefan Hajnoczi; Lee yang; qemu-devel@nongnu.org; > >> aliguori@amazon.com > >> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io > blocking > >> on tap > >> > >> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote: > >>> Here the detail network: > >>> > >>> +--------------------------------------------+ > >>> | The host add tap1 and eth10 to bridge 'br1'| > +- > >> -------+ > >>> | +------------+ | > | > >> send | > >>> | | VM eth1-+-tap1 --- bridge --- eth10 --+--------------------- > + > >>> | | packets| > >>> | +------------+ | > | > >> | > >>> +--------------------------------------------+ > >>> +--------------------------------------------+ +--------+ > >>> > >>> Qemu start vm by virtio, use tap interface, option is: > >>> -net nic,vlan=101, model=virtio -net > >>> tap,vlan=101,ifname=tap1,script=no,downscript=no > >> Use the newer -netdev/-device syntax to get offload support and > >> slightly better performance: > >> > >> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device > >> virtio-net-pci,netdev=tap0 > >> > >>> And add tap1 and eth10 to bridge br1 in the host: > >>> Brctl addif br1 tap1 > >>> Brctl addif br1 eth10 > >>> > >>> total recv 505387 time 2000925 us: > >>> means call tap_send once dealing 505387 packets, the packet payload > >>> was 300 bytes, and time use for tap_send() was 2,000,925 > >>> micro-seconds, time was measured by record time stamp at function > >> tap_send() start and end. > >>> We just test the performance of VM. > >> That is 150 MB of incoming packets in a single tap_send(). Network > >> rx queues are maybe a few 1000 packets so I wonder what is going on > here. > >> > >> Maybe more packets are arriving while QEMU is reading them and we > >> keep looping. That's strange though because the virtio-net rx > >> virtqueue should fill up (it only has 256 entries). > >> > >> Can you investigate more and find out exactly what is going on? > It's > >> not clear yet that adding a budget is the solution or just hiding a > >> deeper problem. > >> > >> Stefan > > [Wangkai (Kevin,C)] > > > > Hi Stefan, > > > > I think I have found the problem, why 256 entries virtqueue cannot > > prevent packets receiving. > > > > I have start one SMP guest, which have 2 cores, one core was pending > > on Io, and the other core was receiving the packets, and QEMU filling > > the virtqueue while the guest kernel was moving the packets out from > > the queue and process. > > > > They were racing, only if the guest got enough packets and receive > > slower than QEMU sending, the virtqueue full, then finish once > receive. > I hit the similar issue in the past. Using pktgen to inject packets > from tap to guest directly, then guest was slow to respond any other io > event. > > This is similar to the tx and we have tx_burst or tx_timer to solve it. > But for rx, probably we could do some limitation in tap_send() like > this patch since e1000 may have the same issue. > > Generically, we need some mechanism to guarantee the fairness between > devices to prevent one from starving the others. [Wangkai (Kevin,C)] Yes, the QEMU io shared one thread, and locked, fairness should be guaranteed between io events and the io lock should be as small as possible. > > > > And I have tried -netdev/-device syntax start guest again, got very > > few Improvement. > > > > Regards > > Wangkai > >
On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > When used a tap as net driver for vm, if too many packets was delivered to the > guest os via tap interface, the guest os will be blocked on io events for a long > time, while tap driver was busying process packets. Please tweak this description to explain clearly that the tap_send() loop can execute for a very long time if more packets are received by the host during tap_send(). It seems that the guest does not require the QEMU global mutex often during virtio-net receive, otherwise the rx virtqueue would fill up and tap_send() would return. > diff --git a/net/tap.c b/net/tap.c > index a40f7f0..df9a0eb 100644 > --- a/net/tap.c > +++ b/net/tap.c > @@ -189,6 +189,7 @@ static void tap_send(void *opaque) > { > TAPState *s = opaque; > int size; > + int pkt = 0; > > while (qemu_can_send_packet(&s->nc)) { > uint8_t *buf = s->buf; > @@ -210,6 +211,11 @@ static void tap_send(void *opaque) > } else if (size < 0) { > break; > } > + > + /* limit io block time slice for 50 packets */ > + pkt++; > + if (pkt >= 50) > + break; Please use scripts/checkpatch.pl to check coding style. QEMU always uses curlies around if statement bodies, even when they are only one line.
diff --git a/net/tap.c b/net/tap.c index a40f7f0..df9a0eb 100644 --- a/net/tap.c +++ b/net/tap.c @@ -189,6 +189,7 @@ static void tap_send(void *opaque) { TAPState *s = opaque; int size; + int pkt = 0; while (qemu_can_send_packet(&s->nc)) { uint8_t *buf = s->buf; @@ -210,6 +211,11 @@ static void tap_send(void *opaque) } else if (size < 0) { break; } + + /* limit io block time slice for 50 packets */ + pkt++; + if (pkt >= 50) + break; } }
When used a tap as net driver for vm, if too many packets was delivered to the guest os via tap interface, the guest os will be blocked on io events for a long time, while tap driver was busying process packets. kvm vcpu thread block on io lock call trace: __lll_lock_wait _L_lock_1004 __pthread_mutex_lock qemu_mutex_lock kvm_cpu_exec qemu_kvm_cpu_thread_fn start_thread qemu io thread call trace: ... qemu_net_queue_send tap_send qemu_iohandler_poll main_loop_wait main_loop I think the qemu io lock time should be as small as possible, and the io work slice should be limited at a particular ration or time. --- Signed-off-by: Wangkai <wangkai86@huawei.com> --- 2.0.0