| Message ID | 1369227018-27837-1-git-send-email-stefanha@redhat.com |
|---|---|
| State | New |
| Headers | show |
Am 22.05.2013 14:50, schrieb Stefan Hajnoczi: > Net queues support efficient "receive disable". For example, tap's file > descriptor will not be polled while its peer has receive disabled. This > saves CPU cycles for needlessly copying and then dropping packets which > the peer cannot receive. > > rtl8139 is missing the qemu_flush_queued_packets() call that wakes the > queue up when receive becomes possible again. > > As a result, the Windows 7 guest driver reaches a state where the > rtl8139 cannot receive packets. The driver has actually refilled the > receive buffer but we never resume reception. > > The bug can be reproduced by running a large FTP 'get' inside a Windows > 7 guest: > > $ qemu -netdev tap,id=tap0,... > -device rtl8139,netdev=tap0 > > The Linux guest driver does not trigger the bug, probably due to a > different buffer management strategy. > > Reported-by: Oliver Francke <oliver.francke@filoo.de> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Sounds as if we should Cc: qemu-stable@nongnu.org Andreas > --- > hw/net/rtl8139.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/hw/net/rtl8139.c b/hw/net/rtl8139.c > index 9369507..7993f9f 100644 > --- a/hw/net/rtl8139.c > +++ b/hw/net/rtl8139.c > @@ -2575,6 +2575,9 @@ static void rtl8139_RxBufPtr_write(RTL8139State *s, uint32_t val) > /* this value is off by 16 */ > s->RxBufPtr = MOD2(val + 0x10, s->RxBufferSize); > > + /* more buffer space may be available so try to receive */ > + qemu_flush_queued_packets(qemu_get_queue(s->nic)); > + > DPRINTF(" CAPR write: rx buffer length %d head 0x%04x read 0x%04x\n", > s->RxBufferSize, s->RxBufAddr, s->RxBufPtr); > } >
On Wed, May 22, 2013 at 2:53 PM, Andreas Färber <afaerber@suse.de> wrote: > Am 22.05.2013 14:50, schrieb Stefan Hajnoczi: >> Net queues support efficient "receive disable". For example, tap's file >> descriptor will not be polled while its peer has receive disabled. This >> saves CPU cycles for needlessly copying and then dropping packets which >> the peer cannot receive. >> >> rtl8139 is missing the qemu_flush_queued_packets() call that wakes the >> queue up when receive becomes possible again. >> >> As a result, the Windows 7 guest driver reaches a state where the >> rtl8139 cannot receive packets. The driver has actually refilled the >> receive buffer but we never resume reception. >> >> The bug can be reproduced by running a large FTP 'get' inside a Windows >> 7 guest: >> >> $ qemu -netdev tap,id=tap0,... >> -device rtl8139,netdev=tap0 >> >> The Linux guest driver does not trigger the bug, probably due to a >> different buffer management strategy. >> >> Reported-by: Oliver Francke <oliver.francke@filoo.de> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > > Sounds as if we should > > Cc: qemu-stable@nongnu.org Yes, please. Oliver just confirmed that it fixes the issue for him on IRC so this is good for QEMU 1.5.1. Stefan
On Wed, May 22, 2013 at 02:50:18PM +0200, Stefan Hajnoczi wrote: > Net queues support efficient "receive disable". For example, tap's file > descriptor will not be polled while its peer has receive disabled. This > saves CPU cycles for needlessly copying and then dropping packets which > the peer cannot receive. > > rtl8139 is missing the qemu_flush_queued_packets() call that wakes the > queue up when receive becomes possible again. > > As a result, the Windows 7 guest driver reaches a state where the > rtl8139 cannot receive packets. The driver has actually refilled the > receive buffer but we never resume reception. > > The bug can be reproduced by running a large FTP 'get' inside a Windows > 7 guest: > > $ qemu -netdev tap,id=tap0,... > -device rtl8139,netdev=tap0 > > The Linux guest driver does not trigger the bug, probably due to a > different buffer management strategy. > > Reported-by: Oliver Francke <oliver.francke@filoo.de> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > --- > hw/net/rtl8139.c | 3 +++ > 1 file changed, 3 insertions(+) Applied to my net tree: https://github.com/stefanha/qemu/commits/net Stefan
Hi all,
I ocassionally have seen a probably related problem in the past. It mainly happend with rtl8139 under
WinXP where we most likely use rtl8139 due to lack of shipped e1000 drivers.
My question is if you see increasing dropped packets on the tap device if this problem occurs?
tap36 Link encap:Ethernet HWaddr b2:84:23:c0:e2:c0
inet6 addr: fe80::b084:23ff:fec0:e2c0/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:5816096 errors:0 dropped:0 overruns:0 frame:0
TX packets:3878744 errors:0 dropped:13775 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:5161769434 (5.1 GB) TX bytes:380415916 (380.4 MB)
In my case as well the only option to recover without shutting down the whole vServer is Live Migration
to another Node.
However, I also see this problem under qemu-kvm-1.2.0 while Oliver reported it does not happen there.
Thank you,
Peter
On 22.05.2013 14:50, Stefan Hajnoczi wrote:
> Net queues support efficient "receive disable". For example, tap's file
> descriptor will not be polled while its peer has receive disabled. This
> saves CPU cycles for needlessly copying and then dropping packets which
> the peer cannot receive.
>
> rtl8139 is missing the qemu_flush_queued_packets() call that wakes the
> queue up when receive becomes possible again.
>
> As a result, the Windows 7 guest driver reaches a state where the
> rtl8139 cannot receive packets. The driver has actually refilled the
> receive buffer but we never resume reception.
>
> The bug can be reproduced by running a large FTP 'get' inside a Windows
> 7 guest:
>
> $ qemu -netdev tap,id=tap0,...
> -device rtl8139,netdev=tap0
>
> The Linux guest driver does not trigger the bug, probably due to a
> different buffer management strategy.
>
> Reported-by: Oliver Francke <oliver.francke@filoo.de>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> hw/net/rtl8139.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/hw/net/rtl8139.c b/hw/net/rtl8139.c
> index 9369507..7993f9f 100644
> --- a/hw/net/rtl8139.c
> +++ b/hw/net/rtl8139.c
> @@ -2575,6 +2575,9 @@ static void rtl8139_RxBufPtr_write(RTL8139State *s, uint32_t val)
> /* this value is off by 16 */
> s->RxBufPtr = MOD2(val + 0x10, s->RxBufferSize);
>
> + /* more buffer space may be available so try to receive */
> + qemu_flush_queued_packets(qemu_get_queue(s->nic));
> +
> DPRINTF(" CAPR write: rx buffer length %d head 0x%04x read 0x%04x\n",
> s->RxBufferSize, s->RxBufAddr, s->RxBufPtr);
> }
On Mon, May 27, 2013 at 08:15:42AM +0200, Peter Lieven wrote: > I ocassionally have seen a probably related problem in the past. It mainly happend with rtl8139 under > WinXP where we most likely use rtl8139 due to lack of shipped e1000 drivers. > > My question is if you see increasing dropped packets on the tap device if this problem occurs? > > tap36 Link encap:Ethernet HWaddr b2:84:23:c0:e2:c0 > inet6 addr: fe80::b084:23ff:fec0:e2c0/64 Scope:Link > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:5816096 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3878744 errors:0 dropped:13775 overruns:0 carrier:0 > collisions:0 txqueuelen:500 > RX bytes:5161769434 (5.1 GB) TX bytes:380415916 (380.4 MB) My reading of the tun code is that will see TX dropped increase. This is because tun keeps a finite size queue of tx packets. Since QEMU userspace is not monitoring the tap fd anymore we'll never drain the queue and soon enough the TX dropped counter will begin incrementing. > In my case as well the only option to recover without shutting down the whole vServer is Live Migration > to another Node. > > However, I also see this problem under qemu-kvm-1.2.0 while Oliver reported it does not happen there. Yes, the patch that exposes this problem was only merged in 1.2.1. Can you still reproduce the problem now that the patch has been merged into qemu.git/master? Stefan
On 27.05.2013 10:32, Stefan Hajnoczi wrote: > On Mon, May 27, 2013 at 08:15:42AM +0200, Peter Lieven wrote: >> I ocassionally have seen a probably related problem in the past. It mainly happend with rtl8139 under >> WinXP where we most likely use rtl8139 due to lack of shipped e1000 drivers. >> >> My question is if you see increasing dropped packets on the tap device if this problem occurs? >> >> tap36 Link encap:Ethernet HWaddr b2:84:23:c0:e2:c0 >> inet6 addr: fe80::b084:23ff:fec0:e2c0/64 Scope:Link >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> RX packets:5816096 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:3878744 errors:0 dropped:13775 overruns:0 carrier:0 >> collisions:0 txqueuelen:500 >> RX bytes:5161769434 (5.1 GB) TX bytes:380415916 (380.4 MB) > My reading of the tun code is that will see TX dropped increase. This > is because tun keeps a finite size queue of tx packets. Since QEMU > userspace is not monitoring the tap fd anymore we'll never drain the > queue and soon enough the TX dropped counter will begin incrementing. Ok, so this would fit. > >> In my case as well the only option to recover without shutting down the whole vServer is Live Migration >> to another Node. >> >> However, I also see this problem under qemu-kvm-1.2.0 while Oliver reported it does not happen there. > Yes, the patch that exposes this problem was only merged in 1.2.1. Can you say which patch exactly? I cherry-picked some patches by hand. > > Can you still reproduce the problem now that the patch has been merged > into qemu.git/master? Unfortunately, I have no reliable way of reproducing the issue. It only happens from time to time. Peter
Well, Am 27.05.2013 um 08:15 schrieb Peter Lieven <lieven-lists@dlhnet.de>: > Hi all, > > I ocassionally have seen a probably related problem in the past. It mainly happend with rtl8139 under > WinXP where we most likely use rtl8139 due to lack of shipped e1000 drivers. > > My question is if you see increasing dropped packets on the tap device if this problem occurs? > > tap36 Link encap:Ethernet HWaddr b2:84:23:c0:e2:c0 > inet6 addr: fe80::b084:23ff:fec0:e2c0/64 Scope:Link > UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > RX packets:5816096 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3878744 errors:0 dropped:13775 overruns:0 carrier:0 > collisions:0 txqueuelen:500 > RX bytes:5161769434 (5.1 GB) TX bytes:380415916 (380.4 MB) > > In my case as well the only option to recover without shutting down the whole vServer is Live Migration > to another Node. > ACK, tried it and every network-devices might have been re-created into a defined state qemu-wise. > However, I also see this problem under qemu-kvm-1.2.0 while Oliver reported it does not happen there. > Neither me nor any affected customers have ever seen such failures in qemu-1.2.0, so this was my last-known-good ;) Oliver. > Thank you, > Peter > > On 22.05.2013 14:50, Stefan Hajnoczi wrote: >> Net queues support efficient "receive disable". For example, tap's file >> descriptor will not be polled while its peer has receive disabled. This >> saves CPU cycles for needlessly copying and then dropping packets which >> the peer cannot receive. >> >> rtl8139 is missing the qemu_flush_queued_packets() call that wakes the >> queue up when receive becomes possible again. >> >> As a result, the Windows 7 guest driver reaches a state where the >> rtl8139 cannot receive packets. The driver has actually refilled the >> receive buffer but we never resume reception. >> >> The bug can be reproduced by running a large FTP 'get' inside a Windows >> 7 guest: >> >> $ qemu -netdev tap,id=tap0,... >> -device rtl8139,netdev=tap0 >> >> The Linux guest driver does not trigger the bug, probably due to a >> different buffer management strategy. >> >> Reported-by: Oliver Francke <oliver.francke@filoo.de> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> >> --- >> hw/net/rtl8139.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/hw/net/rtl8139.c b/hw/net/rtl8139.c >> index 9369507..7993f9f 100644 >> --- a/hw/net/rtl8139.c >> +++ b/hw/net/rtl8139.c >> @@ -2575,6 +2575,9 @@ static void rtl8139_RxBufPtr_write(RTL8139State *s, uint32_t val) >> /* this value is off by 16 */ >> s->RxBufPtr = MOD2(val + 0x10, s->RxBufferSize); >> + /* more buffer space may be available so try to receive */ >> + qemu_flush_queued_packets(qemu_get_queue(s->nic)); >> + >> DPRINTF(" CAPR write: rx buffer length %d head 0x%04x read 0x%04x\n", >> s->RxBufferSize, s->RxBufAddr, s->RxBufPtr); >> } >
diff --git a/hw/net/rtl8139.c b/hw/net/rtl8139.c index 9369507..7993f9f 100644 --- a/hw/net/rtl8139.c +++ b/hw/net/rtl8139.c @@ -2575,6 +2575,9 @@ static void rtl8139_RxBufPtr_write(RTL8139State *s, uint32_t val) /* this value is off by 16 */ s->RxBufPtr = MOD2(val + 0x10, s->RxBufferSize); + /* more buffer space may be available so try to receive */ + qemu_flush_queued_packets(qemu_get_queue(s->nic)); + DPRINTF(" CAPR write: rx buffer length %d head 0x%04x read 0x%04x\n", s->RxBufferSize, s->RxBufAddr, s->RxBufPtr); }
Net queues support efficient "receive disable". For example, tap's file descriptor will not be polled while its peer has receive disabled. This saves CPU cycles for needlessly copying and then dropping packets which the peer cannot receive. rtl8139 is missing the qemu_flush_queued_packets() call that wakes the queue up when receive becomes possible again. As a result, the Windows 7 guest driver reaches a state where the rtl8139 cannot receive packets. The driver has actually refilled the receive buffer but we never resume reception. The bug can be reproduced by running a large FTP 'get' inside a Windows 7 guest: $ qemu -netdev tap,id=tap0,... -device rtl8139,netdev=tap0 The Linux guest driver does not trigger the bug, probably due to a different buffer management strategy. Reported-by: Oliver Francke <oliver.francke@filoo.de> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- hw/net/rtl8139.c | 3 +++ 1 file changed, 3 insertions(+)