| Message ID | 20140910003532.22d72602@bahia.local |
|---|---|
| State | New |
| Headers | show |
On 09/10/2014 08:35 AM, Greg Kurz wrote: > On Mon, 8 Sep 2014 19:05:02 +0300 > "Michael S. Tsirkin" <mst@redhat.com> wrote: > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 >> pci: Use bus master address space for delivering MSI/MSI-X messages >> breaks virtio-net for rhel6.[56] x86 guests because they don't >> enable bus mastering for virtio PCI devices >> >> Old guests forgot to enable bus mastering, enable it >> automatically on DRIVER_OK. >> >> Note: we should either back out the original patch from >> stable or apply this one on top. >> >> Cc: qemu-stable@nongnu.org >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> >> --- >> hw/virtio/virtio-pci.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c >> index ddb5da1..af937d2 100644 >> --- a/hw/virtio/virtio-pci.c >> +++ b/hw/virtio/virtio-pci.c >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, >> + true); >> } >> break; >> case VIRTIO_MSI_CONFIG_VECTOR: > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. Cc'ing Nikunj - he is SLOF expert. I cannot comment - I have no idea what VIRTIO_PCI_FLAG_BUS_MASTER_BUG does. > > Michael, > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > this fails for rhel6.5 ppc64 because it is never called... I did some debugging: > it looks like the guest kernel calls the OF quisece call to flush pending DMA > and disables bus master on the virtio-blk device (PCI_COMMAND == 0x3). The > guest then continues to boot and hangs... It appears that waiting for the > guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since we need this for > MSI to work, I tried the following and it fixes the issue: > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > index af937d2..3d72aa8 100644 > --- a/hw/virtio/virtio-pci.c > +++ b/hw/virtio/virtio-pci.c > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) > { > VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); > > - if (msix_enabled(&proxy->pci_dev)) > + if (msix_enabled(&proxy->pci_dev)) { > + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > + true); > + } > msix_notify(&proxy->pci_dev, vector); > - else { > + } else { > VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); > pci_set_irq(&proxy->pci_dev, vdev->isr & 1); > } > > If this is acceptable, I'll make it a helper and squash it into your patch. > > Thoughts ? >
Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > On Mon, 8 Sep 2014 19:05:02 +0300 > "Michael S. Tsirkin" <mst@redhat.com> wrote: > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 >> pci: Use bus master address space for delivering MSI/MSI-X messages >> breaks virtio-net for rhel6.[56] x86 guests because they don't >> enable bus mastering for virtio PCI devices >> >> Old guests forgot to enable bus mastering, enable it >> automatically on DRIVER_OK. >> >> Note: we should either back out the original patch from >> stable or apply this one on top. >> >> Cc: qemu-stable@nongnu.org >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> >> --- >> hw/virtio/virtio-pci.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c >> index ddb5da1..af937d2 100644 >> --- a/hw/virtio/virtio-pci.c >> +++ b/hw/virtio/virtio-pci.c >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, >> + true); >> } >> break; >> case VIRTIO_MSI_CONFIG_VECTOR: > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > > Michael, > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > this fails for rhel6.5 ppc64 because it is never called... > I did some debugging: it looks like the guest kernel calls the OF > quisece call to flush pending DMA and disables bus master on the > virtio-blk device (PCI_COMMAND == 0x3). Getting confused, above you are talking about virtio-net and here it is virtio-blk. Anyways, the routines still remains same for both of them. From SLOF during init we set DRIVER_OK, and after using the device during the quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then a VIRTIO_DEVICE_RESET is done. > The guest then continues to boot and hangs... It appears that waiting > for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since > we need this for MSI to work, I tried the following and it fixes the > issue: > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > index af937d2..3d72aa8 100644 > --- a/hw/virtio/virtio-pci.c > +++ b/hw/virtio/virtio-pci.c > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) > { > VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); > > - if (msix_enabled(&proxy->pci_dev)) > + if (msix_enabled(&proxy->pci_dev)) { > + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > + true); > + } > msix_notify(&proxy->pci_dev, vector); > - else { > + } else { > VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); > pci_set_irq(&proxy->pci_dev, vdev->isr & 1); > } > > If this is acceptable, I'll make it a helper and squash it into your patch. > > Thoughts ? > > -- > Gregory Kurz kurzgreg@fr.ibm.com > gkurz@linux.vnet.ibm.com > Software Engineer @ IBM/Meiosys http://www.ibm.com > Tel +33 (0)562 165 496 > > "Anarchy is about taking complete responsibility for yourself." > Alan Moore.
On Wed, 10 Sep 2014 13:44:49 +0530 Nikunj A Dadhania <nikunj@linux.vnet.ibm.com> wrote: > Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > > > On Mon, 8 Sep 2014 19:05:02 +0300 > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 > >> pci: Use bus master address space for delivering MSI/MSI-X messages > >> breaks virtio-net for rhel6.[56] x86 guests because they don't > >> enable bus mastering for virtio PCI devices > >> > >> Old guests forgot to enable bus mastering, enable it > >> automatically on DRIVER_OK. > >> > >> Note: we should either back out the original patch from > >> stable or apply this one on top. > >> > >> Cc: qemu-stable@nongnu.org > >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> > >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > >> --- > >> hw/virtio/virtio-pci.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > >> index ddb5da1..af937d2 100644 > >> --- a/hw/virtio/virtio-pci.c > >> +++ b/hw/virtio/virtio-pci.c > >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) > >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && > >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > >> + true); > >> } > >> break; > >> case VIRTIO_MSI_CONFIG_VECTOR: > > > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > > > > Michael, > > > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > > this fails for rhel6.5 ppc64 because it is never called... > > > I did some debugging: it looks like the guest kernel calls the OF > > quisece call to flush pending DMA and disables bus master on the > > virtio-blk device (PCI_COMMAND == 0x3). > > Getting confused, above you are talking about virtio-net and here it is > virtio-blk. > I tried running rhel6.5 (old kernel that doesn't enable bus mastering on virtio PCI devices), with a virtio-blk based disk and a virtio-net based NIC for both x86_64 and ppc64. Results are as follow: - x86_64: boots well but fails to activate network - ppc64: does not boot because the virtio-blk notification doesn't reach the guest > Anyways, the routines still remains same for both of them. From SLOF > during init we set DRIVER_OK, and after using the device during the > quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then > a VIRTIO_DEVICE_RESET is done. > I chose to debug by attaching gdb to qemu-system-ppc64 itself. It appears that SLOF seems to be enabling bus master during init but at some point bus master gets disabled... unfortunately my SLOF knowledge is limited and I don't know how exactly what's happening in the guest between SLOF and the kernel. > > The guest then continues to boot and hangs... It appears that waiting > > for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since > > we need this for MSI to work, I tried the following and it fixes the > > issue: > > > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > > index af937d2..3d72aa8 100644 > > --- a/hw/virtio/virtio-pci.c > > +++ b/hw/virtio/virtio-pci.c > > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) > > { > > VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); > > > > - if (msix_enabled(&proxy->pci_dev)) > > + if (msix_enabled(&proxy->pci_dev)) { > > + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > > + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > > + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > > + true); > > + } > > msix_notify(&proxy->pci_dev, vector); > > - else { > > + } else { > > VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); > > pci_set_irq(&proxy->pci_dev, vdev->isr & 1); > > } > > > > If this is acceptable, I'll make it a helper and squash it into your patch. > > > > Thoughts ? > > > > -- > > Gregory Kurz kurzgreg@fr.ibm.com > > gkurz@linux.vnet.ibm.com > > Software Engineer @ IBM/Meiosys http://www.ibm.com > > Tel +33 (0)562 165 496 > > > > "Anarchy is about taking complete responsibility for yourself." > > Alan Moore.
On Wed, 10 Sep 2014 12:32:30 +0300 "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Wed, Sep 10, 2014 at 01:44:49PM +0530, Nikunj A Dadhania wrote: > > Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > > > > > On Mon, 8 Sep 2014 19:05:02 +0300 > > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 > > >> pci: Use bus master address space for delivering MSI/MSI-X messages > > >> breaks virtio-net for rhel6.[56] x86 guests because they don't > > >> enable bus mastering for virtio PCI devices > > >> > > >> Old guests forgot to enable bus mastering, enable it > > >> automatically on DRIVER_OK. > > >> > > >> Note: we should either back out the original patch from > > >> stable or apply this one on top. > > >> > > >> Cc: qemu-stable@nongnu.org > > >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> > > >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > > >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > > >> --- > > >> hw/virtio/virtio-pci.c | 2 ++ > > >> 1 file changed, 2 insertions(+) > > >> > > >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > > >> index ddb5da1..af937d2 100644 > > >> --- a/hw/virtio/virtio-pci.c > > >> +++ b/hw/virtio/virtio-pci.c > > >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) > > >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && > > >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > > >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > > >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > > >> + true); > > >> } > > >> break; > > >> case VIRTIO_MSI_CONFIG_VECTOR: > > > > > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > > > > > > Michael, > > > > > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > > > this fails for rhel6.5 ppc64 because it is never called... > > > > > I did some debugging: it looks like the guest kernel calls the OF > > > quisece call to flush pending DMA and disables bus master on the > > > virtio-blk device (PCI_COMMAND == 0x3). > > > > Getting confused, above you are talking about virtio-net and here it is > > virtio-blk. > > > > Anyways, the routines still remains same for both of them. From SLOF > > during init we set DRIVER_OK, and after using the device during the > > quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then > > a VIRTIO_DEVICE_RESET is done. > > BTW, you really should start enabling bus mastering, avoid relying > on the work-around we have for broken guests. > FWIW during my debug session, I see that SLOF enables bus mastering... unfortunately, it gets disabled at some point after the guest kernel is started (around the ppc64 prom_init() call). > > > The guest then continues to boot and hangs... It appears that waiting > > > for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since > > > we need this for MSI to work, I tried the following and it fixes the > > > issue: > > > > > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > > > index af937d2..3d72aa8 100644 > > > --- a/hw/virtio/virtio-pci.c > > > +++ b/hw/virtio/virtio-pci.c > > > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) > > > { > > > VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); > > > > > > - if (msix_enabled(&proxy->pci_dev)) > > > + if (msix_enabled(&proxy->pci_dev)) { > > > + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > > > + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > > > + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > > > + true); > > > + } > > > msix_notify(&proxy->pci_dev, vector); > > > - else { > > > + } else { > > > VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); > > > pci_set_irq(&proxy->pci_dev, vdev->isr & 1); > > > } > > > > > > If this is acceptable, I'll make it a helper and squash it into your patch. > > > > > > Thoughts ? > > > > > > -- > > > Gregory Kurz kurzgreg@fr.ibm.com > > > gkurz@linux.vnet.ibm.com > > > Software Engineer @ IBM/Meiosys http://www.ibm.com > > > Tel +33 (0)562 165 496 > > > > > > "Anarchy is about taking complete responsibility for yourself." > > > Alan Moore. >
"Michael S. Tsirkin" <mst@redhat.com> writes: > On Wed, Sep 10, 2014 at 01:44:49PM +0530, Nikunj A Dadhania wrote: >> Greg Kurz <gkurz@linux.vnet.ibm.com> writes: >> >> > On Mon, 8 Sep 2014 19:05:02 +0300 >> > "Michael S. Tsirkin" <mst@redhat.com> wrote: >> > >> >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 >> >> pci: Use bus master address space for delivering MSI/MSI-X messages >> >> breaks virtio-net for rhel6.[56] x86 guests because they don't >> >> enable bus mastering for virtio PCI devices >> >> >> >> Old guests forgot to enable bus mastering, enable it >> >> automatically on DRIVER_OK. >> >> >> >> Note: we should either back out the original patch from >> >> stable or apply this one on top. >> >> >> >> Cc: qemu-stable@nongnu.org >> >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> >> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> >> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> >> >> --- >> >> hw/virtio/virtio-pci.c | 2 ++ >> >> 1 file changed, 2 insertions(+) >> >> >> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c >> >> index ddb5da1..af937d2 100644 >> >> --- a/hw/virtio/virtio-pci.c >> >> +++ b/hw/virtio/virtio-pci.c >> >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) >> >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && >> >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { >> >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; >> >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, >> >> + true); >> >> } >> >> break; >> >> case VIRTIO_MSI_CONFIG_VECTOR: >> > >> > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. >> > >> > Michael, >> > >> > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, >> > this fails for rhel6.5 ppc64 because it is never called... >> >> > I did some debugging: it looks like the guest kernel calls the OF >> > quisece call to flush pending DMA and disables bus master on the >> > virtio-blk device (PCI_COMMAND == 0x3). >> >> Getting confused, above you are talking about virtio-net and here it is >> virtio-blk. >> >> Anyways, the routines still remains same for both of them. From SLOF >> during init we set DRIVER_OK, and after using the device during the >> quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then >> a VIRTIO_DEVICE_RESET is done. > > BTW, you really should start enabling bus mastering, avoid relying > on the work-around we have for broken guests. In SLOF, we do enable PCI MASTER during device scanning and then later disable it. Regards Nikunj
Greg Kurz <gkurz@linux.vnet.ibm.com> writes: >> > > I did some debugging: it looks like the guest kernel calls the OF >> > > quisece call to flush pending DMA and disables bus master on the >> > > virtio-blk device (PCI_COMMAND == 0x3). >> > >> > Getting confused, above you are talking about virtio-net and here it is >> > virtio-blk. >> > >> > Anyways, the routines still remains same for both of them. From SLOF >> > during init we set DRIVER_OK, and after using the device during the >> > quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then >> > a VIRTIO_DEVICE_RESET is done. >> >> BTW, you really should start enabling bus mastering, avoid relying >> on the work-around we have for broken guests. >> > > FWIW during my debug session, I see that SLOF enables bus mastering... > unfortunately, it gets disabled at some point after the guest kernel > is started (around the ppc64 prom_init() call). Is it before quiesce call? Regards Nikunj
On Wed, Sep 10, 2014 at 01:44:49PM +0530, Nikunj A Dadhania wrote: > Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > > > On Mon, 8 Sep 2014 19:05:02 +0300 > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 > >> pci: Use bus master address space for delivering MSI/MSI-X messages > >> breaks virtio-net for rhel6.[56] x86 guests because they don't > >> enable bus mastering for virtio PCI devices > >> > >> Old guests forgot to enable bus mastering, enable it > >> automatically on DRIVER_OK. > >> > >> Note: we should either back out the original patch from > >> stable or apply this one on top. > >> > >> Cc: qemu-stable@nongnu.org > >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> > >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > >> --- > >> hw/virtio/virtio-pci.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > >> index ddb5da1..af937d2 100644 > >> --- a/hw/virtio/virtio-pci.c > >> +++ b/hw/virtio/virtio-pci.c > >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) > >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && > >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > >> + true); > >> } > >> break; > >> case VIRTIO_MSI_CONFIG_VECTOR: > > > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > > > > Michael, > > > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > > this fails for rhel6.5 ppc64 because it is never called... > > > I did some debugging: it looks like the guest kernel calls the OF > > quisece call to flush pending DMA and disables bus master on the > > virtio-blk device (PCI_COMMAND == 0x3). > > Getting confused, above you are talking about virtio-net and here it is > virtio-blk. > > Anyways, the routines still remains same for both of them. From SLOF > during init we set DRIVER_OK, and after using the device during the > quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then > a VIRTIO_DEVICE_RESET is done. BTW, you really should start enabling bus mastering, avoid relying on the work-around we have for broken guests. > > The guest then continues to boot and hangs... It appears that waiting > > for the guest to issue VIRTIO_CONFIG_S_DRIVER_OK is not enough. Since > > we need this for MSI to work, I tried the following and it fixes the > > issue: > > > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > > index af937d2..3d72aa8 100644 > > --- a/hw/virtio/virtio-pci.c > > +++ b/hw/virtio/virtio-pci.c > > @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) > > { > > VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); > > > > - if (msix_enabled(&proxy->pci_dev)) > > + if (msix_enabled(&proxy->pci_dev)) { > > + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > > + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > > + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > > + true); > > + } > > msix_notify(&proxy->pci_dev, vector); > > - else { > > + } else { > > VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); > > pci_set_irq(&proxy->pci_dev, vdev->isr & 1); > > } > > > > If this is acceptable, I'll make it a helper and squash it into your patch. > > > > Thoughts ? > > > > -- > > Gregory Kurz kurzgreg@fr.ibm.com > > gkurz@linux.vnet.ibm.com > > Software Engineer @ IBM/Meiosys http://www.ibm.com > > Tel +33 (0)562 165 496 > > > > "Anarchy is about taking complete responsibility for yourself." > > Alan Moore.
On Wed, Sep 10, 2014 at 02:45:51PM +0530, Nikunj A Dadhania wrote: > "Michael S. Tsirkin" <mst@redhat.com> writes: > > > On Wed, Sep 10, 2014 at 01:44:49PM +0530, Nikunj A Dadhania wrote: > >> Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > >> > >> > On Mon, 8 Sep 2014 19:05:02 +0300 > >> > "Michael S. Tsirkin" <mst@redhat.com> wrote: > >> > > >> >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 > >> >> pci: Use bus master address space for delivering MSI/MSI-X messages > >> >> breaks virtio-net for rhel6.[56] x86 guests because they don't > >> >> enable bus mastering for virtio PCI devices > >> >> > >> >> Old guests forgot to enable bus mastering, enable it > >> >> automatically on DRIVER_OK. > >> >> > >> >> Note: we should either back out the original patch from > >> >> stable or apply this one on top. > >> >> > >> >> Cc: qemu-stable@nongnu.org > >> >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> > >> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > >> >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > >> >> --- > >> >> hw/virtio/virtio-pci.c | 2 ++ > >> >> 1 file changed, 2 insertions(+) > >> >> > >> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > >> >> index ddb5da1..af937d2 100644 > >> >> --- a/hw/virtio/virtio-pci.c > >> >> +++ b/hw/virtio/virtio-pci.c > >> >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) > >> >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && > >> >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > >> >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > >> >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > >> >> + true); > >> >> } > >> >> break; > >> >> case VIRTIO_MSI_CONFIG_VECTOR: > >> > > >> > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > >> > > >> > Michael, > >> > > >> > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > >> > this fails for rhel6.5 ppc64 because it is never called... > >> > >> > I did some debugging: it looks like the guest kernel calls the OF > >> > quisece call to flush pending DMA and disables bus master on the > >> > virtio-blk device (PCI_COMMAND == 0x3). > >> > >> Getting confused, above you are talking about virtio-net and here it is > >> virtio-blk. > >> > >> Anyways, the routines still remains same for both of them. From SLOF > >> during init we set DRIVER_OK, and after using the device during the > >> quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then > >> a VIRTIO_DEVICE_RESET is done. > > > > BTW, you really should start enabling bus mastering, avoid relying > > on the work-around we have for broken guests. > > In SLOF, we do enable PCI MASTER during device scanning and then later > disable it. > > Regards > Nikunj But device is then reset, right Greg? You get as far as reset? If yes I doubt something that happens before reset matters, unless we are leaking some state across reset which would be a bug in itself.
On Wed, Sep 10, 2014 at 11:01:48AM +0200, Greg Kurz wrote: > On Wed, 10 Sep 2014 12:32:30 +0300 > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > On Wed, Sep 10, 2014 at 01:44:49PM +0530, Nikunj A Dadhania wrote: > > > Greg Kurz <gkurz@linux.vnet.ibm.com> writes: > > > > > > > On Mon, 8 Sep 2014 19:05:02 +0300 > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > > >> commit cc943c36faa192cd4b32af8fe5edb31894017d35 > > > >> pci: Use bus master address space for delivering MSI/MSI-X messages > > > >> breaks virtio-net for rhel6.[56] x86 guests because they don't > > > >> enable bus mastering for virtio PCI devices > > > >> > > > >> Old guests forgot to enable bus mastering, enable it > > > >> automatically on DRIVER_OK. > > > >> > > > >> Note: we should either back out the original patch from > > > >> stable or apply this one on top. > > > >> > > > >> Cc: qemu-stable@nongnu.org > > > >> Reported-by: Greg Kurz <gkurz@linux.vnet.ibm.com> > > > >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > > > >> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > > > >> --- > > > >> hw/virtio/virtio-pci.c | 2 ++ > > > >> 1 file changed, 2 insertions(+) > > > >> > > > >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c > > > >> index ddb5da1..af937d2 100644 > > > >> --- a/hw/virtio/virtio-pci.c > > > >> +++ b/hw/virtio/virtio-pci.c > > > >> @@ -320,6 +320,8 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) > > > >> if ((val & VIRTIO_CONFIG_S_DRIVER_OK) && > > > >> !(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { > > > >> proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; > > > >> + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, > > > >> + true); > > > >> } > > > >> break; > > > >> case VIRTIO_MSI_CONFIG_VECTOR: > > > > > > > > Cc'ing Alexey for some SLOF and early boot of the ppc64 kernel expertise. > > > > > > > > Michael, > > > > > > > > This was enough to fix virtio-net in the rhel6.5 x86 guest case. Unfortunately, > > > > this fails for rhel6.5 ppc64 because it is never called... > > > > > > > I did some debugging: it looks like the guest kernel calls the OF > > > > quisece call to flush pending DMA and disables bus master on the > > > > virtio-blk device (PCI_COMMAND == 0x3). > > > > > > Getting confused, above you are talking about virtio-net and here it is > > > virtio-blk. > > > > > > Anyways, the routines still remains same for both of them. From SLOF > > > during init we set DRIVER_OK, and after using the device during the > > > quiesce, called from linux kernel VIRTIO_CONFIG_S_FAILED is set and then > > > a VIRTIO_DEVICE_RESET is done. > > > > BTW, you really should start enabling bus mastering, avoid relying > > on the work-around we have for broken guests. > > > > FWIW during my debug session, I see that SLOF enables bus mastering... > unfortunately, it gets disabled at some point after the guest kernel > is started (around the ppc64 prom_init() call). OK I'm not sure I have all the details but does the patch I sent help you?
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index af937d2..3d72aa8 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -111,9 +111,14 @@ static void virtio_pci_notify(DeviceState *d, uint16_t vector) { VirtIOPCIProxy *proxy = to_virtio_pci_proxy_fast(d); - if (msix_enabled(&proxy->pci_dev)) + if (msix_enabled(&proxy->pci_dev)) { + if (!(proxy->pci_dev.config[PCI_COMMAND] & PCI_COMMAND_MASTER)) { + proxy->flags |= VIRTIO_PCI_FLAG_BUS_MASTER_BUG; + memory_region_set_enabled(&proxy->pci_dev.bus_master_enable_region, + true); + } msix_notify(&proxy->pci_dev, vector); - else { + } else { VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); pci_set_irq(&proxy->pci_dev, vdev->isr & 1); }