| Message ID | 1373614639-15523-1-git-send-email-aik@ozlabs.ru |
|---|---|
| State | New |
| Headers | show |
Hi! Got a small lesson from Ben how to post stuff to the list and here are more details :) Every PCI device which uses IO ports (at least vga, e1000, virtio-pci, rtl8139) is broken in the master branch of QEMU for powerpc-kvm. The problem is exactly with endianness. For example, setup_vq() from drivers/virtio/virtio_pci.c (guest kernel) fails on: iowrite16(msix_vec, vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR); as QEMU receives msix_vec==0x100 instead of 0x1 in virtio_ioport_write(). The patch which broke it is b40acf99bef69fa8ab0f9092ff162fde945eec12 "ioport: Switch dispatching to memory core layer". The patch below fixes this but I am not really sure it is correct. Is it? Thanks. On 07/12/2013 05:37 PM, Alexey Kardashevskiy wrote: > sPAPR PHB emulates IO ports on PCI via a special memory region which > routes all reads/writes further via cpu_in*/cpu_out* which are eventually > processed by MemoryRegionOps implemented by devices. > > As devices normally take care of endianness themselves by setting > correct MemoryRegionOps::endianness for their memory regions, it is > better to set endianness in spapr_io_ops to NATIVE. > > Tested on sPAPR KVM with e1000, rtl8139, virtio-net. > > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> > --- > > I would really appreciate if someone told me what exactly changed > in QEMU and broke spapr-pci. It used to work... Thanks! > > --- > hw/ppc/spapr_pci.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c > index ca588aa..dfe4d04 100644 > --- a/hw/ppc/spapr_pci.c > +++ b/hw/ppc/spapr_pci.c > @@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr, > } > > static const MemoryRegionOps spapr_io_ops = { > - .endianness = DEVICE_LITTLE_ENDIAN, > + .endianness = DEVICE_NATIVE_ENDIAN, > .read = spapr_io_read, > .write = spapr_io_write > }; >
Am 12.07.2013 um 10:59 schrieb Alexey Kardashevskiy <aik@ozlabs.ru>: > Hi! > > Got a small lesson from Ben how to post stuff to the list and here are more > details :) > > Every PCI device which uses IO ports (at least vga, e1000, virtio-pci, > rtl8139) is broken in the master branch of QEMU for powerpc-kvm. The > problem is exactly with endianness. For example, setup_vq() from > drivers/virtio/virtio_pci.c (guest kernel) fails on: > > iowrite16(msix_vec, vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR); > > as QEMU receives msix_vec==0x100 instead of 0x1 in virtio_ioport_write(). > > The patch which broke it is b40acf99bef69fa8ab0f9092ff162fde945eec12 > "ioport: Switch dispatching to memory core layer". > > The patch below fixes this but I am not really sure it is correct. Is it? It is not. Please read my mail on the topic. Alex > Thanks. > > > On 07/12/2013 05:37 PM, Alexey Kardashevskiy wrote: >> sPAPR PHB emulates IO ports on PCI via a special memory region which >> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >> processed by MemoryRegionOps implemented by devices. >> >> As devices normally take care of endianness themselves by setting >> correct MemoryRegionOps::endianness for their memory regions, it is >> better to set endianness in spapr_io_ops to NATIVE. >> >> Tested on sPAPR KVM with e1000, rtl8139, virtio-net. >> >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >> --- >> >> I would really appreciate if someone told me what exactly changed >> in QEMU and broke spapr-pci. It used to work... Thanks! >> >> --- >> hw/ppc/spapr_pci.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c >> index ca588aa..dfe4d04 100644 >> --- a/hw/ppc/spapr_pci.c >> +++ b/hw/ppc/spapr_pci.c >> @@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr, >> } >> >> static const MemoryRegionOps spapr_io_ops = { >> - .endianness = DEVICE_LITTLE_ENDIAN, >> + .endianness = DEVICE_NATIVE_ENDIAN, >> .read = spapr_io_read, >> .write = spapr_io_write >> }; > > > -- > Alexey
On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: > sPAPR PHB emulates IO ports on PCI via a special memory region which > routes all reads/writes further via cpu_in*/cpu_out* which are eventually > processed by MemoryRegionOps implemented by devices. Hrm. That double dispatch was a workaround for bugs in the plain memory region dispatching which meant we couldn't directly map regions in memory space to IO areas. It would be worth checking if that workaround is still necessary.
On 07/13/2013 06:03 PM, David Gibson wrote: > On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: >> sPAPR PHB emulates IO ports on PCI via a special memory region which >> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >> processed by MemoryRegionOps implemented by devices. > Hrm. That double dispatch was a workaround for bugs in the plain > memory region dispatching which meant we couldn't directly map regions > in memory space to IO areas. > > It would be worth checking if that workaround is still necessary. Hm. Good point, thanks! It seems memory_region_init_io is not necessary any more. Will make a patch for it.
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > On 07/13/2013 06:03 PM, David Gibson wrote: >> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: >>> sPAPR PHB emulates IO ports on PCI via a special memory region which >>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >>> processed by MemoryRegionOps implemented by devices. > >> Hrm. That double dispatch was a workaround for bugs in the plain >> memory region dispatching which meant we couldn't directly map regions >> in memory space to IO areas. >> >> It would be worth checking if that workaround is still necessary. > > Hm. Good point, thanks! It seems memory_region_init_io is not necessary any > more. Will make a patch for it. You should try the latest qemu.git commit. There shouldn't be a problem anymore. Regards, Anthony Liguori > > > -- > Alexey
On 07/16/2013 01:02 AM, Anthony Liguori wrote: > Alexey Kardashevskiy <aik@ozlabs.ru> writes: > >> On 07/13/2013 06:03 PM, David Gibson wrote: >>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: >>>> sPAPR PHB emulates IO ports on PCI via a special memory region which >>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >>>> processed by MemoryRegionOps implemented by devices. >> >>> Hrm. That double dispatch was a workaround for bugs in the plain >>> memory region dispatching which meant we couldn't directly map regions >>> in memory space to IO areas. >>> >>> It would be worth checking if that workaround is still necessary. >> >> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any >> more. Will make a patch for it. > > You should try the latest qemu.git commit. There shouldn't be a problem > anymore. Does this mean sPAPR still needs an additional IO memory region? It looks redundand and everything (almost) works without it...
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > On 07/16/2013 01:02 AM, Anthony Liguori wrote: >> Alexey Kardashevskiy <aik@ozlabs.ru> writes: >> >>> On 07/13/2013 06:03 PM, David Gibson wrote: >>>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: >>>>> sPAPR PHB emulates IO ports on PCI via a special memory region which >>>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >>>>> processed by MemoryRegionOps implemented by devices. >>> >>>> Hrm. That double dispatch was a workaround for bugs in the plain >>>> memory region dispatching which meant we couldn't directly map regions >>>> in memory space to IO areas. >>>> >>>> It would be worth checking if that workaround is still necessary. >>> >>> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any >>> more. Will make a patch for it. >> >> You should try the latest qemu.git commit. There shouldn't be a problem >> anymore. > > > Does this mean sPAPR still needs an additional IO memory region? It looks > redundand and everything (almost) works without it... There's more brokenness... Some ISA devices mark themselves as "little endian" whereas others mark themselves as "native endian". "little endian" really means "do byte lane swapping during dispatch" if host endian != target endian. So on sPAPR, what you're getting is the redundant IO memory region causing a byte lane swap which is then negated by the ISA devices that mark themselves as little endian (such as VGA). The right solution is to drop the additional IO region on sPAPR and remove the ISA devices marking themselves as "little endian". But that requires careful testing and fixing the other platforms that also are relying on the doube byte lane swapping. Regards, Anthony Liguori > > > -- > Alexey
On 07/16/2013 10:28 AM, Anthony Liguori wrote: > Alexey Kardashevskiy <aik@ozlabs.ru> writes: > >> On 07/16/2013 01:02 AM, Anthony Liguori wrote: >>> Alexey Kardashevskiy <aik@ozlabs.ru> writes: >>> >>>> On 07/13/2013 06:03 PM, David Gibson wrote: >>>>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote: >>>>>> sPAPR PHB emulates IO ports on PCI via a special memory region which >>>>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually >>>>>> processed by MemoryRegionOps implemented by devices. >>>> >>>>> Hrm. That double dispatch was a workaround for bugs in the plain >>>>> memory region dispatching which meant we couldn't directly map regions >>>>> in memory space to IO areas. >>>>> >>>>> It would be worth checking if that workaround is still necessary. >>>> >>>> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any >>>> more. Will make a patch for it. >>> >>> You should try the latest qemu.git commit. There shouldn't be a problem >>> anymore. >> >> >> Does this mean sPAPR still needs an additional IO memory region? It looks >> redundand and everything (almost) works without it... > > There's more brokenness... > > Some ISA devices mark themselves as "little endian" whereas others mark > themselves as "native endian". > > "little endian" really means "do byte lane swapping during dispatch" if > host endian != target endian. > > So on sPAPR, what you're getting is the redundant IO memory region > causing a byte lane swap which is then negated by the ISA devices that > mark themselves as little endian (such as VGA). > > The right solution is to drop the additional IO region on sPAPR and > remove the ISA devices marking themselves as "little endian". No, this is not endianness, this is something different caused by a difference in IO port registration (subpage? section? memoryregion? I am going to draw a graph and try realize what is what here :) ). Even very first IO port access does not reach VGA, fails somehow in address_space_translate_internal() but devices other than VGA (well, at least network devices) work perfectly. > But that requires careful testing and fixing the other platforms that > also are relying on the doube byte lane swapping.
On Mon, Jul 15, 2013 at 7:39 PM, Alexey Kardashevskiy <aik@ozlabs.ru> wrote: > On 07/16/2013 10:28 AM, Anthony Liguori wrote: >> Alexey Kardashevskiy <aik@ozlabs.ru> writes: >> >> The right solution is to drop the additional IO region on sPAPR and >> remove the ISA devices marking themselves as "little endian". > > > No, this is not endianness, this is something different caused by a > difference in IO port registration (subpage? section? memoryregion? I am > going to draw a graph and try realize what is what here :) ). > > Even very first IO port access does not reach VGA, fails somehow in > address_space_translate_internal() but devices other than VGA (well, at > least network devices) work perfectly. Can you be more specific about what's failing? VGA works fine for me with TCG. Regards, Anthony Liguori > > >> But that requires careful testing and fixing the other platforms that >> also are relying on the doube byte lane swapping. > > > -- > Alexey
On 07/16/2013 12:05 PM, Anthony Liguori wrote: > On Mon, Jul 15, 2013 at 7:39 PM, Alexey Kardashevskiy <aik@ozlabs.ru> wrote: >> On 07/16/2013 10:28 AM, Anthony Liguori wrote: >>> Alexey Kardashevskiy <aik@ozlabs.ru> writes: >>> >>> The right solution is to drop the additional IO region on sPAPR and >>> remove the ISA devices marking themselves as "little endian". >> >> >> No, this is not endianness, this is something different caused by a >> difference in IO port registration (subpage? section? memoryregion? I am >> going to draw a graph and try realize what is what here :) ). >> >> Even very first IO port access does not reach VGA, fails somehow in >> address_space_translate_internal() but devices other than VGA (well, at >> least network devices) work perfectly. > > Can you be more specific about what's failing? VGA works fine for me with TCG. With the patch we are replying to? Wow. Does not work for me (full emulation or kvm) - does not show even a single symbol.
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index ca588aa..dfe4d04 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr, } static const MemoryRegionOps spapr_io_ops = { - .endianness = DEVICE_LITTLE_ENDIAN, + .endianness = DEVICE_NATIVE_ENDIAN, .read = spapr_io_read, .write = spapr_io_write };
sPAPR PHB emulates IO ports on PCI via a special memory region which routes all reads/writes further via cpu_in*/cpu_out* which are eventually processed by MemoryRegionOps implemented by devices. As devices normally take care of endianness themselves by setting correct MemoryRegionOps::endianness for their memory regions, it is better to set endianness in spapr_io_ops to NATIVE. Tested on sPAPR KVM with e1000, rtl8139, virtio-net. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> --- I would really appreciate if someone told me what exactly changed in QEMU and broke spapr-pci. It used to work... Thanks! --- hw/ppc/spapr_pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)