| Message ID | 20200323084617.1782-2-bbhushan2@marvell.com |
|---|---|
| State | New |
| Headers | show |
| Series | virtio-iommu: VFIO integration | expand |
[Cc +dwg who originated this warning] On Mon, 23 Mar 2020 14:16:09 +0530 Bharat Bhushan <bbhushan2@marvell.com> wrote: > On ARM, the MSI doorbell is translated by the virtual IOMMU. > As such address_space_translate() returns the MSI controller > MMIO region and we get an "iommu map to non memory area" > message. Let's remove this latter. > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > --- > hw/vfio/common.c | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/hw/vfio/common.c b/hw/vfio/common.c > index 5ca11488d6..c586edf47a 100644 > --- a/hw/vfio/common.c > +++ b/hw/vfio/common.c > @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void **vaddr, > &xlat, &len, writable, > MEMTXATTRS_UNSPECIFIED); > if (!memory_region_is_ram(mr)) { > - error_report("iommu map to non memory area %"HWADDR_PRIx"", > - xlat); > return false; > } > I'm a bit confused here, I think we need more justification beyond "we hit this warning and we don't want to because it's ok in this one special case, therefore remove it". I assume the special case is that the device MSI address is managed via the SET_IRQS ioctl and therefore we won't actually get DMAs to this range. But I imagine the case that was in mind when adding this warning was general peer-to-peer between and assigned and emulated device. Maybe there's an argument to be made that such a p2p mapping might also be used in a non-vIOMMU case. We skip creating those mappings and drivers continue to work, maybe because nobody attempts to do p2p DMA with the types of devices we emulate, maybe because p2p DMA is not absolutely reliable on bare metal and drivers test it before using it. Anyway, I need a better argument why this is ok. Thanks, Alex
Hi Alex, On 3/24/20 12:08 AM, Alex Williamson wrote: > [Cc +dwg who originated this warning] > > On Mon, 23 Mar 2020 14:16:09 +0530 > Bharat Bhushan <bbhushan2@marvell.com> wrote: > >> On ARM, the MSI doorbell is translated by the virtual IOMMU. >> As such address_space_translate() returns the MSI controller >> MMIO region and we get an "iommu map to non memory area" >> message. Let's remove this latter. >> >> Signed-off-by: Eric Auger <eric.auger@redhat.com> >> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> >> --- >> hw/vfio/common.c | 2 -- >> 1 file changed, 2 deletions(-) >> >> diff --git a/hw/vfio/common.c b/hw/vfio/common.c >> index 5ca11488d6..c586edf47a 100644 >> --- a/hw/vfio/common.c >> +++ b/hw/vfio/common.c >> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void **vaddr, >> &xlat, &len, writable, >> MEMTXATTRS_UNSPECIFIED); >> if (!memory_region_is_ram(mr)) { >> - error_report("iommu map to non memory area %"HWADDR_PRIx"", >> - xlat); >> return false; >> } >> > > I'm a bit confused here, I think we need more justification beyond "we > hit this warning and we don't want to because it's ok in this one > special case, therefore remove it". I assume the special case is that > the device MSI address is managed via the SET_IRQS ioctl and therefore > we won't actually get DMAs to this range. Yes exactly. The guest creates a mapping between one giova and this gpa (corresponding to the MSI controller doorbell) because MSIs are mapped on ARM. But practically the physical device is programmed with an host chosen iova that maps onto the physical MSI controller's doorbell. so the device never performs DMA accesses to this range. But I imagine the case that > was in mind when adding this warning was general peer-to-peer between > and assigned and emulated device. yes makes sense. Maybe there's an argument to be made > that such a p2p mapping might also be used in a non-vIOMMU case. We > skip creating those mappings and drivers continue to work, maybe > because nobody attempts to do p2p DMA with the types of devices we > emulate, maybe because p2p DMA is not absolutely reliable on bare metal > and drivers test it before using it. MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c iommu_dma_get_msi_page). One idea could be to pass that flag through the IOMMU Notifier mechanism into the iotlb->perm. Eventually when we get this in vfio_get_vaddr() we would not print the warning. Could that make sense? Thanks Eric Anyway, I need a better argument > why this is ok. Thanks, > > Alex >
On Thu, 26 Mar 2020 18:35:48 +0100 Auger Eric <eric.auger@redhat.com> wrote: > Hi Alex, > > On 3/24/20 12:08 AM, Alex Williamson wrote: > > [Cc +dwg who originated this warning] > > > > On Mon, 23 Mar 2020 14:16:09 +0530 > > Bharat Bhushan <bbhushan2@marvell.com> wrote: > > > >> On ARM, the MSI doorbell is translated by the virtual IOMMU. > >> As such address_space_translate() returns the MSI controller > >> MMIO region and we get an "iommu map to non memory area" > >> message. Let's remove this latter. > >> > >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > >> --- > >> hw/vfio/common.c | 2 -- > >> 1 file changed, 2 deletions(-) > >> > >> diff --git a/hw/vfio/common.c b/hw/vfio/common.c > >> index 5ca11488d6..c586edf47a 100644 > >> --- a/hw/vfio/common.c > >> +++ b/hw/vfio/common.c > >> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void **vaddr, > >> &xlat, &len, writable, > >> MEMTXATTRS_UNSPECIFIED); > >> if (!memory_region_is_ram(mr)) { > >> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > >> - xlat); > >> return false; > >> } > >> > > > > I'm a bit confused here, I think we need more justification beyond "we > > hit this warning and we don't want to because it's ok in this one > > special case, therefore remove it". I assume the special case is that > > the device MSI address is managed via the SET_IRQS ioctl and therefore > > we won't actually get DMAs to this range. > Yes exactly. The guest creates a mapping between one giova and this gpa > (corresponding to the MSI controller doorbell) because MSIs are mapped > on ARM. But practically the physical device is programmed with an host > chosen iova that maps onto the physical MSI controller's doorbell. so > the device never performs DMA accesses to this range. > > But I imagine the case that > > was in mind when adding this warning was general peer-to-peer between > > and assigned and emulated device. > yes makes sense. > > Maybe there's an argument to be made > > that such a p2p mapping might also be used in a non-vIOMMU case. We > > skip creating those mappings and drivers continue to work, maybe > > because nobody attempts to do p2p DMA with the types of devices we > > emulate, maybe because p2p DMA is not absolutely reliable on bare metal > > and drivers test it before using it. > MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > iommu_dma_get_msi_page). > One idea could be to pass that flag through the IOMMU Notifier mechanism > into the iotlb->perm. Eventually when we get this in vfio_get_vaddr() we > would not print the warning. Could that make sense? Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. Thanks, Alex
Hi Alex, Eric, > -----Original Message----- > From: Alex Williamson <alex.williamson@redhat.com> > Sent: Thursday, March 26, 2020 11:23 PM > To: Auger Eric <eric.auger@redhat.com> > Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; > peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; > drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- > arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; > yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> > Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio > region translation by viommu > > External Email > > ---------------------------------------------------------------------- > On Thu, 26 Mar 2020 18:35:48 +0100 > Auger Eric <eric.auger@redhat.com> wrote: > > > Hi Alex, > > > > On 3/24/20 12:08 AM, Alex Williamson wrote: > > > [Cc +dwg who originated this warning] > > > > > > On Mon, 23 Mar 2020 14:16:09 +0530 > > > Bharat Bhushan <bbhushan2@marvell.com> wrote: > > > > > >> On ARM, the MSI doorbell is translated by the virtual IOMMU. > > >> As such address_space_translate() returns the MSI controller MMIO > > >> region and we get an "iommu map to non memory area" > > >> message. Let's remove this latter. > > >> > > >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > > >> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > > >> --- > > >> hw/vfio/common.c | 2 -- > > >> 1 file changed, 2 deletions(-) > > >> > > >> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index > > >> 5ca11488d6..c586edf47a 100644 > > >> --- a/hw/vfio/common.c > > >> +++ b/hw/vfio/common.c > > >> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, > void **vaddr, > > >> &xlat, &len, writable, > > >> MEMTXATTRS_UNSPECIFIED); > > >> if (!memory_region_is_ram(mr)) { > > >> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > > >> - xlat); > > >> return false; > > >> } > > >> > > > > > > I'm a bit confused here, I think we need more justification beyond > > > "we hit this warning and we don't want to because it's ok in this > > > one special case, therefore remove it". I assume the special case > > > is that the device MSI address is managed via the SET_IRQS ioctl and > > > therefore we won't actually get DMAs to this range. > > Yes exactly. The guest creates a mapping between one giova and this > > gpa (corresponding to the MSI controller doorbell) because MSIs are > > mapped on ARM. But practically the physical device is programmed with > > an host chosen iova that maps onto the physical MSI controller's > > doorbell. so the device never performs DMA accesses to this range. > > > > But I imagine the case that > > > was in mind when adding this warning was general peer-to-peer > > > between and assigned and emulated device. > > yes makes sense. > > > > Maybe there's an argument to be made > > > that such a p2p mapping might also be used in a non-vIOMMU case. We > > > skip creating those mappings and drivers continue to work, maybe > > > because nobody attempts to do p2p DMA with the types of devices we > > > emulate, maybe because p2p DMA is not absolutely reliable on bare > > > metal and drivers test it before using it. > > MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > > iommu_dma_get_msi_page). > > One idea could be to pass that flag through the IOMMU Notifier > > mechanism into the iotlb->perm. Eventually when we get this in > > vfio_get_vaddr() we would not print the warning. Could that make sense? > > Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. > Thanks, Will change as per above suggestion by Eric. Thanks -Bharat > > Alex
Hi Eric/Alex, > -----Original Message----- > From: Alex Williamson <alex.williamson@redhat.com> > Sent: Thursday, March 26, 2020 11:23 PM > To: Auger Eric <eric.auger@redhat.com> > Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; > peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; > drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- > arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; > yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> > Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio > region translation by viommu > > External Email > > ---------------------------------------------------------------------- > On Thu, 26 Mar 2020 18:35:48 +0100 > Auger Eric <eric.auger@redhat.com> wrote: > > > Hi Alex, > > > > On 3/24/20 12:08 AM, Alex Williamson wrote: > > > [Cc +dwg who originated this warning] > > > > > > On Mon, 23 Mar 2020 14:16:09 +0530 > > > Bharat Bhushan <bbhushan2@marvell.com> wrote: > > > > > >> On ARM, the MSI doorbell is translated by the virtual IOMMU. > > >> As such address_space_translate() returns the MSI controller MMIO > > >> region and we get an "iommu map to non memory area" > > >> message. Let's remove this latter. > > >> > > >> Signed-off-by: Eric Auger <eric.auger@redhat.com> > > >> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > > >> --- > > >> hw/vfio/common.c | 2 -- > > >> 1 file changed, 2 deletions(-) > > >> > > >> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index > > >> 5ca11488d6..c586edf47a 100644 > > >> --- a/hw/vfio/common.c > > >> +++ b/hw/vfio/common.c > > >> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, > void **vaddr, > > >> &xlat, &len, writable, > > >> MEMTXATTRS_UNSPECIFIED); > > >> if (!memory_region_is_ram(mr)) { > > >> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > > >> - xlat); > > >> return false; > > >> } > > >> > > > > > > I'm a bit confused here, I think we need more justification beyond > > > "we hit this warning and we don't want to because it's ok in this > > > one special case, therefore remove it". I assume the special case > > > is that the device MSI address is managed via the SET_IRQS ioctl and > > > therefore we won't actually get DMAs to this range. > > Yes exactly. The guest creates a mapping between one giova and this > > gpa (corresponding to the MSI controller doorbell) because MSIs are > > mapped on ARM. But practically the physical device is programmed with > > an host chosen iova that maps onto the physical MSI controller's > > doorbell. so the device never performs DMA accesses to this range. > > > > But I imagine the case that > > > was in mind when adding this warning was general peer-to-peer > > > between and assigned and emulated device. > > yes makes sense. > > > > Maybe there's an argument to be made > > > that such a p2p mapping might also be used in a non-vIOMMU case. We > > > skip creating those mappings and drivers continue to work, maybe > > > because nobody attempts to do p2p DMA with the types of devices we > > > emulate, maybe because p2p DMA is not absolutely reliable on bare > > > metal and drivers test it before using it. > > MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > > iommu_dma_get_msi_page). > > One idea could be to pass that flag through the IOMMU Notifier > > mechanism into the iotlb->perm. Eventually when we get this in > > vfio_get_vaddr() we would not print the warning. Could that make sense? > > Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. > Thanks, Let me know if I understood the proposal correctly: virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning. Is above correct? Thanks -Bharat > > Alex
Hi Bharat, On 4/2/20 11:01 AM, Bharat Bhushan wrote: > Hi Eric/Alex, > >> -----Original Message----- >> From: Alex Williamson <alex.williamson@redhat.com> >> Sent: Thursday, March 26, 2020 11:23 PM >> To: Auger Eric <eric.auger@redhat.com> >> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; >> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; >> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; >> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- >> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; >> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> >> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio >> region translation by viommu >> >> External Email >> >> ---------------------------------------------------------------------- >> On Thu, 26 Mar 2020 18:35:48 +0100 >> Auger Eric <eric.auger@redhat.com> wrote: >> >>> Hi Alex, >>> >>> On 3/24/20 12:08 AM, Alex Williamson wrote: >>>> [Cc +dwg who originated this warning] >>>> >>>> On Mon, 23 Mar 2020 14:16:09 +0530 >>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: >>>> >>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. >>>>> As such address_space_translate() returns the MSI controller MMIO >>>>> region and we get an "iommu map to non memory area" >>>>> message. Let's remove this latter. >>>>> >>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> >>>>> --- >>>>> hw/vfio/common.c | 2 -- >>>>> 1 file changed, 2 deletions(-) >>>>> >>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index >>>>> 5ca11488d6..c586edf47a 100644 >>>>> --- a/hw/vfio/common.c >>>>> +++ b/hw/vfio/common.c >>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, >> void **vaddr, >>>>> &xlat, &len, writable, >>>>> MEMTXATTRS_UNSPECIFIED); >>>>> if (!memory_region_is_ram(mr)) { >>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", >>>>> - xlat); >>>>> return false; >>>>> } >>>>> >>>> >>>> I'm a bit confused here, I think we need more justification beyond >>>> "we hit this warning and we don't want to because it's ok in this >>>> one special case, therefore remove it". I assume the special case >>>> is that the device MSI address is managed via the SET_IRQS ioctl and >>>> therefore we won't actually get DMAs to this range. >>> Yes exactly. The guest creates a mapping between one giova and this >>> gpa (corresponding to the MSI controller doorbell) because MSIs are >>> mapped on ARM. But practically the physical device is programmed with >>> an host chosen iova that maps onto the physical MSI controller's >>> doorbell. so the device never performs DMA accesses to this range. >>> >>> But I imagine the case that >>>> was in mind when adding this warning was general peer-to-peer >>>> between and assigned and emulated device. >>> yes makes sense. >>> >>> Maybe there's an argument to be made >>>> that such a p2p mapping might also be used in a non-vIOMMU case. We >>>> skip creating those mappings and drivers continue to work, maybe >>>> because nobody attempts to do p2p DMA with the types of devices we >>>> emulate, maybe because p2p DMA is not absolutely reliable on bare >>>> metal and drivers test it before using it. >>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >>> iommu_dma_get_msi_page). >>> One idea could be to pass that flag through the IOMMU Notifier >>> mechanism into the iotlb->perm. Eventually when we get this in >>> vfio_get_vaddr() we would not print the warning. Could that make sense? >> >> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. >> Thanks, > > Let me know if I understood the proposal correctly: > > virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. > In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> > Is above correct? Yes that's what I had in mind. Thanks Eric > > Thanks > -Bharat > >> >> Alex > >
Hi Eric, On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: > > Hi Bharat, > > On 4/2/20 11:01 AM, Bharat Bhushan wrote: > > Hi Eric/Alex, > > > >> -----Original Message----- > >> From: Alex Williamson <alex.williamson@redhat.com> > >> Sent: Thursday, March 26, 2020 11:23 PM > >> To: Auger Eric <eric.auger@redhat.com> > >> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; > >> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > >> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; > >> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- > >> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; > >> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> > >> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio > >> region translation by viommu > >> > >> External Email > >> > >> ---------------------------------------------------------------------- > >> On Thu, 26 Mar 2020 18:35:48 +0100 > >> Auger Eric <eric.auger@redhat.com> wrote: > >> > >>> Hi Alex, > >>> > >>> On 3/24/20 12:08 AM, Alex Williamson wrote: > >>>> [Cc +dwg who originated this warning] > >>>> > >>>> On Mon, 23 Mar 2020 14:16:09 +0530 > >>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: > >>>> > >>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. > >>>>> As such address_space_translate() returns the MSI controller MMIO > >>>>> region and we get an "iommu map to non memory area" > >>>>> message. Let's remove this latter. > >>>>> > >>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > >>>>> --- > >>>>> hw/vfio/common.c | 2 -- > >>>>> 1 file changed, 2 deletions(-) > >>>>> > >>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index > >>>>> 5ca11488d6..c586edf47a 100644 > >>>>> --- a/hw/vfio/common.c > >>>>> +++ b/hw/vfio/common.c > >>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, > >> void **vaddr, > >>>>> &xlat, &len, writable, > >>>>> MEMTXATTRS_UNSPECIFIED); > >>>>> if (!memory_region_is_ram(mr)) { > >>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > >>>>> - xlat); > >>>>> return false; > >>>>> } > >>>>> > >>>> > >>>> I'm a bit confused here, I think we need more justification beyond > >>>> "we hit this warning and we don't want to because it's ok in this > >>>> one special case, therefore remove it". I assume the special case > >>>> is that the device MSI address is managed via the SET_IRQS ioctl and > >>>> therefore we won't actually get DMAs to this range. > >>> Yes exactly. The guest creates a mapping between one giova and this > >>> gpa (corresponding to the MSI controller doorbell) because MSIs are > >>> mapped on ARM. But practically the physical device is programmed with > >>> an host chosen iova that maps onto the physical MSI controller's > >>> doorbell. so the device never performs DMA accesses to this range. > >>> > >>> But I imagine the case that > >>>> was in mind when adding this warning was general peer-to-peer > >>>> between and assigned and emulated device. > >>> yes makes sense. > >>> > >>> Maybe there's an argument to be made > >>>> that such a p2p mapping might also be used in a non-vIOMMU case. We > >>>> skip creating those mappings and drivers continue to work, maybe > >>>> because nobody attempts to do p2p DMA with the types of devices we > >>>> emulate, maybe because p2p DMA is not absolutely reliable on bare > >>>> metal and drivers test it before using it. > >>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > >>> iommu_dma_get_msi_page). > >>> One idea could be to pass that flag through the IOMMU Notifier > >>> mechanism into the iotlb->perm. Eventually when we get this in > >>> vfio_get_vaddr() we would not print the warning. Could that make sense? > >> > >> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. > >> Thanks, > > > > Let me know if I understood the proposal correctly: > > > > virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. > > In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> > > Is above correct? > Yes that's what I had in mind. In that case virtio-iommu driver in guest should not make map (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. Stay Safe Thanks -Bharat > > Thanks > > Eric > > > > Thanks > > -Bharat > > > >> > >> Alex > > > > >
Hi Bharat, On 5/5/20 11:25 AM, Bharat Bhushan wrote: > Hi Eric, > > On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: >> >> Hi Bharat, >> >> On 4/2/20 11:01 AM, Bharat Bhushan wrote: >>> Hi Eric/Alex, >>> >>>> -----Original Message----- >>>> From: Alex Williamson <alex.williamson@redhat.com> >>>> Sent: Thursday, March 26, 2020 11:23 PM >>>> To: Auger Eric <eric.auger@redhat.com> >>>> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; >>>> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; >>>> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; >>>> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- >>>> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; >>>> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> >>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio >>>> region translation by viommu >>>> >>>> External Email >>>> >>>> ---------------------------------------------------------------------- >>>> On Thu, 26 Mar 2020 18:35:48 +0100 >>>> Auger Eric <eric.auger@redhat.com> wrote: >>>> >>>>> Hi Alex, >>>>> >>>>> On 3/24/20 12:08 AM, Alex Williamson wrote: >>>>>> [Cc +dwg who originated this warning] >>>>>> >>>>>> On Mon, 23 Mar 2020 14:16:09 +0530 >>>>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: >>>>>> >>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. >>>>>>> As such address_space_translate() returns the MSI controller MMIO >>>>>>> region and we get an "iommu map to non memory area" >>>>>>> message. Let's remove this latter. >>>>>>> >>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> >>>>>>> --- >>>>>>> hw/vfio/common.c | 2 -- >>>>>>> 1 file changed, 2 deletions(-) >>>>>>> >>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index >>>>>>> 5ca11488d6..c586edf47a 100644 >>>>>>> --- a/hw/vfio/common.c >>>>>>> +++ b/hw/vfio/common.c >>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, >>>> void **vaddr, >>>>>>> &xlat, &len, writable, >>>>>>> MEMTXATTRS_UNSPECIFIED); >>>>>>> if (!memory_region_is_ram(mr)) { >>>>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", >>>>>>> - xlat); >>>>>>> return false; >>>>>>> } >>>>>>> >>>>>> >>>>>> I'm a bit confused here, I think we need more justification beyond >>>>>> "we hit this warning and we don't want to because it's ok in this >>>>>> one special case, therefore remove it". I assume the special case >>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and >>>>>> therefore we won't actually get DMAs to this range. >>>>> Yes exactly. The guest creates a mapping between one giova and this >>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are >>>>> mapped on ARM. But practically the physical device is programmed with >>>>> an host chosen iova that maps onto the physical MSI controller's >>>>> doorbell. so the device never performs DMA accesses to this range. >>>>> >>>>> But I imagine the case that >>>>>> was in mind when adding this warning was general peer-to-peer >>>>>> between and assigned and emulated device. >>>>> yes makes sense. >>>>> >>>>> Maybe there's an argument to be made >>>>>> that such a p2p mapping might also be used in a non-vIOMMU case. We >>>>>> skip creating those mappings and drivers continue to work, maybe >>>>>> because nobody attempts to do p2p DMA with the types of devices we >>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare >>>>>> metal and drivers test it before using it. >>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >>>>> iommu_dma_get_msi_page). >>>>> One idea could be to pass that flag through the IOMMU Notifier >>>>> mechanism into the iotlb->perm. Eventually when we get this in >>>>> vfio_get_vaddr() we would not print the warning. Could that make sense? >>>> >>>> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. >>>> Thanks, >>> >>> Let me know if I understood the proposal correctly: >>> >>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. >>> In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> >>> Is above correct? >> Yes that's what I had in mind. > > In that case virtio-iommu driver in guest should not make map > (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. sorry I don't catch what you meant. Please can you elaborate? Thanks Eric > > Stay Safe > > Thanks > -Bharat > >> >> Thanks >> >> Eric >>> >>> Thanks >>> -Bharat >>> >>>> >>>> Alex >>> >>> >> >
hi Eric, On Tue, May 5, 2020 at 3:00 PM Auger Eric <eric.auger@redhat.com> wrote: > > Hi Bharat, > > On 5/5/20 11:25 AM, Bharat Bhushan wrote: > > Hi Eric, > > > > On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: > >> > >> Hi Bharat, > >> > >> On 4/2/20 11:01 AM, Bharat Bhushan wrote: > >>> Hi Eric/Alex, > >>> > >>>> -----Original Message----- > >>>> From: Alex Williamson <alex.williamson@redhat.com> > >>>> Sent: Thursday, March 26, 2020 11:23 PM > >>>> To: Auger Eric <eric.auger@redhat.com> > >>>> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; > >>>> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > >>>> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; > >>>> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- > >>>> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; > >>>> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> > >>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio > >>>> region translation by viommu > >>>> > >>>> External Email > >>>> > >>>> ---------------------------------------------------------------------- > >>>> On Thu, 26 Mar 2020 18:35:48 +0100 > >>>> Auger Eric <eric.auger@redhat.com> wrote: > >>>> > >>>>> Hi Alex, > >>>>> > >>>>> On 3/24/20 12:08 AM, Alex Williamson wrote: > >>>>>> [Cc +dwg who originated this warning] > >>>>>> > >>>>>> On Mon, 23 Mar 2020 14:16:09 +0530 > >>>>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: > >>>>>> > >>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. > >>>>>>> As such address_space_translate() returns the MSI controller MMIO > >>>>>>> region and we get an "iommu map to non memory area" > >>>>>>> message. Let's remove this latter. > >>>>>>> > >>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> > >>>>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > >>>>>>> --- > >>>>>>> hw/vfio/common.c | 2 -- > >>>>>>> 1 file changed, 2 deletions(-) > >>>>>>> > >>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index > >>>>>>> 5ca11488d6..c586edf47a 100644 > >>>>>>> --- a/hw/vfio/common.c > >>>>>>> +++ b/hw/vfio/common.c > >>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, > >>>> void **vaddr, > >>>>>>> &xlat, &len, writable, > >>>>>>> MEMTXATTRS_UNSPECIFIED); > >>>>>>> if (!memory_region_is_ram(mr)) { > >>>>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > >>>>>>> - xlat); > >>>>>>> return false; > >>>>>>> } > >>>>>>> > >>>>>> > >>>>>> I'm a bit confused here, I think we need more justification beyond > >>>>>> "we hit this warning and we don't want to because it's ok in this > >>>>>> one special case, therefore remove it". I assume the special case > >>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and > >>>>>> therefore we won't actually get DMAs to this range. > >>>>> Yes exactly. The guest creates a mapping between one giova and this > >>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are > >>>>> mapped on ARM. But practically the physical device is programmed with > >>>>> an host chosen iova that maps onto the physical MSI controller's > >>>>> doorbell. so the device never performs DMA accesses to this range. > >>>>> > >>>>> But I imagine the case that > >>>>>> was in mind when adding this warning was general peer-to-peer > >>>>>> between and assigned and emulated device. > >>>>> yes makes sense. > >>>>> > >>>>> Maybe there's an argument to be made > >>>>>> that such a p2p mapping might also be used in a non-vIOMMU case. We > >>>>>> skip creating those mappings and drivers continue to work, maybe > >>>>>> because nobody attempts to do p2p DMA with the types of devices we > >>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare > >>>>>> metal and drivers test it before using it. > >>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > >>>>> iommu_dma_get_msi_page). > >>>>> One idea could be to pass that flag through the IOMMU Notifier > >>>>> mechanism into the iotlb->perm. Eventually when we get this in > >>>>> vfio_get_vaddr() we would not print the warning. Could that make sense? > >>>> > >>>> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. > >>>> Thanks, > >>> > >>> Let me know if I understood the proposal correctly: > >>> > >>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. > >>> In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> > >>> Is above correct? > >> Yes that's what I had in mind. > > > > In that case virtio-iommu driver in guest should not make map > > (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. > sorry I don't catch what you meant. Please can you elaborate? What I understood of the proposal is: Linux: 1) MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c iommu_dma_get_msi_page) 2) virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. GEMU: 3) virtio-iommu device - If VIRTIO_IOMMU_MAP_F_MMIO flag set then will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu() 4. vfio_get_vaddr() will check same flag and will not print the warning. Also vfio_iommu_map_notify() will not do anything. So, rather than going down to step 3 and 4, can we avoid maling map() calling in step-2 itself? Thanks -Bharat > > Thanks > > Eric > > > > Stay Safe > > > > Thanks > > -Bharat > > > >> > >> Thanks > >> > >> Eric > >>> > >>> Thanks > >>> -Bharat > >>> > >>>> > >>>> Alex > >>> > >>> > >> > > >
Hi Eric, On Tue, May 5, 2020 at 3:16 PM Bharat Bhushan <bharatb.linux@gmail.com> wrote: > > hi Eric, > > On Tue, May 5, 2020 at 3:00 PM Auger Eric <eric.auger@redhat.com> wrote: > > > > Hi Bharat, > > > > On 5/5/20 11:25 AM, Bharat Bhushan wrote: > > > Hi Eric, > > > > > > On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: > > >> > > >> Hi Bharat, > > >> > > >> On 4/2/20 11:01 AM, Bharat Bhushan wrote: > > >>> Hi Eric/Alex, > > >>> > > >>>> -----Original Message----- > > >>>> From: Alex Williamson <alex.williamson@redhat.com> > > >>>> Sent: Thursday, March 26, 2020 11:23 PM > > >>>> To: Auger Eric <eric.auger@redhat.com> > > >>>> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; > > >>>> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > > >>>> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; > > >>>> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- > > >>>> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; > > >>>> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> > > >>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio > > >>>> region translation by viommu > > >>>> > > >>>> External Email > > >>>> > > >>>> ---------------------------------------------------------------------- > > >>>> On Thu, 26 Mar 2020 18:35:48 +0100 > > >>>> Auger Eric <eric.auger@redhat.com> wrote: > > >>>> > > >>>>> Hi Alex, > > >>>>> > > >>>>> On 3/24/20 12:08 AM, Alex Williamson wrote: > > >>>>>> [Cc +dwg who originated this warning] > > >>>>>> > > >>>>>> On Mon, 23 Mar 2020 14:16:09 +0530 > > >>>>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: > > >>>>>> > > >>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. > > >>>>>>> As such address_space_translate() returns the MSI controller MMIO > > >>>>>>> region and we get an "iommu map to non memory area" > > >>>>>>> message. Let's remove this latter. > > >>>>>>> > > >>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> > > >>>>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> > > >>>>>>> --- > > >>>>>>> hw/vfio/common.c | 2 -- > > >>>>>>> 1 file changed, 2 deletions(-) > > >>>>>>> > > >>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index > > >>>>>>> 5ca11488d6..c586edf47a 100644 > > >>>>>>> --- a/hw/vfio/common.c > > >>>>>>> +++ b/hw/vfio/common.c > > >>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, > > >>>> void **vaddr, > > >>>>>>> &xlat, &len, writable, > > >>>>>>> MEMTXATTRS_UNSPECIFIED); > > >>>>>>> if (!memory_region_is_ram(mr)) { > > >>>>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", > > >>>>>>> - xlat); > > >>>>>>> return false; > > >>>>>>> } > > >>>>>>> > > >>>>>> > > >>>>>> I'm a bit confused here, I think we need more justification beyond > > >>>>>> "we hit this warning and we don't want to because it's ok in this > > >>>>>> one special case, therefore remove it". I assume the special case > > >>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and > > >>>>>> therefore we won't actually get DMAs to this range. > > >>>>> Yes exactly. The guest creates a mapping between one giova and this > > >>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are > > >>>>> mapped on ARM. But practically the physical device is programmed with > > >>>>> an host chosen iova that maps onto the physical MSI controller's > > >>>>> doorbell. so the device never performs DMA accesses to this range. > > >>>>> > > >>>>> But I imagine the case that > > >>>>>> was in mind when adding this warning was general peer-to-peer > > >>>>>> between and assigned and emulated device. > > >>>>> yes makes sense. > > >>>>> > > >>>>> Maybe there's an argument to be made > > >>>>>> that such a p2p mapping might also be used in a non-vIOMMU case. We > > >>>>>> skip creating those mappings and drivers continue to work, maybe > > >>>>>> because nobody attempts to do p2p DMA with the types of devices we > > >>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare > > >>>>>> metal and drivers test it before using it. > > >>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > > >>>>> iommu_dma_get_msi_page). > > >>>>> One idea could be to pass that flag through the IOMMU Notifier > > >>>>> mechanism into the iotlb->perm. Eventually when we get this in > > >>>>> vfio_get_vaddr() we would not print the warning. Could that make sense? > > >>>> > > >>>> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. > > >>>> Thanks, > > >>> > > >>> Let me know if I understood the proposal correctly: > > >>> > > >>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. > > >>> In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> > > >>> Is above correct? > > >> Yes that's what I had in mind. > > > > > > In that case virtio-iommu driver in guest should not make map > > > (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. > > sorry I don't catch what you meant. Please can you elaborate? > > What I understood of the proposal is: > Linux: > 1) MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c > iommu_dma_get_msi_page) > 2) virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) > with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. > > GEMU: > 3) virtio-iommu device - If VIRTIO_IOMMU_MAP_F_MMIO flag set then will > set a new defined flag (say IOMMU_MMIO) in iotlb->perm in > memory_region_notify_iommu() > 4. vfio_get_vaddr() will check same flag and will not print the > warning. Also vfio_iommu_map_notify() will not do anything. > > So, rather than going down to step 3 and 4, can we avoid maling map() > calling in step-2 itself? ohh, We need to setup msi translation mapping, correct. Thanks -Bharat > > Thanks > -Bharat > > > > > Thanks > > > > Eric > > > > > > Stay Safe > > > > > > Thanks > > > -Bharat > > > > > >> > > >> Thanks > > >> > > >> Eric > > >>> > > >>> Thanks > > >>> -Bharat > > >>> > > >>>> > > >>>> Alex > > >>> > > >>> > > >> > > > > >
Hi Bharat, On 5/5/20 12:18 PM, Bharat Bhushan wrote: > Hi Eric, > > On Tue, May 5, 2020 at 3:16 PM Bharat Bhushan <bharatb.linux@gmail.com> wrote: >> >> hi Eric, >> >> On Tue, May 5, 2020 at 3:00 PM Auger Eric <eric.auger@redhat.com> wrote: >>> >>> Hi Bharat, >>> >>> On 5/5/20 11:25 AM, Bharat Bhushan wrote: >>>> Hi Eric, >>>> >>>> On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: >>>>> >>>>> Hi Bharat, >>>>> >>>>> On 4/2/20 11:01 AM, Bharat Bhushan wrote: >>>>>> Hi Eric/Alex, >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Alex Williamson <alex.williamson@redhat.com> >>>>>>> Sent: Thursday, March 26, 2020 11:23 PM >>>>>>> To: Auger Eric <eric.auger@redhat.com> >>>>>>> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; >>>>>>> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; >>>>>>> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; >>>>>>> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- >>>>>>> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; >>>>>>> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> >>>>>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio >>>>>>> region translation by viommu >>>>>>> >>>>>>> External Email >>>>>>> >>>>>>> ---------------------------------------------------------------------- >>>>>>> On Thu, 26 Mar 2020 18:35:48 +0100 >>>>>>> Auger Eric <eric.auger@redhat.com> wrote: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> On 3/24/20 12:08 AM, Alex Williamson wrote: >>>>>>>>> [Cc +dwg who originated this warning] >>>>>>>>> >>>>>>>>> On Mon, 23 Mar 2020 14:16:09 +0530 >>>>>>>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: >>>>>>>>> >>>>>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. >>>>>>>>>> As such address_space_translate() returns the MSI controller MMIO >>>>>>>>>> region and we get an "iommu map to non memory area" >>>>>>>>>> message. Let's remove this latter. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>>>>>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> >>>>>>>>>> --- >>>>>>>>>> hw/vfio/common.c | 2 -- >>>>>>>>>> 1 file changed, 2 deletions(-) >>>>>>>>>> >>>>>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index >>>>>>>>>> 5ca11488d6..c586edf47a 100644 >>>>>>>>>> --- a/hw/vfio/common.c >>>>>>>>>> +++ b/hw/vfio/common.c >>>>>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, >>>>>>> void **vaddr, >>>>>>>>>> &xlat, &len, writable, >>>>>>>>>> MEMTXATTRS_UNSPECIFIED); >>>>>>>>>> if (!memory_region_is_ram(mr)) { >>>>>>>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", >>>>>>>>>> - xlat); >>>>>>>>>> return false; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'm a bit confused here, I think we need more justification beyond >>>>>>>>> "we hit this warning and we don't want to because it's ok in this >>>>>>>>> one special case, therefore remove it". I assume the special case >>>>>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and >>>>>>>>> therefore we won't actually get DMAs to this range. >>>>>>>> Yes exactly. The guest creates a mapping between one giova and this >>>>>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are >>>>>>>> mapped on ARM. But practically the physical device is programmed with >>>>>>>> an host chosen iova that maps onto the physical MSI controller's >>>>>>>> doorbell. so the device never performs DMA accesses to this range. >>>>>>>> >>>>>>>> But I imagine the case that >>>>>>>>> was in mind when adding this warning was general peer-to-peer >>>>>>>>> between and assigned and emulated device. >>>>>>>> yes makes sense. >>>>>>>> >>>>>>>> Maybe there's an argument to be made >>>>>>>>> that such a p2p mapping might also be used in a non-vIOMMU case. We >>>>>>>>> skip creating those mappings and drivers continue to work, maybe >>>>>>>>> because nobody attempts to do p2p DMA with the types of devices we >>>>>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare >>>>>>>>> metal and drivers test it before using it. >>>>>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >>>>>>>> iommu_dma_get_msi_page). >>>>>>>> One idea could be to pass that flag through the IOMMU Notifier >>>>>>>> mechanism into the iotlb->perm. Eventually when we get this in >>>>>>>> vfio_get_vaddr() we would not print the warning. Could that make sense? >>>>>>> >>>>>>> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. >>>>>>> Thanks, >>>>>> >>>>>> Let me know if I understood the proposal correctly: >>>>>> >>>>>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. >>>>>> In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> >>>>>> Is above correct? >>>>> Yes that's what I had in mind. >>>> >>>> In that case virtio-iommu driver in guest should not make map >>>> (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. >>> sorry I don't catch what you meant. Please can you elaborate? >> >> What I understood of the proposal is: >> Linux: >> 1) MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >> iommu_dma_get_msi_page) >> 2) virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) >> with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. >> >> GEMU: >> 3) virtio-iommu device - If VIRTIO_IOMMU_MAP_F_MMIO flag set then will >> set a new defined flag (say IOMMU_MMIO) in iotlb->perm in >> memory_region_notify_iommu() >> 4. vfio_get_vaddr() will check same flag and will not print the >> warning. Also vfio_iommu_map_notify() will not do anything. >> >> So, rather than going down to step 3 and 4, can we avoid maling map() >> calling in step-2 itself? > > ohh, We need to setup msi translation mapping, correct. The virtio-iommu driver is supposed to obey the map() operation argument and cascade this to the device. I don't see why it would filter this out. Now we should have means to declare the vITS GPA as an HW MSI reserved region and avoid calling the map(). I will try this out. Thanks Eric > > Thanks > -Bharat > >> >> Thanks >> -Bharat >> >>> >>> Thanks >>> >>> Eric >>>> >>>> Stay Safe >>>> >>>> Thanks >>>> -Bharat >>>> >>>>> >>>>> Thanks >>>>> >>>>> Eric >>>>>> >>>>>> Thanks >>>>>> -Bharat >>>>>> >>>>>>> >>>>>>> Alex >>>>>> >>>>>> >>>>> >>>> >>> >
Hi Bharat, On 5/5/20 12:18 PM, Bharat Bhushan wrote: > Hi Eric, > > On Tue, May 5, 2020 at 3:16 PM Bharat Bhushan <bharatb.linux@gmail.com> wrote: >> >> hi Eric, >> >> On Tue, May 5, 2020 at 3:00 PM Auger Eric <eric.auger@redhat.com> wrote: >>> >>> Hi Bharat, >>> >>> On 5/5/20 11:25 AM, Bharat Bhushan wrote: >>>> Hi Eric, >>>> >>>> On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <eric.auger@redhat.com> wrote: >>>>> >>>>> Hi Bharat, >>>>> >>>>> On 4/2/20 11:01 AM, Bharat Bhushan wrote: >>>>>> Hi Eric/Alex, >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Alex Williamson <alex.williamson@redhat.com> >>>>>>> Sent: Thursday, March 26, 2020 11:23 PM >>>>>>> To: Auger Eric <eric.auger@redhat.com> >>>>>>> Cc: Bharat Bhushan <bbhushan2@marvell.com>; peter.maydell@linaro.org; >>>>>>> peterx@redhat.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; >>>>>>> mst@redhat.com; Tomasz Nowicki [C] <tnowicki@marvell.com>; >>>>>>> drjones@redhat.com; linuc.decode@gmail.com; qemu-devel@nongnu.org; qemu- >>>>>>> arm@nongnu.org; bharatb.linux@gmail.com; jean-philippe@linaro.org; >>>>>>> yang.zhong@intel.com; David Gibson <david@gibson.dropbear.id.au> >>>>>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio >>>>>>> region translation by viommu >>>>>>> >>>>>>> External Email >>>>>>> >>>>>>> ---------------------------------------------------------------------- >>>>>>> On Thu, 26 Mar 2020 18:35:48 +0100 >>>>>>> Auger Eric <eric.auger@redhat.com> wrote: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> On 3/24/20 12:08 AM, Alex Williamson wrote: >>>>>>>>> [Cc +dwg who originated this warning] >>>>>>>>> >>>>>>>>> On Mon, 23 Mar 2020 14:16:09 +0530 >>>>>>>>> Bharat Bhushan <bbhushan2@marvell.com> wrote: >>>>>>>>> >>>>>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU. >>>>>>>>>> As such address_space_translate() returns the MSI controller MMIO >>>>>>>>>> region and we get an "iommu map to non memory area" >>>>>>>>>> message. Let's remove this latter. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com> >>>>>>>>>> Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com> >>>>>>>>>> --- >>>>>>>>>> hw/vfio/common.c | 2 -- >>>>>>>>>> 1 file changed, 2 deletions(-) >>>>>>>>>> >>>>>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index >>>>>>>>>> 5ca11488d6..c586edf47a 100644 >>>>>>>>>> --- a/hw/vfio/common.c >>>>>>>>>> +++ b/hw/vfio/common.c >>>>>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, >>>>>>> void **vaddr, >>>>>>>>>> &xlat, &len, writable, >>>>>>>>>> MEMTXATTRS_UNSPECIFIED); >>>>>>>>>> if (!memory_region_is_ram(mr)) { >>>>>>>>>> - error_report("iommu map to non memory area %"HWADDR_PRIx"", >>>>>>>>>> - xlat); >>>>>>>>>> return false; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'm a bit confused here, I think we need more justification beyond >>>>>>>>> "we hit this warning and we don't want to because it's ok in this >>>>>>>>> one special case, therefore remove it". I assume the special case >>>>>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and >>>>>>>>> therefore we won't actually get DMAs to this range. >>>>>>>> Yes exactly. The guest creates a mapping between one giova and this >>>>>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are >>>>>>>> mapped on ARM. But practically the physical device is programmed with >>>>>>>> an host chosen iova that maps onto the physical MSI controller's >>>>>>>> doorbell. so the device never performs DMA accesses to this range. >>>>>>>> >>>>>>>> But I imagine the case that >>>>>>>>> was in mind when adding this warning was general peer-to-peer >>>>>>>>> between and assigned and emulated device. >>>>>>>> yes makes sense. >>>>>>>> >>>>>>>> Maybe there's an argument to be made >>>>>>>>> that such a p2p mapping might also be used in a non-vIOMMU case. We >>>>>>>>> skip creating those mappings and drivers continue to work, maybe >>>>>>>>> because nobody attempts to do p2p DMA with the types of devices we >>>>>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare >>>>>>>>> metal and drivers test it before using it. >>>>>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >>>>>>>> iommu_dma_get_msi_page). >>>>>>>> One idea could be to pass that flag through the IOMMU Notifier >>>>>>>> mechanism into the iotlb->perm. Eventually when we get this in >>>>>>>> vfio_get_vaddr() we would not print the warning. Could that make sense? >>>>>>> >>>>>>> Yeah, if we can identify a valid case that doesn't need a warning, that's fine by me. >>>>>>> Thanks, >>>>>> >>>>>> Let me know if I understood the proposal correctly: >>>>>> >>>>>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. >>>>>> In qemu, virtio-iommu device will set a new defined flag (say IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). vfio_get_vaddr() will check same flag and will not print the warning.> >>>>>> Is above correct? >>>>> Yes that's what I had in mind. >>>> >>>> In that case virtio-iommu driver in guest should not make map >>>> (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped. >>> sorry I don't catch what you meant. Please can you elaborate? >> >> What I understood of the proposal is: >> Linux: >> 1) MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c >> iommu_dma_get_msi_page) >> 2) virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) >> with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping. >> >> GEMU: >> 3) virtio-iommu device - If VIRTIO_IOMMU_MAP_F_MMIO flag set then will >> set a new defined flag (say IOMMU_MMIO) in iotlb->perm in >> memory_region_notify_iommu() >> 4. vfio_get_vaddr() will check same flag and will not print the >> warning. Also vfio_iommu_map_notify() will not do anything. >> >> So, rather than going down to step 3 and 4, can we avoid maling map() >> calling in step-2 itself? > > ohh, We need to setup msi translation mapping, correct. I sent [PATCH 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM With this series, the guest does not attempt to map the ITS MSI doorbell anymore. So you shouldn't get "iommu map to non memory area" anymore if you rebase on it and remove this patch. At the moment I only bypass the ITS doorbell but I think it is safe to do the same with GICv2M doorbell as well. Anyway all assigned devices will be able to access this latter. Thanks Eric > > Thanks > -Bharat > >> >> Thanks >> -Bharat >> >>> >>> Thanks >>> >>> Eric >>>> >>>> Stay Safe >>>> >>>> Thanks >>>> -Bharat >>>> >>>>> >>>>> Thanks >>>>> >>>>> Eric >>>>>> >>>>>> Thanks >>>>>> -Bharat >>>>>> >>>>>>> >>>>>>> Alex >>>>>> >>>>>> >>>>> >>>> >>> >
diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 5ca11488d6..c586edf47a 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void **vaddr, &xlat, &len, writable, MEMTXATTRS_UNSPECIFIED); if (!memory_region_is_ram(mr)) { - error_report("iommu map to non memory area %"HWADDR_PRIx"", - xlat); return false; }