| Message ID | 1555038815-31916-1-git-send-email-srinath.mannam@broadcom.com |
|---|---|
| Headers | show |
| Series | PCIe Host request to reserve IOVA | expand |
On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > Few SOCs have limitation that their PCIe host can't allow few inbound > address ranges. Allowed inbound address ranges are listed in dma-ranges > DT property and this address ranges are required to do IOVA mapping. > Remaining address ranges have to be reserved in IOVA mapping. If I understand correctly, devices below these PCIe host bridges can DMA only to the listed address ranges, and you prevent devices from doing DMA to the holes between the listed ranges by reserving the holes in dma-iommu. Apparently there's something that makes sure driver dma_map_*() always goes through dma-iommu? I traced as far as seeing that dma-iommu depends on CONFIG_IOMMU_DMA, and that arm64 selects CONFIG_IOMMU_DMA if CONFIG_IOMMU_SUPPORT, but then the trail got cold. I didn't see what selects CONFIG_IOMMU_SUPPORT. This does look like what Robin suggested, as far as I can tell. Hopefully he'll take a look and give his reviewed-by. Thanks for persevering! > PCIe Host driver of those SOCs has to list resource entries of allowed > address ranges given in dma-ranges DT property in sorted order. This > sorted list of resources will be processed and reserve IOVA address for > inaccessible address holes while initializing IOMMU domain. > > This patch set is based on Linux-5.0-rc2. > > Changes from v3: > - Addressed Robin Murphy review comments. > - pcie-iproc: parse dma-ranges and make sorted resource list. > - dma-iommu: process list and reserve gaps between entries > > Changes from v2: > - Patch set rebased to Linux-5.0-rc2 > > Changes from v1: > - Addressed Oza review comments. > > Srinath Mannam (3): > PCI: Add dma_ranges window list > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > drivers/pci/probe.c | 3 +++ > include/linux/pci.h | 1 + > 4 files changed, 66 insertions(+), 1 deletion(-) > > -- > 2.7.4 >
Hi Bjorn, Thanks for review. Please find my reply below. On Sat, Apr 13, 2019 at 4:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > Few SOCs have limitation that their PCIe host can't allow few inbound > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > DT property and this address ranges are required to do IOVA mapping. > > Remaining address ranges have to be reserved in IOVA mapping. > > If I understand correctly, devices below these PCIe host bridges can > DMA only to the listed address ranges, and you prevent devices from > doing DMA to the holes between the listed ranges by reserving the > holes in dma-iommu. Yes, devices below these PCIe host bridges can DMA only to the listed address ranges, and this patch prevents to allocate DMA(IOVA) addresses in the holes of listed ranges. > > Apparently there's something that makes sure driver dma_map_*() always > goes through dma-iommu? I traced as far as seeing that dma-iommu > depends on CONFIG_IOMMU_DMA, and that arm64 selects CONFIG_IOMMU_DMA > if CONFIG_IOMMU_SUPPORT, but then the trail got cold. I didn't see > what selects CONFIG_IOMMU_SUPPORT. IOMMU_SUPPORT depends on MMU. > > This does look like what Robin suggested, as far as I can tell. > Hopefully he'll take a look and give his reviewed-by. Thanks for > persevering! Thank you. Regards, Srinath. > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > address ranges given in dma-ranges DT property in sorted order. This > > sorted list of resources will be processed and reserve IOVA address for > > inaccessible address holes while initializing IOMMU domain. > > > > This patch set is based on Linux-5.0-rc2. > > > > Changes from v3: > > - Addressed Robin Murphy review comments. > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > - dma-iommu: process list and reserve gaps between entries > > > > Changes from v2: > > - Patch set rebased to Linux-5.0-rc2 > > > > Changes from v1: > > - Addressed Oza review comments. > > > > Srinath Mannam (3): > > PCI: Add dma_ranges window list > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > drivers/pci/probe.c | 3 +++ > > include/linux/pci.h | 1 + > > 4 files changed, 66 insertions(+), 1 deletion(-) > > > > -- > > 2.7.4 > >
On Tue, Apr 16, 2019 at 05:28:36PM +0530, Srinath Mannam wrote: > On Sat, Apr 13, 2019 at 4:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > DT property and this address ranges are required to do IOVA mapping. > > > Remaining address ranges have to be reserved in IOVA mapping. > > > If I understand correctly, devices below these PCIe host bridges can > > DMA only to the listed address ranges, and you prevent devices from > > doing DMA to the holes between the listed ranges by reserving the > > holes in dma-iommu. > > Yes, devices below these PCIe host bridges can DMA only to the listed > address ranges, > and this patch prevents to allocate DMA(IOVA) addresses in the holes > of listed ranges. > > > Apparently there's something that makes sure driver dma_map_*() always > > goes through dma-iommu? I traced as far as seeing that dma-iommu > > depends on CONFIG_IOMMU_DMA, and that arm64 selects CONFIG_IOMMU_DMA > > if CONFIG_IOMMU_SUPPORT, but then the trail got cold. I didn't see > > what selects CONFIG_IOMMU_SUPPORT. > > IOMMU_SUPPORT depends on MMU. Yes, I see that IOMMU_SUPPORT depends on MMU (in drivers/iommu/Kconfig). But that doesn't *select* IOMMU_SUPPORT; it only means you *can't* select it unless MMU has already been selected. I think you only get dma-iommu if you choose to select IOMMU_SUPPORT via menuconfig or whatever, and the current config rules allow you to turn that off. Maybe that's OK, I dunno. If you do turn it off, I guess we'll ignore the holes in "dma-ranges" and devices will be able to DMA to the holes. > > This does look like what Robin suggested, as far as I can tell. > > Hopefully he'll take a look and give his reviewed-by. Thanks for > > persevering! > Thank you. > > Regards, > Srinath. > > > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > > address ranges given in dma-ranges DT property in sorted order. This > > > sorted list of resources will be processed and reserve IOVA address for > > > inaccessible address holes while initializing IOMMU domain. > > > > > > This patch set is based on Linux-5.0-rc2. > > > > > > Changes from v3: > > > - Addressed Robin Murphy review comments. > > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > > - dma-iommu: process list and reserve gaps between entries > > > > > > Changes from v2: > > > - Patch set rebased to Linux-5.0-rc2 > > > > > > Changes from v1: > > > - Addressed Oza review comments. > > > > > > Srinath Mannam (3): > > > PCI: Add dma_ranges window list > > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > > drivers/pci/probe.c | 3 +++ > > > include/linux/pci.h | 1 + > > > 4 files changed, 66 insertions(+), 1 deletion(-) > > > > > > -- > > > 2.7.4 > > >
[+cc Scott] On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > Few SOCs have limitation that their PCIe host can't allow few inbound > address ranges. Allowed inbound address ranges are listed in dma-ranges > DT property and this address ranges are required to do IOVA mapping. > Remaining address ranges have to be reserved in IOVA mapping. > > PCIe Host driver of those SOCs has to list resource entries of allowed > address ranges given in dma-ranges DT property in sorted order. This > sorted list of resources will be processed and reserve IOVA address for > inaccessible address holes while initializing IOMMU domain. > > This patch set is based on Linux-5.0-rc2. > > Changes from v3: > - Addressed Robin Murphy review comments. > - pcie-iproc: parse dma-ranges and make sorted resource list. > - dma-iommu: process list and reserve gaps between entries > > Changes from v2: > - Patch set rebased to Linux-5.0-rc2 > > Changes from v1: > - Addressed Oza review comments. > > Srinath Mannam (3): > PCI: Add dma_ranges window list > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > drivers/pci/probe.c | 3 +++ > include/linux/pci.h | 1 + > 4 files changed, 66 insertions(+), 1 deletion(-) To make progress on this, I think we need an ack from Joerg for the dma-iommu.c part, an ack from Ray or Scott for the pcie-iproc.c part, and an ack from Robin for the thing as a whole. Then I would say Lorenzo should take a look and merge if he approves, since pcie-iproc.c contains the most changes. Bjorn
On Thu, Apr 18, 2019 at 06:48:28PM -0500, Bjorn Helgaas wrote: > To make progress on this, I think we need an ack from Joerg for the > dma-iommu.c part, an ack from Ray or Scott for the pcie-iproc.c part, > and an ack from Robin for the thing as a whole. I wait for Robin to review the dma-iommu change. If he is okay with it, I am too. Joerg
On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > Few SOCs have limitation that their PCIe host can't allow few inbound > address ranges. Allowed inbound address ranges are listed in dma-ranges > DT property and this address ranges are required to do IOVA mapping. > Remaining address ranges have to be reserved in IOVA mapping. > > PCIe Host driver of those SOCs has to list resource entries of allowed > address ranges given in dma-ranges DT property in sorted order. This > sorted list of resources will be processed and reserve IOVA address for > inaccessible address holes while initializing IOMMU domain. > > This patch set is based on Linux-5.0-rc2. > > Changes from v3: > - Addressed Robin Murphy review comments. > - pcie-iproc: parse dma-ranges and make sorted resource list. > - dma-iommu: process list and reserve gaps between entries > > Changes from v2: > - Patch set rebased to Linux-5.0-rc2 > > Changes from v1: > - Addressed Oza review comments. > > Srinath Mannam (3): > PCI: Add dma_ranges window list > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > drivers/pci/probe.c | 3 +++ > include/linux/pci.h | 1 + > 4 files changed, 66 insertions(+), 1 deletion(-) Bjorn, Joerg, this series should not affect anything in the mainline other than its consumer (ie patch 3); if that's the case should we consider it for v5.2 and if yes how are we going to merge it ? Thanks, Lorenzo
On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > Few SOCs have limitation that their PCIe host can't allow few inbound > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > DT property and this address ranges are required to do IOVA mapping. > > Remaining address ranges have to be reserved in IOVA mapping. > > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > address ranges given in dma-ranges DT property in sorted order. This > > sorted list of resources will be processed and reserve IOVA address for > > inaccessible address holes while initializing IOMMU domain. > > > > This patch set is based on Linux-5.0-rc2. > > > > Changes from v3: > > - Addressed Robin Murphy review comments. > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > - dma-iommu: process list and reserve gaps between entries > > > > Changes from v2: > > - Patch set rebased to Linux-5.0-rc2 > > > > Changes from v1: > > - Addressed Oza review comments. > > > > Srinath Mannam (3): > > PCI: Add dma_ranges window list > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > drivers/pci/probe.c | 3 +++ > > include/linux/pci.h | 1 + > > 4 files changed, 66 insertions(+), 1 deletion(-) > > Bjorn, Joerg, > > this series should not affect anything in the mainline other than its > consumer (ie patch 3); if that's the case should we consider it for v5.2 > and if yes how are we going to merge it ? I acked the first one Robin reviewed the second (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK with it if Robin was (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). Eric reviewed the third (and pointed out a typo). My Kconfiggery never got fully answered -- it looks to me as though it's possible to build pcie-iproc without the DMA hole support, and I thought the whole point of this series was to deal with those holes (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would have expected something like making pcie-iproc depend on IOMMU_SUPPORT. But Srinath didn't respond to that, so maybe it's not an issue and it should only affect pcie-iproc anyway. So bottom line, I'm fine with merging it for v5.2. Do you want to merge it, Lorenzo, or ...? Bjorn
On 2019-05-01 1:55 pm, Bjorn Helgaas wrote: > On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: >> On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: >>> Few SOCs have limitation that their PCIe host can't allow few inbound >>> address ranges. Allowed inbound address ranges are listed in dma-ranges >>> DT property and this address ranges are required to do IOVA mapping. >>> Remaining address ranges have to be reserved in IOVA mapping. >>> >>> PCIe Host driver of those SOCs has to list resource entries of allowed >>> address ranges given in dma-ranges DT property in sorted order. This >>> sorted list of resources will be processed and reserve IOVA address for >>> inaccessible address holes while initializing IOMMU domain. >>> >>> This patch set is based on Linux-5.0-rc2. >>> >>> Changes from v3: >>> - Addressed Robin Murphy review comments. >>> - pcie-iproc: parse dma-ranges and make sorted resource list. >>> - dma-iommu: process list and reserve gaps between entries >>> >>> Changes from v2: >>> - Patch set rebased to Linux-5.0-rc2 >>> >>> Changes from v1: >>> - Addressed Oza review comments. >>> >>> Srinath Mannam (3): >>> PCI: Add dma_ranges window list >>> iommu/dma: Reserve IOVA for PCIe inaccessible DMA address >>> PCI: iproc: Add sorted dma ranges resource entries to host bridge >>> >>> drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ >>> drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- >>> drivers/pci/probe.c | 3 +++ >>> include/linux/pci.h | 1 + >>> 4 files changed, 66 insertions(+), 1 deletion(-) >> >> Bjorn, Joerg, >> >> this series should not affect anything in the mainline other than its >> consumer (ie patch 3); if that's the case should we consider it for v5.2 >> and if yes how are we going to merge it ? > > I acked the first one > > Robin reviewed the second > (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) > (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK > with it if Robin was > (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). > > Eric reviewed the third (and pointed out a typo). > > My Kconfiggery never got fully answered -- it looks to me as though it's > possible to build pcie-iproc without the DMA hole support, and I thought > the whole point of this series was to deal with those holes > (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would > have expected something like making pcie-iproc depend on IOMMU_SUPPORT. > But Srinath didn't respond to that, so maybe it's not an issue and it > should only affect pcie-iproc anyway. Hmm, I'm sure I had at least half-written a reply on that point, but I can't seem to find it now... anyway, the gist is that these inbound windows are generally set up to cover the physical address ranges of DRAM and anything else that devices might need to DMA to. Thus if you're not using an IOMMU, the fact that devices can't access the gaps in between doesn't matter because there won't be anything there anyway; it only needs mitigating if you do use an IOMMU and start giving arbitrary non-physical addresses to the endpoint. > So bottom line, I'm fine with merging it for v5.2. Do you want to merge > it, Lorenzo, or ...? This doesn't look like it will conflict with the other DMA ops and MSI mapping changes currently in-flight for iommu-dma, so I have no objection to it going through the PCI tree for 5.2. Robin.
On Wed, May 01, 2019 at 02:20:56PM +0100, Robin Murphy wrote: > On 2019-05-01 1:55 pm, Bjorn Helgaas wrote: > > On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: > > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > > DT property and this address ranges are required to do IOVA mapping. > > > > Remaining address ranges have to be reserved in IOVA mapping. > > > > > > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > > > address ranges given in dma-ranges DT property in sorted order. This > > > > sorted list of resources will be processed and reserve IOVA address for > > > > inaccessible address holes while initializing IOMMU domain. > > > > > > > > This patch set is based on Linux-5.0-rc2. > > > > > > > > Changes from v3: > > > > - Addressed Robin Murphy review comments. > > > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > > > - dma-iommu: process list and reserve gaps between entries > > > > > > > > Changes from v2: > > > > - Patch set rebased to Linux-5.0-rc2 > > > > > > > > Changes from v1: > > > > - Addressed Oza review comments. > > > > > > > > Srinath Mannam (3): > > > > PCI: Add dma_ranges window list > > > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > > > drivers/pci/probe.c | 3 +++ > > > > include/linux/pci.h | 1 + > > > > 4 files changed, 66 insertions(+), 1 deletion(-) > > > > > > Bjorn, Joerg, > > > > > > this series should not affect anything in the mainline other than its > > > consumer (ie patch 3); if that's the case should we consider it for v5.2 > > > and if yes how are we going to merge it ? > > > > I acked the first one > > > > Robin reviewed the second > > (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) > > (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK > > with it if Robin was > > (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). > > > > Eric reviewed the third (and pointed out a typo). > > > > My Kconfiggery never got fully answered -- it looks to me as though it's > > possible to build pcie-iproc without the DMA hole support, and I thought > > the whole point of this series was to deal with those holes > > (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would > > have expected something like making pcie-iproc depend on IOMMU_SUPPORT. > > But Srinath didn't respond to that, so maybe it's not an issue and it > > should only affect pcie-iproc anyway. > > Hmm, I'm sure I had at least half-written a reply on that point, but I > can't seem to find it now... anyway, the gist is that these inbound > windows are generally set up to cover the physical address ranges of DRAM > and anything else that devices might need to DMA to. Thus if you're not > using an IOMMU, the fact that devices can't access the gaps in between > doesn't matter because there won't be anything there anyway; it only > needs mitigating if you do use an IOMMU and start giving arbitrary > non-physical addresses to the endpoint. So basically there is no strict IOMMU_SUPPORT dependency. > > So bottom line, I'm fine with merging it for v5.2. Do you want to merge > > it, Lorenzo, or ...? > > This doesn't look like it will conflict with the other DMA ops and MSI > mapping changes currently in-flight for iommu-dma, so I have no > objection to it going through the PCI tree for 5.2. I will update the DMA_BIT_MASK() according to your review and fix the typo Eric pointed out and push out a branch - we shall see if we can include it for v5.2. Thanks, Lorenzo
Hi Bjorn, Thank you. Please find my reply below. On Wed, May 1, 2019 at 6:25 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > DT property and this address ranges are required to do IOVA mapping. > > > Remaining address ranges have to be reserved in IOVA mapping. > > > > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > > address ranges given in dma-ranges DT property in sorted order. This > > > sorted list of resources will be processed and reserve IOVA address for > > > inaccessible address holes while initializing IOMMU domain. > > > > > > This patch set is based on Linux-5.0-rc2. > > > > > > Changes from v3: > > > - Addressed Robin Murphy review comments. > > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > > - dma-iommu: process list and reserve gaps between entries > > > > > > Changes from v2: > > > - Patch set rebased to Linux-5.0-rc2 > > > > > > Changes from v1: > > > - Addressed Oza review comments. > > > > > > Srinath Mannam (3): > > > PCI: Add dma_ranges window list > > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > > drivers/pci/probe.c | 3 +++ > > > include/linux/pci.h | 1 + > > > 4 files changed, 66 insertions(+), 1 deletion(-) > > > > Bjorn, Joerg, > > > > this series should not affect anything in the mainline other than its > > consumer (ie patch 3); if that's the case should we consider it for v5.2 > > and if yes how are we going to merge it ? > > I acked the first one I will send new patch with change in commit message as per your comment. "s/bridge This list/bridge, this list/" > > Robin reviewed the second > (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) > (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK > with it if Robin was > (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). > I will send patch, for "DMA_BIT_MASK(sizeof(dma_addr_t) * BITS_PER_BYTE)" change to "~(dma_addr_t)0". > Eric reviewed the third (and pointed out a typo). I will send a new patch to address this typo. > > My Kconfiggery never got fully answered -- it looks to me as though it's > possible to build pcie-iproc without the DMA hole support, and I thought > the whole point of this series was to deal with those holes > (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would > have expected something like making pcie-iproc depend on IOMMU_SUPPORT. > But Srinath didn't respond to that, so maybe it's not an issue and it > should only affect pcie-iproc anyway. I am sorry to miss to respond.. In NO-IOMMU case, All inbound addresses allocated from dma APIs are physical addresses of DDR. All DDR physical addresses are fall inside given dma-ranges. so that, without IOMMU_SUPPORT, will not be any issue. Regards, Srinath. > > So bottom line, I'm fine with merging it for v5.2. Do you want to merge > it, Lorenzo, or ...? > > Bjorn
Hi Robin, Thank you so much for all the information. Regards, Srinath. On Wed, May 1, 2019 at 6:51 PM Robin Murphy <robin.murphy@arm.com> wrote: > > On 2019-05-01 1:55 pm, Bjorn Helgaas wrote: > > On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: > >> On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > >>> Few SOCs have limitation that their PCIe host can't allow few inbound > >>> address ranges. Allowed inbound address ranges are listed in dma-ranges > >>> DT property and this address ranges are required to do IOVA mapping. > >>> Remaining address ranges have to be reserved in IOVA mapping. > >>> > >>> PCIe Host driver of those SOCs has to list resource entries of allowed > >>> address ranges given in dma-ranges DT property in sorted order. This > >>> sorted list of resources will be processed and reserve IOVA address for > >>> inaccessible address holes while initializing IOMMU domain. > >>> > >>> This patch set is based on Linux-5.0-rc2. > >>> > >>> Changes from v3: > >>> - Addressed Robin Murphy review comments. > >>> - pcie-iproc: parse dma-ranges and make sorted resource list. > >>> - dma-iommu: process list and reserve gaps between entries > >>> > >>> Changes from v2: > >>> - Patch set rebased to Linux-5.0-rc2 > >>> > >>> Changes from v1: > >>> - Addressed Oza review comments. > >>> > >>> Srinath Mannam (3): > >>> PCI: Add dma_ranges window list > >>> iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > >>> PCI: iproc: Add sorted dma ranges resource entries to host bridge > >>> > >>> drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > >>> drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > >>> drivers/pci/probe.c | 3 +++ > >>> include/linux/pci.h | 1 + > >>> 4 files changed, 66 insertions(+), 1 deletion(-) > >> > >> Bjorn, Joerg, > >> > >> this series should not affect anything in the mainline other than its > >> consumer (ie patch 3); if that's the case should we consider it for v5.2 > >> and if yes how are we going to merge it ? > > > > I acked the first one > > > > Robin reviewed the second > > (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) > > (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK > > with it if Robin was > > (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). > > > > Eric reviewed the third (and pointed out a typo). > > > > My Kconfiggery never got fully answered -- it looks to me as though it's > > possible to build pcie-iproc without the DMA hole support, and I thought > > the whole point of this series was to deal with those holes > > (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would > > have expected something like making pcie-iproc depend on IOMMU_SUPPORT. > > But Srinath didn't respond to that, so maybe it's not an issue and it > > should only affect pcie-iproc anyway. > > Hmm, I'm sure I had at least half-written a reply on that point, but I > can't seem to find it now... anyway, the gist is that these inbound > windows are generally set up to cover the physical address ranges of > DRAM and anything else that devices might need to DMA to. Thus if you're > not using an IOMMU, the fact that devices can't access the gaps in > between doesn't matter because there won't be anything there anyway; it > only needs mitigating if you do use an IOMMU and start giving arbitrary > non-physical addresses to the endpoint. > > > So bottom line, I'm fine with merging it for v5.2. Do you want to merge > > it, Lorenzo, or ...? > > This doesn't look like it will conflict with the other DMA ops and MSI > mapping changes currently in-flight for iommu-dma, so I have no > objection to it going through the PCI tree for 5.2. > > Robin.
Hi Lorenzo, Thanks a lot. Please see my reply below. On Wed, May 1, 2019 at 7:24 PM Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> wrote: > > On Wed, May 01, 2019 at 02:20:56PM +0100, Robin Murphy wrote: > > On 2019-05-01 1:55 pm, Bjorn Helgaas wrote: > > > On Wed, May 01, 2019 at 12:30:38PM +0100, Lorenzo Pieralisi wrote: > > > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > > > DT property and this address ranges are required to do IOVA mapping. > > > > > Remaining address ranges have to be reserved in IOVA mapping. > > > > > > > > > > PCIe Host driver of those SOCs has to list resource entries of allowed > > > > > address ranges given in dma-ranges DT property in sorted order. This > > > > > sorted list of resources will be processed and reserve IOVA address for > > > > > inaccessible address holes while initializing IOMMU domain. > > > > > > > > > > This patch set is based on Linux-5.0-rc2. > > > > > > > > > > Changes from v3: > > > > > - Addressed Robin Murphy review comments. > > > > > - pcie-iproc: parse dma-ranges and make sorted resource list. > > > > > - dma-iommu: process list and reserve gaps between entries > > > > > > > > > > Changes from v2: > > > > > - Patch set rebased to Linux-5.0-rc2 > > > > > > > > > > Changes from v1: > > > > > - Addressed Oza review comments. > > > > > > > > > > Srinath Mannam (3): > > > > > PCI: Add dma_ranges window list > > > > > iommu/dma: Reserve IOVA for PCIe inaccessible DMA address > > > > > PCI: iproc: Add sorted dma ranges resource entries to host bridge > > > > > > > > > > drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ > > > > > drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- > > > > > drivers/pci/probe.c | 3 +++ > > > > > include/linux/pci.h | 1 + > > > > > 4 files changed, 66 insertions(+), 1 deletion(-) > > > > > > > > Bjorn, Joerg, > > > > > > > > this series should not affect anything in the mainline other than its > > > > consumer (ie patch 3); if that's the case should we consider it for v5.2 > > > > and if yes how are we going to merge it ? > > > > > > I acked the first one > > > > > > Robin reviewed the second > > > (https://lore.kernel.org/lkml/e6c812d6-0cad-4cfd-defd-d7ec427a6538@arm.com) > > > (though I do agree with his comment about DMA_BIT_MASK()), Joerg was OK > > > with it if Robin was > > > (https://lore.kernel.org/lkml/20190423145721.GH29810@8bytes.org). > > > > > > Eric reviewed the third (and pointed out a typo). > > > > > > My Kconfiggery never got fully answered -- it looks to me as though it's > > > possible to build pcie-iproc without the DMA hole support, and I thought > > > the whole point of this series was to deal with those holes > > > (https://lore.kernel.org/lkml/20190418234241.GF126710@google.com). I would > > > have expected something like making pcie-iproc depend on IOMMU_SUPPORT. > > > But Srinath didn't respond to that, so maybe it's not an issue and it > > > should only affect pcie-iproc anyway. > > > > Hmm, I'm sure I had at least half-written a reply on that point, but I > > can't seem to find it now... anyway, the gist is that these inbound > > windows are generally set up to cover the physical address ranges of DRAM > > and anything else that devices might need to DMA to. Thus if you're not > > using an IOMMU, the fact that devices can't access the gaps in between > > doesn't matter because there won't be anything there anyway; it only > > needs mitigating if you do use an IOMMU and start giving arbitrary > > non-physical addresses to the endpoint. > > So basically there is no strict IOMMU_SUPPORT dependency. Yes, without IOMMU_SUPPORT, all inbound addresses will fall inside dma-ranges. Issue is only in the case of IOMMU enable, this patch will address by reserving non-allowed address (holes of dma-ranges) by reserving them. > > > > So bottom line, I'm fine with merging it for v5.2. Do you want to merge > > > it, Lorenzo, or ...? > > > > This doesn't look like it will conflict with the other DMA ops and MSI > > mapping changes currently in-flight for iommu-dma, so I have no > > objection to it going through the PCI tree for 5.2. > > I will update the DMA_BIT_MASK() according to your review and fix the > typo Eric pointed out and push out a branch - we shall see if we can > include it for v5.2. I will send new patches with the change DMA_BIT_MASK() and typo along with Bjorn's comment in PATCH-1. Regards, Srinath. > > Thanks, > Lorenzo
From: Srinath Mannam > Sent: 01 May 2019 16:23 ... > > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > > DT property and this address ranges are required to do IOVA mapping. > > > > Remaining address ranges have to be reserved in IOVA mapping. You probably want to fix the english in the first sentence. The sense is reversed. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Hi David, Thanks for review, I will fix in next version of this patch set. Regards, Srinath. On Thu, May 2, 2019 at 3:24 PM David Laight <David.Laight@aculab.com> wrote: > > From: Srinath Mannam > > Sent: 01 May 2019 16:23 > ... > > > > On Fri, Apr 12, 2019 at 08:43:32AM +0530, Srinath Mannam wrote: > > > > > Few SOCs have limitation that their PCIe host can't allow few inbound > > > > > address ranges. Allowed inbound address ranges are listed in dma-ranges > > > > > DT property and this address ranges are required to do IOVA mapping. > > > > > Remaining address ranges have to be reserved in IOVA mapping. > > You probably want to fix the english in the first sentence. > The sense is reversed. > > David > > - > Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK > Registration No: 1397386 (Wales)
Few SOCs have limitation that their PCIe host can't allow few inbound address ranges. Allowed inbound address ranges are listed in dma-ranges DT property and this address ranges are required to do IOVA mapping. Remaining address ranges have to be reserved in IOVA mapping. PCIe Host driver of those SOCs has to list resource entries of allowed address ranges given in dma-ranges DT property in sorted order. This sorted list of resources will be processed and reserve IOVA address for inaccessible address holes while initializing IOMMU domain. This patch set is based on Linux-5.0-rc2. Changes from v3: - Addressed Robin Murphy review comments. - pcie-iproc: parse dma-ranges and make sorted resource list. - dma-iommu: process list and reserve gaps between entries Changes from v2: - Patch set rebased to Linux-5.0-rc2 Changes from v1: - Addressed Oza review comments. Srinath Mannam (3): PCI: Add dma_ranges window list iommu/dma: Reserve IOVA for PCIe inaccessible DMA address PCI: iproc: Add sorted dma ranges resource entries to host bridge drivers/iommu/dma-iommu.c | 19 ++++++++++++++++ drivers/pci/controller/pcie-iproc.c | 44 ++++++++++++++++++++++++++++++++++++- drivers/pci/probe.c | 3 +++ include/linux/pci.h | 1 + 4 files changed, 66 insertions(+), 1 deletion(-)