[RFC] PCI: rcar: Add bus notifier so we can limit the DMA range

Message ID 20180521220514.30256-1-marek.vasut+renesas@gmail.com
State New
Delegated to: Lorenzo Pieralisi
Headers show
Series
  • [RFC] PCI: rcar: Add bus notifier so we can limit the DMA range
Related show

Commit Message

Marek Vasut May 21, 2018, 10:05 p.m.
From: Phil Edworthy <phil.edworthy@renesas.com>

The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
so limit the DMA range to 32bit.

Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Geert Uytterhoeven <geert+renesas@glider.be>
Cc: Phil Edworthy <phil.edworthy@renesas.com>
Cc: Simon Horman <horms+renesas@verge.net.au>
Cc: Wolfram Sang <wsa@the-dreams.de>
Cc: linux-renesas-soc@vger.kernel.org
To: linux-pci@vger.kernel.org
---
NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
      discussion seems to have gone way off, so I'm sending this as a
      RFC. Any feedback on how to do this limiting properly would be nice.
---
 drivers/pci/host/pcie-rcar.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

Comments

Arnd Bergmann May 22, 2018, 8:10 a.m. | #1
On Tue, May 22, 2018 at 12:05 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> From: Phil Edworthy <phil.edworthy@renesas.com>
>
> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
> so limit the DMA range to 32bit.
>
> Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
> Cc: Phil Edworthy <phil.edworthy@renesas.com>
> Cc: Simon Horman <horms+renesas@verge.net.au>
> Cc: Wolfram Sang <wsa@the-dreams.de>
> Cc: linux-renesas-soc@vger.kernel.org
> To: linux-pci@vger.kernel.org
> ---
> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
>       discussion seems to have gone way off, so I'm sending this as a
>       RFC. Any feedback on how to do this limiting properly would be nice.

Doing it in the driver is clearly not appropriate, we must do this in
common code. If I remember correctly, it's specifically ARM64 that is
broken here, it incorrectly allows setting a DMA mask to 64 bit
when that is not available.

      Arnd
Marek Vasut May 22, 2018, 9:52 a.m. | #2
On 05/22/2018 10:10 AM, Arnd Bergmann wrote:
> On Tue, May 22, 2018 at 12:05 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
>> From: Phil Edworthy <phil.edworthy@renesas.com>
>>
>> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
>> so limit the DMA range to 32bit.
>>
>> Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
>> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
>> Cc: Arnd Bergmann <arnd@arndb.de>
>> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
>> Cc: Phil Edworthy <phil.edworthy@renesas.com>
>> Cc: Simon Horman <horms+renesas@verge.net.au>
>> Cc: Wolfram Sang <wsa@the-dreams.de>
>> Cc: linux-renesas-soc@vger.kernel.org
>> To: linux-pci@vger.kernel.org
>> ---
>> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
>>       discussion seems to have gone way off, so I'm sending this as a
>>       RFC. Any feedback on how to do this limiting properly would be nice.
> 
> Doing it in the driver is clearly not appropriate, we must do this in
> common code. If I remember correctly, it's specifically ARM64 that is
> broken here, it incorrectly allows setting a DMA mask to 64 bit
> when that is not available.

Yep, that's correct. ARM64 with devices mapping tremendous amounts of
memory. So did anything change since that discussion references in the NOTE?
Marek Vasut June 4, 2018, 9:09 p.m. | #3
On 05/22/2018 11:52 AM, Marek Vasut wrote:
> On 05/22/2018 10:10 AM, Arnd Bergmann wrote:
>> On Tue, May 22, 2018 at 12:05 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
>>> From: Phil Edworthy <phil.edworthy@renesas.com>
>>>
>>> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
>>> so limit the DMA range to 32bit.
>>>
>>> Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
>>> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
>>> Cc: Arnd Bergmann <arnd@arndb.de>
>>> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
>>> Cc: Phil Edworthy <phil.edworthy@renesas.com>
>>> Cc: Simon Horman <horms+renesas@verge.net.au>
>>> Cc: Wolfram Sang <wsa@the-dreams.de>
>>> Cc: linux-renesas-soc@vger.kernel.org
>>> To: linux-pci@vger.kernel.org
>>> ---
>>> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
>>>       discussion seems to have gone way off, so I'm sending this as a
>>>       RFC. Any feedback on how to do this limiting properly would be nice.
>>
>> Doing it in the driver is clearly not appropriate, we must do this in
>> common code. If I remember correctly, it's specifically ARM64 that is
>> broken here, it incorrectly allows setting a DMA mask to 64 bit
>> when that is not available.
> 
> Yep, that's correct. ARM64 with devices mapping tremendous amounts of
> memory. So did anything change since that discussion references in the NOTE?

Bump ? I think it'd be nice to sort this out once and for all, since I
doubt it's just this controller that has such a limitation.
Bjorn Helgaas June 4, 2018, 10:30 p.m. | #4
[+cc ARM64 folks, linux-kernel]

The original patch under discussion is:
https://lkml.kernel.org/r/20180521220514.30256-1-marek.vasut+renesas@gmail.com

On Tue, May 22, 2018 at 11:52:21AM +0200, Marek Vasut wrote:
> On 05/22/2018 10:10 AM, Arnd Bergmann wrote:
> > On Tue, May 22, 2018 at 12:05 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> >> From: Phil Edworthy <phil.edworthy@renesas.com>
> >>
> >> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
> >> so limit the DMA range to 32bit.
> >>
> >> Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
> >> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> >> Cc: Arnd Bergmann <arnd@arndb.de>
> >> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
> >> Cc: Phil Edworthy <phil.edworthy@renesas.com>
> >> Cc: Simon Horman <horms+renesas@verge.net.au>
> >> Cc: Wolfram Sang <wsa@the-dreams.de>
> >> Cc: linux-renesas-soc@vger.kernel.org
> >> To: linux-pci@vger.kernel.org
> >> ---
> >> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
> >>       discussion seems to have gone way off, so I'm sending this as a
> >>       RFC. Any feedback on how to do this limiting properly would be nice.
> > 
> > Doing it in the driver is clearly not appropriate, we must do this in
> > common code. If I remember correctly, it's specifically ARM64 that is
> > broken here, it incorrectly allows setting a DMA mask to 64 bit
> > when that is not available.
> 
> Yep, that's correct. ARM64 with devices mapping tremendous amounts of
> memory. So did anything change since that discussion references in the NOTE?
> 
> -- 
> Best regards,
> Marek Vasut
Will Deacon June 6, 2018, 3:45 p.m. | #5
On Mon, Jun 04, 2018 at 05:30:48PM -0500, Bjorn Helgaas wrote:
> [+cc ARM64 folks, linux-kernel]
> 
> The original patch under discussion is:
> https://lkml.kernel.org/r/20180521220514.30256-1-marek.vasut+renesas@gmail.com
> 
> On Tue, May 22, 2018 at 11:52:21AM +0200, Marek Vasut wrote:
> > On 05/22/2018 10:10 AM, Arnd Bergmann wrote:
> > > On Tue, May 22, 2018 at 12:05 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> > >> From: Phil Edworthy <phil.edworthy@renesas.com>
> > >>
> > >> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus,
> > >> so limit the DMA range to 32bit.
> > >>
> > >> Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
> > >> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> > >> Cc: Arnd Bergmann <arnd@arndb.de>
> > >> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
> > >> Cc: Phil Edworthy <phil.edworthy@renesas.com>
> > >> Cc: Simon Horman <horms+renesas@verge.net.au>
> > >> Cc: Wolfram Sang <wsa@the-dreams.de>
> > >> Cc: linux-renesas-soc@vger.kernel.org
> > >> To: linux-pci@vger.kernel.org
> > >> ---
> > >> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the
> > >>       discussion seems to have gone way off, so I'm sending this as a
> > >>       RFC. Any feedback on how to do this limiting properly would be nice.
> > > 
> > > Doing it in the driver is clearly not appropriate, we must do this in
> > > common code. If I remember correctly, it's specifically ARM64 that is
> > > broken here, it incorrectly allows setting a DMA mask to 64 bit
> > > when that is not available.
> > 
> > Yep, that's correct. ARM64 with devices mapping tremendous amounts of
> > memory. So did anything change since that discussion references in the NOTE?

Is it specifically arm64 that's broken here? If so, why and how do other
archs (e.g. riscv) handle this? I thought all of this behaviour was driven
by the DMA ops, and we're just using generic code for that afaict.

Will

Patch

diff --git a/drivers/pci/host/pcie-rcar.c b/drivers/pci/host/pcie-rcar.c
index c3eab0b95290..db2b16f40bc1 100644
--- a/drivers/pci/host/pcie-rcar.c
+++ b/drivers/pci/host/pcie-rcar.c
@@ -1325,3 +1325,31 @@  static struct platform_driver rcar_pcie_driver = {
 	.probe = rcar_pcie_probe,
 };
 builtin_platform_driver(rcar_pcie_driver);
+
+static int rcar_pcie_pci_notifier(struct notifier_block *nb,
+			    unsigned long action, void *data)
+{
+	struct device *dev = data;
+
+	switch (action) {
+	case BUS_NOTIFY_BOUND_DRIVER:
+		/* Force the DMA mask to lower 32-bits */
+		dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+		break;
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block device_nb = {
+	.notifier_call = rcar_pcie_pci_notifier,
+};
+
+static int __init register_rcar_pcie_pci_notifier(void)
+{
+	return bus_register_notifier(&pci_bus_type, &device_nb);
+}
+
+arch_initcall(register_rcar_pcie_pci_notifier);