[V3,2/3] PCI: rcar: Do not abort on too many inbound dma-ranges
diff mbox series

Message ID 20190809175741.7066-2-marek.vasut@gmail.com
State New
Delegated to: Lorenzo Pieralisi
Headers show
Series
  • [V3,1/3] PCI: rcar: Move the inbound index check
Related show

Commit Message

Marek Vasut Aug. 9, 2019, 5:57 p.m. UTC
From: Marek Vasut <marek.vasut+renesas@gmail.com>

In case the "dma-ranges" DT property contains either too many ranges
or the range start address is unaligned in such a way that populating
the range into the controller requires multiple entries, a situation
may occur where all ranges cannot be loaded into the controller.

Currently, the driver refuses to probe in such a situation. Relax this
behavior, load as many ranges as possible and warn if some ranges do
not fit anymore.

Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Geert Uytterhoeven <geert+renesas@glider.be>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Wolfram Sang <wsa@the-dreams.de>
Cc: linux-renesas-soc@vger.kernel.org
To: linux-pci@vger.kernel.org
---
V2: Update on top of 1/3
V3: No change
---
 drivers/pci/controller/pcie-rcar.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Comments

Simon Horman Aug. 16, 2019, 1:23 p.m. UTC | #1
On Fri, Aug 09, 2019 at 07:57:40PM +0200, marek.vasut@gmail.com wrote:
> From: Marek Vasut <marek.vasut+renesas@gmail.com>
> 
> In case the "dma-ranges" DT property contains either too many ranges
> or the range start address is unaligned in such a way that populating
> the range into the controller requires multiple entries, a situation
> may occur where all ranges cannot be loaded into the controller.
> 
> Currently, the driver refuses to probe in such a situation. Relax this
> behavior, load as many ranges as possible and warn if some ranges do
> not fit anymore.

What is the motivation for relaxing this?

> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Wolfram Sang <wsa@the-dreams.de>
> Cc: linux-renesas-soc@vger.kernel.org
> To: linux-pci@vger.kernel.org
> ---
> V2: Update on top of 1/3
> V3: No change
> ---
>  drivers/pci/controller/pcie-rcar.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
> index 56a6433eb70b..e2735005ffd3 100644
> --- a/drivers/pci/controller/pcie-rcar.c
> +++ b/drivers/pci/controller/pcie-rcar.c
> @@ -1049,8 +1049,9 @@ static int rcar_pcie_inbound_ranges(struct rcar_pcie *pcie,
>  
>  	while (cpu_addr < cpu_end) {
>  		if (idx >= MAX_NR_INBOUND_MAPS - 1) {
> -			dev_err(pcie->dev, "Failed to map inbound regions!\n");
> -			return -EINVAL;
> +			dev_warn(pcie->dev,
> +				 "Too many inbound regions, not all are mapped.\n");
> +			break;
>  		}
>  		/*
>  		 * Set up 64-bit inbound regions as the range parser doesn't
> -- 
> 2.20.1
>
Marek Vasut Aug. 16, 2019, 1:28 p.m. UTC | #2
On 8/16/19 3:23 PM, Simon Horman wrote:
> On Fri, Aug 09, 2019 at 07:57:40PM +0200, marek.vasut@gmail.com wrote:
>> From: Marek Vasut <marek.vasut+renesas@gmail.com>
>>
>> In case the "dma-ranges" DT property contains either too many ranges
>> or the range start address is unaligned in such a way that populating
>> the range into the controller requires multiple entries, a situation
>> may occur where all ranges cannot be loaded into the controller.
>>
>> Currently, the driver refuses to probe in such a situation. Relax this
>> behavior, load as many ranges as possible and warn if some ranges do
>> not fit anymore.
> 
> What is the motivation for relaxing this?

U-Boot can fill the ranges in properly now, the list would be longer in
such a case and the driver would fail to probe (because the list is
longer than what the hardware can support).
Simon Horman Aug. 16, 2019, 1:38 p.m. UTC | #3
On Fri, Aug 16, 2019 at 03:28:04PM +0200, Marek Vasut wrote:
> On 8/16/19 3:23 PM, Simon Horman wrote:
> > On Fri, Aug 09, 2019 at 07:57:40PM +0200, marek.vasut@gmail.com wrote:
> >> From: Marek Vasut <marek.vasut+renesas@gmail.com>
> >>
> >> In case the "dma-ranges" DT property contains either too many ranges
> >> or the range start address is unaligned in such a way that populating
> >> the range into the controller requires multiple entries, a situation
> >> may occur where all ranges cannot be loaded into the controller.
> >>
> >> Currently, the driver refuses to probe in such a situation. Relax this
> >> behavior, load as many ranges as possible and warn if some ranges do
> >> not fit anymore.
> > 
> > What is the motivation for relaxing this?
> 
> U-Boot can fill the ranges in properly now, the list would be longer in
> such a case and the driver would fail to probe (because the list is
> longer than what the hardware can support).

Thanks, I think that would be worth adding to the changelog.

Regardless,

Reviewed-by: Simon Horman <horms+renesas@verge.net.au>
Marek Vasut Aug. 16, 2019, 5:41 p.m. UTC | #4
On 8/16/19 3:38 PM, Simon Horman wrote:
> On Fri, Aug 16, 2019 at 03:28:04PM +0200, Marek Vasut wrote:
>> On 8/16/19 3:23 PM, Simon Horman wrote:
>>> On Fri, Aug 09, 2019 at 07:57:40PM +0200, marek.vasut@gmail.com wrote:
>>>> From: Marek Vasut <marek.vasut+renesas@gmail.com>
>>>>
>>>> In case the "dma-ranges" DT property contains either too many ranges
>>>> or the range start address is unaligned in such a way that populating
>>>> the range into the controller requires multiple entries, a situation
>>>> may occur where all ranges cannot be loaded into the controller.
>>>>
>>>> Currently, the driver refuses to probe in such a situation. Relax this
>>>> behavior, load as many ranges as possible and warn if some ranges do
>>>> not fit anymore.
>>>
>>> What is the motivation for relaxing this?
>>
>> U-Boot can fill the ranges in properly now, the list would be longer in
>> such a case and the driver would fail to probe (because the list is
>> longer than what the hardware can support).
> 
> Thanks, I think that would be worth adding to the changelog.

It does describe exactly what I just said -- if there are too many
ranges or they start in a way that cannot be easily fully programmed
into the HW, this patch applies.

> Regardless,
> 
> Reviewed-by: Simon Horman <horms+renesas@verge.net.au>
>
Lorenzo Pieralisi Oct. 16, 2019, 3 p.m. UTC | #5
On Fri, Aug 09, 2019 at 07:57:40PM +0200, marek.vasut@gmail.com wrote:
> From: Marek Vasut <marek.vasut+renesas@gmail.com>
> 
> In case the "dma-ranges" DT property contains either too many ranges
> or the range start address is unaligned in such a way that populating
> the range into the controller requires multiple entries, a situation
> may occur where all ranges cannot be loaded into the controller.
> 
> Currently, the driver refuses to probe in such a situation. Relax this
> behavior, load as many ranges as possible and warn if some ranges do
> not fit anymore.

Patches (1) and (3) are fine but I do not think this one is.

Firmware (DT) should provide dma-ranges according to what HW can handle,
more so given that other subsystems (eg IOMMU) rely on the dma-ranges
value to set-up eg DMA - if there is a mismatch between PCI host inbound
regions and software structures describing DMA'able ranges all bets are
off.

I would not merge this specific patch but let me know what you think.

Thanks,
Lorenzo

> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Geert Uytterhoeven <geert+renesas@glider.be>
> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Wolfram Sang <wsa@the-dreams.de>
> Cc: linux-renesas-soc@vger.kernel.org
> To: linux-pci@vger.kernel.org
> ---
> V2: Update on top of 1/3
> V3: No change
> ---
>  drivers/pci/controller/pcie-rcar.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
> index 56a6433eb70b..e2735005ffd3 100644
> --- a/drivers/pci/controller/pcie-rcar.c
> +++ b/drivers/pci/controller/pcie-rcar.c
> @@ -1049,8 +1049,9 @@ static int rcar_pcie_inbound_ranges(struct rcar_pcie *pcie,
>  
>  	while (cpu_addr < cpu_end) {
>  		if (idx >= MAX_NR_INBOUND_MAPS - 1) {
> -			dev_err(pcie->dev, "Failed to map inbound regions!\n");
> -			return -EINVAL;
> +			dev_warn(pcie->dev,
> +				 "Too many inbound regions, not all are mapped.\n");
> +			break;
>  		}
>  		/*
>  		 * Set up 64-bit inbound regions as the range parser doesn't
> -- 
> 2.20.1
>
Marek Vasut Oct. 16, 2019, 3:10 p.m. UTC | #6
On 10/16/19 5:00 PM, Lorenzo Pieralisi wrote:
> On Fri, Aug 09, 2019 at 07:57:40PM +0200, Marek Vasut wrote:
>> From: Marek Vasut <marek.vasut+renesas@gmail.com>
>>
>> In case the "dma-ranges" DT property contains either too many ranges
>> or the range start address is unaligned in such a way that populating
>> the range into the controller requires multiple entries, a situation
>> may occur where all ranges cannot be loaded into the controller.
>>
>> Currently, the driver refuses to probe in such a situation. Relax this
>> behavior, load as many ranges as possible and warn if some ranges do
>> not fit anymore.
> 
> Patches (1) and (3) are fine but I do not think this one is.
> 
> Firmware (DT) should provide dma-ranges according to what HW can handle,
> more so given that other subsystems (eg IOMMU) rely on the dma-ranges
> value to set-up eg DMA - if there is a mismatch between PCI host inbound
> regions and software structures describing DMA'able ranges all bets are
> off.

The firmware provides all the ranges which are available and usable,
that's the hardware description and that should be in the DT. The
firmware cannot decide the policy for the next stage (Linux in this
case) on which ranges are better to use for Linux and which are less
good. Linux can then decide which ranges are best suited for it and
ignore the other ones.
Lorenzo Pieralisi Oct. 16, 2019, 3:26 p.m. UTC | #7
On Wed, Oct 16, 2019 at 05:10:02PM +0200, Marek Vasut wrote:
> On 10/16/19 5:00 PM, Lorenzo Pieralisi wrote:
> > On Fri, Aug 09, 2019 at 07:57:40PM +0200, Marek Vasut wrote:
> >> From: Marek Vasut <marek.vasut+renesas@gmail.com>
> >>
> >> In case the "dma-ranges" DT property contains either too many ranges
> >> or the range start address is unaligned in such a way that populating
> >> the range into the controller requires multiple entries, a situation
> >> may occur where all ranges cannot be loaded into the controller.
> >>
> >> Currently, the driver refuses to probe in such a situation. Relax this
> >> behavior, load as many ranges as possible and warn if some ranges do
> >> not fit anymore.
> > 
> > Patches (1) and (3) are fine but I do not think this one is.
> > 
> > Firmware (DT) should provide dma-ranges according to what HW can handle,
> > more so given that other subsystems (eg IOMMU) rely on the dma-ranges
> > value to set-up eg DMA - if there is a mismatch between PCI host inbound
> > regions and software structures describing DMA'able ranges all bets are
> > off.
> 
> The firmware provides all the ranges which are available and usable,
> that's the hardware description and that should be in the DT.

If the HW (given that those dma-ranges are declared for the PCI host
controller) can't be programmed to enable those DMA ranges - those
ranges are neither available nor usable, ergo DT is broken.

> The firmware cannot decide the policy for the next stage (Linux in
> this case) on which ranges are better to use for Linux and which are
> less good. Linux can then decide which ranges are best suited for it
> and ignore the other ones.

dma-ranges is a property that is used by other kernel subsystems eg
IOMMU other than the RCAR host controller driver. The policy, provided
there is one should be shared across them. You can't leave a PCI
host controller half-programmed and expect other subsystems (that
*expect* those ranges to be DMA'ble) to work.

I reiterate my point: if firmware is broken it is better to fail
the probe rather than limp on hoping that things will keep on
working.

Lorenzo
Marek Vasut Oct. 16, 2019, 3:29 p.m. UTC | #8
On 10/16/19 5:26 PM, Lorenzo Pieralisi wrote:
> On Wed, Oct 16, 2019 at 05:10:02PM +0200, Marek Vasut wrote:
>> On 10/16/19 5:00 PM, Lorenzo Pieralisi wrote:
>>> On Fri, Aug 09, 2019 at 07:57:40PM +0200, Marek Vasut wrote:
>>>> From: Marek Vasut <marek.vasut+renesas@gmail.com>
>>>>
>>>> In case the "dma-ranges" DT property contains either too many ranges
>>>> or the range start address is unaligned in such a way that populating
>>>> the range into the controller requires multiple entries, a situation
>>>> may occur where all ranges cannot be loaded into the controller.
>>>>
>>>> Currently, the driver refuses to probe in such a situation. Relax this
>>>> behavior, load as many ranges as possible and warn if some ranges do
>>>> not fit anymore.
>>>
>>> Patches (1) and (3) are fine but I do not think this one is.
>>>
>>> Firmware (DT) should provide dma-ranges according to what HW can handle,
>>> more so given that other subsystems (eg IOMMU) rely on the dma-ranges
>>> value to set-up eg DMA - if there is a mismatch between PCI host inbound
>>> regions and software structures describing DMA'able ranges all bets are
>>> off.
>>
>> The firmware provides all the ranges which are available and usable,
>> that's the hardware description and that should be in the DT.
> 
> If the HW (given that those dma-ranges are declared for the PCI host
> controller) can't be programmed to enable those DMA ranges - those
> ranges are neither available nor usable, ergo DT is broken.

The hardware can be programmed to enable those DMA ranges, just not all
of them at the same time.

It's not the job of the bootloader to guess which ranges might the next
stage like best.

>> The firmware cannot decide the policy for the next stage (Linux in
>> this case) on which ranges are better to use for Linux and which are
>> less good. Linux can then decide which ranges are best suited for it
>> and ignore the other ones.
> 
> dma-ranges is a property that is used by other kernel subsystems eg
> IOMMU other than the RCAR host controller driver. The policy, provided
> there is one should be shared across them. You can't leave a PCI
> host controller half-programmed and expect other subsystems (that
> *expect* those ranges to be DMA'ble) to work.
> 
> I reiterate my point: if firmware is broken it is better to fail
> the probe rather than limp on hoping that things will keep on
> working.

But the firmware is not broken ?
Lorenzo Pieralisi Oct. 16, 2019, 4:18 p.m. UTC | #9
[+RobH, Robin]

On Wed, Oct 16, 2019 at 05:29:50PM +0200, Marek Vasut wrote:

[...]

> >> The firmware provides all the ranges which are available and usable,
> >> that's the hardware description and that should be in the DT.
> > 
> > If the HW (given that those dma-ranges are declared for the PCI host
> > controller) can't be programmed to enable those DMA ranges - those
> > ranges are neither available nor usable, ergo DT is broken.
> 
> The hardware can be programmed to enable those DMA ranges, just not all
> of them at the same time.

Ok, we are down to DT bindings interpretation then.

> It's not the job of the bootloader to guess which ranges might the next
> stage like best.

By the time this series:

https://patchwork.ozlabs.org/user/todo/linux-pci/?series=132419

is merged, your policy will require the host controller driver to
remove the DMA ranges that could not be programmed in the inbound
address decoders from the dma_ranges list, otherwise things will
fall apart.

> >> The firmware cannot decide the policy for the next stage (Linux in
> >> this case) on which ranges are better to use for Linux and which are
> >> less good. Linux can then decide which ranges are best suited for it
> >> and ignore the other ones.
> > 
> > dma-ranges is a property that is used by other kernel subsystems eg
> > IOMMU other than the RCAR host controller driver. The policy, provided
> > there is one should be shared across them. You can't leave a PCI
> > host controller half-programmed and expect other subsystems (that
> > *expect* those ranges to be DMA'ble) to work.
> > 
> > I reiterate my point: if firmware is broken it is better to fail
> > the probe rather than limp on hoping that things will keep on
> > working.
> 
> But the firmware is not broken ?

See above, it depends on how the dma-ranges property is interpreted,
hopefully we can reach consensus in this thread, I won't merge a patch
that can backfire later unless we all agree that what it does is
correct.

Lorenzo
Rob Herring Oct. 16, 2019, 6:12 p.m. UTC | #10
On Wed, Oct 16, 2019 at 11:18 AM Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
>
> [+RobH, Robin]
>
> On Wed, Oct 16, 2019 at 05:29:50PM +0200, Marek Vasut wrote:
>
> [...]
>
> > >> The firmware provides all the ranges which are available and usable,
> > >> that's the hardware description and that should be in the DT.
> > >
> > > If the HW (given that those dma-ranges are declared for the PCI host
> > > controller) can't be programmed to enable those DMA ranges - those
> > > ranges are neither available nor usable, ergo DT is broken.
> >
> > The hardware can be programmed to enable those DMA ranges, just not all
> > of them at the same time.
>
> Ok, we are down to DT bindings interpretation then.
>
> > It's not the job of the bootloader to guess which ranges might the next
> > stage like best.
>
> By the time this series:
>
> https://patchwork.ozlabs.org/user/todo/linux-pci/?series=132419
>
> is merged, your policy will require the host controller driver to
> remove the DMA ranges that could not be programmed in the inbound
> address decoders from the dma_ranges list, otherwise things will
> fall apart.

I don't think the above series has too much impact on this. It's my
other series dealing with dma masks that's relevant because for dma
masks we only ever look at the first dma-ranges entry. We either have
to support multiple addresses and sizes per device (the only way to
really support any possible dma-ranges), merge entries to single
offset/mask or have some way to select which range entry to use.

So things are broken to some extent regardless unless MAX_NR_INBOUND_MAPS == 1.

> > >> The firmware cannot decide the policy for the next stage (Linux in
> > >> this case) on which ranges are better to use for Linux and which are
> > >> less good. Linux can then decide which ranges are best suited for it
> > >> and ignore the other ones.
> > >
> > > dma-ranges is a property that is used by other kernel subsystems eg
> > > IOMMU other than the RCAR host controller driver. The policy, provided
> > > there is one should be shared across them. You can't leave a PCI
> > > host controller half-programmed and expect other subsystems (that
> > > *expect* those ranges to be DMA'ble) to work.
> > >
> > > I reiterate my point: if firmware is broken it is better to fail
> > > the probe rather than limp on hoping that things will keep on
> > > working.
> >
> > But the firmware is not broken ?
>
> See above, it depends on how the dma-ranges property is interpreted,
> hopefully we can reach consensus in this thread, I won't merge a patch
> that can backfire later unless we all agree that what it does is
> correct.

Defining more dma-ranges entries than the h/w has inbound windows for
sounds like a broken DT to me.

What exactly does dma-ranges contain in this case? I'm not really
visualizing how different clients would pick different dma-ranges
entries.

Rob
Marek Vasut Oct. 16, 2019, 6:17 p.m. UTC | #11
On 10/16/19 8:12 PM, Rob Herring wrote:
> On Wed, Oct 16, 2019 at 11:18 AM Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
>>
>> [+RobH, Robin]
>>
>> On Wed, Oct 16, 2019 at 05:29:50PM +0200, Marek Vasut wrote:
>>
>> [...]
>>
>>>>> The firmware provides all the ranges which are available and usable,
>>>>> that's the hardware description and that should be in the DT.
>>>>
>>>> If the HW (given that those dma-ranges are declared for the PCI host
>>>> controller) can't be programmed to enable those DMA ranges - those
>>>> ranges are neither available nor usable, ergo DT is broken.
>>>
>>> The hardware can be programmed to enable those DMA ranges, just not all
>>> of them at the same time.
>>
>> Ok, we are down to DT bindings interpretation then.
>>
>>> It's not the job of the bootloader to guess which ranges might the next
>>> stage like best.
>>
>> By the time this series:
>>
>> https://patchwork.ozlabs.org/user/todo/linux-pci/?series=132419
>>
>> is merged, your policy will require the host controller driver to
>> remove the DMA ranges that could not be programmed in the inbound
>> address decoders from the dma_ranges list, otherwise things will
>> fall apart.
> 
> I don't think the above series has too much impact on this. It's my
> other series dealing with dma masks that's relevant because for dma
> masks we only ever look at the first dma-ranges entry. We either have
> to support multiple addresses and sizes per device (the only way to
> really support any possible dma-ranges), merge entries to single
> offset/mask or have some way to select which range entry to use.
> 
> So things are broken to some extent regardless unless MAX_NR_INBOUND_MAPS == 1.
> 
>>>>> The firmware cannot decide the policy for the next stage (Linux in
>>>>> this case) on which ranges are better to use for Linux and which are
>>>>> less good. Linux can then decide which ranges are best suited for it
>>>>> and ignore the other ones.
>>>>
>>>> dma-ranges is a property that is used by other kernel subsystems eg
>>>> IOMMU other than the RCAR host controller driver. The policy, provided
>>>> there is one should be shared across them. You can't leave a PCI
>>>> host controller half-programmed and expect other subsystems (that
>>>> *expect* those ranges to be DMA'ble) to work.
>>>>
>>>> I reiterate my point: if firmware is broken it is better to fail
>>>> the probe rather than limp on hoping that things will keep on
>>>> working.
>>>
>>> But the firmware is not broken ?
>>
>> See above, it depends on how the dma-ranges property is interpreted,
>> hopefully we can reach consensus in this thread, I won't merge a patch
>> that can backfire later unless we all agree that what it does is
>> correct.
> 
> Defining more dma-ranges entries than the h/w has inbound windows for
> sounds like a broken DT to me.
> 
> What exactly does dma-ranges contain in this case? I'm not really
> visualizing how different clients would pick different dma-ranges
> entries.

You can have multiple non-continuous DRAM banks for example. And an
entry for SRAM optionally. Each DRAM bank and/or the SRAM should have a
separate dma-ranges entry, right ?
Rob Herring Oct. 16, 2019, 8:25 p.m. UTC | #12
On Wed, Oct 16, 2019 at 1:18 PM Marek Vasut <marek.vasut@gmail.com> wrote:
>
> On 10/16/19 8:12 PM, Rob Herring wrote:
> > On Wed, Oct 16, 2019 at 11:18 AM Lorenzo Pieralisi
> > <lorenzo.pieralisi@arm.com> wrote:
> >>
> >> [+RobH, Robin]
> >>
> >> On Wed, Oct 16, 2019 at 05:29:50PM +0200, Marek Vasut wrote:
> >>
> >> [...]
> >>
> >>>>> The firmware provides all the ranges which are available and usable,
> >>>>> that's the hardware description and that should be in the DT.
> >>>>
> >>>> If the HW (given that those dma-ranges are declared for the PCI host
> >>>> controller) can't be programmed to enable those DMA ranges - those
> >>>> ranges are neither available nor usable, ergo DT is broken.
> >>>
> >>> The hardware can be programmed to enable those DMA ranges, just not all
> >>> of them at the same time.
> >>
> >> Ok, we are down to DT bindings interpretation then.
> >>
> >>> It's not the job of the bootloader to guess which ranges might the next
> >>> stage like best.
> >>
> >> By the time this series:
> >>
> >> https://patchwork.ozlabs.org/user/todo/linux-pci/?series=132419
> >>
> >> is merged, your policy will require the host controller driver to
> >> remove the DMA ranges that could not be programmed in the inbound
> >> address decoders from the dma_ranges list, otherwise things will
> >> fall apart.
> >
> > I don't think the above series has too much impact on this. It's my
> > other series dealing with dma masks that's relevant because for dma
> > masks we only ever look at the first dma-ranges entry. We either have
> > to support multiple addresses and sizes per device (the only way to
> > really support any possible dma-ranges), merge entries to single
> > offset/mask or have some way to select which range entry to use.
> >
> > So things are broken to some extent regardless unless MAX_NR_INBOUND_MAPS == 1.
> >
> >>>>> The firmware cannot decide the policy for the next stage (Linux in
> >>>>> this case) on which ranges are better to use for Linux and which are
> >>>>> less good. Linux can then decide which ranges are best suited for it
> >>>>> and ignore the other ones.
> >>>>
> >>>> dma-ranges is a property that is used by other kernel subsystems eg
> >>>> IOMMU other than the RCAR host controller driver. The policy, provided
> >>>> there is one should be shared across them. You can't leave a PCI
> >>>> host controller half-programmed and expect other subsystems (that
> >>>> *expect* those ranges to be DMA'ble) to work.
> >>>>
> >>>> I reiterate my point: if firmware is broken it is better to fail
> >>>> the probe rather than limp on hoping that things will keep on
> >>>> working.
> >>>
> >>> But the firmware is not broken ?
> >>
> >> See above, it depends on how the dma-ranges property is interpreted,
> >> hopefully we can reach consensus in this thread, I won't merge a patch
> >> that can backfire later unless we all agree that what it does is
> >> correct.
> >
> > Defining more dma-ranges entries than the h/w has inbound windows for
> > sounds like a broken DT to me.
> >
> > What exactly does dma-ranges contain in this case? I'm not really
> > visualizing how different clients would pick different dma-ranges
> > entries.
>
> You can have multiple non-continuous DRAM banks for example. And an
> entry for SRAM optionally. Each DRAM bank and/or the SRAM should have a
> separate dma-ranges entry, right ?

Not necessarily. We really only want to define the minimum we have to.
The ideal system is no dma-ranges. Is each bank at a different
relative position compared to the CPU's view of the system. That would
seem doubtful for just DRAM banks. Perhaps DRAM and SRAM could change.

I suppose if your intent is to use inbound windows as a poor man's
IOMMU to prevent accesses to the holes, then yes you would list them
out. But I think that's wrong and difficult to maintain. You'd also
need to deal with reserved-memory regions too.

Rob
Marek Vasut Oct. 16, 2019, 9:15 p.m. UTC | #13
On 10/16/19 10:25 PM, Rob Herring wrote:
[...]
>>>>>>> The firmware cannot decide the policy for the next stage (Linux in
>>>>>>> this case) on which ranges are better to use for Linux and which are
>>>>>>> less good. Linux can then decide which ranges are best suited for it
>>>>>>> and ignore the other ones.
>>>>>>
>>>>>> dma-ranges is a property that is used by other kernel subsystems eg
>>>>>> IOMMU other than the RCAR host controller driver. The policy, provided
>>>>>> there is one should be shared across them. You can't leave a PCI
>>>>>> host controller half-programmed and expect other subsystems (that
>>>>>> *expect* those ranges to be DMA'ble) to work.
>>>>>>
>>>>>> I reiterate my point: if firmware is broken it is better to fail
>>>>>> the probe rather than limp on hoping that things will keep on
>>>>>> working.
>>>>>
>>>>> But the firmware is not broken ?
>>>>
>>>> See above, it depends on how the dma-ranges property is interpreted,
>>>> hopefully we can reach consensus in this thread, I won't merge a patch
>>>> that can backfire later unless we all agree that what it does is
>>>> correct.
>>>
>>> Defining more dma-ranges entries than the h/w has inbound windows for
>>> sounds like a broken DT to me.
>>>
>>> What exactly does dma-ranges contain in this case? I'm not really
>>> visualizing how different clients would pick different dma-ranges
>>> entries.
>>
>> You can have multiple non-continuous DRAM banks for example. And an
>> entry for SRAM optionally. Each DRAM bank and/or the SRAM should have a
>> separate dma-ranges entry, right ?
> 
> Not necessarily. We really only want to define the minimum we have to.
> The ideal system is no dma-ranges. Is each bank at a different
> relative position compared to the CPU's view of the system. That would
> seem doubtful for just DRAM banks. Perhaps DRAM and SRAM could change.

Is that a question ? Anyway, yes, there is a bit of DRAM below the 32bit
boundary and some more above the 32bit boundary. These two banks don't
need to be continuous. And then you could add the SRAM into the mix.

> I suppose if your intent is to use inbound windows as a poor man's
> IOMMU to prevent accesses to the holes, then yes you would list them
> out. But I think that's wrong and difficult to maintain. You'd also
> need to deal with reserved-memory regions too.

What's the problem with that? The bootloader has all that information
and can patch the DT correctly. In fact, in my specific case, I have
platform which can be populated with differently sized DRAM, so the
holes are also dynamically calculated ; there is no one DT then, the
bootloader is responsible to generate the dma-ranges accordingly.
Rob Herring Oct. 16, 2019, 10:26 p.m. UTC | #14
On Wed, Oct 16, 2019 at 4:16 PM Marek Vasut <marek.vasut@gmail.com> wrote:
>
> On 10/16/19 10:25 PM, Rob Herring wrote:
> [...]
> >>>>>>> The firmware cannot decide the policy for the next stage (Linux in
> >>>>>>> this case) on which ranges are better to use for Linux and which are
> >>>>>>> less good. Linux can then decide which ranges are best suited for it
> >>>>>>> and ignore the other ones.
> >>>>>>
> >>>>>> dma-ranges is a property that is used by other kernel subsystems eg
> >>>>>> IOMMU other than the RCAR host controller driver. The policy, provided
> >>>>>> there is one should be shared across them. You can't leave a PCI
> >>>>>> host controller half-programmed and expect other subsystems (that
> >>>>>> *expect* those ranges to be DMA'ble) to work.
> >>>>>>
> >>>>>> I reiterate my point: if firmware is broken it is better to fail
> >>>>>> the probe rather than limp on hoping that things will keep on
> >>>>>> working.
> >>>>>
> >>>>> But the firmware is not broken ?
> >>>>
> >>>> See above, it depends on how the dma-ranges property is interpreted,
> >>>> hopefully we can reach consensus in this thread, I won't merge a patch
> >>>> that can backfire later unless we all agree that what it does is
> >>>> correct.
> >>>
> >>> Defining more dma-ranges entries than the h/w has inbound windows for
> >>> sounds like a broken DT to me.
> >>>
> >>> What exactly does dma-ranges contain in this case? I'm not really
> >>> visualizing how different clients would pick different dma-ranges
> >>> entries.
> >>
> >> You can have multiple non-continuous DRAM banks for example. And an
> >> entry for SRAM optionally. Each DRAM bank and/or the SRAM should have a
> >> separate dma-ranges entry, right ?
> >
> > Not necessarily. We really only want to define the minimum we have to.
> > The ideal system is no dma-ranges. Is each bank at a different
> > relative position compared to the CPU's view of the system. That would
> > seem doubtful for just DRAM banks. Perhaps DRAM and SRAM could change.
>
> Is that a question ? Anyway, yes, there is a bit of DRAM below the 32bit
> boundary and some more above the 32bit boundary. These two banks don't
> need to be continuous. And then you could add the SRAM into the mix.

Continuous is irrelevant. My question was in more specific terms is
(bank1 addr - bank0 addr) different for CPU's view (i.e phys addr) vs.
PCI host view (i.e. bus addr)? If not, then that is 1 translation and
1 dma-ranges entry.

> > I suppose if your intent is to use inbound windows as a poor man's
> > IOMMU to prevent accesses to the holes, then yes you would list them
> > out. But I think that's wrong and difficult to maintain. You'd also
> > need to deal with reserved-memory regions too.
>
> What's the problem with that? The bootloader has all that information
> and can patch the DT correctly. In fact, in my specific case, I have
> platform which can be populated with differently sized DRAM, so the
> holes are also dynamically calculated ; there is no one DT then, the
> bootloader is responsible to generate the dma-ranges accordingly.

The problems are it doesn't work:

Your dma-mask and offset are not going to be correct.

You are running out of inbound windows. Your patch does nothing to
solve that. The solution would be merging multiple dma-ranges entries
to a single inbound window. We'd have to do that both for dma-mask and
inbound windows. The former would also have to figure out which
entries apply to setting up dma-mask. I'm simply suggesting just do
that up front and avoid any pointless splits.

You are setting up random inbound windows. The bootloader can't assume
what order the OS parses dma-ranges, and the OS can't assume what
order the bootloader writes the entries.

Rob
Marek Vasut Oct. 16, 2019, 10:33 p.m. UTC | #15
On 10/17/19 12:26 AM, Rob Herring wrote:
[...]
>>>> You can have multiple non-continuous DRAM banks for example. And an
>>>> entry for SRAM optionally. Each DRAM bank and/or the SRAM should have a
>>>> separate dma-ranges entry, right ?
>>>
>>> Not necessarily. We really only want to define the minimum we have to.
>>> The ideal system is no dma-ranges. Is each bank at a different
>>> relative position compared to the CPU's view of the system. That would
>>> seem doubtful for just DRAM banks. Perhaps DRAM and SRAM could change.
>>
>> Is that a question ? Anyway, yes, there is a bit of DRAM below the 32bit
>> boundary and some more above the 32bit boundary. These two banks don't
>> need to be continuous. And then you could add the SRAM into the mix.
> 
> Continuous is irrelevant. My question was in more specific terms is
> (bank1 addr - bank0 addr) different for CPU's view (i.e phys addr) vs.
> PCI host view (i.e. bus addr)? If not, then that is 1 translation and
> 1 dma-ranges entry.

I don't think it's different in that aspect. Except the bus has this
32bit limitation, where it only sees subset of the DRAM.

Why should the DMA ranges incorrectly cover also the DRAM which is not
present ?

>>> I suppose if your intent is to use inbound windows as a poor man's
>>> IOMMU to prevent accesses to the holes, then yes you would list them
>>> out. But I think that's wrong and difficult to maintain. You'd also
>>> need to deal with reserved-memory regions too.
>>
>> What's the problem with that? The bootloader has all that information
>> and can patch the DT correctly. In fact, in my specific case, I have
>> platform which can be populated with differently sized DRAM, so the
>> holes are also dynamically calculated ; there is no one DT then, the
>> bootloader is responsible to generate the dma-ranges accordingly.
> 
> The problems are it doesn't work:
> 
> Your dma-mask and offset are not going to be correct.
> 
> You are running out of inbound windows. Your patch does nothing to
> solve that. The solution would be merging multiple dma-ranges entries
> to a single inbound window. We'd have to do that both for dma-mask and
> inbound windows. The former would also have to figure out which
> entries apply to setting up dma-mask. I'm simply suggesting just do
> that up front and avoid any pointless splits.

But then the PCI device can trigger a transaction to non-existent DRAM
and cause undefined behavior. Surely we do not want that ?

> You are setting up random inbound windows. The bootloader can't assume
> what order the OS parses dma-ranges, and the OS can't assume what
> order the bootloader writes the entries.

But the OS can assume the ranges are correct and cover only valid
memory, right ? That is, memory into which the PCI controller can safely
access.

Patch
diff mbox series

diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
index 56a6433eb70b..e2735005ffd3 100644
--- a/drivers/pci/controller/pcie-rcar.c
+++ b/drivers/pci/controller/pcie-rcar.c
@@ -1049,8 +1049,9 @@  static int rcar_pcie_inbound_ranges(struct rcar_pcie *pcie,
 
 	while (cpu_addr < cpu_end) {
 		if (idx >= MAX_NR_INBOUND_MAPS - 1) {
-			dev_err(pcie->dev, "Failed to map inbound regions!\n");
-			return -EINVAL;
+			dev_warn(pcie->dev,
+				 "Too many inbound regions, not all are mapped.\n");
+			break;
 		}
 		/*
 		 * Set up 64-bit inbound regions as the range parser doesn't