Patchwork katmai.dts: extend DMA ranges; add dma/sysace nodes

login
register
mail settings
Submitter Yuri Tikhonov
Date Nov. 13, 2008, 8:49 a.m.
Message ID <200811131149.14715.yur@emcraft.com>
Download mbox | patch
Permalink /patch/8544/
State Changes Requested, archived
Delegated to: Josh Boyer
Headers show

Comments

Yuri Tikhonov - Nov. 13, 2008, 8:49 a.m.
Hello,

This patch extends DMA ranges for PCI(X) to 4GB, so that it could
work on Katmais with 4GB RAM installed.

Add new nodes for the PPC440SPe DMA, XOR engines to
be used in the PPC440SPe ADMA driver, and the SysACE
controller, which connects Compact Flash to Katmai.

Signed-off-by: Ilya Yanok <yanok@emcraft.com>
Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
---
 arch/powerpc/boot/dts/katmai.dts |   46 +++++++++++++++++++++++++++++++------
 1 files changed, 38 insertions(+), 8 deletions(-)
Benjamin Herrenschmidt - Nov. 13, 2008, 9:25 a.m.
On Thu, 2008-11-13 at 11:49 +0300, Yuri Tikhonov wrote:
> Hello,
> 
> This patch extends DMA ranges for PCI(X) to 4GB, so that it could
> work on Katmais with 4GB RAM installed.

And where do you put MMIO ?

The 32 bit part of the PCI space need to be split between MMIO and DMA. 

Ideally, for 64-bit capable DMA, device, we should create a second DMA
range mapping the whole memory at something like 1T or so on PCI, and
keep a smaller (ie. 2G) DMA range for 32-bit only devices backed with
swiotlb.

Cheers,
Ben.
Josh Boyer - Nov. 13, 2008, 11:45 a.m.
On Thu, 13 Nov 2008 11:49:14 +0300
Yuri Tikhonov <yur@emcraft.com> wrote:

> Hello,
> 
> This patch extends DMA ranges for PCI(X) to 4GB, so that it could
> work on Katmais with 4GB RAM installed.
> 
> Add new nodes for the PPC440SPe DMA, XOR engines to
> be used in the PPC440SPe ADMA driver, and the SysACE
> controller, which connects Compact Flash to Katmai.
> 
> Signed-off-by: Ilya Yanok <yanok@emcraft.com>
> Signed-off-by: Yuri Tikhonov <yur@emcraft.com>
> ---
> +		DMA0: dma0 {
> +			interrupt-parent = <&DMA0>;
> +			interrupts = <0 1>;
> +			#interrupt-cells = <1>;
> +			#address-cells = <0>;
> +			#size-cells = <0>;
> +			interrupt-map = <
> +				0 &UIC0 0x14 4
> +				1 &UIC1 0x16 4>;
> +		};
> +		DMA1: dma1 {
> +			interrupt-parent = <&DMA1>;
> +			interrupts = <0 1>;
> +			#interrupt-cells = <1>;
> +			#address-cells = <0>;
> +			#size-cells = <0>;
> +			interrupt-map = <
> +				0 &UIC0 0x16 4
> +				1 &UIC1 0x16 4>;
> +		};
> +		xor {
> +			interrupt-parent = <&UIC1>;
> +			interrupts = <0x1f 4>;
> +		};

You have no compatible property in these 3 nodes.  How are drivers
supposed to bind to them?

You also have no reg or dcr-reg properties.  What exactly are these
nodes for?

> +		sysacecf@4fe000000 {
> +			compatible = "xlnx,opb-sysace-1.00.b";

Odd.  This isn't a xilinx board by any means.  This should probably
look something like:

	compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b";

Though I'm curious about it in general.  The xilinx bindings have the
versioning numbers on them to match particular bit-streams in the FPGAs
if I remember correctly.  Does that really apply here?

josh
Yuri Tikhonov - Nov. 14, 2008, 4:45 a.m.
Hello Ben,

On Thursday, November 13, 2008 you wrote:

> On Thu, 2008-11-13 at 11:49 +0300, Yuri Tikhonov wrote:
>> Hello,
>> 
>> This patch extends DMA ranges for PCI(X) to 4GB, so that it could
>> work on Katmais with 4GB RAM installed.

> And where do you put MMIO ?

> The 32 bit part of the PCI space need to be split between MMIO and DMA.

 My understanding was that the dma-ranges property is responsible for 
setting up the inbound ranges of RAM's physical addresses, where PCI 
could DMA to/from. As regarding the outbound property, this patch 
doesn't change this, and there we have the PCI space split (2 GB of 
memory, and 64K of I/O spaces mapped from the 64-bit physical 
addresses into 32-bit PCI address space). Am I missing something ?

 With the default 2GB dma-ranges we just get the following on Katmai 
with 4GB of SDRAM installed:

...
PCIE0: Checking link...
PCIE0: Device detected, waiting for link...
PCIE0: link is up !
PCI host bridge /plb/pciex@d00000000 (primary) ranges:
 MEM 0x0000000e00000000..0x0000000e7fffffff -> 0x0000000080000000 
  IO 0x0000000f80000000..0x0000000f8000ffff -> 0x0000000000000000
/plb/pciex@d00000000: dma-ranges too small (size=80000000 total_memory=100000000)
PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
 MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000 
  IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
/plb/pciex@d20000000: dma-ranges too small (size=80000000 total_memory=100000000)
PCIE2: Checking link...
PCIE2: Device detected, waiting for link...
PCIE2: link is up !
PCI host bridge /plb/pciex@d40000000 (primary) ranges:
 MEM 0x0000000f00000000..0x0000000f7fffffff -> 0x0000000080000000 
  IO 0x0000000f80020000..0x0000000f8002ffff -> 0x0000000000000000
/plb/pciex@d40000000: dma-ranges too small (size=80000000 total_memory=100000000)
PCI host bridge /plb/pci@c0ec00000 (primary) ranges:
 MEM 0x0000000d80000000..0x0000000dffffffff -> 0x0000000080000000 
  IO 0x0000000c08000000..0x0000000c0800ffff -> 0x0000000000000000
/plb/pci@c0ec00000: dma-ranges too small (size=80000000 total_memory=100000000)
...


> Ideally, for 64-bit capable DMA, device, we should create a second DMA
> range mapping the whole memory at something like 1T or so on PCI, and
> keep a smaller (ie. 2G) DMA range for 32-bit only devices backed with
> swiotlb.

> Cheers,
> Ben.

 Regards, Yuri

 --
 Yuri Tikhonov, Senior Software Engineer
 Emcraft Systems, www.emcraft.com
Grant Likely - Nov. 14, 2008, 5 a.m.
On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote:
> On Thu, 13 Nov 2008 11:49:14 +0300
> Yuri Tikhonov <yur@emcraft.com> wrote:
> > +		sysacecf@4fe000000 {
> > +			compatible = "xlnx,opb-sysace-1.00.b";
> 
> Odd.  This isn't a xilinx board by any means.  This should probably
> look something like:
> 
> 	compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b";

Actually, if there is a sysace, it is definitely a xilinx part.  It
won't be on the SoC.  However, compatible = "xlnx,opb-sysace-1.00.b"
isn't really accurate.  It should really be compatible = "xlnx,sysace"
and the driver modified to accept this string.  "xlnx,opb-sysace-1.00.b"
is an FPGA block used to interface to the system ace which is
definitely not in use here.

So while it does get things to work, it is not a clean description of
the hardware.

g.
Yuri Tikhonov - Nov. 14, 2008, 5:21 a.m.
Hello Josh,

On Thursday, November 13, 2008 you wrote:

[snip]

> You have no compatible property in these 3 nodes.  How are drivers
> supposed to bind to them?

> You also have no reg or dcr-reg properties.  What exactly are these
> nodes for?

 Probably we (me and Ilya) overdone with posting katmai.dts-related 
changes to ML, and duplicated the same patches in different posts. 
Sorry for the confusion. These nodes are necessary for the ppc440spe 
ADMA driver:

http://www.nabble.com/-PATCH-11-11--ppc440spe-adma:-ADMA-driver-for-PPC440SP(e)-td20488049.html

>> +             sysacecf@4fe000000 {
>> +                     compatible = "xlnx,opb-sysace-1.00.b";

> Odd.  This isn't a xilinx board by any means.  This should probably
> look something like:

>         compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b";

> Though I'm curious about it in general.  The xilinx bindings have the
> versioning numbers on them to match particular bit-streams in the FPGAs
> if I remember correctly.  Does that really apply here?

 No, we just selected the description which looked more appropriate 
for our case: SysAce is connected to the External Bus Controller of 
440SPe, which in turns is attached as a slave to OPB (on-chip 
peripheral).

 Regards, Yuri

 --
 Yuri Tikhonov, Senior Software Engineer
 Emcraft Systems, www.emcraft.com
Yuri Tikhonov - Nov. 14, 2008, 5:27 a.m.
Hello Grant,

On Friday, November 14, 2008 you wrote:

> On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote:
>> On Thu, 13 Nov 2008 11:49:14 +0300
>> Yuri Tikhonov <yur@emcraft.com> wrote:
>> > +           sysacecf@4fe000000 {
>> > +                   compatible = "xlnx,opb-sysace-1.00.b";
>> 
>> Odd.  This isn't a xilinx board by any means.  This should probably
>> look something like:
>> 
>>       compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b";

> Actually, if there is a sysace, it is definitely a xilinx part.  It
> won't be on the SoC.  However, compatible = "xlnx,opb-sysace-1.00.b"
> isn't really accurate.  It should really be compatible = "xlnx,sysace"
> and the driver modified to accept this string.

 OK, we'll do this, and then re-post together with the cleaned-up 
"[PATCH] xsysace: use resource_size_t instead of unsigned long" patch.


>   "xlnx,opb-sysace-1.00.b"
> is an FPGA block used to interface to the system ace which is
> definitely not in use here.

> So while it does get things to work, it is not a clean description of
> the hardware.

> g.




 Regards, Yuri

 --
 Yuri Tikhonov, Senior Software Engineer
 Emcraft Systems, www.emcraft.com
Grant Likely - Nov. 14, 2008, 5:30 a.m.
On Thu, Nov 13, 2008 at 10:27 PM, Yuri Tikhonov <yur@emcraft.com> wrote:
>
> Hello Grant,
>
> On Friday, November 14, 2008 you wrote:
>
>> On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote:
>>> On Thu, 13 Nov 2008 11:49:14 +0300
>>> Yuri Tikhonov <yur@emcraft.com> wrote:
>>> > +           sysacecf@4fe000000 {
>>> > +                   compatible = "xlnx,opb-sysace-1.00.b";
>>>
>>> Odd.  This isn't a xilinx board by any means.  This should probably
>>> look something like:
>>>
>>>       compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b";
>
>> Actually, if there is a sysace, it is definitely a xilinx part.  It
>> won't be on the SoC.  However, compatible = "xlnx,opb-sysace-1.00.b"
>> isn't really accurate.  It should really be compatible = "xlnx,sysace"
>> and the driver modified to accept this string.
>
>  OK, we'll do this, and then re-post together with the cleaned-up
> "[PATCH] xsysace: use resource_size_t instead of unsigned long" patch.

Please cc: me on the next version of the xsysace patch.

g.
Benjamin Herrenschmidt - Nov. 14, 2008, 5:41 a.m.
>  My understanding was that the dma-ranges property is responsible for 
> setting up the inbound ranges of RAM's physical addresses, where PCI 
> could DMA to/from. As regarding the outbound property, this patch 
> doesn't change this, and there we have the PCI space split (2 GB of 
> memory, and 64K of I/O spaces mapped from the 64-bit physical 
> addresses into 32-bit PCI address space). Am I missing something ?

The PCI memory space doesn't differenciate inbound and outbound.

For example, if you take the example of a PCI 2.0 or PCI-X setup (I'm
leaving PCI-E alone in the discussion because it's a bit special in that
area, but the problem is fundamentally the same), and you have on the
PCI bus of that thing a device X configured to respond to MMIO at
address, for example, 0x80000000.

Now, what happens if another device, let's call it Y, tries to DMA to
that same address ? Who will pick up the memory write at address
0x80000000 ? The host controller or device X ? 

There is no concept of "inbound" here.. it's just a memory cycle to
address A that gets decoded by whoever tries to decode it on that bus
segment.

Thus DMA ranges must -not- overlap areas of memory used for MMIO, it
just won't work.

>  With the default 2GB dma-ranges we just get the following on Katmai 
> with 4GB of SDRAM installed:

Because that is simply not supported at the moment unless we add what
I've talked about earlier, ie basically, swiotlb support (which Becky is
working on at the moment).

You might be able to work around it somewhat if your PCI device supports
64-bit BARs in which case you can set your outbound regions above the
32-bit space. Note that I haven't verified whether the 32-bit linux
kernel supports that tho. There used to be issues with >32-bit PCI BARs
on 32-bit kernel, at least it wouldn't assign them but that may have
been fixed.

Another option you can try if you device only does 32-bit BARs but
supports 64-bit DMA, is to move the DMA window away from the 32-bit MMIO
space. Stick it at 1T or so in PCI space. You might need a little bit of
kernel tweaking to make the dma-ops pick that up properly but that
shouldnt be too hard.

If you really need 32-bit DMA support, you'll have to wait for swiotlb
from Becky or work with her in bringing it to powerpc so that we can do
bounce buffering for those devices.

Ben.

Patch

diff --git a/arch/powerpc/boot/dts/katmai.dts b/arch/powerpc/boot/dts/katmai.dts
index 077819b..7749478 100644
--- a/arch/powerpc/boot/dts/katmai.dts
+++ b/arch/powerpc/boot/dts/katmai.dts
@@ -245,8 +245,8 @@ 
 			ranges = <0x02000000 0x00000000 0x80000000 0x0000000d 0x80000000 0x00000000 0x80000000
 				  0x01000000 0x00000000 0x00000000 0x0000000c 0x08000000 0x00000000 0x00010000>;
 
-			/* Inbound 2GB range starting at 0 */
-			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>;
+			/* Inbound 4GB range starting at 0 */
+			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>;
 
 			/* This drives busses 0 to 0xf */
 			bus-range = <0x0 0xf>;
@@ -289,8 +289,8 @@ 
 			ranges = <0x02000000 0x00000000 0x80000000 0x0000000e 0x00000000 0x00000000 0x80000000
 				  0x01000000 0x00000000 0x00000000 0x0000000f 0x80000000 0x00000000 0x00010000>;
 
-			/* Inbound 2GB range starting at 0 */
-			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>;
+			/* Inbound 4GB range starting at 0 */
+			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>;
 
 			/* This drives busses 10 to 0x1f */
 			bus-range = <0x10 0x1f>;
@@ -330,8 +330,8 @@ 
 			ranges = <0x02000000 0x00000000 0x80000000 0x0000000e 0x80000000 0x00000000 0x80000000
 				  0x01000000 0x00000000 0x00000000 0x0000000f 0x80010000 0x00000000 0x00010000>;
 
-			/* Inbound 2GB range starting at 0 */
-			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>;
+			/* Inbound 4GB range starting at 0 */
+			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>;
 
 			/* This drives busses 10 to 0x1f */
 			bus-range = <0x20 0x2f>;
@@ -371,8 +371,8 @@ 
 			ranges = <0x02000000 0x00000000 0x80000000 0x0000000f 0x00000000 0x00000000 0x80000000
 				  0x01000000 0x00000000 0x00000000 0x0000000f 0x80020000 0x00000000 0x00010000>;
 
-			/* Inbound 2GB range starting at 0 */
-			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>;
+			/* Inbound 4GB range starting at 0 */
+			dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>;
 
 			/* This drives busses 10 to 0x1f */
 			bus-range = <0x30 0x3f>;
@@ -392,6 +392,36 @@ 
 				0x0 0x0 0x0 0x3 &UIC3 0xa 0x4 /* swizzled int C */
 				0x0 0x0 0x0 0x4 &UIC3 0xb 0x4 /* swizzled int D */>;
 		};
+		DMA0: dma0 {
+			interrupt-parent = <&DMA0>;
+			interrupts = <0 1>;
+			#interrupt-cells = <1>;
+			#address-cells = <0>;
+			#size-cells = <0>;
+			interrupt-map = <
+				0 &UIC0 0x14 4
+				1 &UIC1 0x16 4>;
+		};
+		DMA1: dma1 {
+			interrupt-parent = <&DMA1>;
+			interrupts = <0 1>;
+			#interrupt-cells = <1>;
+			#address-cells = <0>;
+			#size-cells = <0>;
+			interrupt-map = <
+				0 &UIC0 0x16 4
+				1 &UIC1 0x16 4>;
+		};
+		xor {
+			interrupt-parent = <&UIC1>;
+			interrupts = <0x1f 4>;
+		};
+		sysacecf@4fe000000 {
+			compatible = "xlnx,opb-sysace-1.00.b";
+			interrupt-parent = <&UIC2>;
+			interrupts = <0x19 4>;
+			reg = <0x00000004 0xfe000000 0x100>;
+		};
 	};
 
 	chosen {