| Message ID | 200811131149.14715.yur@emcraft.com (mailing list archive) |
|---|---|
| State | Changes Requested, archived |
| Delegated to: | Josh Boyer |
| Headers | show |
On Thu, 2008-11-13 at 11:49 +0300, Yuri Tikhonov wrote: > Hello, > > This patch extends DMA ranges for PCI(X) to 4GB, so that it could > work on Katmais with 4GB RAM installed. And where do you put MMIO ? The 32 bit part of the PCI space need to be split between MMIO and DMA. Ideally, for 64-bit capable DMA, device, we should create a second DMA range mapping the whole memory at something like 1T or so on PCI, and keep a smaller (ie. 2G) DMA range for 32-bit only devices backed with swiotlb. Cheers, Ben.
On Thu, 13 Nov 2008 11:49:14 +0300 Yuri Tikhonov <yur@emcraft.com> wrote: > Hello, > > This patch extends DMA ranges for PCI(X) to 4GB, so that it could > work on Katmais with 4GB RAM installed. > > Add new nodes for the PPC440SPe DMA, XOR engines to > be used in the PPC440SPe ADMA driver, and the SysACE > controller, which connects Compact Flash to Katmai. > > Signed-off-by: Ilya Yanok <yanok@emcraft.com> > Signed-off-by: Yuri Tikhonov <yur@emcraft.com> > --- > + DMA0: dma0 { > + interrupt-parent = <&DMA0>; > + interrupts = <0 1>; > + #interrupt-cells = <1>; > + #address-cells = <0>; > + #size-cells = <0>; > + interrupt-map = < > + 0 &UIC0 0x14 4 > + 1 &UIC1 0x16 4>; > + }; > + DMA1: dma1 { > + interrupt-parent = <&DMA1>; > + interrupts = <0 1>; > + #interrupt-cells = <1>; > + #address-cells = <0>; > + #size-cells = <0>; > + interrupt-map = < > + 0 &UIC0 0x16 4 > + 1 &UIC1 0x16 4>; > + }; > + xor { > + interrupt-parent = <&UIC1>; > + interrupts = <0x1f 4>; > + }; You have no compatible property in these 3 nodes. How are drivers supposed to bind to them? You also have no reg or dcr-reg properties. What exactly are these nodes for? > + sysacecf@4fe000000 { > + compatible = "xlnx,opb-sysace-1.00.b"; Odd. This isn't a xilinx board by any means. This should probably look something like: compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b"; Though I'm curious about it in general. The xilinx bindings have the versioning numbers on them to match particular bit-streams in the FPGAs if I remember correctly. Does that really apply here? josh
Hello Ben, On Thursday, November 13, 2008 you wrote: > On Thu, 2008-11-13 at 11:49 +0300, Yuri Tikhonov wrote: >> Hello, >> >> This patch extends DMA ranges for PCI(X) to 4GB, so that it could >> work on Katmais with 4GB RAM installed. > And where do you put MMIO ? > The 32 bit part of the PCI space need to be split between MMIO and DMA. My understanding was that the dma-ranges property is responsible for setting up the inbound ranges of RAM's physical addresses, where PCI could DMA to/from. As regarding the outbound property, this patch doesn't change this, and there we have the PCI space split (2 GB of memory, and 64K of I/O spaces mapped from the 64-bit physical addresses into 32-bit PCI address space). Am I missing something ? With the default 2GB dma-ranges we just get the following on Katmai with 4GB of SDRAM installed: ... PCIE0: Checking link... PCIE0: Device detected, waiting for link... PCIE0: link is up ! PCI host bridge /plb/pciex@d00000000 (primary) ranges: MEM 0x0000000e00000000..0x0000000e7fffffff -> 0x0000000080000000 IO 0x0000000f80000000..0x0000000f8000ffff -> 0x0000000000000000 /plb/pciex@d00000000: dma-ranges too small (size=80000000 total_memory=100000000) PCIE1: Checking link... PCIE1: Device detected, waiting for link... PCIE1: link is up ! PCI host bridge /plb/pciex@d20000000 (primary) ranges: MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000 IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000 /plb/pciex@d20000000: dma-ranges too small (size=80000000 total_memory=100000000) PCIE2: Checking link... PCIE2: Device detected, waiting for link... PCIE2: link is up ! PCI host bridge /plb/pciex@d40000000 (primary) ranges: MEM 0x0000000f00000000..0x0000000f7fffffff -> 0x0000000080000000 IO 0x0000000f80020000..0x0000000f8002ffff -> 0x0000000000000000 /plb/pciex@d40000000: dma-ranges too small (size=80000000 total_memory=100000000) PCI host bridge /plb/pci@c0ec00000 (primary) ranges: MEM 0x0000000d80000000..0x0000000dffffffff -> 0x0000000080000000 IO 0x0000000c08000000..0x0000000c0800ffff -> 0x0000000000000000 /plb/pci@c0ec00000: dma-ranges too small (size=80000000 total_memory=100000000) ... > Ideally, for 64-bit capable DMA, device, we should create a second DMA > range mapping the whole memory at something like 1T or so on PCI, and > keep a smaller (ie. 2G) DMA range for 32-bit only devices backed with > swiotlb. > Cheers, > Ben. Regards, Yuri -- Yuri Tikhonov, Senior Software Engineer Emcraft Systems, www.emcraft.com
On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote: > On Thu, 13 Nov 2008 11:49:14 +0300 > Yuri Tikhonov <yur@emcraft.com> wrote: > > + sysacecf@4fe000000 { > > + compatible = "xlnx,opb-sysace-1.00.b"; > > Odd. This isn't a xilinx board by any means. This should probably > look something like: > > compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b"; Actually, if there is a sysace, it is definitely a xilinx part. It won't be on the SoC. However, compatible = "xlnx,opb-sysace-1.00.b" isn't really accurate. It should really be compatible = "xlnx,sysace" and the driver modified to accept this string. "xlnx,opb-sysace-1.00.b" is an FPGA block used to interface to the system ace which is definitely not in use here. So while it does get things to work, it is not a clean description of the hardware. g.
Hello Josh, On Thursday, November 13, 2008 you wrote: [snip] > You have no compatible property in these 3 nodes. How are drivers > supposed to bind to them? > You also have no reg or dcr-reg properties. What exactly are these > nodes for? Probably we (me and Ilya) overdone with posting katmai.dts-related changes to ML, and duplicated the same patches in different posts. Sorry for the confusion. These nodes are necessary for the ppc440spe ADMA driver: http://www.nabble.com/-PATCH-11-11--ppc440spe-adma:-ADMA-driver-for-PPC440SP(e)-td20488049.html >> + sysacecf@4fe000000 { >> + compatible = "xlnx,opb-sysace-1.00.b"; > Odd. This isn't a xilinx board by any means. This should probably > look something like: > compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b"; > Though I'm curious about it in general. The xilinx bindings have the > versioning numbers on them to match particular bit-streams in the FPGAs > if I remember correctly. Does that really apply here? No, we just selected the description which looked more appropriate for our case: SysAce is connected to the External Bus Controller of 440SPe, which in turns is attached as a slave to OPB (on-chip peripheral). Regards, Yuri -- Yuri Tikhonov, Senior Software Engineer Emcraft Systems, www.emcraft.com
Hello Grant, On Friday, November 14, 2008 you wrote: > On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote: >> On Thu, 13 Nov 2008 11:49:14 +0300 >> Yuri Tikhonov <yur@emcraft.com> wrote: >> > + sysacecf@4fe000000 { >> > + compatible = "xlnx,opb-sysace-1.00.b"; >> >> Odd. This isn't a xilinx board by any means. This should probably >> look something like: >> >> compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b"; > Actually, if there is a sysace, it is definitely a xilinx part. It > won't be on the SoC. However, compatible = "xlnx,opb-sysace-1.00.b" > isn't really accurate. It should really be compatible = "xlnx,sysace" > and the driver modified to accept this string. OK, we'll do this, and then re-post together with the cleaned-up "[PATCH] xsysace: use resource_size_t instead of unsigned long" patch. > "xlnx,opb-sysace-1.00.b" > is an FPGA block used to interface to the system ace which is > definitely not in use here. > So while it does get things to work, it is not a clean description of > the hardware. > g. Regards, Yuri -- Yuri Tikhonov, Senior Software Engineer Emcraft Systems, www.emcraft.com
On Thu, Nov 13, 2008 at 10:27 PM, Yuri Tikhonov <yur@emcraft.com> wrote: > > Hello Grant, > > On Friday, November 14, 2008 you wrote: > >> On Thu, Nov 13, 2008 at 06:45:33AM -0500, Josh Boyer wrote: >>> On Thu, 13 Nov 2008 11:49:14 +0300 >>> Yuri Tikhonov <yur@emcraft.com> wrote: >>> > + sysacecf@4fe000000 { >>> > + compatible = "xlnx,opb-sysace-1.00.b"; >>> >>> Odd. This isn't a xilinx board by any means. This should probably >>> look something like: >>> >>> compatible = "amcc,sysace-440spe", "xlnx,opb-sysace-1.00.b"; > >> Actually, if there is a sysace, it is definitely a xilinx part. It >> won't be on the SoC. However, compatible = "xlnx,opb-sysace-1.00.b" >> isn't really accurate. It should really be compatible = "xlnx,sysace" >> and the driver modified to accept this string. > > OK, we'll do this, and then re-post together with the cleaned-up > "[PATCH] xsysace: use resource_size_t instead of unsigned long" patch. Please cc: me on the next version of the xsysace patch. g.
> My understanding was that the dma-ranges property is responsible for > setting up the inbound ranges of RAM's physical addresses, where PCI > could DMA to/from. As regarding the outbound property, this patch > doesn't change this, and there we have the PCI space split (2 GB of > memory, and 64K of I/O spaces mapped from the 64-bit physical > addresses into 32-bit PCI address space). Am I missing something ? The PCI memory space doesn't differenciate inbound and outbound. For example, if you take the example of a PCI 2.0 or PCI-X setup (I'm leaving PCI-E alone in the discussion because it's a bit special in that area, but the problem is fundamentally the same), and you have on the PCI bus of that thing a device X configured to respond to MMIO at address, for example, 0x80000000. Now, what happens if another device, let's call it Y, tries to DMA to that same address ? Who will pick up the memory write at address 0x80000000 ? The host controller or device X ? There is no concept of "inbound" here.. it's just a memory cycle to address A that gets decoded by whoever tries to decode it on that bus segment. Thus DMA ranges must -not- overlap areas of memory used for MMIO, it just won't work. > With the default 2GB dma-ranges we just get the following on Katmai > with 4GB of SDRAM installed: Because that is simply not supported at the moment unless we add what I've talked about earlier, ie basically, swiotlb support (which Becky is working on at the moment). You might be able to work around it somewhat if your PCI device supports 64-bit BARs in which case you can set your outbound regions above the 32-bit space. Note that I haven't verified whether the 32-bit linux kernel supports that tho. There used to be issues with >32-bit PCI BARs on 32-bit kernel, at least it wouldn't assign them but that may have been fixed. Another option you can try if you device only does 32-bit BARs but supports 64-bit DMA, is to move the DMA window away from the 32-bit MMIO space. Stick it at 1T or so in PCI space. You might need a little bit of kernel tweaking to make the dma-ops pick that up properly but that shouldnt be too hard. If you really need 32-bit DMA support, you'll have to wait for swiotlb from Becky or work with her in bringing it to powerpc so that we can do bounce buffering for those devices. Ben.
diff --git a/arch/powerpc/boot/dts/katmai.dts b/arch/powerpc/boot/dts/katmai.dts index 077819b..7749478 100644 --- a/arch/powerpc/boot/dts/katmai.dts +++ b/arch/powerpc/boot/dts/katmai.dts @@ -245,8 +245,8 @@ ranges = <0x02000000 0x00000000 0x80000000 0x0000000d 0x80000000 0x00000000 0x80000000 0x01000000 0x00000000 0x00000000 0x0000000c 0x08000000 0x00000000 0x00010000>; - /* Inbound 2GB range starting at 0 */ - dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>; + /* Inbound 4GB range starting at 0 */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>; /* This drives busses 0 to 0xf */ bus-range = <0x0 0xf>; @@ -289,8 +289,8 @@ ranges = <0x02000000 0x00000000 0x80000000 0x0000000e 0x00000000 0x00000000 0x80000000 0x01000000 0x00000000 0x00000000 0x0000000f 0x80000000 0x00000000 0x00010000>; - /* Inbound 2GB range starting at 0 */ - dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>; + /* Inbound 4GB range starting at 0 */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>; /* This drives busses 10 to 0x1f */ bus-range = <0x10 0x1f>; @@ -330,8 +330,8 @@ ranges = <0x02000000 0x00000000 0x80000000 0x0000000e 0x80000000 0x00000000 0x80000000 0x01000000 0x00000000 0x00000000 0x0000000f 0x80010000 0x00000000 0x00010000>; - /* Inbound 2GB range starting at 0 */ - dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>; + /* Inbound 4GB range starting at 0 */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>; /* This drives busses 10 to 0x1f */ bus-range = <0x20 0x2f>; @@ -371,8 +371,8 @@ ranges = <0x02000000 0x00000000 0x80000000 0x0000000f 0x00000000 0x00000000 0x80000000 0x01000000 0x00000000 0x00000000 0x0000000f 0x80020000 0x00000000 0x00010000>; - /* Inbound 2GB range starting at 0 */ - dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x0 0x80000000>; + /* Inbound 4GB range starting at 0 */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x1 0x00000000>; /* This drives busses 10 to 0x1f */ bus-range = <0x30 0x3f>; @@ -392,6 +392,36 @@ 0x0 0x0 0x0 0x3 &UIC3 0xa 0x4 /* swizzled int C */ 0x0 0x0 0x0 0x4 &UIC3 0xb 0x4 /* swizzled int D */>; }; + DMA0: dma0 { + interrupt-parent = <&DMA0>; + interrupts = <0 1>; + #interrupt-cells = <1>; + #address-cells = <0>; + #size-cells = <0>; + interrupt-map = < + 0 &UIC0 0x14 4 + 1 &UIC1 0x16 4>; + }; + DMA1: dma1 { + interrupt-parent = <&DMA1>; + interrupts = <0 1>; + #interrupt-cells = <1>; + #address-cells = <0>; + #size-cells = <0>; + interrupt-map = < + 0 &UIC0 0x16 4 + 1 &UIC1 0x16 4>; + }; + xor { + interrupt-parent = <&UIC1>; + interrupts = <0x1f 4>; + }; + sysacecf@4fe000000 { + compatible = "xlnx,opb-sysace-1.00.b"; + interrupt-parent = <&UIC2>; + interrupts = <0x19 4>; + reg = <0x00000004 0xfe000000 0x100>; + }; }; chosen {