mbox series

[v4,0/6] Add support for PCIe controller to work in endpoint mode on R-Car SoCs

Message ID 20200208183641.6674-1-prabhakar.mahadev-lad.rj@bp.renesas.com
Headers show
Series Add support for PCIe controller to work in endpoint mode on R-Car SoCs | expand

Message

Lad, Prabhakar Feb. 8, 2020, 6:36 p.m. UTC
This patch series adds support for PCIe controller on rcar to work in
endpoint mode, this also extends the epf framework to handle base region
for mapping PCI address locally..

Note:
The cadence/rockchip/designware endpoint drivers are build tested only.

Changes for v4:
1] Fixed dtb_check error reported by Rob
2] Fixed review comments reported by Kishon
   a] Dropped pci_epc_find_best_fit_window()
   b] Fixed initializing mem ptr in __pci_epc_mem_init()
   c] Dropped map_size from pci_epc_mem_window structure

Changes for v3:
1] Fixed review comments from Bjorn and Kishon.
3] Converted to DT schema

Changes for v2:
1] Fixed review comments from Biju for dt-bindings to include an example
   for a tested platform.
2] Fixed review comments from Kishon to extend the features of outbound
   regions in epf framework.
3] Added support to parse outbound-ranges in OF.

lspci output on host:
=====================

01:00.0 Unassigned class [ff00]: Renesas Technology Corp. Device 002d
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 152
        Region 0: Memory at fe200200 (64-bit, non-prefetchable) [size=128]
        Region 2: Memory at fe200000 (64-bit, non-prefetchable) [size=256]
        Region 4: Memory at fe200100 (64-bit, non-prefetchable) [size=256]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 00000004fa36d000  Data: 0001
                Masking: fffffffe  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Kernel driver in use: pci-endpoint-test

BAR Test
========
root@g2e:~# pcitest -b 0
BAR0:           OKAY
root@g2e:~# pcitest -b 1
BAR1:           NOT OKAY
root@g2e:~# pcitest -b 2
BAR2:           OKAY
root@g2e:~# pcitest -b 3
BAR3:           NOT OKAY
root@g2e:~# pcitest -b 4
BAR4:           OKAY
root@g2e:~# pcitest -b 5
BAR5:           NOT OKAY

Note: BAR test for 1/3/5 fail because they are configured to be 64bits

Interrupt Test
==============
root@g2e:~# pcitest -i 0
SET IRQ TYPE TO LEGACY:         OKAY
root@g2e:~# pcitest -l
LEGACY IRQ:     OKAY

Read Test
=========
root@g2e:~# pcitest -r -s 1
READ (      1 bytes):           OKAY
root@g2e:~# pcitest -r -s 1024
READ (   1024 bytes):           OKAY
root@g2e:~# pcitest -r -s 1025
READ (   1025 bytes):           OKAY
root@g2e:~# pcitest -r -s 1024000
READ (1024000 bytes):           OKAY
root@g2e:~# pcitest -r -s 1024001
READ (1024001 bytes):           OKAY

Write Test
==========
root@g2e:~# pcitest -w -s 1
WRITE (      1 bytes):          OKAY
root@g2e:~# pcitest -w -s 1024
WRITE (   1024 bytes):          OKAY
root@g2e:~# pcitest -w -s 1025
WRITE (   1025 bytes):          OKAY
root@g2e:~# pcitest -w -s 1024000
WRITE (1024000 bytes):          OKAY
root@g2e:~# pcitest -w -s 1024001
WRITE (1024001 bytes):          OKAY

Copy Test
=========
root@g2e:~# pcitest -c -s 1
COPY (      1 bytes):           OKAY
root@g2e:~# pcitest -c -s 1024
COPY (   1024 bytes):           OKAY
root@g2e:~# pcitest -c -s 1025
COPY (   1025 bytes):           OKAY
root@g2e:~# pcitest -c -s 1024000
COPY (1024000 bytes):           OKAY
root@g2e:~# pcitest -c -s 1024001
COPY (1024001 bytes):           OKAY

Lad Prabhakar (6):
  PCI: rcar: Preparation for adding endpoint support
  PCI: rcar: Fix calculating mask for PCIEPAMR register
  PCI: endpoint: Add support to handle multiple base for mapping
    outbound memory
  dt-bindings: PCI: rcar: Add bindings for R-Car PCIe endpoint
    controller
  PCI: rcar: Add support for rcar PCIe controller in endpoint mode
  misc: pci_endpoint_test: Add Device ID for RZ/G2E PCIe controller

 .../devicetree/bindings/pci/rcar-pci-ep.yaml       |   76 ++
 arch/arm64/configs/defconfig                       |    2 +-
 drivers/misc/pci_endpoint_test.c                   |    3 +
 drivers/pci/controller/Kconfig                     |   11 +-
 drivers/pci/controller/Makefile                    |    3 +-
 drivers/pci/controller/cadence/pcie-cadence-ep.c   |    7 +-
 drivers/pci/controller/dwc/pcie-designware-ep.c    |   29 +-
 drivers/pci/controller/pcie-rcar-ep.c              |  492 ++++++++
 drivers/pci/controller/pcie-rcar-host.c            | 1044 +++++++++++++++++
 drivers/pci/controller/pcie-rcar.c                 | 1232 +-------------------
 drivers/pci/controller/pcie-rcar.h                 |  132 +++
 drivers/pci/controller/pcie-rockchip-ep.c          |    7 +-
 drivers/pci/endpoint/pci-epc-mem.c                 |  166 ++-
 include/linux/pci-epc.h                            |   39 +-
 14 files changed, 1982 insertions(+), 1261 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/rcar-pci-ep.yaml
 create mode 100644 drivers/pci/controller/pcie-rcar-ep.c
 create mode 100644 drivers/pci/controller/pcie-rcar-host.c
 create mode 100644 drivers/pci/controller/pcie-rcar.h

Comments

Sergei Shtylyov Feb. 8, 2020, 7:07 p.m. UTC | #1
Hello!

On 02/08/2020 09:36 PM, Lad Prabhakar wrote:

> The mask value was calculated incorrectly for PCIEPAMR register if the
> size was less the 128bytes, this patch fixes the above by adding a check

   Less than, perhaps?

> on size.
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
[...]

MBR, Sergei
Lad, Prabhakar Feb. 8, 2020, 7:16 p.m. UTC | #2
Hi Sergei,

Thank you for the review.

On Sat, Feb 8, 2020 at 7:07 PM Sergei Shtylyov
<sergei.shtylyov@cogentembedded.com> wrote:
>
> Hello!
>
> On 02/08/2020 09:36 PM, Lad Prabhakar wrote:
>
> > The mask value was calculated incorrectly for PCIEPAMR register if the
> > size was less the 128bytes, this patch fixes the above by adding a check
>
>    Less than, perhaps?
>
Oops shall fix that.

Cheers,
--Prabhakar Lad

> > on size.
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> [...]
>
> MBR, Sergei
Nathan Chancellor Feb. 12, 2020, 3:42 a.m. UTC | #3
Hi Lad,

On Sat, Feb 08, 2020 at 06:36:40PM +0000, Lad Prabhakar wrote:
> This patch adds support for rcar PCIe controller to work in endpoint mode.
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
>  drivers/pci/controller/Kconfig        |   7 +
>  drivers/pci/controller/Makefile       |   1 +
>  drivers/pci/controller/pcie-rcar-ep.c | 492 ++++++++++++++++++++++++++++++++++
>  drivers/pci/controller/pcie-rcar.h    |   6 +
>  4 files changed, 506 insertions(+)
>  create mode 100644 drivers/pci/controller/pcie-rcar-ep.c
> 

<snip>

> diff --git a/drivers/pci/controller/pcie-rcar-ep.c b/drivers/pci/controller/pcie-rcar-ep.c
> new file mode 100644
> index 0000000..32a7fca
> --- /dev/null
> +++ b/drivers/pci/controller/pcie-rcar-ep.c

<snip>

> +static int rcar_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
> +				struct pci_epf_bar *epf_bar)
> +{
> +	struct rcar_pcie *ep = epc_get_drvdata(epc);
> +	dma_addr_t cpu_addr = epf_bar->phys_addr;
> +	int flags = epf_bar->flags | LAR_ENABLE | LAM_64BIT;
> +	enum pci_barno bar = epf_bar->barno;
> +	u64 size = 1ULL << fls64(epf_bar->size - 1);
> +	u32 mask;
> +	int idx;
> +	int err;
> +
> +	idx = find_first_zero_bit(ep->ib_window_map, ep->num_ib_windows);
> +	if (idx >= ep->num_ib_windows) {
> +		dev_err(ep->dev, "no free inbound window\n");
> +		return -EINVAL;
> +	}
> +
> +	if ((flags & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO)
> +		flags |= IO_SPACE;
> +
> +	ep->bar_to_atu[bar] = idx;
> +	/* use 64 bit bars */
> +	set_bit(idx, ep->ib_window_map);
> +	set_bit(idx + 1, ep->ib_window_map);
> +
> +	if (cpu_addr > 0) {
> +		unsigned long nr_zeros = __ffs64(cpu_addr);
> +		u64 alignment = 1ULL << nr_zeros;
> +
> +		size = min(size, alignment);
> +	} else {
> +		size = size;
> +	}

We received a report from the 0day bot that clang warns that this is
unnecessary. Would you mind removing it if you have to spin up a new
version?

You can view the full report here:

https://groups.google.com/d/msg/clang-built-linux/KHUKw5L8yxw/Mb7KRMG7BQAJ

Cheers,
Nathan
Lad, Prabhakar Feb. 12, 2020, 7:38 a.m. UTC | #4
Hi Nathan,

On Wed, Feb 12, 2020 at 3:42 AM Nathan Chancellor
<natechancellor@gmail.com> wrote:
>
> Hi Lad,
>
> On Sat, Feb 08, 2020 at 06:36:40PM +0000, Lad Prabhakar wrote:
> > This patch adds support for rcar PCIe controller to work in endpoint mode.
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > ---
> >  drivers/pci/controller/Kconfig        |   7 +
> >  drivers/pci/controller/Makefile       |   1 +
> >  drivers/pci/controller/pcie-rcar-ep.c | 492 ++++++++++++++++++++++++++++++++++
> >  drivers/pci/controller/pcie-rcar.h    |   6 +
> >  4 files changed, 506 insertions(+)
> >  create mode 100644 drivers/pci/controller/pcie-rcar-ep.c
> >
>
> <snip>
>
> > diff --git a/drivers/pci/controller/pcie-rcar-ep.c b/drivers/pci/controller/pcie-rcar-ep.c
> > new file mode 100644
> > index 0000000..32a7fca
> > --- /dev/null
> > +++ b/drivers/pci/controller/pcie-rcar-ep.c
>
> <snip>
>
> > +static int rcar_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
> > +                             struct pci_epf_bar *epf_bar)
> > +{
> > +     struct rcar_pcie *ep = epc_get_drvdata(epc);
> > +     dma_addr_t cpu_addr = epf_bar->phys_addr;
> > +     int flags = epf_bar->flags | LAR_ENABLE | LAM_64BIT;
> > +     enum pci_barno bar = epf_bar->barno;
> > +     u64 size = 1ULL << fls64(epf_bar->size - 1);
> > +     u32 mask;
> > +     int idx;
> > +     int err;
> > +
> > +     idx = find_first_zero_bit(ep->ib_window_map, ep->num_ib_windows);
> > +     if (idx >= ep->num_ib_windows) {
> > +             dev_err(ep->dev, "no free inbound window\n");
> > +             return -EINVAL;
> > +     }
> > +
> > +     if ((flags & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO)
> > +             flags |= IO_SPACE;
> > +
> > +     ep->bar_to_atu[bar] = idx;
> > +     /* use 64 bit bars */
> > +     set_bit(idx, ep->ib_window_map);
> > +     set_bit(idx + 1, ep->ib_window_map);
> > +
> > +     if (cpu_addr > 0) {
> > +             unsigned long nr_zeros = __ffs64(cpu_addr);
> > +             u64 alignment = 1ULL << nr_zeros;
> > +
> > +             size = min(size, alignment);
> > +     } else {
> > +             size = size;
> > +     }
>
> We received a report from the 0day bot that clang warns that this is
> unnecessary. Would you mind removing it if you have to spin up a new
> version?
>
Sure ill fix that.

Cheers,
--Prabhakar Lad

> You can view the full report here:
>
> https://groups.google.com/d/msg/clang-built-linux/KHUKw5L8yxw/Mb7KRMG7BQAJ
>
> Cheers,
> Nathan
Lad, Prabhakar Feb. 19, 2020, 9:14 a.m. UTC | #5
Hi Kishon,

On Sat, Feb 8, 2020 at 6:36 PM Lad Prabhakar <prabhakar.csengg@gmail.com> wrote:
>
> R-Car PCIe controller has support to map multiple memory regions for
> mapping the outbound memory in local system also the controller limits
> single allocation for each region (that is, once a chunk is used from the
> region it cannot be used to allocate a new one). This features inspires to
> add support for handling multiple memory bases in endpoint framework.
>
> With this patch pci_epc_mem_init() now accepts multiple regions, also
> page_size for each memory region is passed during initialization so as
> to handle single allocation for each region by setting the page_size to
> window_size.
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
>  drivers/pci/controller/cadence/pcie-cadence-ep.c |   7 +-
>  drivers/pci/controller/dwc/pcie-designware-ep.c  |  29 ++--
>  drivers/pci/controller/pcie-rockchip-ep.c        |   7 +-
>  drivers/pci/endpoint/pci-epc-mem.c               | 166 ++++++++++++++++-------
>  include/linux/pci-epc.h                          |  39 ++++--
>  5 files changed, 168 insertions(+), 80 deletions(-)
>
Could you please review the patch, as I intend to post a v5 with minor
tweaks for patch 01/06.
It would be good if I get your Ack on this patch.

Cheers,
--Prabhakar

> diff --git a/drivers/pci/controller/cadence/pcie-cadence-ep.c b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> index 1c173da..90e32438 100644
> --- a/drivers/pci/controller/cadence/pcie-cadence-ep.c
> +++ b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> @@ -401,6 +401,7 @@ int cdns_pcie_ep_setup(struct cdns_pcie_ep *ep)
>         struct device *dev = ep->pcie.dev;
>         struct platform_device *pdev = to_platform_device(dev);
>         struct device_node *np = dev->of_node;
> +       struct pci_epc_mem_window mem_window;
>         struct cdns_pcie *pcie = &ep->pcie;
>         struct resource *res;
>         struct pci_epc *epc;
> @@ -449,8 +450,10 @@ int cdns_pcie_ep_setup(struct cdns_pcie_ep *ep)
>         if (of_property_read_u8(np, "max-functions", &epc->max_functions) < 0)
>                 epc->max_functions = 1;
>
> -       ret = pci_epc_mem_init(epc, pcie->mem_res->start,
> -                              resource_size(pcie->mem_res));
> +       mem_window.phys_base = pcie->mem_res->start;
> +       mem_window.size = resource_size(pcie->mem_res);
> +       mem_window.page_size = PAGE_SIZE;
> +       ret = pci_epc_mem_init(epc, &mem_window, 1);
>         if (ret < 0) {
>                 dev_err(dev, "failed to initialize the memory space\n");
>                 goto err_init;
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index cfeccd7..b150ef3 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -195,8 +195,7 @@ static void dw_pcie_ep_unmap_addr(struct pci_epc *epc, u8 func_no,
>  }
>
>  static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no,
> -                              phys_addr_t addr,
> -                              u64 pci_addr, size_t size)
> +                              phys_addr_t addr, u64 pci_addr, size_t size)
>  {
>         int ret;
>         struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> @@ -367,6 +366,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>         unsigned int aligned_offset;
>         u16 msg_ctrl, msg_data;
>         u32 msg_addr_lower, msg_addr_upper, reg;
> +       int window = PCI_EPC_DEFAULT_WINDOW;
>         u64 msg_addr;
>         bool has_upper;
>         int ret;
> @@ -390,11 +390,11 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>                 reg = ep->msi_cap + PCI_MSI_DATA_32;
>                 msg_data = dw_pcie_readw_dbi(pci, reg);
>         }
> -       aligned_offset = msg_addr_lower & (epc->mem->page_size - 1);
> +       aligned_offset = msg_addr_lower & (epc->mem[window]->page_size - 1);
>         msg_addr = ((u64)msg_addr_upper) << 32 |
>                         (msg_addr_lower & ~aligned_offset);
> -       ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys, msg_addr,
> -                                 epc->mem->page_size);
> +       ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys,
> +                                 msg_addr, epc->mem[window]->page_size);
>         if (ret)
>                 return ret;
>
> @@ -416,6 +416,7 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>         u32 reg, msg_data, vec_ctrl;
>         u64 tbl_addr, msg_addr, reg_u64;
>         void __iomem *msix_tbl;
> +       int window = PCI_EPC_DEFAULT_WINDOW;
>         int ret;
>
>         reg = ep->msix_cap + PCI_MSIX_TABLE;
> @@ -452,8 +453,8 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>                 return -EPERM;
>         }
>
> -       ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys, msg_addr,
> -                                 epc->mem->page_size);
> +       ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys,
> +                                 msg_addr, epc->mem[window]->page_size);
>         if (ret)
>                 return ret;
>
> @@ -466,10 +467,11 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>
>  void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
>  {
> +       int window = PCI_EPC_DEFAULT_WINDOW;
>         struct pci_epc *epc = ep->epc;
>
>         pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
> -                             epc->mem->page_size);
> +                             epc->mem[window]->page_size);
>
>         pci_epc_mem_exit(epc);
>  }
> @@ -502,6 +504,8 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>         unsigned int nbars;
>         unsigned int offset;
>         struct pci_epc *epc;
> +       size_t msi_page_size;
> +       struct pci_epc_mem_window mem_window;
>         struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>         struct device *dev = pci->dev;
>         struct device_node *np = dev->of_node;
> @@ -574,15 +578,18 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>         if (ret < 0)
>                 epc->max_functions = 1;
>
> -       ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
> -                                ep->page_size);
> +       mem_window.phys_base = ep->phys_base;
> +       mem_window.size = ep->addr_size;
> +       mem_window.page_size = ep->page_size;
> +       ret = __pci_epc_mem_init(epc, &mem_window, 1);
>         if (ret < 0) {
>                 dev_err(dev, "Failed to initialize address space\n");
>                 return ret;
>         }
>
> +       msi_page_size = epc->mem[PCI_EPC_DEFAULT_WINDOW]->page_size;
>         ep->msi_mem = pci_epc_mem_alloc_addr(epc, &ep->msi_mem_phys,
> -                                            epc->mem->page_size);
> +                                            msi_page_size);
>         if (!ep->msi_mem) {
>                 dev_err(dev, "Failed to reserve memory for MSI/MSI-X\n");
>                 return -ENOMEM;
> diff --git a/drivers/pci/controller/pcie-rockchip-ep.c b/drivers/pci/controller/pcie-rockchip-ep.c
> index d743b0a..5a97390 100644
> --- a/drivers/pci/controller/pcie-rockchip-ep.c
> +++ b/drivers/pci/controller/pcie-rockchip-ep.c
> @@ -562,6 +562,7 @@ static const struct of_device_id rockchip_pcie_ep_of_match[] = {
>
>  static int rockchip_pcie_ep_probe(struct platform_device *pdev)
>  {
> +       struct pci_epc_mem_window mem_window;
>         struct device *dev = &pdev->dev;
>         struct rockchip_pcie_ep *ep;
>         struct rockchip_pcie *rockchip;
> @@ -614,8 +615,10 @@ static int rockchip_pcie_ep_probe(struct platform_device *pdev)
>         /* Only enable function 0 by default */
>         rockchip_pcie_write(rockchip, BIT(0), PCIE_CORE_PHY_FUNC_CFG);
>
> -       err = pci_epc_mem_init(epc, rockchip->mem_res->start,
> -                              resource_size(rockchip->mem_res));
> +       mem_window.phys_base = rockchip->mem_res->start;
> +       mem_window.size = resource_size(rockchip->mem_res);
> +       mem_window.page_size = PAGE_SIZE;
> +       err = pci_epc_mem_init(epc, &mem_window, 1);
>         if (err < 0) {
>                 dev_err(dev, "failed to initialize the memory space\n");
>                 goto err_uninit_port;
> diff --git a/drivers/pci/endpoint/pci-epc-mem.c b/drivers/pci/endpoint/pci-epc-mem.c
> index d2b174c..b3eedee 100644
> --- a/drivers/pci/endpoint/pci-epc-mem.c
> +++ b/drivers/pci/endpoint/pci-epc-mem.c
> @@ -38,57 +38,76 @@ static int pci_epc_mem_get_order(struct pci_epc_mem *mem, size_t size)
>  /**
>   * __pci_epc_mem_init() - initialize the pci_epc_mem structure
>   * @epc: the EPC device that invoked pci_epc_mem_init
> - * @phys_base: the physical address of the base
> - * @size: the size of the address space
> - * @page_size: size of each page
> + * @windows: pointer to windows supported by the device
> + * @num_windows: number of windows device supports
>   *
>   * Invoke to initialize the pci_epc_mem structure used by the
>   * endpoint functions to allocate mapped PCI address.
>   */
> -int __pci_epc_mem_init(struct pci_epc *epc, phys_addr_t phys_base, size_t size,
> -                      size_t page_size)
> +int __pci_epc_mem_init(struct pci_epc *epc, struct pci_epc_mem_window *windows,
> +                      int num_windows)
>  {
> -       int ret;
> -       struct pci_epc_mem *mem;
> -       unsigned long *bitmap;
> +       struct pci_epc_mem *mem = NULL;
> +       unsigned long *bitmap = NULL;
>         unsigned int page_shift;
> -       int pages;
> +       size_t page_size;
>         int bitmap_size;
> -
> -       if (page_size < PAGE_SIZE)
> -               page_size = PAGE_SIZE;
> -
> -       page_shift = ilog2(page_size);
> -       pages = size >> page_shift;
> -       bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> -
> -       mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> -       if (!mem) {
> -               ret = -ENOMEM;
> -               goto err;
> -       }
> -
> -       bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> -       if (!bitmap) {
> -               ret = -ENOMEM;
> -               goto err_mem;
> +       int pages;
> +       int ret;
> +       int i;
> +
> +       epc->mem_windows = 0;
> +
> +       if (!windows)
> +               return -EINVAL;
> +
> +       if (num_windows <= 0)
> +               return -EINVAL;
> +
> +       epc->mem = kcalloc(num_windows, sizeof(*mem), GFP_KERNEL);
> +       if (!epc->mem)
> +               return -EINVAL;
> +
> +       for (i = 0; i < num_windows; i++) {
> +               page_size = windows[i].page_size;
> +               if (page_size < PAGE_SIZE)
> +                       page_size = PAGE_SIZE;
> +               page_shift = ilog2(page_size);
> +               pages = windows[i].size >> page_shift;
> +               bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> +
> +               mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> +               if (!mem) {
> +                       ret = -ENOMEM;
> +                       goto err_mem;
> +               }
> +
> +               bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> +               if (!bitmap) {
> +                       ret = -ENOMEM;
> +                       goto err_mem;
> +               }
> +
> +               mem->bitmap = bitmap;
> +               mem->window.phys_base = windows[i].phys_base;
> +               mem->page_size = page_size;
> +               mem->pages = pages;
> +               mem->window.size = windows[i].size;
> +               epc->mem[i] = mem;
>         }
> -
> -       mem->bitmap = bitmap;
> -       mem->phys_base = phys_base;
> -       mem->page_size = page_size;
> -       mem->pages = pages;
> -       mem->size = size;
> -
> -       epc->mem = mem;
> +       epc->mem_windows = num_windows;
>
>         return 0;
>
>  err_mem:
> -       kfree(mem);
> +       for (; i >= 0; i--) {
> +               mem = epc->mem[i];
> +               kfree(mem->bitmap);
> +               kfree(mem);
> +       }
> +       kfree(epc->mem);
>
> -err:
> -return ret;
> +       return ret;
>  }
>  EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
>
> @@ -101,11 +120,21 @@ EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
>   */
>  void pci_epc_mem_exit(struct pci_epc *epc)
>  {
> -       struct pci_epc_mem *mem = epc->mem;
> +       struct pci_epc_mem *mem;
> +       int i;
> +
> +       if (!epc->mem_windows)
> +               return;
> +
> +       for (i = 0; i <= epc->mem_windows; i++) {
> +               mem = epc->mem[i];
> +               kfree(mem->bitmap);
> +               kfree(mem);
> +       }
> +       kfree(epc->mem);
>
>         epc->mem = NULL;
> -       kfree(mem->bitmap);
> -       kfree(mem);
> +       epc->mem_windows = 0;
>  }
>  EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
>
> @@ -121,20 +150,30 @@ EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
>  void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
>                                      phys_addr_t *phys_addr, size_t size)
>  {
> -       int pageno;
> -       void __iomem *virt_addr;
> -       struct pci_epc_mem *mem = epc->mem;
> -       unsigned int page_shift = ilog2(mem->page_size);
> +       void __iomem *virt_addr = NULL;
> +       struct pci_epc_mem *mem;
> +       unsigned int page_shift;
> +       int pageno = -EINVAL;
>         int order;
> +       int i;
>
> -       size = ALIGN(size, mem->page_size);
> -       order = pci_epc_mem_get_order(mem, size);
> +       for (i = 0; i < epc->mem_windows; i++) {
> +               mem = epc->mem[i];
> +               size = ALIGN(size, mem->page_size);
> +               order = pci_epc_mem_get_order(mem, size);
> +
> +               pageno = bitmap_find_free_region(mem->bitmap, mem->pages,
> +                                                order);
> +               if (pageno >= 0)
> +                       break;
> +       }
>
> -       pageno = bitmap_find_free_region(mem->bitmap, mem->pages, order);
>         if (pageno < 0)
>                 return NULL;
>
> -       *phys_addr = mem->phys_base + ((phys_addr_t)pageno << page_shift);
> +       page_shift = ilog2(mem->page_size);
> +       *phys_addr = mem->window.phys_base +
> +                    ((phys_addr_t)pageno << page_shift);
>         virt_addr = ioremap(*phys_addr, size);
>         if (!virt_addr)
>                 bitmap_release_region(mem->bitmap, pageno, order);
> @@ -143,6 +182,22 @@ void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
>  }
>  EXPORT_SYMBOL_GPL(pci_epc_mem_alloc_addr);
>
> +struct pci_epc_mem *pci_epc_get_matching_window(struct pci_epc *epc,
> +                                               phys_addr_t phys_addr)
> +{
> +       struct pci_epc_mem *mem;
> +       int i;
> +
> +       for (i = 0; i < epc->mem_windows; i++) {
> +               mem = epc->mem[i];
> +
> +               if (mem->window.phys_base == phys_addr)
> +                       return mem;
> +       }
> +
> +       return NULL;
> +}
> +
>  /**
>   * pci_epc_mem_free_addr() - free the allocated memory address
>   * @epc: the EPC device on which memory was allocated
> @@ -155,13 +210,20 @@ EXPORT_SYMBOL_GPL(pci_epc_mem_alloc_addr);
>  void pci_epc_mem_free_addr(struct pci_epc *epc, phys_addr_t phys_addr,
>                            void __iomem *virt_addr, size_t size)
>  {
> +       struct pci_epc_mem *mem;
> +       unsigned int page_shift;
>         int pageno;
> -       struct pci_epc_mem *mem = epc->mem;
> -       unsigned int page_shift = ilog2(mem->page_size);
>         int order;
>
> +       mem = pci_epc_get_matching_window(epc, phys_addr);
> +       if (!mem) {
> +               pr_err("failed to get matching window\n");
> +               return;
> +       }
> +
> +       page_shift = ilog2(mem->page_size);
>         iounmap(virt_addr);
> -       pageno = (phys_addr - mem->phys_base) >> page_shift;
> +       pageno = (phys_addr - mem->window.phys_base) >> page_shift;
>         size = ALIGN(size, mem->page_size);
>         order = pci_epc_mem_get_order(mem, size);
>         bitmap_release_region(mem->bitmap, pageno, order);
> diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> index 56f1846..dde42e5 100644
> --- a/include/linux/pci-epc.h
> +++ b/include/linux/pci-epc.h
> @@ -64,17 +64,29 @@ struct pci_epc_ops {
>         struct module *owner;
>  };
>
> +#define PCI_EPC_DEFAULT_WINDOW         0
> +
> +/**
> + * struct pci_epc_mem_window - address window of the endpoint controller
> + * @phys_base: physical base address of the PCI address window
> + * @size: the size of the PCI address window
> + * @page_size: size of each page
> + */
> +struct pci_epc_mem_window {
> +       phys_addr_t     phys_base;
> +       size_t          size;
> +       size_t          page_size;
> +};
> +
>  /**
>   * struct pci_epc_mem - address space of the endpoint controller
> - * @phys_base: physical base address of the PCI address space
> - * @size: the size of the PCI address space
> + * @window: address window of the endpoint controller
>   * @bitmap: bitmap to manage the PCI address space
> - * @pages: number of bits representing the address region
>   * @page_size: size of each page
> + * @pages: number of bits representing the address region
>   */
>  struct pci_epc_mem {
> -       phys_addr_t     phys_base;
> -       size_t          size;
> +       struct pci_epc_mem_window window;
>         unsigned long   *bitmap;
>         size_t          page_size;
>         int             pages;
> @@ -85,7 +97,8 @@ struct pci_epc_mem {
>   * @dev: PCI EPC device
>   * @pci_epf: list of endpoint functions present in this EPC device
>   * @ops: function pointers for performing endpoint operations
> - * @mem: address space of the endpoint controller
> + * @mem: array of address space of the endpoint controller
> + * @mem_windows: number of windows supported by device
>   * @max_functions: max number of functions that can be configured in this EPC
>   * @group: configfs group representing the PCI EPC device
>   * @lock: spinlock to protect pci_epc ops
> @@ -94,7 +107,8 @@ struct pci_epc {
>         struct device                   dev;
>         struct list_head                pci_epf;
>         const struct pci_epc_ops        *ops;
> -       struct pci_epc_mem              *mem;
> +       struct pci_epc_mem              **mem;
> +       unsigned int                    mem_windows;
>         u8                              max_functions;
>         struct config_group             *group;
>         /* spinlock to protect against concurrent access of EP controller */
> @@ -128,8 +142,8 @@ struct pci_epc_features {
>  #define devm_pci_epc_create(dev, ops)    \
>                 __devm_pci_epc_create((dev), (ops), THIS_MODULE)
>
> -#define pci_epc_mem_init(epc, phys_addr, size) \
> -               __pci_epc_mem_init((epc), (phys_addr), (size), PAGE_SIZE)
> +#define pci_epc_mem_init(epc, windows, num_windows)    \
> +               __pci_epc_mem_init((epc), windows, num_windows)
>
>  static inline void epc_set_drvdata(struct pci_epc *epc, void *data)
>  {
> @@ -159,8 +173,7 @@ int pci_epc_set_bar(struct pci_epc *epc, u8 func_no,
>  void pci_epc_clear_bar(struct pci_epc *epc, u8 func_no,
>                        struct pci_epf_bar *epf_bar);
>  int pci_epc_map_addr(struct pci_epc *epc, u8 func_no,
> -                    phys_addr_t phys_addr,
> -                    u64 pci_addr, size_t size);
> +                    phys_addr_t phys_addr, u64 pci_addr, size_t size);
>  void pci_epc_unmap_addr(struct pci_epc *epc, u8 func_no,
>                         phys_addr_t phys_addr);
>  int pci_epc_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts);
> @@ -178,8 +191,8 @@ unsigned int pci_epc_get_first_free_bar(const struct pci_epc_features
>  struct pci_epc *pci_epc_get(const char *epc_name);
>  void pci_epc_put(struct pci_epc *epc);
>
> -int __pci_epc_mem_init(struct pci_epc *epc, phys_addr_t phys_addr, size_t size,
> -                      size_t page_size);
> +int __pci_epc_mem_init(struct pci_epc *epc, struct pci_epc_mem_window *window,
> +                      int num_windows);
>  void pci_epc_mem_exit(struct pci_epc *epc);
>  void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
>                                      phys_addr_t *phys_addr, size_t size);
> --
> 2.7.4
>
Kishon Vijay Abraham I Feb. 21, 2020, 11:40 a.m. UTC | #6
Hi Prabhakar,

On 09/02/20 12:06 am, Lad Prabhakar wrote:
> R-Car PCIe controller has support to map multiple memory regions for
> mapping the outbound memory in local system also the controller limits
> single allocation for each region (that is, once a chunk is used from the
> region it cannot be used to allocate a new one). This features inspires to
> add support for handling multiple memory bases in endpoint framework.
> 
> With this patch pci_epc_mem_init() now accepts multiple regions, also
> page_size for each memory region is passed during initialization so as
> to handle single allocation for each region by setting the page_size to
> window_size.

This patch looks much better now except for one comment below..
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
>  drivers/pci/controller/cadence/pcie-cadence-ep.c |   7 +-
>  drivers/pci/controller/dwc/pcie-designware-ep.c  |  29 ++--
>  drivers/pci/controller/pcie-rockchip-ep.c        |   7 +-
>  drivers/pci/endpoint/pci-epc-mem.c               | 166 ++++++++++++++++-------
>  include/linux/pci-epc.h                          |  39 ++++--
>  5 files changed, 168 insertions(+), 80 deletions(-)
> 
.
.
<snip>
.
.
> diff --git a/drivers/pci/endpoint/pci-epc-mem.c b/drivers/pci/endpoint/pci-epc-mem.c
> index d2b174c..b3eedee 100644
> --- a/drivers/pci/endpoint/pci-epc-mem.c
> +++ b/drivers/pci/endpoint/pci-epc-mem.c
> @@ -38,57 +38,76 @@ static int pci_epc_mem_get_order(struct pci_epc_mem *mem, size_t size)
>  /**
>   * __pci_epc_mem_init() - initialize the pci_epc_mem structure
>   * @epc: the EPC device that invoked pci_epc_mem_init
> - * @phys_base: the physical address of the base
> - * @size: the size of the address space
> - * @page_size: size of each page
> + * @windows: pointer to windows supported by the device
> + * @num_windows: number of windows device supports
>   *
>   * Invoke to initialize the pci_epc_mem structure used by the
>   * endpoint functions to allocate mapped PCI address.
>   */
> -int __pci_epc_mem_init(struct pci_epc *epc, phys_addr_t phys_base, size_t size,
> -		       size_t page_size)
> +int __pci_epc_mem_init(struct pci_epc *epc, struct pci_epc_mem_window *windows,
> +		       int num_windows)
>  {
> -	int ret;
> -	struct pci_epc_mem *mem;
> -	unsigned long *bitmap;
> +	struct pci_epc_mem *mem = NULL;
> +	unsigned long *bitmap = NULL;
>  	unsigned int page_shift;
> -	int pages;
> +	size_t page_size;
>  	int bitmap_size;
> -
> -	if (page_size < PAGE_SIZE)
> -		page_size = PAGE_SIZE;
> -
> -	page_shift = ilog2(page_size);
> -	pages = size >> page_shift;
> -	bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> -
> -	mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> -	if (!mem) {
> -		ret = -ENOMEM;
> -		goto err;
> -	}
> -
> -	bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> -	if (!bitmap) {
> -		ret = -ENOMEM;
> -		goto err_mem;
> +	int pages;
> +	int ret;
> +	int i;
> +
> +	epc->mem_windows = 0;
> +
> +	if (!windows)
> +		return -EINVAL;
> +
> +	if (num_windows <= 0)
> +		return -EINVAL;
> +
> +	epc->mem = kcalloc(num_windows, sizeof(*mem), GFP_KERNEL);
> +	if (!epc->mem)
> +		return -EINVAL;
> +
> +	for (i = 0; i < num_windows; i++) {
> +		page_size = windows[i].page_size;
> +		if (page_size < PAGE_SIZE)
> +			page_size = PAGE_SIZE;
> +		page_shift = ilog2(page_size);
> +		pages = windows[i].size >> page_shift;
> +		bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> +
> +		mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> +		if (!mem) {
> +			ret = -ENOMEM;
> +			goto err_mem;
> +		}
> +
> +		bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> +		if (!bitmap) {
> +			ret = -ENOMEM;
> +			goto err_mem;
> +		}
> +
> +		mem->bitmap = bitmap;
> +		mem->window.phys_base = windows[i].phys_base;
> +		mem->page_size = page_size;
> +		mem->pages = pages;
> +		mem->window.size = windows[i].size;
> +		epc->mem[i] = mem;
>  	}
> -
> -	mem->bitmap = bitmap;
> -	mem->phys_base = phys_base;
> -	mem->page_size = page_size;
> -	mem->pages = pages;
> -	mem->size = size;
> -
> -	epc->mem = mem;
> +	epc->mem_windows = num_windows;
>  
>  	return 0;
>  
>  err_mem:
> -	kfree(mem);
> +	for (; i >= 0; i--) {
> +		mem = epc->mem[i];
> +		kfree(mem->bitmap);
> +		kfree(mem);
> +	}
> +	kfree(epc->mem);
>  
> -err:
> -return ret;
> +	return ret;
>  }
>  EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
>  
> @@ -101,11 +120,21 @@ EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
>   */
>  void pci_epc_mem_exit(struct pci_epc *epc)
>  {
> -	struct pci_epc_mem *mem = epc->mem;
> +	struct pci_epc_mem *mem;
> +	int i;
> +
> +	if (!epc->mem_windows)
> +		return;
> +
> +	for (i = 0; i <= epc->mem_windows; i++) {
> +		mem = epc->mem[i];
> +		kfree(mem->bitmap);
> +		kfree(mem);
> +	}
> +	kfree(epc->mem);
>  
>  	epc->mem = NULL;
> -	kfree(mem->bitmap);
> -	kfree(mem);
> +	epc->mem_windows = 0;
>  }
>  EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
>  
> @@ -121,20 +150,30 @@ EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
>  void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
>  				     phys_addr_t *phys_addr, size_t size)
>  {
> -	int pageno;
> -	void __iomem *virt_addr;
> -	struct pci_epc_mem *mem = epc->mem;
> -	unsigned int page_shift = ilog2(mem->page_size);
> +	void __iomem *virt_addr = NULL;
> +	struct pci_epc_mem *mem;
> +	unsigned int page_shift;
> +	int pageno = -EINVAL;
>  	int order;
> +	int i;
>  
> -	size = ALIGN(size, mem->page_size);
> -	order = pci_epc_mem_get_order(mem, size);
> +	for (i = 0; i < epc->mem_windows; i++) {
> +		mem = epc->mem[i];
> +		size = ALIGN(size, mem->page_size);
> +		order = pci_epc_mem_get_order(mem, size);
> +
> +		pageno = bitmap_find_free_region(mem->bitmap, mem->pages,
> +						 order);
> +		if (pageno >= 0)
> +			break;
> +	}
>  
> -	pageno = bitmap_find_free_region(mem->bitmap, mem->pages, order);
>  	if (pageno < 0)
>  		return NULL;
>  
> -	*phys_addr = mem->phys_base + ((phys_addr_t)pageno << page_shift);
> +	page_shift = ilog2(mem->page_size);
> +	*phys_addr = mem->window.phys_base +
> +		     ((phys_addr_t)pageno << page_shift);
>  	virt_addr = ioremap(*phys_addr, size);
>  	if (!virt_addr)
>  		bitmap_release_region(mem->bitmap, pageno, order);
> @@ -143,6 +182,22 @@ void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
>  }
>  EXPORT_SYMBOL_GPL(pci_epc_mem_alloc_addr);
>  
> +struct pci_epc_mem *pci_epc_get_matching_window(struct pci_epc *epc,
> +						phys_addr_t phys_addr)
> +{
> +	struct pci_epc_mem *mem;
> +	int i;
> +
> +	for (i = 0; i < epc->mem_windows; i++) {
> +		mem = epc->mem[i];
> +
> +		if (mem->window.phys_base == phys_addr)
> +			return mem;

This will work only if the phys_addr is same as start of windows base.
This need not be true for all the platforms and will fail for all the
allocations except the first allocation.

Thanks
Kishon
Lad, Prabhakar Feb. 22, 2020, 1:32 p.m. UTC | #7
Hi Kishon,

Thank you for the review.

On Fri, Feb 21, 2020 at 11:36 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Prabhakar,
>
> On 09/02/20 12:06 am, Lad Prabhakar wrote:
> > R-Car PCIe controller has support to map multiple memory regions for
> > mapping the outbound memory in local system also the controller limits
> > single allocation for each region (that is, once a chunk is used from the
> > region it cannot be used to allocate a new one). This features inspires to
> > add support for handling multiple memory bases in endpoint framework.
> >
> > With this patch pci_epc_mem_init() now accepts multiple regions, also
> > page_size for each memory region is passed during initialization so as
> > to handle single allocation for each region by setting the page_size to
> > window_size.
>
> This patch looks much better now except for one comment below..
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > ---
> >  drivers/pci/controller/cadence/pcie-cadence-ep.c |   7 +-
> >  drivers/pci/controller/dwc/pcie-designware-ep.c  |  29 ++--
> >  drivers/pci/controller/pcie-rockchip-ep.c        |   7 +-
> >  drivers/pci/endpoint/pci-epc-mem.c               | 166 ++++++++++++++++-------
> >  include/linux/pci-epc.h                          |  39 ++++--
> >  5 files changed, 168 insertions(+), 80 deletions(-)
> >
> .
> .
> <snip>
> .
> .
> > diff --git a/drivers/pci/endpoint/pci-epc-mem.c b/drivers/pci/endpoint/pci-epc-mem.c
> > index d2b174c..b3eedee 100644
> > --- a/drivers/pci/endpoint/pci-epc-mem.c
> > +++ b/drivers/pci/endpoint/pci-epc-mem.c
> > @@ -38,57 +38,76 @@ static int pci_epc_mem_get_order(struct pci_epc_mem *mem, size_t size)
> >  /**
> >   * __pci_epc_mem_init() - initialize the pci_epc_mem structure
> >   * @epc: the EPC device that invoked pci_epc_mem_init
> > - * @phys_base: the physical address of the base
> > - * @size: the size of the address space
> > - * @page_size: size of each page
> > + * @windows: pointer to windows supported by the device
> > + * @num_windows: number of windows device supports
> >   *
> >   * Invoke to initialize the pci_epc_mem structure used by the
> >   * endpoint functions to allocate mapped PCI address.
> >   */
> > -int __pci_epc_mem_init(struct pci_epc *epc, phys_addr_t phys_base, size_t size,
> > -                    size_t page_size)
> > +int __pci_epc_mem_init(struct pci_epc *epc, struct pci_epc_mem_window *windows,
> > +                    int num_windows)
> >  {
> > -     int ret;
> > -     struct pci_epc_mem *mem;
> > -     unsigned long *bitmap;
> > +     struct pci_epc_mem *mem = NULL;
> > +     unsigned long *bitmap = NULL;
> >       unsigned int page_shift;
> > -     int pages;
> > +     size_t page_size;
> >       int bitmap_size;
> > -
> > -     if (page_size < PAGE_SIZE)
> > -             page_size = PAGE_SIZE;
> > -
> > -     page_shift = ilog2(page_size);
> > -     pages = size >> page_shift;
> > -     bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> > -
> > -     mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> > -     if (!mem) {
> > -             ret = -ENOMEM;
> > -             goto err;
> > -     }
> > -
> > -     bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> > -     if (!bitmap) {
> > -             ret = -ENOMEM;
> > -             goto err_mem;
> > +     int pages;
> > +     int ret;
> > +     int i;
> > +
> > +     epc->mem_windows = 0;
> > +
> > +     if (!windows)
> > +             return -EINVAL;
> > +
> > +     if (num_windows <= 0)
> > +             return -EINVAL;
> > +
> > +     epc->mem = kcalloc(num_windows, sizeof(*mem), GFP_KERNEL);
> > +     if (!epc->mem)
> > +             return -EINVAL;
> > +
> > +     for (i = 0; i < num_windows; i++) {
> > +             page_size = windows[i].page_size;
> > +             if (page_size < PAGE_SIZE)
> > +                     page_size = PAGE_SIZE;
> > +             page_shift = ilog2(page_size);
> > +             pages = windows[i].size >> page_shift;
> > +             bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
> > +
> > +             mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> > +             if (!mem) {
> > +                     ret = -ENOMEM;
> > +                     goto err_mem;
> > +             }
> > +
> > +             bitmap = kzalloc(bitmap_size, GFP_KERNEL);
> > +             if (!bitmap) {
> > +                     ret = -ENOMEM;
> > +                     goto err_mem;
> > +             }
> > +
> > +             mem->bitmap = bitmap;
> > +             mem->window.phys_base = windows[i].phys_base;
> > +             mem->page_size = page_size;
> > +             mem->pages = pages;
> > +             mem->window.size = windows[i].size;
> > +             epc->mem[i] = mem;
> >       }
> > -
> > -     mem->bitmap = bitmap;
> > -     mem->phys_base = phys_base;
> > -     mem->page_size = page_size;
> > -     mem->pages = pages;
> > -     mem->size = size;
> > -
> > -     epc->mem = mem;
> > +     epc->mem_windows = num_windows;
> >
> >       return 0;
> >
> >  err_mem:
> > -     kfree(mem);
> > +     for (; i >= 0; i--) {
> > +             mem = epc->mem[i];
> > +             kfree(mem->bitmap);
> > +             kfree(mem);
> > +     }
> > +     kfree(epc->mem);
> >
> > -err:
> > -return ret;
> > +     return ret;
> >  }
> >  EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
> >
> > @@ -101,11 +120,21 @@ EXPORT_SYMBOL_GPL(__pci_epc_mem_init);
> >   */
> >  void pci_epc_mem_exit(struct pci_epc *epc)
> >  {
> > -     struct pci_epc_mem *mem = epc->mem;
> > +     struct pci_epc_mem *mem;
> > +     int i;
> > +
> > +     if (!epc->mem_windows)
> > +             return;
> > +
> > +     for (i = 0; i <= epc->mem_windows; i++) {
> > +             mem = epc->mem[i];
> > +             kfree(mem->bitmap);
> > +             kfree(mem);
> > +     }
> > +     kfree(epc->mem);
> >
> >       epc->mem = NULL;
> > -     kfree(mem->bitmap);
> > -     kfree(mem);
> > +     epc->mem_windows = 0;
> >  }
> >  EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
> >
> > @@ -121,20 +150,30 @@ EXPORT_SYMBOL_GPL(pci_epc_mem_exit);
> >  void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
> >                                    phys_addr_t *phys_addr, size_t size)
> >  {
> > -     int pageno;
> > -     void __iomem *virt_addr;
> > -     struct pci_epc_mem *mem = epc->mem;
> > -     unsigned int page_shift = ilog2(mem->page_size);
> > +     void __iomem *virt_addr = NULL;
> > +     struct pci_epc_mem *mem;
> > +     unsigned int page_shift;
> > +     int pageno = -EINVAL;
> >       int order;
> > +     int i;
> >
> > -     size = ALIGN(size, mem->page_size);
> > -     order = pci_epc_mem_get_order(mem, size);
> > +     for (i = 0; i < epc->mem_windows; i++) {
> > +             mem = epc->mem[i];
> > +             size = ALIGN(size, mem->page_size);
> > +             order = pci_epc_mem_get_order(mem, size);
> > +
> > +             pageno = bitmap_find_free_region(mem->bitmap, mem->pages,
> > +                                              order);
> > +             if (pageno >= 0)
> > +                     break;
> > +     }
> >
> > -     pageno = bitmap_find_free_region(mem->bitmap, mem->pages, order);
> >       if (pageno < 0)
> >               return NULL;
> >
> > -     *phys_addr = mem->phys_base + ((phys_addr_t)pageno << page_shift);
> > +     page_shift = ilog2(mem->page_size);
> > +     *phys_addr = mem->window.phys_base +
> > +                  ((phys_addr_t)pageno << page_shift);
> >       virt_addr = ioremap(*phys_addr, size);
> >       if (!virt_addr)
> >               bitmap_release_region(mem->bitmap, pageno, order);
> > @@ -143,6 +182,22 @@ void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
> >  }
> >  EXPORT_SYMBOL_GPL(pci_epc_mem_alloc_addr);
> >
> > +struct pci_epc_mem *pci_epc_get_matching_window(struct pci_epc *epc,
> > +                                             phys_addr_t phys_addr)
> > +{
> > +     struct pci_epc_mem *mem;
> > +     int i;
> > +
> > +     for (i = 0; i < epc->mem_windows; i++) {
> > +             mem = epc->mem[i];
> > +
> > +             if (mem->window.phys_base == phys_addr)
> > +                     return mem;
>
> This will work only if the phys_addr is same as start of windows base.
> This need not be true for all the platforms and will fail for all the
> allocations except the first allocation.
>
Agreed, this worked for me because different windows were used for allocation.
If you are OK with below changes Ill post a V5 soon.

for (i = 0; i < epc->mem_windows; i++) {
    mem = epc->mem[i];

    if (phys_addr >= mem->window.phys_base &&
        phys_addr < (mem->window.phys_base + mem->window.size))
        return mem;
....
...
}

Cheers,
--Prabhakar Lad

> Thanks
> Kishon