[RESEND:,RFC,3/3] pcie: keystone: add pcie driver based on designware core driver

keystone pcie hardware is based on designware version 3.65. This
doesn't support standard MSI controller and ATU port available
on newer version of the designware hardware. In keystone dw hardware
the application register space implements the legacy and MSI
irq controller registers and has registers to allow outbound and
inbound translations. So the outbound/inbound configuration is
done by the keystone pcie driver.

Driver needs to handle irq as there are EOI registers available to
ack the interrupt after the same is acked by the end point device
driver. This requires irqchip implementation for legacy and MSI
irq handling. The driver however re-uses functions from dw core
wherever possible by making their prototype available in the dw
header file and calling the same from the keystone pcie driver.

Also add dts binding documentation for configuration details.

CC: Jingoo Han <jg1.han@samsung.com>
CC: Bjorn Helgaas <bhelgaas@google.com>
CC: Kukjin Kim <kgene.kim@samsung.com>
CC: Richard Zhu <r65037@freescale.com>
CC: Shawn Guo <shawn.guo@linaro.org>
CC: Mohit Kumar <mohit.kumar@st.com>
CC: Santosh Shilimkar <santosh.shilimkar@ti.com>

Signed-off-by: Murali Karicheri <m-karicheri2@ti.com>
---
 .../devicetree/bindings/pci/pcie-keystone.txt      |   32 +
 drivers/pci/host/Kconfig                           |    5 +
 drivers/pci/host/Makefile                          |    1 +
 drivers/pci/host/k2-platform.c                     |  191 +++++
 drivers/pci/host/pcie-keystone.c                   |  860 ++++++++++++++++++++
 drivers/pci/host/pcie-ks-pdata.h                   |   19 +
 drivers/pci/quirks.c                               |   12 +
 7 files changed, 1120 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/pcie-keystone.txt
 create mode 100644 drivers/pci/host/k2-platform.c
 create mode 100644 drivers/pci/host/pcie-keystone.c
 create mode 100644 drivers/pci/host/pcie-ks-pdata.h

On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
> +
> +int k2_pcie_platform_setup(struct platform_device *pdev)
> +{
> +	struct resource *phy_base_r, *devstat_r;
> +	void __iomem *phy_base, *devstat;
> +	u32 val;
> +	int i;
> +
> +	devstat_r = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> +						 "reg_devcfg");
> +	if (!devstat_r)
> +		return -ENODEV;

It seems you have a distinct register set for the PHY that you are
driving here. Why not make this part generic PHY API driver?

> +static int ks_pcie_establish_link(struct keystone_pcie *ks_pcie)
> +{
> +	struct pcie_port *pp = &ks_pcie->pp;
> +	int count = 0;
> +
> +	dw_pcie_setup_rc(pp);
> +
> +	/* check if the link is up or not */
> +	while (!dw_pcie_link_up(pp)) {
> +		mdelay(100);
> +		count++;
> +		if (count == 10)
> +			return -EINVAL;
> +	}
> +	return 0;
> +}

You are blocking the CPU for up to one second here, which is really
nasty. Please find a way to move the code into a context where you
can sleep and use msleep() instead, or better use an interrupt if the
hardware supports that and use wait_for_completion().

> +
> +static void ks_pcie_msi_irq_handler(unsigned int irq, struct irq_desc *desc)
> +{
> +	struct keystone_pcie *ks_pcie = irq_desc_get_handler_data(desc);
> +	u32 offset = irq - ks_pcie->msi_host_irqs[0], pending, vector;
> +	struct pcie_port *pp = &ks_pcie->pp;
> +	struct irq_chip *chip = irq_desc_get_chip(desc);
> +	int src, virq;

Did I understand this right that the MSI implementation can be used
by any dw-pcie hardware of the same version? If so, it would be good
to move it into an extra file so it can be shared with the next host
driver that uses this version.

> +/**
> + * ks_pcie_set_outbound_trans() - Set PHYADDR <-> BUSADDR
> + * mapping for outbound
> + */
> +static void ks_pcie_set_outbound_trans(struct keystone_pcie *ks_pcie)

> +static void ks_pcie_set_inbound_trans(struct pcie_port *pp)

Why do you need to set this up from the kernel? Please move the
translation window setup into the boot loader if you can.

> +static struct hw_pci keystone_pcie_hw = {
> +	.nr_controllers	= 1,
> +	.setup		= dw_pcie_setup,
> +	.scan		= ks_pcie_scan_bus,
> +	.swizzle        = pci_common_swizzle,
> +	.add_bus	= dw_pcie_add_bus,
> +	.map_irq	= ks_pcie_map_irq,
> +};

This can just be a local variable in the probe function.

> +
> +static int ks_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
> +{
> +	struct pcie_port *pp = sys_to_pcie(dev->bus->sysdata);
> +	struct keystone_pcie *ks_pcie = to_keystone_pcie(pp);
> +
> +	dev_info(pp->dev, "ks_pcie_map_irq: pin %d\n", pin);
> +
> +	if (!pin || pin > ks_pcie->num_legacy_host_irqs) {
> +		dev_err(pp->dev, "pci irq pin out of range\n");
> +		return -1;
> +	}
> +
> +	/* pin has values from 1-4 */
> +	return (ks_pcie->virqs[pin - 1] >= 0) ?
> +		ks_pcie->virqs[pin - 1] : -1;
> +}
> +
> +
> +static void ack_irq(struct irq_data *d)
> +{
> +}
> +
> +static void mask_irq(struct irq_data *d)
> +{
> +}
> +
> +static void unmask_irq(struct irq_data *d)
> +{
> +}
> +
> +static struct irq_chip ks_pcie_legacy_irq_chip = {
> +	.name = "PCIe-LEGACY-IRQ",
> +	.irq_ack = ack_irq,
> +	.irq_mask = mask_irq,
> +	.irq_unmask = unmask_irq,
> +};

Can you use the generic irqchip code here?

> +	/*
> +	 * support upto MAX_LEGACY_HOST_IRQS host and legacy IRQs.
> +	 * In dt from index 0 to 3
> +	 */
> +	for (i = 0; i < MAX_LEGACY_HOST_IRQS; i++) {
> +		ks_pcie->legacy_host_irqs[i] = irq_of_parse_and_map(np, i);
> +		if (ks_pcie->legacy_host_irqs[i] < 0)
> +			break;
> +		ks_pcie->num_legacy_host_irqs++;
> +	}
> +
> +	if (ks_pcie->num_legacy_host_irqs) {
> +		ks_pcie->legacy_irqd = irq_domain_add_linear(np,
> +					ks_pcie->num_legacy_host_irqs,
> +					&irq_domain_simple_ops, NULL);
> +
> +		if (!ks_pcie->legacy_irqd) {
> +			dev_err(dev,
> +				"Failed to add irq domain for legacy irqs\n");
> +			return -EINVAL;
> +		}
> +	}
> +
> +	for (i = 0; i < ks_pcie->num_legacy_host_irqs; i++) {
> +		ks_pcie->virqs[i] =
> +			irq_create_mapping(ks_pcie->legacy_irqd, i);
> +		irq_set_chip_and_handler(ks_pcie->virqs[i],
> +			&ks_pcie_legacy_irq_chip, handle_level_irq);
> +		irq_set_chip_data(ks_pcie->virqs[i], ks_pcie);
> +		set_irq_flags(ks_pcie->virqs[i], IRQF_VALID);
> +
> +		if (ks_pcie->legacy_host_irqs[i] >= 0) {
> +			irq_set_handler_data(ks_pcie->legacy_host_irqs[i],
> +				ks_pcie);
> +			irq_set_chained_handler(ks_pcie->legacy_host_irqs[i],
> +				ks_pcie_legacy_irq_handler);
> +		}
> +		set_irq_flags(ks_pcie->legacy_host_irqs[i], IRQF_VALID);
> +	}

I have no idea how this would work with the standard interrupt-map property,
since the legacy interrupt host is now the same device as the pci host.

Maybe it's better to move the legacy irqchip handling entirely out of
the driver and use a separate device node for the registers so it can
come with its own #interrupt-cells, and then refer to this irqchip from
the interrupt-map.

> +	ks_pcie->clk = devm_clk_get(&pdev->dev, "pcie");
> +	if (IS_ERR(ks_pcie->clk)) {
> +		dev_err(&pdev->dev, "Failed to get pcie rc clock\n");
> +		return PTR_ERR(ks_pcie->clk);
> +	}
> +	ret = clk_prepare_enable(ks_pcie->clk);
> +	if (ret)
> +		return ret;

Could you move the clock handling into the generic dw-pcie code?

> +
> +/* Keystone PCIe driver does not allow module unload */
> +static int __init ks_pcie_init(void)
> +{
> +	return platform_driver_probe(&ks_pcie_driver, ks_pcie_probe);
> +}
> +subsys_initcall(ks_pcie_init);

Why subsys_initcall?

We should probably try to fix unloading soon.

> diff --git a/drivers/pci/host/pcie-ks-pdata.h b/drivers/pci/host/pcie-ks-pdata.h
> new file mode 100644
> index 0000000..a7626de
> --- /dev/null
> +++ b/drivers/pci/host/pcie-ks-pdata.h
> @@ -0,0 +1,19 @@
> +/*
> + * Copyright (C) 2013-2014 Texas Instruments., Ltd.
> + *		http://www.ti.com
> + *
> + * Author: Murali Karicheri <m-karicheri2@ti.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +/* platform specific setup for k2 pcie */
> +int k2_pcie_platform_setup(struct platform_device *pdev);
> +
> +/* keystone pcie pdata configurations */
> +struct keystone_pcie_pdata {
> +	int (*setup)(struct platform_device *pdev);
> +};

This should all go away once you have a proper PHY driver.

> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 3a02717..7c3f777 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -3461,3 +3461,15 @@ int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags)
>  
>  	return -ENOTTY;
>  }
> +
> +#ifdef CONFIG_PCIE_KEYSTONE
> +/*
> + * The KeyStone PCIe controller has maximum read request size of 256 bytes.
> + * Force this configuration for all EP including bridges.
> + */
> +static void quirk_limit_readrequest(struct pci_dev *dev)
> +{
> +	pcie_set_readrq(dev, 256);
> +}
> +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, quirk_limit_readrequest);
> +#endif /* CONFIG_TI_KEYSTONE_PCIE */

You can't do this:

A quirk for a specific hardware must not be enabled by compile-time options.
You have to find a different way to do this.

	Arnd

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Mar 25, 2014 at 08:44:36AM +0100, Arnd Bergmann wrote:
> On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
[...]
> > +/* Keystone PCIe driver does not allow module unload */
> > +static int __init ks_pcie_init(void)
> > +{
> > +	return platform_driver_probe(&ks_pcie_driver, ks_pcie_probe);
> > +}
> > +subsys_initcall(ks_pcie_init);
> 
> Why subsys_initcall?
> 
> We should probably try to fix unloading soon.

I did some work on this a few months ago but never got around to
cleaning up the patches. Let me see if I can resurrect that work.

Thierry

On 25 March 2014 10:35, Thierry Reding <thierry.reding@gmail.com> wrote:
> On Tue, Mar 25, 2014 at 08:44:36AM +0100, Arnd Bergmann wrote:
>> On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
> [...]
>> > +/* Keystone PCIe driver does not allow module unload */
>> > +static int __init ks_pcie_init(void)
>> > +{
>> > +   return platform_driver_probe(&ks_pcie_driver, ks_pcie_probe);
>> > +}
>> > +subsys_initcall(ks_pcie_init);
>>
>> Why subsys_initcall?
>>
>> We should probably try to fix unloading soon.
>
> I did some work on this a few months ago but never got around to
> cleaning up the patches. Let me see if I can resurrect that work.

I think there may be merit in these drivers using subsys_init. I've
not had time to investigate, but as far as I can remember this causes
issues with piceport.

For ARM32 host drivers, pci_fixup_irqs (arch/arm/kernel/bios32.c) must
be called before init_service_irqs (portdrv_core.c) otherwise pcieport
acts on invalid information in dev->irq and breaks. It seems that its
possible for the portbus driver to pick up new devices before bios32
has been able to fixup the irqs. Making the host bridge drivers subsys
will overcome this. I guess this hasn't been an issue in the past as
host bridge drivers were always in the arch/ directories.

In any case it may be worth testing this driver with PCIEPORTBUS enabled.

Andrew Murray
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Mar 25, 2014 at 08:44:36AM +0100, Arnd Bergmann wrote:

> I have no idea how this would work with the standard interrupt-map property,
> since the legacy interrupt host is now the same device as the pci host.
> 
> Maybe it's better to move the legacy irqchip handling entirely out of
> the driver and use a separate device node for the registers so it can
> come with its own #interrupt-cells, and then refer to this irqchip from
> the interrupt-map.

The other DW PCI-E drivers are being fixed to use interrupt-map, so I
think this driver should be fixed before it goes in as well.

Other PCI-E hosts have handled this particular problem with a
construction like this:

       pcie@21800000 {
               compatible = "ti,keystone2-pcie";
               device_type = "pci";
               #address-cells = <3>;
               #size-cells = <2>;
               #interrupt-cells = <1>;
	       reg = ..

               pcie_intc: interrupt-controller {
                  interrupt-controller;
		  #address-cells = <0>;
                  #interrupt-cells = <1>;
               }

               interrupts =  ... enough to decode INTx;

	       interrtup-map = <0 0 0 1 &pcie_intc 1>, // INT A
	                       <0 0 0 2 &pcie_intc 2>, // INT B
                               <0 0 0 3 &pcie_intc 3>, // INT C
                               <0 0 0 4 &pcie_intc 4>; // INT D
       }

When the legacy irq domain is created it should be bound to the
pcie_intc node, not to pcie node.

ks_pcie_map_irq should be removed.

> > +#ifdef CONFIG_PCIE_KEYSTONE
> > +/*
> > + * The KeyStone PCIe controller has maximum read request size of 256 bytes.
> > + * Force this configuration for all EP including bridges.
> > + */
> > +static void quirk_limit_readrequest(struct pci_dev *dev)
> > +{
> > +	pcie_set_readrq(dev, 256);
> > +}
> > +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, quirk_limit_readrequest);
> > +#endif /* CONFIG_TI_KEYSTONE_PCIE */
> 
> You can't do this:
> 
> A quirk for a specific hardware must not be enabled by compile-time options.
> You have to find a different way to do this.

This configuration should happen automatically by the PCI core if the
config spaces are correct. See pcie_write_mrrs, pcie_write_mps,
pcie_bus_configure_settings, etc.

Your host will need to conform to the PCI spec and provide a root
port bridge header for this to work.

Also, even in the keystone case the MRRS should not be globally forced
like this, there may be other bridges in the system with a 128 byte
MPS.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

>-----Original Message-----
>From: Arnd Bergmann [mailto:arnd@arndb.de]
>Sent: Tuesday, March 25, 2014 3:45 AM
>To: linux-arm-kernel@lists.infradead.org
>Cc: Karicheri, Muralidharan; linux-pci@vger.kernel.org; linux-samsung-
>soc@vger.kernel.org; linux-kernel@vger.kernel.org; Richard Zhu; Kukjin Kim; Mohit
>Kumar; Jingoo Han; Shilimkar, Santosh; Bjorn Helgaas; Shawn Guo
>Subject: Re: [RESEND: RFC PATCH 3/3] pcie: keystone: add pcie driver based on
>designware core driver
>
>On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
>> +
>> +int k2_pcie_platform_setup(struct platform_device *pdev) {
>> +	struct resource *phy_base_r, *devstat_r;
>> +	void __iomem *phy_base, *devstat;
>> +	u32 val;
>> +	int i;
>> +
>> +	devstat_r = platform_get_resource_byname(pdev, IORESOURCE_MEM,
>> +						 "reg_devcfg");
>> +	if (!devstat_r)
>> +		return -ENODEV;
>
>It seems you have a distinct register set for the PHY that you are driving here. Why not
>make this part generic PHY API driver?
>
>> +static int ks_pcie_establish_link(struct keystone_pcie *ks_pcie) {
>> +	struct pcie_port *pp = &ks_pcie->pp;
>> +	int count = 0;
>> +
>> +	dw_pcie_setup_rc(pp);
>> +
>> +	/* check if the link is up or not */
>> +	while (!dw_pcie_link_up(pp)) {
>> +		mdelay(100);
>> +		count++;
>> +		if (count == 10)
>> +			return -EINVAL;
>> +	}
>> +	return 0;
>> +}
>
>You are blocking the CPU for up to one second here, which is really nasty. Please find a way
>to move the code into a context where you can sleep and use msleep() instead, or better
>use an interrupt if the hardware supports that and use wait_for_completion().
>
>> +
>> +static void ks_pcie_msi_irq_handler(unsigned int irq, struct irq_desc
>> +*desc) {
>> +	struct keystone_pcie *ks_pcie = irq_desc_get_handler_data(desc);
>> +	u32 offset = irq - ks_pcie->msi_host_irqs[0], pending, vector;
>> +	struct pcie_port *pp = &ks_pcie->pp;
>> +	struct irq_chip *chip = irq_desc_get_chip(desc);
>> +	int src, virq;
>
>Did I understand this right that the MSI implementation can be used by any dw-pcie
>hardware of the same version? If so, it would be good to move it into an extra file so it can
>be shared with the next host driver that uses this version.
>
>> +/**
>> + * ks_pcie_set_outbound_trans() - Set PHYADDR <-> BUSADDR
>> + * mapping for outbound
>> + */
>> +static void ks_pcie_set_outbound_trans(struct keystone_pcie *ks_pcie)
>
>> +static void ks_pcie_set_inbound_trans(struct pcie_port *pp)
>
>Why do you need to set this up from the kernel? Please move the translation window setup
>into the boot loader if you can.
>
>> +static struct hw_pci keystone_pcie_hw = {
>> +	.nr_controllers	= 1,
>> +	.setup		= dw_pcie_setup,
>> +	.scan		= ks_pcie_scan_bus,
>> +	.swizzle        = pci_common_swizzle,
>> +	.add_bus	= dw_pcie_add_bus,
>> +	.map_irq	= ks_pcie_map_irq,
>> +};
>
>This can just be a local variable in the probe function.
>
>> +
>> +static int ks_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8
>> +pin) {
>> +	struct pcie_port *pp = sys_to_pcie(dev->bus->sysdata);
>> +	struct keystone_pcie *ks_pcie = to_keystone_pcie(pp);
>> +
>> +	dev_info(pp->dev, "ks_pcie_map_irq: pin %d\n", pin);
>> +
>> +	if (!pin || pin > ks_pcie->num_legacy_host_irqs) {
>> +		dev_err(pp->dev, "pci irq pin out of range\n");
>> +		return -1;
>> +	}
>> +
>> +	/* pin has values from 1-4 */
>> +	return (ks_pcie->virqs[pin - 1] >= 0) ?
>> +		ks_pcie->virqs[pin - 1] : -1;
>> +}
>> +
>> +
>> +static void ack_irq(struct irq_data *d) { }
>> +
>> +static void mask_irq(struct irq_data *d) { }
>> +
>> +static void unmask_irq(struct irq_data *d) { }
>> +
>> +static struct irq_chip ks_pcie_legacy_irq_chip = {
>> +	.name = "PCIe-LEGACY-IRQ",
>> +	.irq_ack = ack_irq,
>> +	.irq_mask = mask_irq,
>> +	.irq_unmask = unmask_irq,
>> +};
>
>Can you use the generic irqchip code here?
>
>> +	/*
>> +	 * support upto MAX_LEGACY_HOST_IRQS host and legacy IRQs.
>> +	 * In dt from index 0 to 3
>> +	 */
>> +	for (i = 0; i < MAX_LEGACY_HOST_IRQS; i++) {
>> +		ks_pcie->legacy_host_irqs[i] = irq_of_parse_and_map(np, i);
>> +		if (ks_pcie->legacy_host_irqs[i] < 0)
>> +			break;
>> +		ks_pcie->num_legacy_host_irqs++;
>> +	}
>> +
>> +	if (ks_pcie->num_legacy_host_irqs) {
>> +		ks_pcie->legacy_irqd = irq_domain_add_linear(np,
>> +					ks_pcie->num_legacy_host_irqs,
>> +					&irq_domain_simple_ops, NULL);
>> +
>> +		if (!ks_pcie->legacy_irqd) {
>> +			dev_err(dev,
>> +				"Failed to add irq domain for legacy irqs\n");
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	for (i = 0; i < ks_pcie->num_legacy_host_irqs; i++) {
>> +		ks_pcie->virqs[i] =
>> +			irq_create_mapping(ks_pcie->legacy_irqd, i);
>> +		irq_set_chip_and_handler(ks_pcie->virqs[i],
>> +			&ks_pcie_legacy_irq_chip, handle_level_irq);
>> +		irq_set_chip_data(ks_pcie->virqs[i], ks_pcie);
>> +		set_irq_flags(ks_pcie->virqs[i], IRQF_VALID);
>> +
>> +		if (ks_pcie->legacy_host_irqs[i] >= 0) {
>> +			irq_set_handler_data(ks_pcie->legacy_host_irqs[i],
>> +				ks_pcie);
>> +			irq_set_chained_handler(ks_pcie->legacy_host_irqs[i],
>> +				ks_pcie_legacy_irq_handler);
>> +		}
>> +		set_irq_flags(ks_pcie->legacy_host_irqs[i], IRQF_VALID);
>> +	}
>
>I have no idea how this would work with the standard interrupt-map property, since the
>legacy interrupt host is now the same device as the pci host.
>
>Maybe it's better to move the legacy irqchip handling entirely out of the driver and use a
>separate device node for the registers so it can come with its own #interrupt-cells, and then
>refer to this irqchip from the interrupt-map.
>
>> +	ks_pcie->clk = devm_clk_get(&pdev->dev, "pcie");
>> +	if (IS_ERR(ks_pcie->clk)) {
>> +		dev_err(&pdev->dev, "Failed to get pcie rc clock\n");
>> +		return PTR_ERR(ks_pcie->clk);
>> +	}
>> +	ret = clk_prepare_enable(ks_pcie->clk);
>> +	if (ret)
>> +		return ret;
>
>Could you move the clock handling into the generic dw-pcie code?
>
>> +
>> +/* Keystone PCIe driver does not allow module unload */ static int
>> +__init ks_pcie_init(void) {
>> +	return platform_driver_probe(&ks_pcie_driver, ks_pcie_probe); }
>> +subsys_initcall(ks_pcie_init);
>
>Why subsys_initcall?
>
>We should probably try to fix unloading soon.
>
>> diff --git a/drivers/pci/host/pcie-ks-pdata.h
>> b/drivers/pci/host/pcie-ks-pdata.h
>> new file mode 100644
>> index 0000000..a7626de
>> --- /dev/null
>> +++ b/drivers/pci/host/pcie-ks-pdata.h
>> @@ -0,0 +1,19 @@
>> +/*
>> + * Copyright (C) 2013-2014 Texas Instruments., Ltd.
>> + *		http://www.ti.com
>> + *
>> + * Author: Murali Karicheri <m-karicheri2@ti.com>
>> + *
>> + * This program is free software; you can redistribute it and/or
>> +modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + */
>> +
>> +/* platform specific setup for k2 pcie */ int
>> +k2_pcie_platform_setup(struct platform_device *pdev);
>> +
>> +/* keystone pcie pdata configurations */ struct keystone_pcie_pdata {
>> +	int (*setup)(struct platform_device *pdev); };
>
>This should all go away once you have a proper PHY driver.
>
>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index
>> 3a02717..7c3f777 100644
>> --- a/drivers/pci/quirks.c
>> +++ b/drivers/pci/quirks.c
>> @@ -3461,3 +3461,15 @@ int pci_dev_specific_acs_enabled(struct pci_dev
>> *dev, u16 acs_flags)
>>
>>  	return -ENOTTY;
>>  }
>> +
>> +#ifdef CONFIG_PCIE_KEYSTONE
>> +/*
>> + * The KeyStone PCIe controller has maximum read request size of 256 bytes.
>> + * Force this configuration for all EP including bridges.
>> + */
>> +static void quirk_limit_readrequest(struct pci_dev *dev) {
>> +	pcie_set_readrq(dev, 256);
>> +}
>> +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID,
>> +quirk_limit_readrequest); #endif /* CONFIG_TI_KEYSTONE_PCIE */
>
>You can't do this:
>
>A quirk for a specific hardware must not be enabled by compile-time options.
>You have to find a different way to do this.
>
>	Arnd
All,

Thanks for the comments. I will review them when I get a chance and respond.

Murali Karicheri
Linux Kernel, Software Development


--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 2 April 2014 16:43, Murali Karicheri <m-karicheri2@ti.com> wrote:

> Keystone pcie driver is developed based on other dw based pcie drivers
>
> such as pci-exynos that uses subsys_initcall(). I am new to this list,
>
> probably Jingoo (copied) has some history on why we can't use module.
> For now I will keep it as is and can be re-visited in the next revisions.
> Also I will experiment with PCIE port driver as well.
>
>
> BTW, PCIE driver currently uses Legacy or MSI IRQ. Keystone PCI has
> a platform IRQ. Is DT based irq configuration is the appropriate way
> to add this capability?

As far as I am aware - the PCI standards define a particular way for
devices to describe which interrupt will be used for things like
hotplug, AER and PME. These interrupts are always PCI interrupts (i.e.
MSI/MSI-X/legacy). Thus the port services code in the kernel uses
standard configuration space accesses to determine the interrupt to
use. Also note that it's not just the host bridge that can provide
these services but any PCIE device, I guess in this sense a host
bridge is treated like any other device.

If my understanding is correct I don't believe the current port
services code allows exceptions to this, i.e. to say this host bridge
actually uses a platform IRQ for AER rather than an MSI. Though this
may be quite useful as I suspect many host bridges provide interrupts
for things like PME through platform IRQs rather that PCI interrupts.

Does the Keystone have platform IRQs for things like AER? Is that
because the IP makes these events available through platform IRQs in
addition to the standard PCI means?

Andrew Murray
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Arnd,

Thanks for reviewing the RFC patch. Please see below my response.

On 3/25/2014 3:44 AM, Arnd Bergmann wrote:
> On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
>> +
>> +int k2_pcie_platform_setup(struct platform_device *pdev)
>> +{
>> +	struct resource *phy_base_r, *devstat_r;
>> +	void __iomem *phy_base, *devstat;
>> +	u32 val;
>> +	int i;
>> +
>> +	devstat_r = platform_get_resource_byname(pdev, IORESOURCE_MEM,
>> +						 "reg_devcfg");
>> +	if (!devstat_r)
>> +		return -ENODEV;
> It seems you have a distinct register set for the PHY that you are
> driving here. Why not make this part generic PHY API driver?

Will investigate

>> +static int ks_pcie_establish_link(struct keystone_pcie *ks_pcie)
>> +{
>> +	struct pcie_port *pp = &ks_pcie->pp;
>> +	int count = 0;
>> +
>> +	dw_pcie_setup_rc(pp);
>> +
>> +	/* check if the link is up or not */
>> +	while (!dw_pcie_link_up(pp)) {
>> +		mdelay(100);
>> +		count++;
>> +		if (count == 10)
>> +			return -EINVAL;
>> +	}
>> +	return 0;
>> +}
> You are blocking the CPU for up to one second here, which is really
> nasty. Please find a way to move the code into a context where you
> can sleep and use msleep() instead, or better use an interrupt if the
> hardware supports that and use wait_for_completion().
Ok. I think I could use usleep_range() API here
>> +
>> +static void ks_pcie_msi_irq_handler(unsigned int irq, struct irq_desc *desc)
>> +{
>> +	struct keystone_pcie *ks_pcie = irq_desc_get_handler_data(desc);
>> +	u32 offset = irq - ks_pcie->msi_host_irqs[0], pending, vector;
>> +	struct pcie_port *pp = &ks_pcie->pp;
>> +	struct irq_chip *chip = irq_desc_get_chip(desc);
>> +	int src, virq;
> Did I understand this right that the MSI implementation can be used
> by any dw-pcie hardware of the same version? If so, it would be good
> to move it into an extra file so it can be shared with the next host
> driver that uses this version.
Yes. Can be moved to a separate file, dw-msi-v3.65.c ??

>> +/**
>> + * ks_pcie_set_outbound_trans() - Set PHYADDR <-> BUSADDR
>> + * mapping for outbound
>> + */
>> +static void ks_pcie_set_outbound_trans(struct keystone_pcie *ks_pcie)
>> +static void ks_pcie_set_inbound_trans(struct pcie_port *pp)
> Why do you need to set this up from the kernel? Please move the
> translation window setup into the boot loader if you can.

Why to create a u-boot dependency?
>> +static struct hw_pci keystone_pcie_hw = {
>> +	.nr_controllers	= 1,
>> +	.setup		= dw_pcie_setup,
>> +	.scan		= ks_pcie_scan_bus,
>> +	.swizzle        = pci_common_swizzle,
>> +	.add_bus	= dw_pcie_add_bus,
>> +	.map_irq	= ks_pcie_map_irq,
>> +};
> This can just be a local variable in the probe function.

Ok

>
>> +
>> +static int ks_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
>> +{
>> +	struct pcie_port *pp = sys_to_pcie(dev->bus->sysdata);
>> +	struct keystone_pcie *ks_pcie = to_keystone_pcie(pp);
>> +
>> +	dev_info(pp->dev, "ks_pcie_map_irq: pin %d\n", pin);
>> +
>> +	if (!pin || pin > ks_pcie->num_legacy_host_irqs) {
>> +		dev_err(pp->dev, "pci irq pin out of range\n");
>> +		return -1;
>> +	}
>> +
>> +	/* pin has values from 1-4 */
>> +	return (ks_pcie->virqs[pin - 1] >= 0) ?
>> +		ks_pcie->virqs[pin - 1] : -1;
>> +}
>> +
>> +
>> +static void ack_irq(struct irq_data *d)
>> +{
>> +}
>> +
>> +static void mask_irq(struct irq_data *d)
>> +{
>> +}
>> +
>> +static void unmask_irq(struct irq_data *d)
>> +{
>> +}
>> +
>> +static struct irq_chip ks_pcie_legacy_irq_chip = {
>> +	.name = "PCIe-LEGACY-IRQ",
>> +	.irq_ack = ack_irq,
>> +	.irq_mask = mask_irq,
>> +	.irq_unmask = unmask_irq,
>> +};
> Can you use the generic irqchip code here?

Will investigate.

>
>> +	/*
>> +	 * support upto MAX_LEGACY_HOST_IRQS host and legacy IRQs.
>> +	 * In dt from index 0 to 3
>> +	 */
>> +	for (i = 0; i < MAX_LEGACY_HOST_IRQS; i++) {
>> +		ks_pcie->legacy_host_irqs[i] = irq_of_parse_and_map(np, i);
>> +		if (ks_pcie->legacy_host_irqs[i] < 0)
>> +			break;
>> +		ks_pcie->num_legacy_host_irqs++;
>> +	}
>> +
>> +	if (ks_pcie->num_legacy_host_irqs) {
>> +		ks_pcie->legacy_irqd = irq_domain_add_linear(np,
>> +					ks_pcie->num_legacy_host_irqs,
>> +					&irq_domain_simple_ops, NULL);
>> +
>> +		if (!ks_pcie->legacy_irqd) {
>> +			dev_err(dev,
>> +				"Failed to add irq domain for legacy irqs\n");
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	for (i = 0; i < ks_pcie->num_legacy_host_irqs; i++) {
>> +		ks_pcie->virqs[i] =
>> +			irq_create_mapping(ks_pcie->legacy_irqd, i);
>> +		irq_set_chip_and_handler(ks_pcie->virqs[i],
>> +			&ks_pcie_legacy_irq_chip, handle_level_irq);
>> +		irq_set_chip_data(ks_pcie->virqs[i], ks_pcie);
>> +		set_irq_flags(ks_pcie->virqs[i], IRQF_VALID);
>> +
>> +		if (ks_pcie->legacy_host_irqs[i] >= 0) {
>> +			irq_set_handler_data(ks_pcie->legacy_host_irqs[i],
>> +				ks_pcie);
>> +			irq_set_chained_handler(ks_pcie->legacy_host_irqs[i],
>> +				ks_pcie_legacy_irq_handler);
>> +		}
>> +		set_irq_flags(ks_pcie->legacy_host_irqs[i], IRQF_VALID);
>> +	}
> I have no idea how this would work with the standard interrupt-map property,
> since the legacy interrupt host is now the same device as the pci host.
>
> Maybe it's better to move the legacy irqchip handling entirely out of
> the driver and use a separate device node for the registers so it can
> come with its own #interrupt-cells, and then refer to this irqchip from
> the interrupt-map.

Will investigate

>> +	ks_pcie->clk = devm_clk_get(&pdev->dev, "pcie");
>> +	if (IS_ERR(ks_pcie->clk)) {
>> +		dev_err(&pdev->dev, "Failed to get pcie rc clock\n");
>> +		return PTR_ERR(ks_pcie->clk);
>> +	}
>> +	ret = clk_prepare_enable(ks_pcie->clk);
>> +	if (ret)
>> +		return ret;
> Could you move the clock handling into the generic dw-pcie code?

Could be. But currently only pci-exynos.c is using a clock name 
"pcie".pci-imx6.c uses pcie_axi" and
will have to be fixed so that we can move this code to the generic 
dw-pcie code.  Sean Cross is the
author for pci-imx6.c.

Sean,

Is "pcie_axi" is the pcie hw clock? If so, can you rename this to "pcie" 
so that  my patch can move the
clock handling code to pcie designware code?

>
>> +
>> +/* Keystone PCIe driver does not allow module unload */
>> +static int __init ks_pcie_init(void)
>> +{
>> +	return platform_driver_probe(&ks_pcie_driver, ks_pcie_probe);
>> +}
>> +subsys_initcall(ks_pcie_init);
> Why subsys_initcall?
>
> We should probably try to fix unloading soon.

Replied in another thread already from Andrew.

>
>> diff --git a/drivers/pci/host/pcie-ks-pdata.h b/drivers/pci/host/pcie-ks-pdata.h
>> new file mode 100644
>> index 0000000..a7626de
>> --- /dev/null
>> +++ b/drivers/pci/host/pcie-ks-pdata.h
>> @@ -0,0 +1,19 @@
>> +/*
>> + * Copyright (C) 2013-2014 Texas Instruments., Ltd.
>> + *		http://www.ti.com
>> + *
>> + * Author: Murali Karicheri <m-karicheri2@ti.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + */
>> +
>> +/* platform specific setup for k2 pcie */
>> +int k2_pcie_platform_setup(struct platform_device *pdev);
>> +
>> +/* keystone pcie pdata configurations */
>> +struct keystone_pcie_pdata {
>> +	int (*setup)(struct platform_device *pdev);
>> +};
> This should all go away once you have a proper PHY driver.
Will investigate.
>
>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
>> index 3a02717..7c3f777 100644
>> --- a/drivers/pci/quirks.c
>> +++ b/drivers/pci/quirks.c
>> @@ -3461,3 +3461,15 @@ int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags)
>>   
>>   	return -ENOTTY;
>>   }
>> +
>> +#ifdef CONFIG_PCIE_KEYSTONE
>> +/*
>> + * The KeyStone PCIe controller has maximum read request size of 256 bytes.
>> + * Force this configuration for all EP including bridges.
>> + */
>> +static void quirk_limit_readrequest(struct pci_dev *dev)
>> +{
>> +	pcie_set_readrq(dev, 256);
>> +}
>> +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, quirk_limit_readrequest);
>> +#endif /* CONFIG_TI_KEYSTONE_PCIE */
> You can't do this:
>
> A quirk for a specific hardware must not be enabled by compile-time options.
> You have to find a different way to do this.

Will investigate.

>
> 	Arnd
>

Murali

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 4/2/2014 12:47 PM, Andrew Murray wrote:
> On 2 April 2014 16:43, Murali Karicheri <m-karicheri2@ti.com> wrote:
>
>> Keystone pcie driver is developed based on other dw based pcie drivers
>>
>> such as pci-exynos that uses subsys_initcall(). I am new to this list,
>>
>> probably Jingoo (copied) has some history on why we can't use module.
>> For now I will keep it as is and can be re-visited in the next revisions.
>> Also I will experiment with PCIE port driver as well.
>>
>>
>> BTW, PCIE driver currently uses Legacy or MSI IRQ. Keystone PCI has
>> a platform IRQ. Is DT based irq configuration is the appropriate way
>> to add this capability?
> As far as I am aware - the PCI standards define a particular way for
> devices to describe which interrupt will be used for things like
> hotplug, AER and PME. These interrupts are always PCI interrupts (i.e.
> MSI/MSI-X/legacy). Thus the port services code in the kernel uses
> standard configuration space accesses to determine the interrupt to
> use. Also note that it's not just the host bridge that can provide
> these services but any PCIE device, I guess in this sense a host
> bridge is treated like any other device.
>
> If my understanding is correct I don't believe the current port
> services code allows exceptions to this, i.e. to say this host bridge
> actually uses a platform IRQ for AER rather than an MSI. Though this
> may be quite useful as I suspect many host bridges provide interrupts
> for things like PME through platform IRQs rather that PCI interrupts.
>
> Does the Keystone have platform IRQs for things like AER? Is that
> because the IP makes these events available through platform IRQs in
> addition to the standard PCI means?
>
> Andrew Murray
Andrew,

Yes. Keystone PCIe hw is based on designware hw version v3.65 that 
implements
Error interrupt as a platform IRQ only (not in addition) and 
unfortunately standard
PCI means are not supported. The latter versions do support standard PCI 
means.

Murali

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am Mittwoch, den 02.04.2014, 13:17 -0400 schrieb Murali Karicheri:
> Arnd,
> 
> Thanks for reviewing the RFC patch. Please see below my response.
> 
> On 3/25/2014 3:44 AM, Arnd Bergmann wrote:
> > On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
[...]
> 
> >> +	ks_pcie->clk = devm_clk_get(&pdev->dev, "pcie");
> >> +	if (IS_ERR(ks_pcie->clk)) {
> >> +		dev_err(&pdev->dev, "Failed to get pcie rc clock\n");
> >> +		return PTR_ERR(ks_pcie->clk);
> >> +	}
> >> +	ret = clk_prepare_enable(ks_pcie->clk);
> >> +	if (ret)
> >> +		return ret;
> > Could you move the clock handling into the generic dw-pcie code?
> 
> Could be. But currently only pci-exynos.c is using a clock name 
> "pcie".pci-imx6.c uses pcie_axi" and
> will have to be fixed so that we can move this code to the generic 
> dw-pcie code.  Sean Cross is the
> author for pci-imx6.c.
> 
> Sean,
> 
> Is "pcie_axi" is the pcie hw clock? If so, can you rename this to "pcie" 
> so that  my patch can move the
> clock handling code to pcie designware code?
> 
My series "i.MX6 PCIe binding change and MSI support" already changes
the clock name. So we could consolidate the clock handling once this is
in. Only difference is that the i.MX6 needs an additional PHY clock.

Regards,
Lucas

On 4/3/2014 4:32 AM, Lucas Stach wrote:
> Am Mittwoch, den 02.04.2014, 13:17 -0400 schrieb Murali Karicheri:
>> Arnd,
>>
>> Thanks for reviewing the RFC patch. Please see below my response.
>>
>> On 3/25/2014 3:44 AM, Arnd Bergmann wrote:
>>> On Monday 24 March 2014 20:35:26 Murali Karicheri wrote:
> [...]
>>>> +	ks_pcie->clk = devm_clk_get(&pdev->dev, "pcie");
>>>> +	if (IS_ERR(ks_pcie->clk)) {
>>>> +		dev_err(&pdev->dev, "Failed to get pcie rc clock\n");
>>>> +		return PTR_ERR(ks_pcie->clk);
>>>> +	}
>>>> +	ret = clk_prepare_enable(ks_pcie->clk);
>>>> +	if (ret)
>>>> +		return ret;
>>> Could you move the clock handling into the generic dw-pcie code?
>> Could be. But currently only pci-exynos.c is using a clock name
>> "pcie".pci-imx6.c uses pcie_axi" and
>> will have to be fixed so that we can move this code to the generic
>> dw-pcie code.  Sean Cross is the
>> author for pci-imx6.c.
>>
>> Sean,
>>
>> Is "pcie_axi" is the pcie hw clock? If so, can you rename this to "pcie"
>> so that  my patch can move the
>> clock handling code to pcie designware code?
>>
> My series "i.MX6 PCIe binding change and MSI support" already changes
> the clock name. So we could consolidate the clock handling once this is
> in. Only difference is that the i.MX6 needs an additional PHY clock.
>
> Regards,
> Lucas
Lucas,

Thanks. I will send a patch for this.

Murali
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 3/25/2014 12:54 PM, Jason Gunthorpe wrote:
> On Tue, Mar 25, 2014 at 08:44:36AM +0100, Arnd Bergmann wrote:
>
>> I have no idea how this would work with the standard interrupt-map property,
>> since the legacy interrupt host is now the same device as the pci host.
>>
>> Maybe it's better to move the legacy irqchip handling entirely out of
>> the driver and use a separate device node for the registers so it can
>> come with its own #interrupt-cells, and then refer to this irqchip from
>> the interrupt-map.
> The other DW PCI-E drivers are being fixed to use interrupt-map, so I
> think this driver should be fixed before it goes in as well.
Jason,

Could you send me a patch reference or branch that I can review to 
better understand
changes required in my driver to use the bindings you have described below?

Murali
> Other PCI-E hosts have handled this particular problem with a
> construction like this:
>
>         pcie@21800000 {
>                 compatible = "ti,keystone2-pcie";
>                 device_type = "pci";
>                 #address-cells = <3>;
>                 #size-cells = <2>;
>                 #interrupt-cells = <1>;
> 	       reg = ..
>
>                 pcie_intc: interrupt-controller {
>                    interrupt-controller;
> 		  #address-cells = <0>;
>                    #interrupt-cells = <1>;
>                 }
>
>                 interrupts =  ... enough to decode INTx;
>
> 	       interrtup-map = <0 0 0 1 &pcie_intc 1>, // INT A
> 	                       <0 0 0 2 &pcie_intc 2>, // INT B
>                                 <0 0 0 3 &pcie_intc 3>, // INT C
>                                 <0 0 0 4 &pcie_intc 4>; // INT D
>         }
>
> When the legacy irq domain is created it should be bound to the
> pcie_intc node, not to pcie node.
>
> ks_pcie_map_irq should be removed.
The latest dw core driver still had map_irq() being provided by the driver.
So any patch reference will help me understand that you describe here.

Thanks

Murali
>
>>> +#ifdef CONFIG_PCIE_KEYSTONE
>>> +/*
>>> + * The KeyStone PCIe controller has maximum read request size of 256 bytes.
>>> + * Force this configuration for all EP including bridges.
>>> + */
>>> +static void quirk_limit_readrequest(struct pci_dev *dev)
>>> +{
>>> +	pcie_set_readrq(dev, 256);
>>> +}
>>> +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, quirk_limit_readrequest);
>>> +#endif /* CONFIG_TI_KEYSTONE_PCIE */
>> You can't do this:
>>
>> A quirk for a specific hardware must not be enabled by compile-time options.
>> You have to find a different way to do this.
> This configuration should happen automatically by the PCI core if the
> config spaces are correct. See pcie_write_mrrs, pcie_write_mps,
> pcie_bus_configure_settings, etc.
>
> Your host will need to conform to the PCI spec and provide a root
> port bridge header for this to work.
>
> Also, even in the keystone case the MRRS should not be globally forced
> like this, there may be other bridges in the system with a 128 byte
> MPS.
>
> Jason

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Apr 07, 2014 at 12:38:23PM -0400, Murali Karicheri wrote:
> On 3/25/2014 12:54 PM, Jason Gunthorpe wrote:
> >On Tue, Mar 25, 2014 at 08:44:36AM +0100, Arnd Bergmann wrote:
> >
> >>I have no idea how this would work with the standard interrupt-map property,
> >>since the legacy interrupt host is now the same device as the pci host.
> >>
> >>Maybe it's better to move the legacy irqchip handling entirely out of
> >>the driver and use a separate device node for the registers so it can
> >>come with its own #interrupt-cells, and then refer to this irqchip from
> >>the interrupt-map.
> >The other DW PCI-E drivers are being fixed to use interrupt-map, so I
> >think this driver should be fixed before it goes in as well.
> 
> Could you send me a patch reference or branch that I can review to
> better understand changes required in my driver to use the bindings
> you have described below?

Bjorn is now merging it:

[PATCH v2 0/6] PCI irq mapping fixes and cleanups
http://www.spinics.net/lists/linux-pci/msg29589.html

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html