[2/2] aer: add support aer interrupt with none MSI/MSI-X/INTx mode

On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
maybe there is interrupt line for aer pme etc. Search the interrupt
number in the fdt file.

Signed-off-by: Po Liu <po.liu@nxp.com>
---
 drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

[+cc Rob]

Hi Po,

On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> maybe there is interrupt line for aer pme etc. Search the interrupt
> number in the fdt file.

My understanding is that AER interrupt signaling can be done via INTx,
MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
either?  How is the interrupt you're requesting here different from
INTx?

My guess is that your device *does* support INTx, and we should use
that.  ACPI has the generic _PRT that describes INTx routing.  There
must be something similar for DT, but I don't know what it is.  It
seems like this should be described using that DT mechanism (whatever
it is), and we shouldn't need special-case code in the portdrv code.

> Signed-off-by: Po Liu <po.liu@nxp.com>
> ---
>  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>  1 file changed, 28 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
> index 32d4d0a..64833e5 100644
> --- a/drivers/pci/pcie/portdrv_core.c
> +++ b/drivers/pci/pcie/portdrv_core.c
> @@ -15,6 +15,7 @@
>  #include <linux/slab.h>
>  #include <linux/pcieport_if.h>
>  #include <linux/aer.h>
> +#include <linux/of_irq.h>
>  
>  #include "../pci.h"
>  #include "portdrv.h"
> @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev *dev, int *vectors, int mask)
>  static int init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
>  {
>  	int i, irq = -1;
> +	int ret;
> +	struct device_node *np = NULL;
> +
> +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> +		irqs[i] = 0;
> +
> +	if (dev->bus->dev.of_node)
> +		np = dev->bus->dev.of_node;
> +
> +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> +	 * request irq for aer
> +	 */
> +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> +			(mask & PCIE_PORT_SERVICE_PME)) {
> +		ret = of_irq_get_byname(np, "aer");
> +		if (ret > 0) {
> +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> +			if (dev->irq)
> +				irq = dev->irq;
> +			goto no_msi;
> +		}
> +	}
>  
>  	/*
>  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> @@ -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
>  		irq = dev->irq;
>  
>   no_msi:
> -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> -		irqs[i] = irq;
> +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> +		if (!irqs[i])
> +			irqs[i] = irq;
> +	}
>  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>  
> -	if (irq < 0)
> +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>  		return -ENODEV;
>  	return 0;
>  }
> -- 
> 2.1.0.27.g96db324
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Bjorn,


>  -----Original Message-----

>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]

>  Sent: Thursday, June 02, 2016 11:48 AM

>  To: Po Liu

>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;

>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;

>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn

>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring

>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none

>  MSI/MSI-X/INTx mode

>  

>  [+cc Rob]

>  

>  Hi Po,

>  

>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:

>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.

>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,

>  > maybe there is interrupt line for aer pme etc. Search the interrupt

>  > number in the fdt file.

>  

>  My understanding is that AER interrupt signaling can be done via INTx,

>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device

>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx

>  either?  How is the interrupt you're requesting here different from INTx?


Layerscape use none of MSI or MSI-X or INTx to indicate the devices or root error
in RC mode. But use an independent SPI interrupt(arm interrupt controller) line.
And at same time, AER capability list in PCIe register descriptors. And also,
The Root Error Command/status register was proper to enable/disable the AER interrupt.

This hardware implementation make sense in some platforms, and this was described
in the function init_service_irqs() in the drivers/pci/pcie/portdrv_core.c in current
kernel as:

241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
244      */                                                                             

So I think this was the proper place to update the irq number before aer service
driver was loaded.

>  

>  My guess is that your device *does* support INTx, and we should use that.

>  ACPI has the generic _PRT that describes INTx routing.  There must be

>  something similar for DT, but I don't know what it is.  It seems like

>  this should be described using that DT mechanism (whatever it is), and

>  we shouldn't need special-case code in the portdrv code.

>  

>  > Signed-off-by: Po Liu <po.liu@nxp.com>

>  > ---

>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---

>  >  1 file changed, 28 insertions(+), 3 deletions(-)

>  >

>  > diff --git a/drivers/pci/pcie/portdrv_core.c

>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644

>  > --- a/drivers/pci/pcie/portdrv_core.c

>  > +++ b/drivers/pci/pcie/portdrv_core.c

>  > @@ -15,6 +15,7 @@

>  >  #include <linux/slab.h>

>  >  #include <linux/pcieport_if.h>

>  >  #include <linux/aer.h>

>  > +#include <linux/of_irq.h>

>  >

>  >  #include "../pci.h"

>  >  #include "portdrv.h"

>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev

>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct

>  > pci_dev *dev, int *irqs, int mask)  {

>  >  	int i, irq = -1;

>  > +	int ret;

>  > +	struct device_node *np = NULL;

>  > +

>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)

>  > +		irqs[i] = 0;

>  > +

>  > +	if (dev->bus->dev.of_node)

>  > +		np = dev->bus->dev.of_node;

>  > +

>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,

>  > +	 * request irq for aer

>  > +	 */

>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&

>  > +			(mask & PCIE_PORT_SERVICE_PME)) {

>  > +		ret = of_irq_get_byname(np, "aer");

>  > +		if (ret > 0) {

>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;

>  > +			if (dev->irq)

>  > +				irq = dev->irq;

>  > +			goto no_msi;

>  > +		}

>  > +	}

>  >

>  >  	/*

>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use

>  @@

>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,

>  int *irqs, int mask)

>  >  		irq = dev->irq;

>  >

>  >   no_msi:

>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)

>  > -		irqs[i] = irq;

>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {

>  > +		if (!irqs[i])

>  > +			irqs[i] = irq;

>  > +	}

>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;

>  >

>  > -	if (irq < 0)

>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)

>  >  		return -ENODEV;

>  >  	return 0;

>  >  }

>  > --

>  > 2.1.0.27.g96db324

>  >

>  >

>  > _______________________________________________

>  > linux-arm-kernel mailing list

>  > linux-arm-kernel@lists.infradead.org

>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >  -----Original Message-----
> >  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >  Sent: Thursday, June 02, 2016 11:48 AM
> >  To: Po Liu
> >  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >  MSI/MSI-X/INTx mode
> >  
> >  [+cc Rob]
> >  
> >  Hi Po,
> >  
> >  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >  > number in the fdt file.
> >  
> >  My understanding is that AER interrupt signaling can be done via INTx,
> >  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >  either?  How is the interrupt you're requesting here different from INTx?
> 
> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> or root error in RC mode. But use an independent SPI interrupt(arm
> interrupt controller) line.  

The Root Port is a PCI device and should follow the normal PCI rules
for interrupts.  As far as I understand, that means it should use MSI,
MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
That's all from the PCI point of view, of course.

What's on the other end of those interrupts is outside the scope of
PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
something else.  Drivers should not care about how it is connected,
and that's why I don't think this code really fits in portdrv.

Maybe your Root Port is non-compliant in some way and maybe we need a
quirk or something to work around that hardware defect.  But I'm not
convinced yet that we have that sort of problem.  It seems like we
just don't have the correct DT description.

> And at same time, AER capability list
> in PCIe register descriptors. And also, The Root Error
> Command/status register was proper to enable/disable the AER
> interrupt.

I'm not sure what you're saying here.  Here's what I think you said;
please correct me if I'm wrong:

  - Your Root Port has an AER capability.

  - Your Root Port does not support MSI or MSI-X.  (This would
    actually be a spec violation because the PCIe spec r3.0, sec 7.7
    says "All PCI Express device Functions that are capable of
    generating interrupts must implement MSI or MSI-X or both.")
  
  - The three enable bits in the Root Error Command Register enable or
    disable generation of interrupts.  These enable bits control all
    interrupts, whether MSI, MSI-X, or INTx.

  - The Root Error Status Register contains an "Advanced Error
    Interrupt Message Number" field, but that's only applicable if MSI
    or MSI-X is enabled, and if your device doesn't support MSI or
    MSI-X, this field doesn't apply.

> This hardware implementation make sense in some platforms, and this
> was described in the function init_service_irqs() in the
> drivers/pci/pcie/portdrv_core.c in current kernel as:

> 
> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
> 244      */                                                                             
> 
> So I think this was the proper place to update the irq number before aer service
> driver was loaded.
> 
> >  
> >  My guess is that your device *does* support INTx, and we should use that.
> >  ACPI has the generic _PRT that describes INTx routing.  There must be
> >  something similar for DT, but I don't know what it is.  It seems like
> >  this should be described using that DT mechanism (whatever it is), and
> >  we shouldn't need special-case code in the portdrv code.
> >  
> >  > Signed-off-by: Po Liu <po.liu@nxp.com>
> >  > ---
> >  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
> >  >  1 file changed, 28 insertions(+), 3 deletions(-)
> >  >
> >  > diff --git a/drivers/pci/pcie/portdrv_core.c
> >  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
> >  > --- a/drivers/pci/pcie/portdrv_core.c
> >  > +++ b/drivers/pci/pcie/portdrv_core.c
> >  > @@ -15,6 +15,7 @@
> >  >  #include <linux/slab.h>
> >  >  #include <linux/pcieport_if.h>
> >  >  #include <linux/aer.h>
> >  > +#include <linux/of_irq.h>
> >  >
> >  >  #include "../pci.h"
> >  >  #include "portdrv.h"
> >  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
> >  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
> >  > pci_dev *dev, int *irqs, int mask)  {
> >  >  	int i, irq = -1;
> >  > +	int ret;
> >  > +	struct device_node *np = NULL;
> >  > +
> >  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >  > +		irqs[i] = 0;
> >  > +
> >  > +	if (dev->bus->dev.of_node)
> >  > +		np = dev->bus->dev.of_node;
> >  > +
> >  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> >  > +	 * request irq for aer
> >  > +	 */
> >  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> >  > +			(mask & PCIE_PORT_SERVICE_PME)) {
> >  > +		ret = of_irq_get_byname(np, "aer");
> >  > +		if (ret > 0) {
> >  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> >  > +			if (dev->irq)
> >  > +				irq = dev->irq;
> >  > +			goto no_msi;
> >  > +		}
> >  > +	}
> >  >
> >  >  	/*
> >  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> >  @@
> >  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
> >  int *irqs, int mask)
> >  >  		irq = dev->irq;
> >  >
> >  >   no_msi:
> >  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >  > -		irqs[i] = irq;
> >  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> >  > +		if (!irqs[i])
> >  > +			irqs[i] = irq;
> >  > +	}
> >  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
> >  >
> >  > -	if (irq < 0)
> >  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
> >  >  		return -ENODEV;
> >  >  	return 0;
> >  >  }
> >  > --
> >  > 2.1.0.27.g96db324
> >  >
> >  >
> >  > _______________________________________________
> >  > linux-arm-kernel mailing list
> >  > linux-arm-kernel@lists.infradead.org
> >  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>>  -----Original Message-----
>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>>  To: Po Liu
>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>>  MSI/MSI-X/INTx mode
>>>  
>>>  [+cc Rob]
>>>  
>>>  Hi Po,
>>>  
>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
>>>  > number in the fdt file.
>>>  
>>>  My understanding is that AER interrupt signaling can be done via INTx,
>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
>>>  either?  How is the interrupt you're requesting here different from INTx?
>>
>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
>> or root error in RC mode. But use an independent SPI interrupt(arm
>> interrupt controller) line.  
> 
> The Root Port is a PCI device and should follow the normal PCI rules
> for interrupts.  As far as I understand, that means it should use MSI,
> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> That's all from the PCI point of view, of course.
> 
Bjorn.

I am faced with the same issue on Keystone PCI hardware and it has been
on my TODO list  for quite some time. Keystone PCI hardware also doesn't
use MSI or MSI-X or INTx for reporting errors received at the root port,
but use a platform interrupt instead (not complaint to PCI standard as
per PCI base spec). So I would need similar change to have the error
interrupt passed to the aer driver. So there are hardware out there
like Keystone which requires to support this through platform IRQ.

> What's on the other end of those interrupts is outside the scope of
> PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
> virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
> something else.  Drivers should not care about how it is connected,
> and that's why I don't think this code really fits in portdrv.
> 
> Maybe your Root Port is non-compliant in some way and maybe we need a
> quirk or something to work around that hardware defect.  But I'm not
> convinced yet that we have that sort of problem.  It seems like we
> just don't have the correct DT description.

Is quirk is what is suggested here to support this?

Murali
> 
>> And at same time, AER capability list
>> in PCIe register descriptors. And also, The Root Error
>> Command/status register was proper to enable/disable the AER
>> interrupt.
> 
> I'm not sure what you're saying here.  Here's what I think you said;
> please correct me if I'm wrong:
> 
>   - Your Root Port has an AER capability.
> 
>   - Your Root Port does not support MSI or MSI-X.  (This would
>     actually be a spec violation because the PCIe spec r3.0, sec 7.7
>     says "All PCI Express device Functions that are capable of
>     generating interrupts must implement MSI or MSI-X or both.")
>   
>   - The three enable bits in the Root Error Command Register enable or
>     disable generation of interrupts.  These enable bits control all
>     interrupts, whether MSI, MSI-X, or INTx.
> 
>   - The Root Error Status Register contains an "Advanced Error
>     Interrupt Message Number" field, but that's only applicable if MSI
>     or MSI-X is enabled, and if your device doesn't support MSI or
>     MSI-X, this field doesn't apply.
> 
>> This hardware implementation make sense in some platforms, and this
>> was described in the function init_service_irqs() in the
>> drivers/pci/pcie/portdrv_core.c in current kernel as:
> 
>>
>> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
>> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
>> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
>> 244      */                                                                             
>>
>> So I think this was the proper place to update the irq number before aer service
>> driver was loaded.
>>
>>>  
>>>  My guess is that your device *does* support INTx, and we should use that.
>>>  ACPI has the generic _PRT that describes INTx routing.  There must be
>>>  something similar for DT, but I don't know what it is.  It seems like
>>>  this should be described using that DT mechanism (whatever it is), and
>>>  we shouldn't need special-case code in the portdrv code.
>>>  
>>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
>>>  > ---
>>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
>>>  >
>>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
>>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
>>>  > --- a/drivers/pci/pcie/portdrv_core.c
>>>  > +++ b/drivers/pci/pcie/portdrv_core.c
>>>  > @@ -15,6 +15,7 @@
>>>  >  #include <linux/slab.h>
>>>  >  #include <linux/pcieport_if.h>
>>>  >  #include <linux/aer.h>
>>>  > +#include <linux/of_irq.h>
>>>  >
>>>  >  #include "../pci.h"
>>>  >  #include "portdrv.h"
>>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
>>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
>>>  > pci_dev *dev, int *irqs, int mask)  {
>>>  >  	int i, irq = -1;
>>>  > +	int ret;
>>>  > +	struct device_node *np = NULL;
>>>  > +
>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>  > +		irqs[i] = 0;
>>>  > +
>>>  > +	if (dev->bus->dev.of_node)
>>>  > +		np = dev->bus->dev.of_node;
>>>  > +
>>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
>>>  > +	 * request irq for aer
>>>  > +	 */
>>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
>>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
>>>  > +		ret = of_irq_get_byname(np, "aer");
>>>  > +		if (ret > 0) {
>>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
>>>  > +			if (dev->irq)
>>>  > +				irq = dev->irq;
>>>  > +			goto no_msi;
>>>  > +		}
>>>  > +	}
>>>  >
>>>  >  	/*
>>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
>>>  @@
>>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
>>>  int *irqs, int mask)
>>>  >  		irq = dev->irq;
>>>  >
>>>  >   no_msi:
>>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>  > -		irqs[i] = irq;
>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
>>>  > +		if (!irqs[i])
>>>  > +			irqs[i] = irq;
>>>  > +	}
>>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>>>  >
>>>  > -	if (irq < 0)
>>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>>>  >  		return -ENODEV;
>>>  >  	return 0;
>>>  >  }
>>>  > --
>>>  > 2.1.0.27.g96db324
>>>  >
>>>  >
>>>  > _______________________________________________
>>>  > linux-arm-kernel mailing list
>>>  > linux-arm-kernel@lists.infradead.org
>>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> > On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>>  -----Original Message-----
> >>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>>  To: Po Liu
> >>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>>  MSI/MSI-X/INTx mode
> >>>  
> >>>  [+cc Rob]
> >>>  
> >>>  Hi Po,
> >>>  
> >>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >>>  > number in the fdt file.
> >>>  
> >>>  My understanding is that AER interrupt signaling can be done via INTx,
> >>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >>>  either?  How is the interrupt you're requesting here different from INTx?
> >>
> >> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> >> or root error in RC mode. But use an independent SPI interrupt(arm
> >> interrupt controller) line.  
> > 
> > The Root Port is a PCI device and should follow the normal PCI rules
> > for interrupts.  As far as I understand, that means it should use MSI,
> > MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> > use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> > INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> > That's all from the PCI point of view, of course.
> 
> I am faced with the same issue on Keystone PCI hardware and it has been
> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
> use MSI or MSI-X or INTx for reporting errors received at the root port,
> but use a platform interrupt instead (not complaint to PCI standard as
> per PCI base spec). So I would need similar change to have the error
> interrupt passed to the aer driver. So there are hardware out there
> like Keystone which requires to support this through platform IRQ.

This is not a new area of the spec, and it's hard for me to believe
that these two new PCIe controllers are both broken the same way
(although I guess both are DesignWare-based, so maybe this is the same
underlying problem in both cases?).  I think it's more likely that we
just haven't figured out the right way to describe this in the DT.

I assume you have a Root Port with an AER capability, no MSI
capability, and no MSI-X capability, right?  What does its Interrupt
Pin register contain?  If it's zero, it doesn't use INTx either, so
according to the spec it should generate no interrupts.

But if Interrupt Pin is non-zero, that means the Root Port should be
able to generate virtual INTx interrupts.  Presumably the Root Complex
connects those interrupts to something; maybe to your platform
interrupt?

PCI doesn't say anything about an interrupt after it leaves the Root
Complex, so the fact that it's connected to a "platform interrupt" or
"SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
Shouldn't we be able to use the interrupt-map and related DT
properties to express the fact that Root Port virtual INTx is routed
to platform interrupt Y, even without any special-case code in
portdrv?

> > What's on the other end of those interrupts is outside the scope of
> > PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
> > virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
> > something else.  Drivers should not care about how it is connected,
> > and that's why I don't think this code really fits in portdrv.
> > 
> > Maybe your Root Port is non-compliant in some way and maybe we need a
> > quirk or something to work around that hardware defect.  But I'm not
> > convinced yet that we have that sort of problem.  It seems like we
> > just don't have the correct DT description.
> 
> Is quirk is what is suggested here to support this?

No, I don't understand the problem yet, so I'm not ready to suggest a
solution.

> >> And at same time, AER capability list
> >> in PCIe register descriptors. And also, The Root Error
> >> Command/status register was proper to enable/disable the AER
> >> interrupt.
> > 
> > I'm not sure what you're saying here.  Here's what I think you said;
> > please correct me if I'm wrong:
> > 
> >   - Your Root Port has an AER capability.
> > 
> >   - Your Root Port does not support MSI or MSI-X.  (This would
> >     actually be a spec violation because the PCIe spec r3.0, sec 7.7
> >     says "All PCI Express device Functions that are capable of
> >     generating interrupts must implement MSI or MSI-X or both.")
> >   
> >   - The three enable bits in the Root Error Command Register enable or
> >     disable generation of interrupts.  These enable bits control all
> >     interrupts, whether MSI, MSI-X, or INTx.
> > 
> >   - The Root Error Status Register contains an "Advanced Error
> >     Interrupt Message Number" field, but that's only applicable if MSI
> >     or MSI-X is enabled, and if your device doesn't support MSI or
> >     MSI-X, this field doesn't apply.
> > 
> >> This hardware implementation make sense in some platforms, and this
> >> was described in the function init_service_irqs() in the
> >> drivers/pci/pcie/portdrv_core.c in current kernel as:
> > 
> >>
> >> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
> >> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
> >> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
> >> 244      */                                                                             
> >>
> >> So I think this was the proper place to update the irq number before aer service
> >> driver was loaded.
> >>
> >>>  
> >>>  My guess is that your device *does* support INTx, and we should use that.
> >>>  ACPI has the generic _PRT that describes INTx routing.  There must be
> >>>  something similar for DT, but I don't know what it is.  It seems like
> >>>  this should be described using that DT mechanism (whatever it is), and
> >>>  we shouldn't need special-case code in the portdrv code.
> >>>  
> >>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
> >>>  > ---
> >>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
> >>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
> >>>  >
> >>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
> >>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
> >>>  > --- a/drivers/pci/pcie/portdrv_core.c
> >>>  > +++ b/drivers/pci/pcie/portdrv_core.c
> >>>  > @@ -15,6 +15,7 @@
> >>>  >  #include <linux/slab.h>
> >>>  >  #include <linux/pcieport_if.h>
> >>>  >  #include <linux/aer.h>
> >>>  > +#include <linux/of_irq.h>
> >>>  >
> >>>  >  #include "../pci.h"
> >>>  >  #include "portdrv.h"
> >>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
> >>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
> >>>  > pci_dev *dev, int *irqs, int mask)  {
> >>>  >  	int i, irq = -1;
> >>>  > +	int ret;
> >>>  > +	struct device_node *np = NULL;
> >>>  > +
> >>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >>>  > +		irqs[i] = 0;
> >>>  > +
> >>>  > +	if (dev->bus->dev.of_node)
> >>>  > +		np = dev->bus->dev.of_node;
> >>>  > +
> >>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> >>>  > +	 * request irq for aer
> >>>  > +	 */
> >>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> >>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
> >>>  > +		ret = of_irq_get_byname(np, "aer");
> >>>  > +		if (ret > 0) {
> >>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> >>>  > +			if (dev->irq)
> >>>  > +				irq = dev->irq;
> >>>  > +			goto no_msi;
> >>>  > +		}
> >>>  > +	}
> >>>  >
> >>>  >  	/*
> >>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> >>>  @@
> >>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
> >>>  int *irqs, int mask)
> >>>  >  		irq = dev->irq;
> >>>  >
> >>>  >   no_msi:
> >>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >>>  > -		irqs[i] = irq;
> >>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> >>>  > +		if (!irqs[i])
> >>>  > +			irqs[i] = irq;
> >>>  > +	}
> >>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
> >>>  >
> >>>  > -	if (irq < 0)
> >>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
> >>>  >  		return -ENODEV;
> >>>  >  	return 0;
> >>>  >  }
> >>>  > --
> >>>  > 2.1.0.27.g96db324
> >>>  >
> >>>  >
> >>>  > _______________________________________________
> >>>  > linux-arm-kernel mailing list
> >>>  > linux-arm-kernel@lists.infradead.org
> >>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> -- 
> Murali Karicheri
> Linux Kernel, Keystone
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Po,

Sorry to hijack your discussion, but the problem seems to be same for
Keystone PCI controller which is also designware (old version) based.

On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
>> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>>>>  -----Original Message-----
>>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>>>>  To: Po Liu
>>>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
>>>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
>>>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
>>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>>>>  MSI/MSI-X/INTx mode
>>>>>  
>>>>>  [+cc Rob]
>>>>>  
>>>>>  Hi Po,
>>>>>  
>>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
>>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
>>>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
>>>>>  > number in the fdt file.
>>>>>  
>>>>>  My understanding is that AER interrupt signaling can be done via INTx,
>>>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
>>>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
>>>>>  either?  How is the interrupt you're requesting here different from INTx?
>>>>
>>>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
>>>> or root error in RC mode. But use an independent SPI interrupt(arm
>>>> interrupt controller) line.  
>>>
>>> The Root Port is a PCI device and should follow the normal PCI rules
>>> for interrupts.  As far as I understand, that means it should use MSI,
>>> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
>>> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
>>> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
>>> That's all from the PCI point of view, of course.
>>
>> I am faced with the same issue on Keystone PCI hardware and it has been
>> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
>> use MSI or MSI-X or INTx for reporting errors received at the root port,
>> but use a platform interrupt instead (not complaint to PCI standard as
>> per PCI base spec). So I would need similar change to have the error
>> interrupt passed to the aer driver. So there are hardware out there
>> like Keystone which requires to support this through platform IRQ.
> 
> This is not a new area of the spec, and it's hard for me to believe
> that these two new PCIe controllers are both broken the same way
> (although I guess both are DesignWare-based, so maybe this is the same
> underlying problem in both cases?).  I think it's more likely that we
> just haven't figured out the right way to describe this in the DT.

Keystone is using an older version of the designware IP and it implements
all of the interrupts in the application register space unlike other
newer version of the hardware. So I assume, the version used on Layerscape
is also an older version and the both have same issue in terms of 
non standard platform interrupt used for error reporting.

> 
> I assume you have a Root Port with an AER capability, no MSI
> capability, and no MSI-X capability, right? 

Has AER capability and both MSI and INTx (legacy) capability

> What does its Interrupt
> Pin register contain?  If it's zero, it doesn't use INTx either, so
> according to the spec it should generate no interrupts.
> 
At address offset 0x3C by default has a value of 1, but it is writable
by software. So default is INTx A.

> But if Interrupt Pin is non-zero, that means the Root Port should be
> able to generate virtual INTx interrupts.  Presumably the Root Complex
> connects those interrupts to something; maybe to your platform
> interrupt?

Probably that is what is happening. Both Power management and Error
interrupts are raised on platform interrupt lines.

12 Error Interrupts

[0] System error (OR of fatal, nonfatal, correctable errors) (RC mode only)
[1] PCIe fatal error (RC mode only)
[2] PCIe non-fatal error (RC mode only)
[3] PCIe correctable error (RC mode only)
[4] AXI Error due to fatal condition in AXI bridge (EP/RC modes)
[5] PCIe advanced error (RC mode only)

13 Power management and reset event interrupts

[0] Power management turn-off message interrupt (EP mode only)
[1] Power management ack message interrupt (RC mode only)
[2] Power management event interrupt (RC mode only)
[3] Link request reset interrupt (hot reset or link down) (RC mode only)

> 
> PCI doesn't say anything about an interrupt after it leaves the Root
> Complex, so the fact that it's connected to a "platform interrupt" or
> "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
> Shouldn't we be able to use the interrupt-map and related DT
> properties to express the fact that Root Port virtual INTx is routed
> to platform interrupt Y, even without any special-case code in
> portdrv?

My understanding is if RC also raise interrupt on INTx, then below 
map make sense, where either EP or RC can raise interrupt and the
line will be muxed for interrupt from EP or RC port.

Here is the DT entry in PCIE keystone for Legacy interrupt mapping
to host interrupt. 
                        #interrupt-cells = <1>;
                        interrupt-map-mask = <0 0 0 7>;
                        interrupt-map = <0 0 0 1 &pcie_intc0 0>, /* INT A */
                                        <0 0 0 2 &pcie_intc0 1>, /* INT B */
                                        <0 0 0 3 &pcie_intc0 2>, /* INT C */
                                        <0 0 0 4 &pcie_intc0 3>; /* INT D */


And  then

                        pcie_intc0: legacy-interrupt-controller {
                                interrupt-controller;
                                #interrupt-cells = <1>;
                                interrupt-parent = <&gic>;
                                interrupts = <GIC_SPI 48 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 49 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 50 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 51 IRQ_TYPE_EDGE_RISING>;
                        };

So if RC interrupt for Power management and Error interrupt is a separate
line, how can software describe that using the above interrupt map?

Murali
> 
>>> What's on the other end of those interrupts is outside the scope of
>>> PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
>>> virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
>>> something else.  Drivers should not care about how it is connected,
>>> and that's why I don't think this code really fits in portdrv.
>>>
>>> Maybe your Root Port is non-compliant in some way and maybe we need a
>>> quirk or something to work around that hardware defect.  But I'm not
>>> convinced yet that we have that sort of problem.  It seems like we
>>> just don't have the correct DT description.
>>
>> Is quirk is what is suggested here to support this?
> 
> No, I don't understand the problem yet, so I'm not ready to suggest a
> solution.
> 
>>>> And at same time, AER capability list
>>>> in PCIe register descriptors. And also, The Root Error
>>>> Command/status register was proper to enable/disable the AER
>>>> interrupt.
>>>
>>> I'm not sure what you're saying here.  Here's what I think you said;
>>> please correct me if I'm wrong:
>>>
>>>   - Your Root Port has an AER capability.
>>>
>>>   - Your Root Port does not support MSI or MSI-X.  (This would
>>>     actually be a spec violation because the PCIe spec r3.0, sec 7.7
>>>     says "All PCI Express device Functions that are capable of
>>>     generating interrupts must implement MSI or MSI-X or both.")
>>>   
>>>   - The three enable bits in the Root Error Command Register enable or
>>>     disable generation of interrupts.  These enable bits control all
>>>     interrupts, whether MSI, MSI-X, or INTx.
>>>
>>>   - The Root Error Status Register contains an "Advanced Error
>>>     Interrupt Message Number" field, but that's only applicable if MSI
>>>     or MSI-X is enabled, and if your device doesn't support MSI or
>>>     MSI-X, this field doesn't apply.
>>>
>>>> This hardware implementation make sense in some platforms, and this
>>>> was described in the function init_service_irqs() in the
>>>> drivers/pci/pcie/portdrv_core.c in current kernel as:
>>>
>>>>
>>>> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
>>>> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
>>>> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
>>>> 244      */                                                                             
>>>>
>>>> So I think this was the proper place to update the irq number before aer service
>>>> driver was loaded.
>>>>
>>>>>  
>>>>>  My guess is that your device *does* support INTx, and we should use that.
>>>>>  ACPI has the generic _PRT that describes INTx routing.  There must be
>>>>>  something similar for DT, but I don't know what it is.  It seems like
>>>>>  this should be described using that DT mechanism (whatever it is), and
>>>>>  we shouldn't need special-case code in the portdrv code.
>>>>>  
>>>>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
>>>>>  > ---
>>>>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>>>>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
>>>>>  >
>>>>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
>>>>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
>>>>>  > --- a/drivers/pci/pcie/portdrv_core.c
>>>>>  > +++ b/drivers/pci/pcie/portdrv_core.c
>>>>>  > @@ -15,6 +15,7 @@
>>>>>  >  #include <linux/slab.h>
>>>>>  >  #include <linux/pcieport_if.h>
>>>>>  >  #include <linux/aer.h>
>>>>>  > +#include <linux/of_irq.h>
>>>>>  >
>>>>>  >  #include "../pci.h"
>>>>>  >  #include "portdrv.h"
>>>>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
>>>>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
>>>>>  > pci_dev *dev, int *irqs, int mask)  {
>>>>>  >  	int i, irq = -1;
>>>>>  > +	int ret;
>>>>>  > +	struct device_node *np = NULL;
>>>>>  > +
>>>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>>>  > +		irqs[i] = 0;
>>>>>  > +
>>>>>  > +	if (dev->bus->dev.of_node)
>>>>>  > +		np = dev->bus->dev.of_node;
>>>>>  > +
>>>>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
>>>>>  > +	 * request irq for aer
>>>>>  > +	 */
>>>>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
>>>>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
>>>>>  > +		ret = of_irq_get_byname(np, "aer");
>>>>>  > +		if (ret > 0) {
>>>>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
>>>>>  > +			if (dev->irq)
>>>>>  > +				irq = dev->irq;
>>>>>  > +			goto no_msi;
>>>>>  > +		}
>>>>>  > +	}
>>>>>  >
>>>>>  >  	/*
>>>>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
>>>>>  @@
>>>>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
>>>>>  int *irqs, int mask)
>>>>>  >  		irq = dev->irq;
>>>>>  >
>>>>>  >   no_msi:
>>>>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>>>  > -		irqs[i] = irq;
>>>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
>>>>>  > +		if (!irqs[i])
>>>>>  > +			irqs[i] = irq;
>>>>>  > +	}
>>>>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>>>>>  >
>>>>>  > -	if (irq < 0)
>>>>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>>>>>  >  		return -ENODEV;
>>>>>  >  	return 0;
>>>>>  >  }
>>>>>  > --
>>>>>  > 2.1.0.27.g96db324
>>>>>  >
>>>>>  >
>>>>>  > _______________________________________________
>>>>>  > linux-arm-kernel mailing list
>>>>>  > linux-arm-kernel@lists.infradead.org
>>>>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>> -- 
>> Murali Karicheri
>> Linux Kernel, Keystone

On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> Po,
> 
> Sorry to hijack your discussion, but the problem seems to be same for
> Keystone PCI controller which is also designware (old version) based.
> 
> On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>>>>  -----Original Message-----
> >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>>>>  To: Po Liu
> >>>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >>>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >>>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>>>>  MSI/MSI-X/INTx mode
> >>>>>  
> >>>>>  [+cc Rob]
> >>>>>  
> >>>>>  Hi Po,
> >>>>>  
> >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >>>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >>>>>  > number in the fdt file.
> >>>>>  
> >>>>>  My understanding is that AER interrupt signaling can be done via INTx,
> >>>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >>>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >>>>>  either?  How is the interrupt you're requesting here different from INTx?
> >>>>
> >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> >>>> or root error in RC mode. But use an independent SPI interrupt(arm
> >>>> interrupt controller) line.  
> >>>
> >>> The Root Port is a PCI device and should follow the normal PCI rules
> >>> for interrupts.  As far as I understand, that means it should use MSI,
> >>> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> >>> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> >>> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> >>> That's all from the PCI point of view, of course.
> >>
> >> I am faced with the same issue on Keystone PCI hardware and it has been
> >> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
> >> use MSI or MSI-X or INTx for reporting errors received at the root port,
> >> but use a platform interrupt instead (not complaint to PCI standard as
> >> per PCI base spec). So I would need similar change to have the error
> >> interrupt passed to the aer driver. So there are hardware out there
> >> like Keystone which requires to support this through platform IRQ.
> > 
> > This is not a new area of the spec, and it's hard for me to believe
> > that these two new PCIe controllers are both broken the same way
> > (although I guess both are DesignWare-based, so maybe this is the same
> > underlying problem in both cases?).  I think it's more likely that we
> > just haven't figured out the right way to describe this in the DT.
> 
> Keystone is using an older version of the designware IP and it implements
> all of the interrupts in the application register space unlike other
> newer version of the hardware. So I assume, the version used on Layerscape
> is also an older version and the both have same issue in terms of 
> non standard platform interrupt used for error reporting.
> 
> > I assume you have a Root Port with an AER capability, no MSI
> > capability, and no MSI-X capability, right? 
> 
> Has AER capability and both MSI and INTx (legacy) capability
> 
> > What does its Interrupt
> > Pin register contain?  If it's zero, it doesn't use INTx either, so
> > according to the spec it should generate no interrupts.
> > 
> At address offset 0x3C by default has a value of 1, but it is writable
> by software. So default is INTx A.

0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
*Pin* is at 0x3d and should be read-only.

Does your Keystone driver support MSI?  If so, since your Root Port
supports MSI, I would think we would use that by default, and the INTx
stuff wouldn't even matter.

> > But if Interrupt Pin is non-zero, that means the Root Port should be
> > able to generate virtual INTx interrupts.  Presumably the Root Complex
> > connects those interrupts to something; maybe to your platform
> > interrupt?
> 
> Probably that is what is happening. Both Power management and Error
> interrupts are raised on platform interrupt lines.
> 
> 12 Error Interrupts
> 
> [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode only)
> [1] PCIe fatal error (RC mode only)
> [2] PCIe non-fatal error (RC mode only)
> [3] PCIe correctable error (RC mode only)
> [4] AXI Error due to fatal condition in AXI bridge (EP/RC modes)
> [5] PCIe advanced error (RC mode only)
> 
> 13 Power management and reset event interrupts
> 
> [0] Power management turn-off message interrupt (EP mode only)
> [1] Power management ack message interrupt (RC mode only)
> [2] Power management event interrupt (RC mode only)
> [3] Link request reset interrupt (hot reset or link down) (RC mode only)
>
> > PCI doesn't say anything about an interrupt after it leaves the Root
> > Complex, so the fact that it's connected to a "platform interrupt" or
> > "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
> > Shouldn't we be able to use the interrupt-map and related DT
> > properties to express the fact that Root Port virtual INTx is routed
> > to platform interrupt Y, even without any special-case code in
> > portdrv?
> 
> My understanding is if RC also raise interrupt on INTx, then below 
> map make sense, where either EP or RC can raise interrupt and the
> line will be muxed for interrupt from EP or RC port.

I'm sorry, I didn't quite catch your meaning here, so let me try to
clarify some terminology.  Maybe we'll eventually blunder into a
common understanding :)

INTx is a PCI concept and only means something in the PCI hierarchy.
The RC would *receive* virtual INTx interrupts from the PCI hierarchy
and turn them into some platform-specific interrupt (not INTx) on the
upstream side.

So strictly speaking, the RC might raise platform-specific interrupts
when it receives INTx interrupts, but it would not raise any INTx
interrupts itself.

> Here is the DT entry in PCIE keystone for Legacy interrupt mapping
> to host interrupt. 
>                         #interrupt-cells = <1>;
>                         interrupt-map-mask = <0 0 0 7>;
>                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /* INT A */
>                                         <0 0 0 2 &pcie_intc0 1>, /* INT B */
>                                         <0 0 0 3 &pcie_intc0 2>, /* INT C */
>                                         <0 0 0 4 &pcie_intc0 3>; /* INT D */

If I understand correctly, this is the mapping from the PCI world
(INTA, INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)

If a Root Port raises a virtual INTA, the RC should turn it into the
corresponding platform interrupt, i.e., GIC_SPI 48 from the
description below.

> And  then
> 
>                         pcie_intc0: legacy-interrupt-controller {
>                                 interrupt-controller;
>                                 #interrupt-cells = <1>;
>                                 interrupt-parent = <&gic>;
>                                 interrupts = <GIC_SPI 48 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 49 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 50 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 51 IRQ_TYPE_EDGE_RISING>;
>                         };
> 
> So if RC interrupt for Power management and Error interrupt is a separate
> line, how can software describe that using the above interrupt map?

I don't know anything about interrupts the RC might generate on its
own behalf.  A lot of the RC behavior is not specified by the PCIe
spec because the RC is on the border between the upstream
platform-specific stuff and the downstream PCIe stuff.  Is there
something in the PCIe spec about the power management and error
interrupts you're talking about?  Of maybe you can point me to a
section of the Keystone spec that talks about interrupts generated by
the RC?

In any event, the AER interrupts we're looking for in portdrv are from
the Root Port, not from the RC.  For INTx, my understanding is that
the RC *transforms* virtual INTx messages from the Root Port in the
PCIe domain into GIC transactions in the platform domain.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Bjorn,
I confirm we met same problem with KeyStone base on DesignWare design.


Best regards,
Liu Po

>  -----Original Message-----

>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]

>  Sent: Saturday, June 04, 2016 11:49 AM

>  To: Murali Karicheri

>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-

>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;

>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;

>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;

>  Mingkai Hu; Rob Herring

>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none

>  MSI/MSI-X/INTx mode

>  

>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:

>  > Po,

>  >

>  > Sorry to hijack your discussion, but the problem seems to be same for

>  > Keystone PCI controller which is also designware (old version) based.

>  >

>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:

>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:

>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:

>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:

>  > >>>>>  -----Original Message-----

>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]

>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM

>  > >>>>>  To: Po Liu

>  > >>>>>  Cc: linux-pci@vger.kernel.org;

>  > >>>>> linux-arm-kernel@lists.infradead.org;

>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd

>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;

>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob

>  > >>>>> Herring

>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with

>  > >>>>> none  MSI/MSI-X/INTx mode

>  > >>>>>

>  > >>>>>  [+cc Rob]

>  > >>>>>

>  > >>>>>  Hi Po,

>  > >>>>>

>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:

>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx

>  in RC mode.

>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx

>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search

>  > >>>>> the interrupt  > number in the fdt file.

>  > >>>>>

>  > >>>>>  My understanding is that AER interrupt signaling can be done

>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).

>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you

>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt

>  you're requesting here different from INTx?

>  > >>>>

>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the

>  > >>>> devices or root error in RC mode. But use an independent SPI

>  > >>>> interrupt(arm interrupt controller) line.

>  > >>>

>  > >>> The Root Port is a PCI device and should follow the normal PCI

>  > >>> rules for interrupts.  As far as I understand, that means it

>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI

>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register

>  > >>> should tell us which (INTA/ INTB/etc.), and

>  PCI_COMMAND_INTX_DISABLE should work to disable it.

>  > >>> That's all from the PCI point of view, of course.

>  > >>

>  > >> I am faced with the same issue on Keystone PCI hardware and it has

>  > >> been on my TODO list  for quite some time. Keystone PCI hardware

>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received

>  > >> at the root port, but use a platform interrupt instead (not

>  > >> complaint to PCI standard as per PCI base spec). So I would need

>  > >> similar change to have the error interrupt passed to the aer

>  > >> driver. So there are hardware out there like Keystone which

>  requires to support this through platform IRQ.

>  > >

>  > > This is not a new area of the spec, and it's hard for me to believe

>  > > that these two new PCIe controllers are both broken the same way

>  > > (although I guess both are DesignWare-based, so maybe this is the

>  > > same underlying problem in both cases?).  I think it's more likely

>  > > that we just haven't figured out the right way to describe this in

>  the DT.

>  >

>  > Keystone is using an older version of the designware IP and it

>  > implements all of the interrupts in the application register space

>  > unlike other newer version of the hardware. So I assume, the version

>  > used on Layerscape is also an older version and the both have same

>  > issue in terms of non standard platform interrupt used for error

>  reporting.

>  >

>  > > I assume you have a Root Port with an AER capability, no MSI

>  > > capability, and no MSI-X capability, right?

>  >

>  > Has AER capability and both MSI and INTx (legacy) capability

>  >

>  > > What does its Interrupt

>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so

>  > > according to the spec it should generate no interrupts.

>  > >

>  > At address offset 0x3C by default has a value of 1, but it is writable

>  > by software. So default is INTx A.

>  

>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt

>  *Pin* is at 0x3d and should be read-only.

>  

>  Does your Keystone driver support MSI?  If so, since your Root Port

>  supports MSI, I would think we would use that by default, and the INTx

>  stuff wouldn't even matter.


Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
But both of them not work for AER interrupt when devices or root port report aer error.
But another GIC interrupt line do.

>  

>  > > But if Interrupt Pin is non-zero, that means the Root Port should be

>  > > able to generate virtual INTx interrupts.  Presumably the Root

>  > > Complex connects those interrupts to something; maybe to your

>  > > platform interrupt?

>  >

>  > Probably that is what is happening. Both Power management and Error

>  > interrupts are raised on platform interrupt lines.

>  >

>  > 12 Error Interrupts

>  >

>  > [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode

>  > only) [1] PCIe fatal error (RC mode only) [2] PCIe non-fatal error (RC

>  > mode only) [3] PCIe correctable error (RC mode only) [4] AXI Error due

>  > to fatal condition in AXI bridge (EP/RC modes) [5] PCIe advanced error

>  > (RC mode only)

>  >

>  > 13 Power management and reset event interrupts

>  >

>  > [0] Power management turn-off message interrupt (EP mode only) [1]

>  > Power management ack message interrupt (RC mode only) [2] Power

>  > management event interrupt (RC mode only) [3] Link request reset

>  > interrupt (hot reset or link down) (RC mode only)

>  >

>  > > PCI doesn't say anything about an interrupt after it leaves the Root

>  > > Complex, so the fact that it's connected to a "platform interrupt"

>  > > or "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-

>  compliant.

>  > > Shouldn't we be able to use the interrupt-map and related DT

>  > > properties to express the fact that Root Port virtual INTx is routed

>  > > to platform interrupt Y, even without any special-case code in

>  > > portdrv?

>  >

>  > My understanding is if RC also raise interrupt on INTx, then below map

>  > make sense, where either EP or RC can raise interrupt and the line

>  > will be muxed for interrupt from EP or RC port.

>  

>  I'm sorry, I didn't quite catch your meaning here, so let me try to

>  clarify some terminology.  Maybe we'll eventually blunder into a common

>  understanding :)

>  

>  INTx is a PCI concept and only means something in the PCI hierarchy.

>  The RC would *receive* virtual INTx interrupts from the PCI hierarchy

>  and turn them into some platform-specific interrupt (not INTx) on the

>  upstream side.

>  

>  So strictly speaking, the RC might raise platform-specific interrupts

>  when it receives INTx interrupts, but it would not raise any INTx

>  interrupts itself.

>  

>  > Here is the DT entry in PCIE keystone for Legacy interrupt mapping to

>  > host interrupt.

>  >                         #interrupt-cells = <1>;

>  >                         interrupt-map-mask = <0 0 0 7>;

>  >                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /*

>  INT A */

>  >                                         <0 0 0 2 &pcie_intc0 1>, /*

>  INT B */

>  >                                         <0 0 0 3 &pcie_intc0 2>, /*

>  INT C */

>  >                                         <0 0 0 4 &pcie_intc0 3>; /*

>  > INT D */

>  

>  If I understand correctly, this is the mapping from the PCI world (INTA,

>  INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)

>  

>  If a Root Port raises a virtual INTA, the RC should turn it into the

>  corresponding platform interrupt, i.e., GIC_SPI 48 from the description

>  below.

>  

>  > And  then

>  >

>  >                         pcie_intc0: legacy-interrupt-controller {

>  >                                 interrupt-controller;

>  >                                 #interrupt-cells = <1>;

>  >                                 interrupt-parent = <&gic>;

>  >                                 interrupts = <GIC_SPI 48

>  IRQ_TYPE_EDGE_RISING>,

>  >                                         <GIC_SPI 49

>  IRQ_TYPE_EDGE_RISING>,

>  >                                         <GIC_SPI 50

>  IRQ_TYPE_EDGE_RISING>,

>  >                                         <GIC_SPI 51

>  IRQ_TYPE_EDGE_RISING>;

>  >                         };

>  >

>  > So if RC interrupt for Power management and Error interrupt is a

>  > separate line, how can software describe that using the above

>  interrupt map?

>  

>  I don't know anything about interrupts the RC might generate on its own

>  behalf.  A lot of the RC behavior is not specified by the PCIe spec

>  because the RC is on the border between the upstream platform-specific

>  stuff and the downstream PCIe stuff.  Is there something in the PCIe

>  spec about the power management and error interrupts you're talking

>  about?  Of maybe you can point me to a section of the Keystone spec that

>  talks about interrupts generated by the RC?


Below is one of the PCIE controller interrupts list: 

142 PEX1 INTA 
143 PEX1 INTB 
144 PEX1 INTC 
145 PEX1 INTD 
146-147 Reserved
148 PEX1 MSI 
149 PEX1 PME 
150 PEX1 CFG err interrupt

Only the "150 PEX1 CFG err interrupt" routing to the aer interrupt. 


>  

>  In any event, the AER interrupts we're looking for in portdrv are from

>  the Root Port, not from the RC.  For INTx, my understanding is that the

>  RC *transforms* virtual INTx messages from the Root Port in the PCIe

>  domain into GIC transactions in the platform domain.

>  

>  Bjorn

On 06/06/2016 03:32 AM, Po Liu wrote:
> Hi Bjorn,
> I confirm we met same problem with KeyStone base on DesignWare design.
> 
> 
> Best regards,
> Liu Po
> 
>>  -----Original Message-----
>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>  Sent: Saturday, June 04, 2016 11:49 AM
>>  To: Murali Karicheri
>>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
>>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
>>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
>>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
>>  Mingkai Hu; Rob Herring
>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>  MSI/MSI-X/INTx mode
>>  
>>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
>>  > Po,
>>  >
>>  > Sorry to hijack your discussion, but the problem seems to be same for
>>  > Keystone PCI controller which is also designware (old version) based.
>>  >
>>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
>>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
>>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>  > >>>>>  -----Original Message-----
>>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>  > >>>>>  To: Po Liu
>>  > >>>>>  Cc: linux-pci@vger.kernel.org;
>>  > >>>>> linux-arm-kernel@lists.infradead.org;
>>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd
>>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;
>>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob
>>  > >>>>> Herring
>>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with
>>  > >>>>> none  MSI/MSI-X/INTx mode
>>  > >>>>>
>>  > >>>>>  [+cc Rob]
>>  > >>>>>
>>  > >>>>>  Hi Po,
>>  > >>>>>
>>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx
>>  in RC mode.
>>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx
>>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search
>>  > >>>>> the interrupt  > number in the fdt file.
>>  > >>>>>
>>  > >>>>>  My understanding is that AER interrupt signaling can be done
>>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).
>>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you
>>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt
>>  you're requesting here different from INTx?
>>  > >>>>
>>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
>>  > >>>> devices or root error in RC mode. But use an independent SPI
>>  > >>>> interrupt(arm interrupt controller) line.
>>  > >>>
>>  > >>> The Root Port is a PCI device and should follow the normal PCI
>>  > >>> rules for interrupts.  As far as I understand, that means it
>>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI
>>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register
>>  > >>> should tell us which (INTA/ INTB/etc.), and
>>  PCI_COMMAND_INTX_DISABLE should work to disable it.
>>  > >>> That's all from the PCI point of view, of course.
>>  > >>
>>  > >> I am faced with the same issue on Keystone PCI hardware and it has
>>  > >> been on my TODO list  for quite some time. Keystone PCI hardware
>>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received
>>  > >> at the root port, but use a platform interrupt instead (not
>>  > >> complaint to PCI standard as per PCI base spec). So I would need
>>  > >> similar change to have the error interrupt passed to the aer
>>  > >> driver. So there are hardware out there like Keystone which
>>  requires to support this through platform IRQ.
>>  > >
>>  > > This is not a new area of the spec, and it's hard for me to believe
>>  > > that these two new PCIe controllers are both broken the same way
>>  > > (although I guess both are DesignWare-based, so maybe this is the
>>  > > same underlying problem in both cases?).  I think it's more likely
>>  > > that we just haven't figured out the right way to describe this in
>>  the DT.
>>  >
>>  > Keystone is using an older version of the designware IP and it
>>  > implements all of the interrupts in the application register space
>>  > unlike other newer version of the hardware. So I assume, the version
>>  > used on Layerscape is also an older version and the both have same
>>  > issue in terms of non standard platform interrupt used for error
>>  reporting.
>>  >
>>  > > I assume you have a Root Port with an AER capability, no MSI
>>  > > capability, and no MSI-X capability, right?
>>  >
>>  > Has AER capability and both MSI and INTx (legacy) capability
>>  >
>>  > > What does its Interrupt
>>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so
>>  > > according to the spec it should generate no interrupts.
>>  > >
>>  > At address offset 0x3C by default has a value of 1, but it is writable
>>  > by software. So default is INTx A.
>>  
>>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
>>  *Pin* is at 0x3d and should be read-only.
>>  

You are right. But default is 1 at this address.

>>  Does your Keystone driver support MSI?  If so, since your Root Port
>>  supports MSI, I would think we would use that by default, and the INTx
>>  stuff wouldn't even matter.
> 
> Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
> But both of them not work for AER interrupt when devices or root port report aer error.
> But another GIC interrupt line do.

Same with Keystone. Even though both MSI and INTx are supported
error interrupt at root port is reported on a different interrupt
line than MSI/INTx. So for Power Management event interrupt is also
different line.

Murali
> 
>>  
>>  > > But if Interrupt Pin is non-zero, that means the Root Port should be
>>  > > able to generate virtual INTx interrupts.  Presumably the Root
>>  > > Complex connects those interrupts to something; maybe to your
>>  > > platform interrupt?
>>  >
>>  > Probably that is what is happening. Both Power management and Error
>>  > interrupts are raised on platform interrupt lines.
>>  >
>>  > 12 Error Interrupts
>>  >
>>  > [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode
>>  > only) [1] PCIe fatal error (RC mode only) [2] PCIe non-fatal error (RC
>>  > mode only) [3] PCIe correctable error (RC mode only) [4] AXI Error due
>>  > to fatal condition in AXI bridge (EP/RC modes) [5] PCIe advanced error
>>  > (RC mode only)
>>  >
>>  > 13 Power management and reset event interrupts
>>  >
>>  > [0] Power management turn-off message interrupt (EP mode only) [1]
>>  > Power management ack message interrupt (RC mode only) [2] Power
>>  > management event interrupt (RC mode only) [3] Link request reset
>>  > interrupt (hot reset or link down) (RC mode only)
>>  >
>>  > > PCI doesn't say anything about an interrupt after it leaves the Root
>>  > > Complex, so the fact that it's connected to a "platform interrupt"
>>  > > or "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-
>>  compliant.
>>  > > Shouldn't we be able to use the interrupt-map and related DT
>>  > > properties to express the fact that Root Port virtual INTx is routed
>>  > > to platform interrupt Y, even without any special-case code in
>>  > > portdrv?
>>  >
>>  > My understanding is if RC also raise interrupt on INTx, then below map
>>  > make sense, where either EP or RC can raise interrupt and the line
>>  > will be muxed for interrupt from EP or RC port.
>>  
>>  I'm sorry, I didn't quite catch your meaning here, so let me try to
>>  clarify some terminology.  Maybe we'll eventually blunder into a common
>>  understanding :)
>>  
>>  INTx is a PCI concept and only means something in the PCI hierarchy.
>>  The RC would *receive* virtual INTx interrupts from the PCI hierarchy
>>  and turn them into some platform-specific interrupt (not INTx) on the
>>  upstream side.
>>  
>>  So strictly speaking, the RC might raise platform-specific interrupts
>>  when it receives INTx interrupts, but it would not raise any INTx
>>  interrupts itself.
>>  
>>  > Here is the DT entry in PCIE keystone for Legacy interrupt mapping to
>>  > host interrupt.
>>  >                         #interrupt-cells = <1>;
>>  >                         interrupt-map-mask = <0 0 0 7>;
>>  >                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /*
>>  INT A */
>>  >                                         <0 0 0 2 &pcie_intc0 1>, /*
>>  INT B */
>>  >                                         <0 0 0 3 &pcie_intc0 2>, /*
>>  INT C */
>>  >                                         <0 0 0 4 &pcie_intc0 3>; /*
>>  > INT D */
>>  
>>  If I understand correctly, this is the mapping from the PCI world (INTA,
>>  INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)
>>  
>>  If a Root Port raises a virtual INTA, the RC should turn it into the
>>  corresponding platform interrupt, i.e., GIC_SPI 48 from the description
>>  below.
>>  
>>  > And  then
>>  >
>>  >                         pcie_intc0: legacy-interrupt-controller {
>>  >                                 interrupt-controller;
>>  >                                 #interrupt-cells = <1>;
>>  >                                 interrupt-parent = <&gic>;
>>  >                                 interrupts = <GIC_SPI 48
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 49
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 50
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 51
>>  IRQ_TYPE_EDGE_RISING>;
>>  >                         };
>>  >
>>  > So if RC interrupt for Power management and Error interrupt is a
>>  > separate line, how can software describe that using the above
>>  interrupt map?
>>  
>>  I don't know anything about interrupts the RC might generate on its own
>>  behalf.  A lot of the RC behavior is not specified by the PCIe spec
>>  because the RC is on the border between the upstream platform-specific
>>  stuff and the downstream PCIe stuff.  Is there something in the PCIe
>>  spec about the power management and error interrupts you're talking
>>  about?  Of maybe you can point me to a section of the Keystone spec that
>>  talks about interrupts generated by the RC?
> 
> Below is one of the PCIE controller interrupts list: 
> 
> 142 PEX1 INTA 
> 143 PEX1 INTB 
> 144 PEX1 INTC 
> 145 PEX1 INTD 
> 146-147 Reserved
> 148 PEX1 MSI 
> 149 PEX1 PME 
> 150 PEX1 CFG err interrupt
> 
> Only the "150 PEX1 CFG err interrupt" routing to the aer interrupt. 
> 
> 
>>  
>>  In any event, the AER interrupts we're looking for in portdrv are from
>>  the Root Port, not from the RC.  For INTx, my understanding is that the
>>  RC *transforms* virtual INTx messages from the Root Port in the PCIe
>>  domain into GIC transactions in the platform domain.
>>  
>>  Bjorn

On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
> On 06/06/2016 03:32 AM, Po Liu wrote:
> > Hi Bjorn,
> > I confirm we met same problem with KeyStone base on DesignWare design.
> > 
> > 
> > Best regards,
> > Liu Po
> > 
> >>  -----Original Message-----
> >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>  Sent: Saturday, June 04, 2016 11:49 AM
> >>  To: Murali Karicheri
> >>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
> >>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> >>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
> >>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
> >>  Mingkai Hu; Rob Herring
> >>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>  MSI/MSI-X/INTx mode
> >>  
> >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> >>  > Po,
> >>  >
> >>  > Sorry to hijack your discussion, but the problem seems to be same for
> >>  > Keystone PCI controller which is also designware (old version) based.
> >>  >
> >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>  > >>>>>  -----Original Message-----
> >>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>  > >>>>>  To: Po Liu
> >>  > >>>>>  Cc: linux-pci@vger.kernel.org;
> >>  > >>>>> linux-arm-kernel@lists.infradead.org;
> >>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd
> >>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;
> >>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob
> >>  > >>>>> Herring
> >>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with
> >>  > >>>>> none  MSI/MSI-X/INTx mode
> >>  > >>>>>
> >>  > >>>>>  [+cc Rob]
> >>  > >>>>>
> >>  > >>>>>  Hi Po,
> >>  > >>>>>
> >>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx
> >>  in RC mode.
> >>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx
> >>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search
> >>  > >>>>> the interrupt  > number in the fdt file.
> >>  > >>>>>
> >>  > >>>>>  My understanding is that AER interrupt signaling can be done
> >>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).
> >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you
> >>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt
> >>  you're requesting here different from INTx?
> >>  > >>>>
> >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
> >>  > >>>> devices or root error in RC mode. But use an independent SPI
> >>  > >>>> interrupt(arm interrupt controller) line.
> >>  > >>>
> >>  > >>> The Root Port is a PCI device and should follow the normal PCI
> >>  > >>> rules for interrupts.  As far as I understand, that means it
> >>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI
> >>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register
> >>  > >>> should tell us which (INTA/ INTB/etc.), and
> >>  PCI_COMMAND_INTX_DISABLE should work to disable it.
> >>  > >>> That's all from the PCI point of view, of course.
> >>  > >>
> >>  > >> I am faced with the same issue on Keystone PCI hardware and it has
> >>  > >> been on my TODO list  for quite some time. Keystone PCI hardware
> >>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received
> >>  > >> at the root port, but use a platform interrupt instead (not
> >>  > >> complaint to PCI standard as per PCI base spec). So I would need
> >>  > >> similar change to have the error interrupt passed to the aer
> >>  > >> driver. So there are hardware out there like Keystone which
> >>  requires to support this through platform IRQ.
> >>  > >
> >>  > > This is not a new area of the spec, and it's hard for me to believe
> >>  > > that these two new PCIe controllers are both broken the same way
> >>  > > (although I guess both are DesignWare-based, so maybe this is the
> >>  > > same underlying problem in both cases?).  I think it's more likely
> >>  > > that we just haven't figured out the right way to describe this in
> >>  the DT.
> >>  >
> >>  > Keystone is using an older version of the designware IP and it
> >>  > implements all of the interrupts in the application register space
> >>  > unlike other newer version of the hardware. So I assume, the version
> >>  > used on Layerscape is also an older version and the both have same
> >>  > issue in terms of non standard platform interrupt used for error
> >>  reporting.
> >>  >
> >>  > > I assume you have a Root Port with an AER capability, no MSI
> >>  > > capability, and no MSI-X capability, right?
> >>  >
> >>  > Has AER capability and both MSI and INTx (legacy) capability
> >>  >
> >>  > > What does its Interrupt
> >>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so
> >>  > > according to the spec it should generate no interrupts.
> >>  > >
> >>  > At address offset 0x3C by default has a value of 1, but it is writable
> >>  > by software. So default is INTx A.
> >>  
> >>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
> >>  *Pin* is at 0x3d and should be read-only.
> >>  
> 
> You are right. But default is 1 at this address.
> 
> >>  Does your Keystone driver support MSI?  If so, since your Root Port
> >>  supports MSI, I would think we would use that by default, and the INTx
> >>  stuff wouldn't even matter.
> > 
> > Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
> > But both of them not work for AER interrupt when devices or root port report aer error.
> > But another GIC interrupt line do.
> 
> Same with Keystone. Even though both MSI and INTx are supported
> error interrupt at root port is reported on a different interrupt
> line than MSI/INTx. So for Power Management event interrupt is also
> different line.

I'm looking at the "Error Message Controls" diagram in the PCIe spec
r3.0, sec 6.2.6.  Does this hardware fit into the platform-specific
"System Error" case there?  Do the Root Control enable bits (in the
PCIe Capability) control this interrupt?  If so, maybe this makes more
sense than I thought.

We have to assume both notification paths work (the normal
INTx/MSI/MSI-X AER interrupts as well as the platform-specific System
Errors), so if we add support for System Errors, it should be
structured so we prefer INTx/MSI/MSI-X, and only fall back to System
Errors if they don't work.  AER would have to know which path it's
using so aer_enable_rootport() can disable the other one (currently it
always disables System Errors).

Then you'd have to add quirks to mark MSI/MSI-X/INTx as being broken
on your devices to force the fallback to System Errors.

How much would this screw up other PCIe services (PME, hotplug, VC,
etc)?  Does MSI work for them?

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

SGkgQmpvcm4sDQoNCj4gIC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ICANCj4gIE9uIE1v
biwgSnVuIDA2LCAyMDE2IGF0IDEwOjAxOjQ0QU0gLTA0MDAsIE11cmFsaSBLYXJpY2hlcmkgd3Jv
dGU6DQo+ICA+IE9uIDA2LzA2LzIwMTYgMDM6MzIgQU0sIFBvIExpdSB3cm90ZToNCj4gID4gPiBI
aSBCam9ybiwNCj4gID4gPiBJIGNvbmZpcm0gd2UgbWV0IHNhbWUgcHJvYmxlbSB3aXRoIEtleVN0
b25lIGJhc2Ugb24gRGVzaWduV2FyZQ0KPiAgZGVzaWduLg0KPiAgPiA+DQo+ICA+ID4NCj4gID4g
PiBCZXN0IHJlZ2FyZHMsDQo+ICA+ID4gTGl1IFBvDQo+ICA+ID4NCj4gID4gPj4gIC0tLS0tT3Jp
Z2luYWwgTWVzc2FnZS0tLS0tDQo+ICA+ID4+ICBGcm9tOiBCam9ybiBIZWxnYWFzIFttYWlsdG86
aGVsZ2Fhc0BrZXJuZWwub3JnXQ0KPiAgPiA+PiAgU2VudDogU2F0dXJkYXksIEp1bmUgMDQsIDIw
MTYgMTE6NDkgQU0NCj4gID4gPj4gIFRvOiBNdXJhbGkgS2FyaWNoZXJpDQo+ICA+ID4+ICBDYzog
UG8gTGl1OyBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBsaW51eC1hcm0tDQo+ICA+ID4+IGtl
cm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnOyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOw0K
PiAgPiA+PiBkZXZpY2V0cmVlQHZnZXIua2VybmVsLm9yZzsgQXJuZCBCZXJnbWFubjsgUm95IFph
bmc7IE1hcmMgWnluZ2llcjsNCj4gID4gPj4gU3R1YXJ0IFlvZGVyOyBZYW5nLUxlbyBMaTsgTWlu
Z2h1YW4gTGlhbjsgQmpvcm4gSGVsZ2FhczsgU2hhd24gR3VvOw0KPiAgPiA+PiBNaW5na2FpIEh1
OyBSb2IgSGVycmluZw0KPiAgPiA+PiAgU3ViamVjdDogUmU6IFtQQVRDSCAyLzJdIGFlcjogYWRk
IHN1cHBvcnQgYWVyIGludGVycnVwdCB3aXRoIG5vbmUNCj4gID4gPj4gTVNJL01TSS1YL0lOVHgg
bW9kZQ0KPiAgPiA+Pg0KPiAgPiA+PiAgT24gRnJpLCBKdW4gMDMsIDIwMTYgYXQgMDE6MzE6MTFQ
TSAtMDQwMCwgTXVyYWxpIEthcmljaGVyaSB3cm90ZToNCj4gID4gPj4gID4gUG8sDQo+ICA+ID4+
ICA+DQo+ICA+ID4+ICA+IFNvcnJ5IHRvIGhpamFjayB5b3VyIGRpc2N1c3Npb24sIGJ1dCB0aGUg
cHJvYmxlbSBzZWVtcyB0byBiZQ0KPiAgPiA+PiBzYW1lIGZvciAgPiBLZXlzdG9uZSBQQ0kgY29u
dHJvbGxlciB3aGljaCBpcyBhbHNvIGRlc2lnbndhcmUgKG9sZA0KPiAgdmVyc2lvbikgYmFzZWQu
DQo+ICA+ID4+ICA+DQo+ICA+ID4+ICA+IE9uIDA2LzAzLzIwMTYgMTI6MDkgQU0sIEJqb3JuIEhl
bGdhYXMgd3JvdGU6DQo+ICA+ID4+ICA+ID4gT24gVGh1LCBKdW4gMDIsIDIwMTYgYXQgMTE6Mzc6
MjhBTSAtMDQwMCwgTXVyYWxpIEthcmljaGVyaQ0KPiAgd3JvdGU6DQo+ICA+ID4+ICA+ID4+IE9u
IDA2LzAyLzIwMTYgMDk6NTUgQU0sIEJqb3JuIEhlbGdhYXMgd3JvdGU6DQo+ICA+ID4+ICA+ID4+
PiBPbiBUaHUsIEp1biAwMiwgMjAxNiBhdCAwNTowMToxOUFNICswMDAwLCBQbyBMaXUgd3JvdGU6
DQo+ICA+ID4+ICA+ID4+Pj4+ICAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLSAgPiA+Pj4+PiAg
RnJvbTogQmpvcm4gSGVsZ2Fhcw0KPiAgPiA+PiBbbWFpbHRvOmhlbGdhYXNAa2VybmVsLm9yZ10g
ID4gPj4+Pj4gIFNlbnQ6IFRodXJzZGF5LCBKdW5lIDAyLCAyMDE2DQo+ICA+ID4+IDExOjQ4IEFN
ICA+ID4+Pj4+ICBUbzogUG8gTGl1ICA+ID4+Pj4+ICBDYzoNCj4gID4gPj4gbGludXgtcGNpQHZn
ZXIua2VybmVsLm9yZzsgID4gPj4+Pj4NCj4gID4gPj4gbGludXgtYXJtLWtlcm5lbEBsaXN0cy5p
bmZyYWRlYWQub3JnOw0KPiAgPiA+PiAgPiA+Pj4+PiAgbGludXgta2VybmVsQHZnZXIua2VybmVs
Lm9yZzsgZGV2aWNldHJlZUB2Z2VyLmtlcm5lbC5vcmc7DQo+ICA+ID4+IEFybmQgID4gPj4+Pj4g
QmVyZ21hbm47ICBSb3kgWmFuZzsgTWFyYyBaeW5naWVyOyBTdHVhcnQgWW9kZXI7DQo+ICA+ID4+
IFlhbmctTGVvIExpOyAgPiA+Pj4+PiBNaW5naHVhbiBMaWFuOyBCam9ybiAgSGVsZ2FhczsgU2hh
d24gR3VvOw0KPiAgPiA+PiBNaW5na2FpIEh1OyBSb2IgID4gPj4+Pj4gSGVycmluZyAgPiA+Pj4+
PiAgU3ViamVjdDogUmU6IFtQQVRDSCAyLzJdDQo+ICA+ID4+IGFlcjogYWRkIHN1cHBvcnQgYWVy
IGludGVycnVwdCB3aXRoICA+ID4+Pj4+IG5vbmUgIE1TSS9NU0ktWC9JTlR4DQo+ICA+ID4+IG1v
ZGUgID4gPj4+Pj4gID4gPj4+Pj4gIFsrY2MgUm9iXSAgPiA+Pj4+PiAgPiA+Pj4+PiAgSGkgUG8s
ICA+DQo+ICA+ID4+ID4+Pj4+ICA+ID4+Pj4+ICBPbiBUaHUsIE1heSAyNiwgMjAxNiBhdCAwMjow
MDowNlBNICswODAwLCBQbyBMaXUNCj4gID4gPj4gd3JvdGU6DQo+ICA+ID4+ICA+ID4+Pj4+ICA+
IE9uIHNvbWUgcGxhdGZvcm1zLCByb290IHBvcnQgZG9lc24ndCBzdXBwb3J0DQo+ICA+ID4+IE1T
SS9NU0ktWC9JTlR4ICBpbiBSQyBtb2RlLg0KPiAgPiA+PiAgPiA+Pj4+PiAgPiBXaGVuIGNoaXAg
c3VwcG9ydCB0aGUgYWVyIGludGVycnVwdCB3aXRoIG5vbmUNCj4gID4gPj4gTVNJL01TSS1YL0lO
VHggID4gPj4+Pj4gbW9kZSwgID4gbWF5YmUgdGhlcmUgaXMgaW50ZXJydXB0IGxpbmUgZm9yDQo+
ICA+ID4+IGFlciBwbWUgZXRjLiBTZWFyY2ggID4gPj4+Pj4gdGhlIGludGVycnVwdCAgPiBudW1i
ZXIgaW4gdGhlIGZkdA0KPiAgZmlsZS4NCj4gID4gPj4gID4gPj4+Pj4NCj4gID4gPj4gID4gPj4+
Pj4gIE15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCBBRVIgaW50ZXJydXB0IHNpZ25hbGluZyBjYW4g
YmUNCj4gID4gPj4gZG9uZSAgPiA+Pj4+PiB2aWEgSU5UeCwgIE1TSSwgb3IgTVNJLVggKFBDSWUg
c3BlYyByMy4wLCBzZWMNCj4gIDYuMi40LjEuMikuDQo+ICA+ID4+ICA+ID4+Pj4+IEFwcGFyZW50
bHkgeW91ciBkZXZpY2UgIGRvZXNuJ3Qgc3VwcG9ydCBNU0kgb3IgTVNJLVguICBBcmUNCj4gID4g
Pj4geW91ICA+ID4+Pj4+IHNheWluZyBpdCBkb2Vzbid0IHN1cHBvcnQgSU5UeCAgZWl0aGVyPyAg
SG93IGlzIHRoZQ0KPiAgPiA+PiBpbnRlcnJ1cHQgIHlvdSdyZSByZXF1ZXN0aW5nIGhlcmUgZGlm
ZmVyZW50IGZyb20gSU5UeD8NCj4gID4gPj4gID4gPj4+Pg0KPiAgPiA+PiAgPiA+Pj4+IExheWVy
c2NhcGUgdXNlIG5vbmUgb2YgTVNJIG9yIE1TSS1YIG9yIElOVHggdG8gaW5kaWNhdGUgdGhlDQo+
ICA+ID4+ID4gPj4+PiBkZXZpY2VzIG9yIHJvb3QgZXJyb3IgaW4gUkMgbW9kZS4gQnV0IHVzZSBh
biBpbmRlcGVuZGVudCBTUEkNCj4gID4gPj4gPiA+Pj4+IGludGVycnVwdChhcm0gaW50ZXJydXB0
IGNvbnRyb2xsZXIpIGxpbmUuDQo+ICA+ID4+ICA+ID4+Pg0KPiAgPiA+PiAgPiA+Pj4gVGhlIFJv
b3QgUG9ydCBpcyBhIFBDSSBkZXZpY2UgYW5kIHNob3VsZCBmb2xsb3cgdGhlIG5vcm1hbA0KPiAg
PiA+PiBQQ0kgID4gPj4+IHJ1bGVzIGZvciBpbnRlcnJ1cHRzLiAgQXMgZmFyIGFzIEkgdW5kZXJz
dGFuZCwgdGhhdA0KPiAgPiA+PiBtZWFucyBpdCAgPiA+Pj4gc2hvdWxkIHVzZSBNU0ksIE1TSS1Y
LCBvciBJTlR4LiAgSWYgeW91ciBSb290IFBvcnQNCj4gID4gPj4gZG9lc24ndCB1c2UgTVNJICA+
ID4+PiBvciBNU0ktWCwgaXQgc2hvdWxkIHVzZSBJTlR4LCB0aGUNCj4gID4gPj4gUENJX0lOVEVS
UlVQVF9QSU4gcmVnaXN0ZXIgID4gPj4+IHNob3VsZCB0ZWxsIHVzIHdoaWNoIChJTlRBLw0KPiAg
PiA+PiBJTlRCL2V0Yy4pLCBhbmQgIFBDSV9DT01NQU5EX0lOVFhfRElTQUJMRSBzaG91bGQgd29y
ayB0byBkaXNhYmxlIGl0Lg0KPiAgPiA+PiAgPiA+Pj4gVGhhdCdzIGFsbCBmcm9tIHRoZSBQQ0kg
cG9pbnQgb2Ygdmlldywgb2YgY291cnNlLg0KPiAgPiA+PiAgPiA+Pg0KPiAgPiA+PiAgPiA+PiBJ
IGFtIGZhY2VkIHdpdGggdGhlIHNhbWUgaXNzdWUgb24gS2V5c3RvbmUgUENJIGhhcmR3YXJlIGFu
ZA0KPiAgPiA+PiBpdCBoYXMgID4gPj4gYmVlbiBvbiBteSBUT0RPIGxpc3QgIGZvciBxdWl0ZSBz
b21lIHRpbWUuIEtleXN0b25lDQo+ICA+ID4+IFBDSSBoYXJkd2FyZSAgPiA+PiBhbHNvIGRvZXNu
J3QgdXNlIE1TSSBvciBNU0ktWCBvciBJTlR4IGZvcg0KPiAgPiA+PiByZXBvcnRpbmcgZXJyb3Jz
IHJlY2VpdmVkICA+ID4+IGF0IHRoZSByb290IHBvcnQsIGJ1dCB1c2UgYQ0KPiAgPiA+PiBwbGF0
Zm9ybSBpbnRlcnJ1cHQgaW5zdGVhZCAobm90ICA+ID4+IGNvbXBsYWludCB0byBQQ0kgc3RhbmRh
cmQgYXMNCj4gID4gPj4gcGVyIFBDSSBiYXNlIHNwZWMpLiBTbyBJIHdvdWxkIG5lZWQgID4gPj4g
c2ltaWxhciBjaGFuZ2UgdG8gaGF2ZQ0KPiAgPiA+PiB0aGUgZXJyb3IgaW50ZXJydXB0IHBhc3Nl
ZCB0byB0aGUgYWVyICA+ID4+IGRyaXZlci4gU28gdGhlcmUgYXJlDQo+ICA+ID4+IGhhcmR3YXJl
IG91dCB0aGVyZSBsaWtlIEtleXN0b25lIHdoaWNoICByZXF1aXJlcyB0byBzdXBwb3J0IHRoaXMN
Cj4gIHRocm91Z2ggcGxhdGZvcm0gSVJRLg0KPiAgPiA+PiAgPiA+DQo+ICA+ID4+ICA+ID4gVGhp
cyBpcyBub3QgYSBuZXcgYXJlYSBvZiB0aGUgc3BlYywgYW5kIGl0J3MgaGFyZCBmb3IgbWUgdG8N
Cj4gID4gPj4gYmVsaWV2ZSAgPiA+IHRoYXQgdGhlc2UgdHdvIG5ldyBQQ0llIGNvbnRyb2xsZXJz
IGFyZSBib3RoIGJyb2tlbg0KPiAgPiA+PiB0aGUgc2FtZSB3YXkgID4gPiAoYWx0aG91Z2ggSSBn
dWVzcyBib3RoIGFyZSBEZXNpZ25XYXJlLWJhc2VkLCBzbw0KPiAgPiA+PiBtYXliZSB0aGlzIGlz
IHRoZSAgPiA+IHNhbWUgdW5kZXJseWluZyBwcm9ibGVtIGluIGJvdGggY2FzZXM/KS4gIEkNCj4g
ID4gPj4gdGhpbmsgaXQncyBtb3JlIGxpa2VseSAgPiA+IHRoYXQgd2UganVzdCBoYXZlbid0IGZp
Z3VyZWQgb3V0IHRoZQ0KPiAgPiA+PiByaWdodCB3YXkgdG8gZGVzY3JpYmUgdGhpcyBpbiAgdGhl
IERULg0KPiAgPiA+PiAgPg0KPiAgPiA+PiAgPiBLZXlzdG9uZSBpcyB1c2luZyBhbiBvbGRlciB2
ZXJzaW9uIG9mIHRoZSBkZXNpZ253YXJlIElQIGFuZCBpdA0KPiAgPiA+PiA+IGltcGxlbWVudHMg
YWxsIG9mIHRoZSBpbnRlcnJ1cHRzIGluIHRoZSBhcHBsaWNhdGlvbiByZWdpc3Rlcg0KPiAgPiA+
PiBzcGFjZSAgPiB1bmxpa2Ugb3RoZXIgbmV3ZXIgdmVyc2lvbiBvZiB0aGUgaGFyZHdhcmUuIFNv
IEkgYXNzdW1lLA0KPiAgPiA+PiB0aGUgdmVyc2lvbiAgPiB1c2VkIG9uIExheWVyc2NhcGUgaXMg
YWxzbyBhbiBvbGRlciB2ZXJzaW9uIGFuZCB0aGUNCj4gID4gPj4gYm90aCBoYXZlIHNhbWUgID4g
aXNzdWUgaW4gdGVybXMgb2Ygbm9uIHN0YW5kYXJkIHBsYXRmb3JtIGludGVycnVwdA0KPiAgPiA+
PiB1c2VkIGZvciBlcnJvciAgcmVwb3J0aW5nLg0KPiAgPiA+PiAgPg0KPiAgPiA+PiAgPiA+IEkg
YXNzdW1lIHlvdSBoYXZlIGEgUm9vdCBQb3J0IHdpdGggYW4gQUVSIGNhcGFiaWxpdHksIG5vIE1T
SQ0KPiAgPiA+PiA+ID4gY2FwYWJpbGl0eSwgYW5kIG5vIE1TSS1YIGNhcGFiaWxpdHksIHJpZ2h0
Pw0KPiAgPiA+PiAgPg0KPiAgPiA+PiAgPiBIYXMgQUVSIGNhcGFiaWxpdHkgYW5kIGJvdGggTVNJ
IGFuZCBJTlR4IChsZWdhY3kpIGNhcGFiaWxpdHkgID4NCj4gID4gPj4gPiA+IFdoYXQgZG9lcyBp
dHMgSW50ZXJydXB0ICA+ID4gUGluIHJlZ2lzdGVyIGNvbnRhaW4/ICBJZiBpdCdzDQo+ICA+ID4+
IHplcm8sIGl0IGRvZXNuJ3QgdXNlIElOVHggZWl0aGVyLCBzbyAgPiA+IGFjY29yZGluZyB0byB0
aGUgc3BlYyBpdA0KPiAgPiA+PiBzaG91bGQgZ2VuZXJhdGUgbm8gaW50ZXJydXB0cy4NCj4gID4g
Pj4gID4gPg0KPiAgPiA+PiAgPiBBdCBhZGRyZXNzIG9mZnNldCAweDNDIGJ5IGRlZmF1bHQgaGFz
IGEgdmFsdWUgb2YgMSwgYnV0IGl0IGlzDQo+ICA+ID4+IHdyaXRhYmxlICA+IGJ5IHNvZnR3YXJl
LiBTbyBkZWZhdWx0IGlzIElOVHggQS4NCj4gID4gPj4NCj4gID4gPj4gIDB4M2MgaXMgdGhlIElu
dGVycnVwdCAqTGluZSosIHdoaWNoIGlzIHJlYWQvd3JpdGUuICBUaGUgSW50ZXJydXB0DQo+ICA+
ID4+ICAqUGluKiBpcyBhdCAweDNkIGFuZCBzaG91bGQgYmUgcmVhZC1vbmx5Lg0KPiAgPiA+Pg0K
PiAgPg0KPiAgPiBZb3UgYXJlIHJpZ2h0LiBCdXQgZGVmYXVsdCBpcyAxIGF0IHRoaXMgYWRkcmVz
cy4NCj4gID4NCj4gID4gPj4gIERvZXMgeW91ciBLZXlzdG9uZSBkcml2ZXIgc3VwcG9ydCBNU0k/
ICBJZiBzbywgc2luY2UgeW91ciBSb290DQo+ICA+ID4+IFBvcnQgIHN1cHBvcnRzIE1TSSwgSSB3
b3VsZCB0aGluayB3ZSB3b3VsZCB1c2UgdGhhdCBieSBkZWZhdWx0LCBhbmQNCj4gID4gPj4gdGhl
IElOVHggIHN0dWZmIHdvdWxkbid0IGV2ZW4gbWF0dGVyLg0KPiAgPiA+DQo+ICA+ID4gTGF5ZXJz
Y2FwZSBpcyBhbHNvIHNob3dzICJCb3RoIG1lc3NhZ2Ugc2lnbmFsZWQgaW50ZXJydXB0cyAoTVNJ
KSBhbmQNCj4gIGxlZ2FjeSBJTlR4IGFyZSBzdXBwb3J0ZWQuIg0KPiAgPiA+IEJ1dCBib3RoIG9m
IHRoZW0gbm90IHdvcmsgZm9yIEFFUiBpbnRlcnJ1cHQgd2hlbiBkZXZpY2VzIG9yIHJvb3QNCj4g
IHBvcnQgcmVwb3J0IGFlciBlcnJvci4NCj4gID4gPiBCdXQgYW5vdGhlciBHSUMgaW50ZXJydXB0
IGxpbmUgZG8uDQo+ICA+DQo+ICA+IFNhbWUgd2l0aCBLZXlzdG9uZS4gRXZlbiB0aG91Z2ggYm90
aCBNU0kgYW5kIElOVHggYXJlIHN1cHBvcnRlZCBlcnJvcg0KPiAgPiBpbnRlcnJ1cHQgYXQgcm9v
dCBwb3J0IGlzIHJlcG9ydGVkIG9uIGEgZGlmZmVyZW50IGludGVycnVwdCBsaW5lIHRoYW4NCj4g
ID4gTVNJL0lOVHguIFNvIGZvciBQb3dlciBNYW5hZ2VtZW50IGV2ZW50IGludGVycnVwdCBpcyBh
bHNvIGRpZmZlcmVudA0KPiAgPiBsaW5lLg0KPiAgDQo+ICBJJ20gbG9va2luZyBhdCB0aGUgIkVy
cm9yIE1lc3NhZ2UgQ29udHJvbHMiIGRpYWdyYW0gaW4gdGhlIFBDSWUgc3BlYw0KPiAgcjMuMCwg
c2VjIDYuMi42LiAgRG9lcyB0aGlzIGhhcmR3YXJlIGZpdCBpbnRvIHRoZSBwbGF0Zm9ybS1zcGVj
aWZpYw0KPiAgIlN5c3RlbSBFcnJvciIgY2FzZSB0aGVyZT8gIERvIHRoZSBSb290IENvbnRyb2wg
ZW5hYmxlIGJpdHMgKGluIHRoZSBQQ0llDQo+ICBDYXBhYmlsaXR5KSBjb250cm9sIHRoaXMgaW50
ZXJydXB0PyAgSWYgc28sIG1heWJlIHRoaXMgbWFrZXMgbW9yZSBzZW5zZQ0KPiAgdGhhbiBJIHRo
b3VnaHQuDQoNCkl0IHN1cHBvc2VkbHkgbm90IHRoZSAiU3lzdGVtIEVycm9yIiBjYXNlLiBCdXQg
InRoZSBFcnJvciBJbnRlcnJ1cHQiIGNhc2UuDQpXaGljaCBtZWFucyAiIFJvb3QgRXJyb3IgQ29t
bWFuZCByZWdpc3RlciAiIGNvdWxkIGNvbnRyb2wgdGhlIGludGVycnVwdA0KIGxpbmUgd2UgaGF2
ZSBub3cuIChyZWZlciBQQ0llIHNwZWMgcjMuMCwgc2VjIDYuMi42KQ0KDQpNYXkgdGhpcyBraW5k
IG9mIGhhcmR3YXJlIGRlc2lnbiByb3V0ZSBicm9rZW4gdGhlIHNwZWM/IA0KDQo+ICANCj4gIFdl
IGhhdmUgdG8gYXNzdW1lIGJvdGggbm90aWZpY2F0aW9uIHBhdGhzIHdvcmsgKHRoZSBub3JtYWwg
SU5UeC9NU0kvTVNJLQ0KPiAgWCBBRVIgaW50ZXJydXB0cyBhcyB3ZWxsIGFzIHRoZSBwbGF0Zm9y
bS1zcGVjaWZpYyBTeXN0ZW0gRXJyb3JzKSwgc28gaWYNCj4gIHdlIGFkZCBzdXBwb3J0IGZvciBT
eXN0ZW0gRXJyb3JzLCBpdCBzaG91bGQgYmUgc3RydWN0dXJlZCBzbyB3ZSBwcmVmZXINCj4gIElO
VHgvTVNJL01TSS1YLCBhbmQgb25seSBmYWxsIGJhY2sgdG8gU3lzdGVtIEVycm9ycyBpZiB0aGV5
IGRvbid0IHdvcmsuDQo+ICBBRVIgd291bGQgaGF2ZSB0byBrbm93IHdoaWNoIHBhdGggaXQncyB1
c2luZyBzbyBhZXJfZW5hYmxlX3Jvb3Rwb3J0KCkNCj4gIGNhbiBkaXNhYmxlIHRoZSBvdGhlciBv
bmUgKGN1cnJlbnRseSBpdCBhbHdheXMgZGlzYWJsZXMgU3lzdGVtIEVycm9ycykuDQo+ICANCj4g
IFRoZW4geW91J2QgaGF2ZSB0byBhZGQgcXVpcmtzIHRvIG1hcmsgTVNJL01TSS1YL0lOVHggYXMg
YmVpbmcgYnJva2VuIG9uDQo+ICB5b3VyIGRldmljZXMgdG8gZm9yY2UgdGhlIGZhbGxiYWNrIHRv
IFN5c3RlbSBFcnJvcnMuDQo+ICANCj4gIEhvdyBtdWNoIHdvdWxkIHRoaXMgc2NyZXcgdXAgb3Ro
ZXIgUENJZSBzZXJ2aWNlcyAoUE1FLCBob3RwbHVnLCBWQywgZXRjKT8NCj4gIERvZXMgTVNJIHdv
cmsgZm9yIHRoZW0/DQoNClBNRSBhbHNvIGxpa2UgdGhlIEFFUi4gSG90cGx1ZyBpcyBub3Qgc3Vw
cG9ydGVkLiBPdGhlcnMgbm90IGtub3duLg0KUG8gTGl1DQoNCj4gIA0KPiAgQmpvcm4NCg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Jun 07, 2016 at 10:07:40AM +0000, Po Liu wrote:
> Hi Bjorn,
> 
> >  -----Original Message-----
> >  
> >  On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
> >  > On 06/06/2016 03:32 AM, Po Liu wrote:
> >  > > Hi Bjorn,
> >  > > I confirm we met same problem with KeyStone base on DesignWare
> >  design.
> >  > >
> >  > >
> >  > > Best regards,
> >  > > Liu Po
> >  > >
> >  > >>  -----Original Message-----
> >  > >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >  > >>  Sent: Saturday, June 04, 2016 11:49 AM
> >  > >>  To: Murali Karicheri
> >  > >>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
> >  > >> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> >  > >> devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
> >  > >> Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
> >  > >> Mingkai Hu; Rob Herring
> >  > >>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >  > >> MSI/MSI-X/INTx mode
> >  > >>
> >  > >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> >  > >>  > Po,
> >  > >>  >
> >  > >>  > Sorry to hijack your discussion, but the problem seems to be
> >  > >> same for  > Keystone PCI controller which is also designware (old
> >  version) based.
> >  > >>  >
> >  > >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> >  > >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri
> >  wrote:
> >  > >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >  > >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >  > >>  > >>>>>  -----Original Message-----  > >>>>>  From: Bjorn Helgaas
> >  > >> [mailto:helgaas@kernel.org]  > >>>>>  Sent: Thursday, June 02, 2016
> >  > >> 11:48 AM  > >>>>>  To: Po Liu  > >>>>>  Cc:
> >  > >> linux-pci@vger.kernel.org;  > >>>>>
> >  > >> linux-arm-kernel@lists.infradead.org;
> >  > >>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org;
> >  > >> Arnd  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder;
> >  > >> Yang-Leo Li;  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo;
> >  > >> Mingkai Hu; Rob  > >>>>> Herring  > >>>>>  Subject: Re: [PATCH 2/2]
> >  > >> aer: add support aer interrupt with  > >>>>> none  MSI/MSI-X/INTx
> >  > >> mode  > >>>>>  > >>>>>  [+cc Rob]  > >>>>>  > >>>>>  Hi Po,  >
> >  > >> >>>>>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu
> >  > >> wrote:
> >  > >>  > >>>>>  > On some platforms, root port doesn't support
> >  > >> MSI/MSI-X/INTx  in RC mode.
> >  > >>  > >>>>>  > When chip support the aer interrupt with none
> >  > >> MSI/MSI-X/INTx  > >>>>> mode,  > maybe there is interrupt line for
> >  > >> aer pme etc. Search  > >>>>> the interrupt  > number in the fdt
> >  file.
> >  > >>  > >>>>>
> >  > >>  > >>>>>  My understanding is that AER interrupt signaling can be
> >  > >> done  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec
> >  6.2.4.1.2).
> >  > >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are
> >  > >> you  > >>>>> saying it doesn't support INTx  either?  How is the
> >  > >> interrupt  you're requesting here different from INTx?
> >  > >>  > >>>>
> >  > >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
> >  > >> > >>>> devices or root error in RC mode. But use an independent SPI
> >  > >> > >>>> interrupt(arm interrupt controller) line.
> >  > >>  > >>>
> >  > >>  > >>> The Root Port is a PCI device and should follow the normal
> >  > >> PCI  > >>> rules for interrupts.  As far as I understand, that
> >  > >> means it  > >>> should use MSI, MSI-X, or INTx.  If your Root Port
> >  > >> doesn't use MSI  > >>> or MSI-X, it should use INTx, the
> >  > >> PCI_INTERRUPT_PIN register  > >>> should tell us which (INTA/
> >  > >> INTB/etc.), and  PCI_COMMAND_INTX_DISABLE should work to disable it.
> >  > >>  > >>> That's all from the PCI point of view, of course.
> >  > >>  > >>
> >  > >>  > >> I am faced with the same issue on Keystone PCI hardware and
> >  > >> it has  > >> been on my TODO list  for quite some time. Keystone
> >  > >> PCI hardware  > >> also doesn't use MSI or MSI-X or INTx for
> >  > >> reporting errors received  > >> at the root port, but use a
> >  > >> platform interrupt instead (not  > >> complaint to PCI standard as
> >  > >> per PCI base spec). So I would need  > >> similar change to have
> >  > >> the error interrupt passed to the aer  > >> driver. So there are
> >  > >> hardware out there like Keystone which  requires to support this
> >  through platform IRQ.
> >  > >>  > >
> >  > >>  > > This is not a new area of the spec, and it's hard for me to
> >  > >> believe  > > that these two new PCIe controllers are both broken
> >  > >> the same way  > > (although I guess both are DesignWare-based, so
> >  > >> maybe this is the  > > same underlying problem in both cases?).  I
> >  > >> think it's more likely  > > that we just haven't figured out the
> >  > >> right way to describe this in  the DT.
> >  > >>  >
> >  > >>  > Keystone is using an older version of the designware IP and it
> >  > >> > implements all of the interrupts in the application register
> >  > >> space  > unlike other newer version of the hardware. So I assume,
> >  > >> the version  > used on Layerscape is also an older version and the
> >  > >> both have same  > issue in terms of non standard platform interrupt
> >  > >> used for error  reporting.
> >  > >>  >
> >  > >>  > > I assume you have a Root Port with an AER capability, no MSI
> >  > >> > > capability, and no MSI-X capability, right?
> >  > >>  >
> >  > >>  > Has AER capability and both MSI and INTx (legacy) capability  >
> >  > >> > > What does its Interrupt  > > Pin register contain?  If it's
> >  > >> zero, it doesn't use INTx either, so  > > according to the spec it
> >  > >> should generate no interrupts.
> >  > >>  > >
> >  > >>  > At address offset 0x3C by default has a value of 1, but it is
> >  > >> writable  > by software. So default is INTx A.
> >  > >>
> >  > >>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
> >  > >>  *Pin* is at 0x3d and should be read-only.
> >  > >>
> >  >
> >  > You are right. But default is 1 at this address.
> >  >
> >  > >>  Does your Keystone driver support MSI?  If so, since your Root
> >  > >> Port  supports MSI, I would think we would use that by default, and
> >  > >> the INTx  stuff wouldn't even matter.
> >  > >
> >  > > Layerscape is also shows "Both message signaled interrupts (MSI) and
> >  legacy INTx are supported."
> >  > > But both of them not work for AER interrupt when devices or root
> >  port report aer error.
> >  > > But another GIC interrupt line do.
> >  >
> >  > Same with Keystone. Even though both MSI and INTx are supported error
> >  > interrupt at root port is reported on a different interrupt line than
> >  > MSI/INTx. So for Power Management event interrupt is also different
> >  > line.
> >  
> >  I'm looking at the "Error Message Controls" diagram in the PCIe spec
> >  r3.0, sec 6.2.6.  Does this hardware fit into the platform-specific
> >  "System Error" case there?  Do the Root Control enable bits (in the PCIe
> >  Capability) control this interrupt?  If so, maybe this makes more sense
> >  than I thought.
> 
> It supposedly not the "System Error" case. But "the Error Interrupt" case.
> Which means " Root Error Command register " could control the interrupt
>  line we have now. (refer PCIe spec r3.0, sec 6.2.6)

Did you actually try this out and verify that the PCIe Root Control
enable bits have no effect and the AER Root Error Command bits do
control it?  The names are very similar, so there's lots of room for
misunderstanding here :)

If the AER Root Error Command does control this interrupt, I think the
PCI_COMMAND_INTX_DISABLE bit in the PCI Command register should also
control it (per sec 6.2.4.1.2).

> May this kind of hardware design route broken the spec? 

If the Reporting Enable bits in the Root Port's AER Root Error Command
register control the interrupt, but the interrupt is not delivered via
the Root Port's INTx or MSI/MSI-X, I think the design is not following
the spec.

All the information needed by the AER driver should be communicated
via the config space mechanisms described in the spec (AER capability,
MSI/MSI-X capabilities, Interrupt Pin, etc.)  That way the driver
works without change on future spec-compliant hardware.

> PME also like the AER. Hotplug is not supported. Others not known.
> Po Liu

Per sec 6.1.6, I think PME *should* be signaled by the Root Port's
INTx or MSI/MSI-X.

In particular, it says "Note that all other interrupt sources within
the same Function will assert the same virtual INTx wire when
requesting service."  To me, that means that if we're using INTx, it
will be the same INTx for AER, PME, hotplug, etc., and it should be
the one indicated by the Interrupt Pin register.

But I think on your Root Port:

  - There is an MSI capability, but MSI doesn't actually work at all
  - Interrupt Pin contains 1, indicating INTA, which is routed to IRQ X
  - AER interrupts are routed to some different IRQ Y
  - PME interrupts are routed to a third IRQ Z

So how should we work around this?  I think you should be able to get
partway there with a quirk that sets:

  dev->no_msi = 1;
  dev->irq = Y;

for this device.  That should make AER work, but of course PME would
not work.

Is there a way to set up your interrupt controller so these three
interrupts (X, Y, Z above) all map to the same Linux IRQ?  If you can
do that, you could set up INTA, the AER interrupt, and the PME
interrupt to all be on the same IRQ and everything should work.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

SGkgQmpvcm4sDQoNClRoYW5rcyBmb3IgdGhlIGtpbmRseSByZXBseS4gQWxsIHRoZXNlIGFyZSBo
ZWxwZnVsLg0KDQo+ICBGcm9tOiBCam9ybiBIZWxnYWFzIFttYWlsdG86aGVsZ2Fhc0BrZXJuZWwu
b3JnXQ0KPiAgT24gV2VkLCBKdW5lIDA4LCAyMDE2IDY6NDcgQU0NCj4gIA0KPiAgT24gVHVlLCBK
dW4gMDcsIDIwMTYgYXQgMTA6MDc6NDBBTSArMDAwMCwgUG8gTGl1IHdyb3RlOg0KPiAgPiBIaSBC
am9ybiwNCj4gID4NCj4gID4gPiAgLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gID4gPg0K
PiAgPiA+ICBPbiBNb24sIEp1biAwNiwgMjAxNiBhdCAxMDowMTo0NEFNIC0wNDAwLCBNdXJhbGkg
S2FyaWNoZXJpIHdyb3RlOg0KPiAgPiA+ICA+IE9uIDA2LzA2LzIwMTYgMDM6MzIgQU0sIFBvIExp
dSB3cm90ZToNCj4gID4gPiAgPiA+IEhpIEJqb3JuLA0KPiAgPiA+ICA+ID4gSSBjb25maXJtIHdl
IG1ldCBzYW1lIHByb2JsZW0gd2l0aCBLZXlTdG9uZSBiYXNlIG9uIERlc2lnbldhcmUNCj4gID4g
PiBkZXNpZ24uDQo+ICA+ID4gID4gPg0KPiAgPiA+ICA+ID4NCj4gID4gPiAgPiA+IEJlc3QgcmVn
YXJkcywNCj4gID4gPiAgPiA+IExpdSBQbw0KPiAgPiA+ICA+ID4NCj4gID4gPiAgPiA+PiAgLS0t
LS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gID4gPiAgPiA+PiAgRnJvbTogQmpvcm4gSGVsZ2Fh
cyBbbWFpbHRvOmhlbGdhYXNAa2VybmVsLm9yZ10gID4gPj4gIFNlbnQ6DQo+ICA+ID4gU2F0dXJk
YXksIEp1bmUgMDQsIDIwMTYgMTE6NDkgQU0gID4gPj4gIFRvOiBNdXJhbGkgS2FyaWNoZXJpICA+
ID4+DQo+ICA+ID4gQ2M6IFBvIExpdTsgbGludXgtcGNpQHZnZXIua2VybmVsLm9yZzsgbGludXgt
YXJtLSAgPiA+Pg0KPiAgPiA+IGtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnOyBsaW51eC1rZXJu
ZWxAdmdlci5rZXJuZWwub3JnOyAgPiA+Pg0KPiAgPiA+IGRldmljZXRyZWVAdmdlci5rZXJuZWwu
b3JnOyBBcm5kIEJlcmdtYW5uOyBSb3kgWmFuZzsgTWFyYyBaeW5naWVyOw0KPiAgPiA+ID4gPj4g
U3R1YXJ0IFlvZGVyOyBZYW5nLUxlbyBMaTsgTWluZ2h1YW4gTGlhbjsgQmpvcm4gSGVsZ2Fhczsg
U2hhd24NCj4gID4gPiBHdW87ICA+ID4+IE1pbmdrYWkgSHU7IFJvYiBIZXJyaW5nICA+ID4+ICBT
dWJqZWN0OiBSZTogW1BBVENIIDIvMl0NCj4gID4gPiBhZXI6IGFkZCBzdXBwb3J0IGFlciBpbnRl
cnJ1cHQgd2l0aCBub25lICA+ID4+IE1TSS9NU0ktWC9JTlR4IG1vZGUNCj4gID4gPiA+ID4+ICA+
ID4+ICBPbiBGcmksIEp1biAwMywgMjAxNiBhdCAwMTozMToxMVBNIC0wNDAwLCBNdXJhbGkNCj4g
ID4gPiBLYXJpY2hlcmkgd3JvdGU6DQo+ICA+ID4gID4gPj4gID4gUG8sDQo+ICA+ID4gID4gPj4g
ID4NCj4gID4gPiAgPiA+PiAgPiBTb3JyeSB0byBoaWphY2sgeW91ciBkaXNjdXNzaW9uLCBidXQg
dGhlIHByb2JsZW0gc2VlbXMgdG8NCj4gID4gPiBiZSAgPiA+PiBzYW1lIGZvciAgPiBLZXlzdG9u
ZSBQQ0kgY29udHJvbGxlciB3aGljaCBpcyBhbHNvDQo+ICA+ID4gZGVzaWdud2FyZSAob2xkDQo+
ICA+ID4gIHZlcnNpb24pIGJhc2VkLg0KPiAgPiA+ICA+ID4+ICA+DQo+ICA+ID4gID4gPj4gID4g
T24gMDYvMDMvMjAxNiAxMjowOSBBTSwgQmpvcm4gSGVsZ2FhcyB3cm90ZToNCj4gID4gPiAgPiA+
PiAgPiA+IE9uIFRodSwgSnVuIDAyLCAyMDE2IGF0IDExOjM3OjI4QU0gLTA0MDAsIE11cmFsaQ0K
PiAgPiA+IEthcmljaGVyaQ0KPiAgPiA+ICB3cm90ZToNCj4gID4gPiAgPiA+PiAgPiA+PiBPbiAw
Ni8wMi8yMDE2IDA5OjU1IEFNLCBCam9ybiBIZWxnYWFzIHdyb3RlOg0KPiAgPiA+ICA+ID4+ICA+
ID4+PiBPbiBUaHUsIEp1biAwMiwgMjAxNiBhdCAwNTowMToxOUFNICswMDAwLCBQbyBMaXUgd3Jv
dGU6DQo+ICA+ID4gID4gPj4gID4gPj4+Pj4gIC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tICA+
ID4+Pj4+ICBGcm9tOiBCam9ybg0KPiAgPiA+IEhlbGdhYXMgID4gPj4gW21haWx0bzpoZWxnYWFz
QGtlcm5lbC5vcmddICA+ID4+Pj4+ICBTZW50OiBUaHVyc2RheSwNCj4gID4gPiBKdW5lIDAyLCAy
MDE2ICA+ID4+IDExOjQ4IEFNICA+ID4+Pj4+ICBUbzogUG8gTGl1ICA+ID4+Pj4+ICBDYzoNCj4g
ID4gPiAgPiA+PiBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyAgPiA+Pj4+PiAgPiA+Pg0KPiAg
PiA+IGxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZzsNCj4gID4gPiAgPiA+PiAg
PiA+Pj4+PiAgbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZzsNCj4gID4gPiBkZXZpY2V0cmVl
QHZnZXIua2VybmVsLm9yZzsgID4gPj4gQXJuZCAgPiA+Pj4+PiBCZXJnbWFubjsgIFJveSBaYW5n
Ow0KPiAgPiA+IE1hcmMgWnluZ2llcjsgU3R1YXJ0IFlvZGVyOyAgPiA+PiBZYW5nLUxlbyBMaTsg
ID4gPj4+Pj4gTWluZ2h1YW4NCj4gID4gPiBMaWFuOyBCam9ybiAgSGVsZ2FhczsgU2hhd24gR3Vv
OyAgPiA+PiBNaW5na2FpIEh1OyBSb2IgID4gPj4+Pj4NCj4gID4gPiBIZXJyaW5nICA+ID4+Pj4+
ICBTdWJqZWN0OiBSZTogW1BBVENIIDIvMl0gID4gPj4gYWVyOiBhZGQgc3VwcG9ydA0KPiAgPiA+
IGFlciBpbnRlcnJ1cHQgd2l0aCAgPiA+Pj4+PiBub25lICBNU0kvTVNJLVgvSU5UeCAgPiA+PiBt
b2RlICA+ID4+Pj4+DQo+ICA+ID4gPiA+Pj4+PiAgWytjYyBSb2JdICA+ID4+Pj4+ICA+ID4+Pj4+
ICBIaSBQbywgID4gID4gPj4gPj4+Pj4gID4gPj4+Pj4NCj4gID4gPiBPbiBUaHUsIE1heSAyNiwg
MjAxNiBhdCAwMjowMDowNlBNICswODAwLCBQbyBMaXUgID4gPj4gd3JvdGU6DQo+ICA+ID4gID4g
Pj4gID4gPj4+Pj4gID4gT24gc29tZSBwbGF0Zm9ybXMsIHJvb3QgcG9ydCBkb2Vzbid0IHN1cHBv
cnQgID4gPj4NCj4gID4gPiBNU0kvTVNJLVgvSU5UeCAgaW4gUkMgbW9kZS4NCj4gID4gPiAgPiA+
PiAgPiA+Pj4+PiAgPiBXaGVuIGNoaXAgc3VwcG9ydCB0aGUgYWVyIGludGVycnVwdCB3aXRoIG5v
bmUgID4NCj4gID4gPiA+PiBNU0kvTVNJLVgvSU5UeCAgPiA+Pj4+PiBtb2RlLCAgPiBtYXliZSB0
aGVyZSBpcyBpbnRlcnJ1cHQgbGluZQ0KPiAgPiA+IGZvciAgPiA+PiBhZXIgcG1lIGV0Yy4gU2Vh
cmNoICA+ID4+Pj4+IHRoZSBpbnRlcnJ1cHQgID4gbnVtYmVyIGluDQo+ICA+ID4gdGhlIGZkdCAg
ZmlsZS4NCj4gID4gPiAgPiA+PiAgPiA+Pj4+Pg0KPiAgPiA+ICA+ID4+ICA+ID4+Pj4+ICBNeSB1
bmRlcnN0YW5kaW5nIGlzIHRoYXQgQUVSIGludGVycnVwdCBzaWduYWxpbmcgY2FuDQo+ICA+ID4g
YmUgID4gPj4gZG9uZSAgPiA+Pj4+PiB2aWEgSU5UeCwgIE1TSSwgb3IgTVNJLVggKFBDSWUgc3Bl
YyByMy4wLCBzZWMNCj4gID4gPiA2LjIuNC4xLjIpLg0KPiAgPiA+ICA+ID4+ICA+ID4+Pj4+IEFw
cGFyZW50bHkgeW91ciBkZXZpY2UgIGRvZXNuJ3Qgc3VwcG9ydCBNU0kgb3IgTVNJLVguDQo+ICA+
ID4gQXJlICA+ID4+IHlvdSAgPiA+Pj4+PiBzYXlpbmcgaXQgZG9lc24ndCBzdXBwb3J0IElOVHgg
IGVpdGhlcj8gIEhvdw0KPiAgPiA+IGlzIHRoZSAgPiA+PiBpbnRlcnJ1cHQgIHlvdSdyZSByZXF1
ZXN0aW5nIGhlcmUgZGlmZmVyZW50IGZyb20gSU5UeD8NCj4gID4gPiAgPiA+PiAgPiA+Pj4+DQo+
ICA+ID4gID4gPj4gID4gPj4+PiBMYXllcnNjYXBlIHVzZSBub25lIG9mIE1TSSBvciBNU0ktWCBv
ciBJTlR4IHRvDQo+ICA+ID4gaW5kaWNhdGUgdGhlICA+ID4+ID4gPj4+PiBkZXZpY2VzIG9yIHJv
b3QgZXJyb3IgaW4gUkMgbW9kZS4gQnV0IHVzZQ0KPiAgPiA+IGFuIGluZGVwZW5kZW50IFNQSSAg
PiA+PiA+ID4+Pj4gaW50ZXJydXB0KGFybSBpbnRlcnJ1cHQgY29udHJvbGxlcikNCj4gIGxpbmUu
DQo+ICA+ID4gID4gPj4gID4gPj4+DQo+ICA+ID4gID4gPj4gID4gPj4+IFRoZSBSb290IFBvcnQg
aXMgYSBQQ0kgZGV2aWNlIGFuZCBzaG91bGQgZm9sbG93IHRoZQ0KPiAgPiA+IG5vcm1hbCAgPiA+
PiBQQ0kgID4gPj4+IHJ1bGVzIGZvciBpbnRlcnJ1cHRzLiAgQXMgZmFyIGFzIEkNCj4gID4gPiB1
bmRlcnN0YW5kLCB0aGF0ICA+ID4+IG1lYW5zIGl0ICA+ID4+PiBzaG91bGQgdXNlIE1TSSwgTVNJ
LVgsIG9yDQo+ICA+ID4gSU5UeC4gIElmIHlvdXIgUm9vdCBQb3J0ICA+ID4+IGRvZXNuJ3QgdXNl
IE1TSSAgPiA+Pj4gb3IgTVNJLVgsIGl0DQo+ICA+ID4gc2hvdWxkIHVzZSBJTlR4LCB0aGUgID4g
Pj4gUENJX0lOVEVSUlVQVF9QSU4gcmVnaXN0ZXIgID4gPj4+IHNob3VsZA0KPiAgPiA+IHRlbGwg
dXMgd2hpY2ggKElOVEEvICA+ID4+IElOVEIvZXRjLiksIGFuZCAgUENJX0NPTU1BTkRfSU5UWF9E
SVNBQkxFDQo+ICBzaG91bGQgd29yayB0byBkaXNhYmxlIGl0Lg0KPiAgPiA+ICA+ID4+ICA+ID4+
PiBUaGF0J3MgYWxsIGZyb20gdGhlIFBDSSBwb2ludCBvZiB2aWV3LCBvZiBjb3Vyc2UuDQo+ICA+
ID4gID4gPj4gID4gPj4NCj4gID4gPiAgPiA+PiAgPiA+PiBJIGFtIGZhY2VkIHdpdGggdGhlIHNh
bWUgaXNzdWUgb24gS2V5c3RvbmUgUENJIGhhcmR3YXJlDQo+ICA+ID4gYW5kICA+ID4+IGl0IGhh
cyAgPiA+PiBiZWVuIG9uIG15IFRPRE8gbGlzdCAgZm9yIHF1aXRlIHNvbWUgdGltZS4NCj4gID4g
PiBLZXlzdG9uZSAgPiA+PiBQQ0kgaGFyZHdhcmUgID4gPj4gYWxzbyBkb2Vzbid0IHVzZSBNU0kg
b3IgTVNJLVggb3INCj4gID4gPiBJTlR4IGZvciAgPiA+PiByZXBvcnRpbmcgZXJyb3JzIHJlY2Vp
dmVkICA+ID4+IGF0IHRoZSByb290IHBvcnQsIGJ1dA0KPiAgPiA+IHVzZSBhICA+ID4+IHBsYXRm
b3JtIGludGVycnVwdCBpbnN0ZWFkIChub3QgID4gPj4gY29tcGxhaW50IHRvIFBDSQ0KPiAgPiA+
IHN0YW5kYXJkIGFzICA+ID4+IHBlciBQQ0kgYmFzZSBzcGVjKS4gU28gSSB3b3VsZCBuZWVkICA+
ID4+IHNpbWlsYXINCj4gID4gPiBjaGFuZ2UgdG8gaGF2ZSAgPiA+PiB0aGUgZXJyb3IgaW50ZXJy
dXB0IHBhc3NlZCB0byB0aGUgYWVyICA+ID4+DQo+ICA+ID4gZHJpdmVyLiBTbyB0aGVyZSBhcmUg
ID4gPj4gaGFyZHdhcmUgb3V0IHRoZXJlIGxpa2UgS2V5c3RvbmUgd2hpY2gNCj4gID4gPiByZXF1
aXJlcyB0byBzdXBwb3J0IHRoaXMgIHRocm91Z2ggcGxhdGZvcm0gSVJRLg0KPiAgPiA+ICA+ID4+
ICA+ID4NCj4gID4gPiAgPiA+PiAgPiA+IFRoaXMgaXMgbm90IGEgbmV3IGFyZWEgb2YgdGhlIHNw
ZWMsIGFuZCBpdCdzIGhhcmQgZm9yIG1lDQo+ICA+ID4gdG8gID4gPj4gYmVsaWV2ZSAgPiA+IHRo
YXQgdGhlc2UgdHdvIG5ldyBQQ0llIGNvbnRyb2xsZXJzIGFyZSBib3RoDQo+ICA+ID4gYnJva2Vu
ICA+ID4+IHRoZSBzYW1lIHdheSAgPiA+IChhbHRob3VnaCBJIGd1ZXNzIGJvdGggYXJlDQo+ICA+
ID4gRGVzaWduV2FyZS1iYXNlZCwgc28gID4gPj4gbWF5YmUgdGhpcyBpcyB0aGUgID4gPiBzYW1l
IHVuZGVybHlpbmcNCj4gID4gPiBwcm9ibGVtIGluIGJvdGggY2FzZXM/KS4gIEkgID4gPj4gdGhp
bmsgaXQncyBtb3JlIGxpa2VseSAgPiA+IHRoYXQNCj4gID4gPiB3ZSBqdXN0IGhhdmVuJ3QgZmln
dXJlZCBvdXQgdGhlICA+ID4+IHJpZ2h0IHdheSB0byBkZXNjcmliZSB0aGlzIGluDQo+ICB0aGUg
RFQuDQo+ICA+ID4gID4gPj4gID4NCj4gID4gPiAgPiA+PiAgPiBLZXlzdG9uZSBpcyB1c2luZyBh
biBvbGRlciB2ZXJzaW9uIG9mIHRoZSBkZXNpZ253YXJlIElQIGFuZA0KPiAgPiA+IGl0ICA+ID4+
ID4gaW1wbGVtZW50cyBhbGwgb2YgdGhlIGludGVycnVwdHMgaW4gdGhlIGFwcGxpY2F0aW9uDQo+
ICA+ID4gcmVnaXN0ZXIgID4gPj4gc3BhY2UgID4gdW5saWtlIG90aGVyIG5ld2VyIHZlcnNpb24g
b2YgdGhlIGhhcmR3YXJlLg0KPiAgPiA+IFNvIEkgYXNzdW1lLCAgPiA+PiB0aGUgdmVyc2lvbiAg
PiB1c2VkIG9uIExheWVyc2NhcGUgaXMgYWxzbyBhbg0KPiAgPiA+IG9sZGVyIHZlcnNpb24gYW5k
IHRoZSAgPiA+PiBib3RoIGhhdmUgc2FtZSAgPiBpc3N1ZSBpbiB0ZXJtcyBvZiBub24NCj4gID4g
PiBzdGFuZGFyZCBwbGF0Zm9ybSBpbnRlcnJ1cHQgID4gPj4gdXNlZCBmb3IgZXJyb3IgIHJlcG9y
dGluZy4NCj4gID4gPiAgPiA+PiAgPg0KPiAgPiA+ICA+ID4+ICA+ID4gSSBhc3N1bWUgeW91IGhh
dmUgYSBSb290IFBvcnQgd2l0aCBhbiBBRVIgY2FwYWJpbGl0eSwgbm8NCj4gID4gPiBNU0kgID4g
Pj4gPiA+IGNhcGFiaWxpdHksIGFuZCBubyBNU0ktWCBjYXBhYmlsaXR5LCByaWdodD8NCj4gID4g
PiAgPiA+PiAgPg0KPiAgPiA+ICA+ID4+ICA+IEhhcyBBRVIgY2FwYWJpbGl0eSBhbmQgYm90aCBN
U0kgYW5kIElOVHggKGxlZ2FjeSkNCj4gID4gPiBjYXBhYmlsaXR5ICA+ICA+ID4+ID4gPiBXaGF0
IGRvZXMgaXRzIEludGVycnVwdCAgPiA+IFBpbiByZWdpc3Rlcg0KPiAgPiA+IGNvbnRhaW4/ICBJ
ZiBpdCdzICA+ID4+IHplcm8sIGl0IGRvZXNuJ3QgdXNlIElOVHggZWl0aGVyLCBzbyAgPiA+DQo+
ICA+ID4gYWNjb3JkaW5nIHRvIHRoZSBzcGVjIGl0ICA+ID4+IHNob3VsZCBnZW5lcmF0ZSBubyBp
bnRlcnJ1cHRzLg0KPiAgPiA+ICA+ID4+ICA+ID4NCj4gID4gPiAgPiA+PiAgPiBBdCBhZGRyZXNz
IG9mZnNldCAweDNDIGJ5IGRlZmF1bHQgaGFzIGEgdmFsdWUgb2YgMSwgYnV0IGl0DQo+ICA+ID4g
aXMgID4gPj4gd3JpdGFibGUgID4gYnkgc29mdHdhcmUuIFNvIGRlZmF1bHQgaXMgSU5UeCBBLg0K
PiAgPiA+ICA+ID4+DQo+ICA+ID4gID4gPj4gIDB4M2MgaXMgdGhlIEludGVycnVwdCAqTGluZSos
IHdoaWNoIGlzIHJlYWQvd3JpdGUuICBUaGUNCj4gID4gPiBJbnRlcnJ1cHQgID4gPj4gICpQaW4q
IGlzIGF0IDB4M2QgYW5kIHNob3VsZCBiZSByZWFkLW9ubHkuDQo+ICA+ID4gID4gPj4NCj4gID4g
PiAgPg0KPiAgPiA+ICA+IFlvdSBhcmUgcmlnaHQuIEJ1dCBkZWZhdWx0IGlzIDEgYXQgdGhpcyBh
ZGRyZXNzLg0KPiAgPiA+ICA+DQo+ICA+ID4gID4gPj4gIERvZXMgeW91ciBLZXlzdG9uZSBkcml2
ZXIgc3VwcG9ydCBNU0k/ICBJZiBzbywgc2luY2UgeW91cg0KPiAgPiA+IFJvb3QgID4gPj4gUG9y
dCAgc3VwcG9ydHMgTVNJLCBJIHdvdWxkIHRoaW5rIHdlIHdvdWxkIHVzZSB0aGF0IGJ5DQo+ICA+
ID4gZGVmYXVsdCwgYW5kICA+ID4+IHRoZSBJTlR4ICBzdHVmZiB3b3VsZG4ndCBldmVuIG1hdHRl
ci4NCj4gID4gPiAgPiA+DQo+ICA+ID4gID4gPiBMYXllcnNjYXBlIGlzIGFsc28gc2hvd3MgIkJv
dGggbWVzc2FnZSBzaWduYWxlZCBpbnRlcnJ1cHRzDQo+ICA+ID4gKE1TSSkgYW5kICBsZWdhY3kg
SU5UeCBhcmUgc3VwcG9ydGVkLiINCj4gID4gPiAgPiA+IEJ1dCBib3RoIG9mIHRoZW0gbm90IHdv
cmsgZm9yIEFFUiBpbnRlcnJ1cHQgd2hlbiBkZXZpY2VzIG9yDQo+ICA+ID4gcm9vdCAgcG9ydCBy
ZXBvcnQgYWVyIGVycm9yLg0KPiAgPiA+ICA+ID4gQnV0IGFub3RoZXIgR0lDIGludGVycnVwdCBs
aW5lIGRvLg0KPiAgPiA+ICA+DQo+ICA+ID4gID4gU2FtZSB3aXRoIEtleXN0b25lLiBFdmVuIHRo
b3VnaCBib3RoIE1TSSBhbmQgSU5UeCBhcmUgc3VwcG9ydGVkDQo+ICA+ID4gZXJyb3IgID4gaW50
ZXJydXB0IGF0IHJvb3QgcG9ydCBpcyByZXBvcnRlZCBvbiBhIGRpZmZlcmVudCBpbnRlcnJ1cHQN
Cj4gID4gPiBsaW5lIHRoYW4gID4gTVNJL0lOVHguIFNvIGZvciBQb3dlciBNYW5hZ2VtZW50IGV2
ZW50IGludGVycnVwdCBpcw0KPiAgPiA+IGFsc28gZGlmZmVyZW50ICA+IGxpbmUuDQo+ICA+ID4N
Cj4gID4gPiAgSSdtIGxvb2tpbmcgYXQgdGhlICJFcnJvciBNZXNzYWdlIENvbnRyb2xzIiBkaWFn
cmFtIGluIHRoZSBQQ0llDQo+ICA+ID4gc3BlYyAgcjMuMCwgc2VjIDYuMi42LiAgRG9lcyB0aGlz
IGhhcmR3YXJlIGZpdCBpbnRvIHRoZQ0KPiAgPiA+IHBsYXRmb3JtLXNwZWNpZmljICAiU3lzdGVt
IEVycm9yIiBjYXNlIHRoZXJlPyAgRG8gdGhlIFJvb3QgQ29udHJvbA0KPiAgPiA+IGVuYWJsZSBi
aXRzIChpbiB0aGUgUENJZQ0KPiAgPiA+ICBDYXBhYmlsaXR5KSBjb250cm9sIHRoaXMgaW50ZXJy
dXB0PyAgSWYgc28sIG1heWJlIHRoaXMgbWFrZXMgbW9yZQ0KPiAgPiA+IHNlbnNlICB0aGFuIEkg
dGhvdWdodC4NCj4gID4NCj4gID4gSXQgc3VwcG9zZWRseSBub3QgdGhlICJTeXN0ZW0gRXJyb3Ii
IGNhc2UuIEJ1dCAidGhlIEVycm9yIEludGVycnVwdCINCj4gIGNhc2UuDQo+ICA+IFdoaWNoIG1l
YW5zICIgUm9vdCBFcnJvciBDb21tYW5kIHJlZ2lzdGVyICIgY291bGQgY29udHJvbCB0aGUNCj4g
ID4gaW50ZXJydXB0ICBsaW5lIHdlIGhhdmUgbm93LiAocmVmZXIgUENJZSBzcGVjIHIzLjAsIHNl
YyA2LjIuNikNCj4gIA0KPiAgRGlkIHlvdSBhY3R1YWxseSB0cnkgdGhpcyBvdXQgYW5kIHZlcmlm
eSB0aGF0IHRoZSBQQ0llIFJvb3QgQ29udHJvbA0KPiAgZW5hYmxlIGJpdHMgaGF2ZSBubyBlZmZl
Y3QgYW5kIHRoZSBBRVIgUm9vdCBFcnJvciBDb21tYW5kIGJpdHMgZG8NCj4gIGNvbnRyb2wgaXQ/
ICBUaGUgbmFtZXMgYXJlIHZlcnkgc2ltaWxhciwgc28gdGhlcmUncyBsb3RzIG9mIHJvb20gZm9y
DQo+ICBtaXN1bmRlcnN0YW5kaW5nIGhlcmUgOikNCg0KWWVzLCBhbGwgdGhlc2UgcmVzdWx0IHdl
cmUgdGVzdGVkIGJlZm9yZSByZXBseS4NCg0KPiAgDQo+ICBJZiB0aGUgQUVSIFJvb3QgRXJyb3Ig
Q29tbWFuZCBkb2VzIGNvbnRyb2wgdGhpcyBpbnRlcnJ1cHQsIEkgdGhpbmsgdGhlDQo+ICBQQ0lf
Q09NTUFORF9JTlRYX0RJU0FCTEUgYml0IGluIHRoZSBQQ0kgQ29tbWFuZCByZWdpc3RlciBzaG91
bGQgYWxzbw0KPiAgY29udHJvbCBpdCAocGVyIHNlYyA2LjIuNC4xLjIpLg0KDQpZZXMsIEkgYW0g
c3VyZSB0aGUgUENJX0NPTU1BTkRfSU5UWF9ESVNBQkxFIGJpdCBjYW4gYWxzbyBjb250cm9sIHRo
aXMgaW50ZXJydXB0Lg0KDQo+ICANCj4gID4gTWF5IHRoaXMga2luZCBvZiBoYXJkd2FyZSBkZXNp
Z24gcm91dGUgYnJva2VuIHRoZSBzcGVjPw0KPiAgDQo+ICBJZiB0aGUgUmVwb3J0aW5nIEVuYWJs
ZSBiaXRzIGluIHRoZSBSb290IFBvcnQncyBBRVIgUm9vdCBFcnJvciBDb21tYW5kDQo+ICByZWdp
c3RlciBjb250cm9sIHRoZSBpbnRlcnJ1cHQsIGJ1dCB0aGUgaW50ZXJydXB0IGlzIG5vdCBkZWxp
dmVyZWQgdmlhDQo+ICB0aGUgUm9vdCBQb3J0J3MgSU5UeCBvciBNU0kvTVNJLVgsIEkgdGhpbmsg
dGhlIGRlc2lnbiBpcyBub3QgZm9sbG93aW5nDQo+ICB0aGUgc3BlYy4NCj4gIA0KPiAgQWxsIHRo
ZSBpbmZvcm1hdGlvbiBuZWVkZWQgYnkgdGhlIEFFUiBkcml2ZXIgc2hvdWxkIGJlIGNvbW11bmlj
YXRlZCB2aWENCj4gIHRoZSBjb25maWcgc3BhY2UgbWVjaGFuaXNtcyBkZXNjcmliZWQgaW4gdGhl
IHNwZWMgKEFFUiBjYXBhYmlsaXR5LA0KPiAgTVNJL01TSS1YIGNhcGFiaWxpdGllcywgSW50ZXJy
dXB0IFBpbiwgZXRjLikgIFRoYXQgd2F5IHRoZSBkcml2ZXIgd29ya3MNCj4gIHdpdGhvdXQgY2hh
bmdlIG9uIGZ1dHVyZSBzcGVjLWNvbXBsaWFudCBoYXJkd2FyZS4NCj4gIA0KPiAgPiBQTUUgYWxz
byBsaWtlIHRoZSBBRVIuIEhvdHBsdWcgaXMgbm90IHN1cHBvcnRlZC4gT3RoZXJzIG5vdCBrbm93
bi4NCj4gID4gUG8gTGl1DQo+ICANCj4gIFBlciBzZWMgNi4xLjYsIEkgdGhpbmsgUE1FICpzaG91
bGQqIGJlIHNpZ25hbGVkIGJ5IHRoZSBSb290IFBvcnQncyBJTlR4DQo+ICBvciBNU0kvTVNJLVgu
DQo+ICANCj4gIEluIHBhcnRpY3VsYXIsIGl0IHNheXMgIk5vdGUgdGhhdCBhbGwgb3RoZXIgaW50
ZXJydXB0IHNvdXJjZXMgd2l0aGluIHRoZQ0KPiAgc2FtZSBGdW5jdGlvbiB3aWxsIGFzc2VydCB0
aGUgc2FtZSB2aXJ0dWFsIElOVHggd2lyZSB3aGVuIHJlcXVlc3RpbmcNCj4gIHNlcnZpY2UuIiAg
VG8gbWUsIHRoYXQgbWVhbnMgdGhhdCBpZiB3ZSdyZSB1c2luZyBJTlR4LCBpdCB3aWxsIGJlIHRo
ZQ0KPiAgc2FtZSBJTlR4IGZvciBBRVIsIFBNRSwgaG90cGx1ZywgZXRjLiwgYW5kIGl0IHNob3Vs
ZCBiZSB0aGUgb25lDQo+ICBpbmRpY2F0ZWQgYnkgdGhlIEludGVycnVwdCBQaW4gcmVnaXN0ZXIu
DQo+ICANCj4gIEJ1dCBJIHRoaW5rIG9uIHlvdXIgUm9vdCBQb3J0Og0KPiAgDQo+ICAgIC0gVGhl
cmUgaXMgYW4gTVNJIGNhcGFiaWxpdHksIGJ1dCBNU0kgZG9lc24ndCBhY3R1YWxseSB3b3JrIGF0
IGFsbA0KPiAgICAtIEludGVycnVwdCBQaW4gY29udGFpbnMgMSwgaW5kaWNhdGluZyBJTlRBLCB3
aGljaCBpcyByb3V0ZWQgdG8gSVJRIFgNCj4gICAgLSBBRVIgaW50ZXJydXB0cyBhcmUgcm91dGVk
IHRvIHNvbWUgZGlmZmVyZW50IElSUSBZDQo+ICAgIC0gUE1FIGludGVycnVwdHMgYXJlIHJvdXRl
ZCB0byBhIHRoaXJkIElSUSBaDQo+ICANCg0KVGhlIGRlc2NyaXB0aW9ucyBhcmUgYWxsIHJpZ2h0
Lg0KDQo+ICBTbyBob3cgc2hvdWxkIHdlIHdvcmsgYXJvdW5kIHRoaXM/ICBJIHRoaW5rIHlvdSBz
aG91bGQgYmUgYWJsZSB0byBnZXQNCj4gIHBhcnR3YXkgdGhlcmUgd2l0aCBhIHF1aXJrIHRoYXQg
c2V0czoNCj4gIA0KPiAgICBkZXYtPm5vX21zaSA9IDE7DQo+ICAgIGRldi0+aXJxID0gWTsNCj4g
IA0KPiAgZm9yIHRoaXMgZGV2aWNlLiAgVGhhdCBzaG91bGQgbWFrZSBBRVIgd29yaywgYnV0IG9m
IGNvdXJzZSBQTUUgd291bGQgbm90DQo+ICB3b3JrLg0KPiAgDQo+ICBJcyB0aGVyZSBhIHdheSB0
byBzZXQgdXAgeW91ciBpbnRlcnJ1cHQgY29udHJvbGxlciBzbyB0aGVzZSB0aHJlZQ0KPiAgaW50
ZXJydXB0cyAoWCwgWSwgWiBhYm92ZSkgYWxsIG1hcCB0byB0aGUgc2FtZSBMaW51eCBJUlE/ICBJ
ZiB5b3UgY2FuIGRvDQo+ICB0aGF0LCB5b3UgY291bGQgc2V0IHVwIElOVEEsIHRoZSBBRVIgaW50
ZXJydXB0LCBhbmQgdGhlIFBNRSBpbnRlcnJ1cHQgdG8NCj4gIGFsbCBiZSBvbiB0aGUgc2FtZSBJ
UlEgYW5kIGV2ZXJ5dGhpbmcgc2hvdWxkIHdvcmsuDQo+ICANCj4gIEJqb3JuDQoNCldlJ2xsIHRo
aW5rIGFib3V0IGFsbCB0aGUgd2F5cy4gSXQgaXMgcmVhbGx5IGhlbHBmdWwsIHRoYW5rcyEgIA0K
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html