diff mbox series

pcie: Add quirk for the Arm Neoverse N1SDP platform

Message ID 20191209160638.141431-1-andre.przywara@arm.com
State New
Headers show
Series pcie: Add quirk for the Arm Neoverse N1SDP platform | expand

Commit Message

Andre Przywara Dec. 9, 2019, 4:06 p.m. UTC
From: Deepak Pandey <Deepak.Pandey@arm.com>

The Arm N1SDP SoC suffers from some PCIe integration issues, most
prominently config space accesses to not existing BDFs being answered
with a bus abort, resulting in an SError.
To mitigate this, the firmware scans the bus before boot (catching the
SErrors) and creates a table with valid BDFs, which acts as a filter for
Linux' config space accesses.

Add code consulting the table as an ACPI PCIe quirk, also register the
corresponding device tree based description of the host controller.
Also fix the other two minor issues on the way, namely not being fully
ECAM compliant and config space accesses being restricted to 32-bit
accesses only.

This allows the Arm Neoverse N1SDP board to boot Linux without crashing
and to access *any* devices (there are no platform devices except UART).

Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com>
[Sudipto: extend to cover the CCIX root port as well]
Signed-off-by: Sudipto Paul <sudipto.paul@arm.com>
[Andre: fix coding style issues, rewrite some parts, add DT support]
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 arch/arm64/configs/defconfig        |   1 +
 drivers/acpi/pci_mcfg.c             |   7 +
 drivers/pci/controller/Kconfig      |  11 ++
 drivers/pci/controller/Makefile     |   1 +
 drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++
 include/linux/pci-ecam.h            |   2 +
 6 files changed, 218 insertions(+)
 create mode 100644 drivers/pci/controller/pcie-n1sdp.c

Comments

Will Deacon Dec. 9, 2019, 4:26 p.m. UTC | #1
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> From: Deepak Pandey <Deepak.Pandey@arm.com>
> 
> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> prominently config space accesses to not existing BDFs being answered
> with a bus abort, resulting in an SError.

"Do as I say, not as I do"?

> To mitigate this, the firmware scans the bus before boot (catching the
> SErrors) and creates a table with valid BDFs, which acts as a filter for
> Linux' config space accesses.
> 
> Add code consulting the table as an ACPI PCIe quirk, also register the
> corresponding device tree based description of the host controller.
> Also fix the other two minor issues on the way, namely not being fully
> ECAM compliant and config space accesses being restricted to 32-bit
> accesses only.
> 
> This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> and to access *any* devices (there are no platform devices except UART).
> 
> Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com>
> [Sudipto: extend to cover the CCIX root port as well]
> Signed-off-by: Sudipto Paul <sudipto.paul@arm.com>
> [Andre: fix coding style issues, rewrite some parts, add DT support]
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/configs/defconfig        |   1 +
>  drivers/acpi/pci_mcfg.c             |   7 +
>  drivers/pci/controller/Kconfig      |  11 ++
>  drivers/pci/controller/Makefile     |   1 +
>  drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++
>  include/linux/pci-ecam.h            |   2 +
>  6 files changed, 218 insertions(+)
>  create mode 100644 drivers/pci/controller/pcie-n1sdp.c

Where can I buy one of these? They're "unreleased" according to:

https://community.arm.com/developer/tools-software/oss-platforms/w/docs/440/neoverse-n1-sdp

and I don't think we should wreck upstream because of a platform that
doesn't exist.

Will
Bjorn Helgaas Dec. 10, 2019, 2:41 p.m. UTC | #2
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> From: Deepak Pandey <Deepak.Pandey@arm.com>
> 
> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> prominently config space accesses to not existing BDFs being answered
> with a bus abort, resulting in an SError.

Can we tease this apart a little more?  Linux doesn't program all the
bits that control error signaling, so even on hardware that works
perfectly, much of this behavior is determined by what firmware did.
I wonder if Linux could be more careful about this.

"Bus abort" is not a term used in PCIe.  IIUC, a config read to a
device that doesn't exist should terminate with an Unsupported Request
completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.

The UR should be an uncorrectable non-fatal error (Table 6-5), and
Figures 6-2 and 6-3 show how it should be handled and when it should
be signaled as a system error.  In case you don't have a copy of the
spec, I extracted those two figures and put them at [1].

Can you collect "lspci -vvxxx" output to see if we can correlate it
with those figures and the behavior you see?

[1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing

> To mitigate this, the firmware scans the bus before boot (catching the
> SErrors) and creates a table with valid BDFs, which acts as a filter for
> Linux' config space accesses.
> 
> Add code consulting the table as an ACPI PCIe quirk, also register the
> corresponding device tree based description of the host controller.
> Also fix the other two minor issues on the way, namely not being fully
> ECAM compliant and config space accesses being restricted to 32-bit
> accesses only.

As I'm sure you've noticed, controllers that support only 32-bit
config writes are not spec compliant and devices may not work
correctly.  The comment in pci_generic_config_write32() explains why.

You may not trip over this problem frequently, but I wouldn't call it
a "minor" issue because when you *do* trip over it, you have no
indication that a register was corrupted.

Even ECAM compliance is not really minor -- if this controller were
fully compliant with the spec, you would need ZERO Linux changes to
support it.  Every quirk like this means additional maintenance
burden, and it's not just a one-time thing.  It means old kernels that
*should* "just work" on your system will not work unless somebody
backports the quirk.

> This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> and to access *any* devices (there are no platform devices except UART).
Andre Przywara Dec. 11, 2019, 11 a.m. UTC | #3
On Tue, 10 Dec 2019 08:41:15 -0600
Bjorn Helgaas <helgaas@kernel.org> wrote:

Hi Bjorn,

> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> > From: Deepak Pandey <Deepak.Pandey@arm.com>
> > 
> > The Arm N1SDP SoC suffers from some PCIe integration issues, most
> > prominently config space accesses to not existing BDFs being answered
> > with a bus abort, resulting in an SError.  
> 
> Can we tease this apart a little more?  Linux doesn't program all the
> bits that control error signaling, so even on hardware that works
> perfectly, much of this behavior is determined by what firmware did.
> I wonder if Linux could be more careful about this.
> 
> "Bus abort" is not a term used in PCIe.

Yes, sorry, that was my sloppy term, also aiming more at the CPU side of the bus, between the cores and the RC.

>  IIUC, a config read to a
> device that doesn't exist should terminate with an Unsupported Request
> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.

Yes, that's what Lorenzo mentioned as well.

> The UR should be an uncorrectable non-fatal error (Table 6-5), and
> Figures 6-2 and 6-3 show how it should be handled and when it should
> be signaled as a system error.  In case you don't have a copy of the
> spec, I extracted those two figures and put them at [1].

Thanks for that.
So in the last few months we tossed several ideas around how to work-around this without kernel intervention, all of them turned out to be not working. There are indeed registers in the RC that influence error reporting to the CPU side, but even if we could suppress (or catch) the SError, we can't recover and fixup the read transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I understood this, there are gates missing which are supposed to translate this specific UR into a valid "all-1s" response.

> Can you collect "lspci -vvxxx" output to see if we can correlate it
> with those figures and the behavior you see?

Attached.

> 
> [1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing
> 
> > To mitigate this, the firmware scans the bus before boot (catching the
> > SErrors) and creates a table with valid BDFs, which acts as a filter for
> > Linux' config space accesses.
> > 
> > Add code consulting the table as an ACPI PCIe quirk, also register the
> > corresponding device tree based description of the host controller.
> > Also fix the other two minor issues on the way, namely not being fully
> > ECAM compliant and config space accesses being restricted to 32-bit
> > accesses only.  
> 
> As I'm sure you've noticed, controllers that support only 32-bit
> config writes are not spec compliant and devices may not work
> correctly.  The comment in pci_generic_config_write32() explains why.

Yes, I understand. Actually it seems to work fine in my experiments without that part of the patch, but the hardware guys insisted that it's needed. I will double check.

> You may not trip over this problem frequently, but I wouldn't call it
> a "minor" issue because when you *do* trip over it, you have no
> indication that a register was corrupted.
> 
> Even ECAM compliance is not really minor -- if this controller were
> fully compliant with the spec, you would need ZERO Linux changes to
> support it.  Every quirk like this means additional maintenance
> burden, and it's not just a one-time thing.  It means old kernels that
> *should* "just work" on your system will not work unless somebody
> backports the quirk.

I am well aware of that, and we had quite some discussions internally, with quite some opposition.
The point is that this board has virtually everything behind PCI (which is good!), so not having working PCI renders this virtually useless. And installing Linux brings back warm and fuzzy memories of 1990's boot floppies - just without the actual disks ;-)
So anything that improves this situation in any way is helpful.

On the technical side this is a quirk, in the ACPI quirk framework(!), so it shouldn't affect anyone without the magic ACPI ID strings. People could even compile it out if needed. Plus the actual code is contained in a single, new file.
So, yes, I see that it's not pretty and we should not have broken hardware in the first place, but this is probably as good as this will get.

Cheers,
Andre

> > This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> > and to access *any* devices (there are no platform devices except UART).
0000:00:00.0 PCI bridge: ARM Device 0100 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 30
	Bus: primary=00, secondary=01, subordinate=19, sec-latency=0
	I/O behind bridge: 00001000-00004fff
	Memory behind bridge: 71200000-71bfffff
	Prefetchable memory behind bridge: 0000000900000000-00000009006fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 00000001  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [274 v1] Transaction Processing Hints
		Interrupt vector mode supported
		Device specific mode supported
		Steering table in TPH capability structure
	Capabilities: [300 v1] #19
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: pcieport
00: b5 13 00 01 07 04 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 19 00 10 40 00 00
20: 20 71 b0 71 01 00 61 00 09 00 00 00 09 00 00 00
30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 1e 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 01 90 03 5a 08 00 00 00 00 00 00 00 00 00 00 00
90: 05 b0 81 01 40 f0 ff ff 00 00 00 00 00 00 00 00
a0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 c0 00 00 00 00 00 00 08 00 00 00 00 00 00 00
c0: 10 00 42 00 23 80 00 00 10 29 00 00 03 ad 61 00
d0: 00 00 02 41 00 00 00 00 00 00 40 00 00 00 00 00
e0: 00 00 00 00 32 18 75 00 00 00 00 00 0e 00 00 00
f0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:01:00.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 24
	Region 0: Memory at 71b00000 (32-bit, non-prefetchable) [size=256K]
	Bus: primary=01, secondary=02, subordinate=08, sec-latency=0
	I/O behind bridge: 00001000-00004fff
	Memory behind bridge: 71200000-71afffff
	Prefetchable memory behind bridge: 0000000900000000-00000009006fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [b00 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 00 10 00 ca 00 04 06 00 00 81 00
10: 00 00 b0 71 00 00 00 00 01 02 08 00 11 41 00 00
20: 20 71 a0 71 01 00 61 00 09 00 00 00 09 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 86 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 52 00 01 80 00 00
70: 10 08 09 00 02 59 41 00 00 00 02 01 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 40 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:01:00.1 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 255
	Region 0: Memory at 71b40000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00
10: 00 00 b4 71 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10
70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:01:00.2 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 255
	Region 0: Memory at 71b42000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00
10: 00 20 b4 71 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10
70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:01:00.3 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 255
	Region 0: Memory at 71b44000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00
10: 00 40 b4 71 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10
70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:01:00.4 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 255
	Region 0: Memory at 71b46000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00
10: 00 60 b4 71 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10
70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:08.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 31
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 71200000-713fffff
	Prefetchable memory behind bridge: 0000000900000000-00000009001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
			Slot #8, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 03 03 00 11 11 00 00
20: 20 71 30 71 01 00 11 00 09 00 00 00 09 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00
70: 10 08 09 00 02 69 79 08 00 00 01 00 df 0c 40 00
80: c0 07 20 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:10.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 32
	Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 5GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #16, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 04 04 00 f1 01 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00
70: 10 08 09 00 82 68 79 10 00 00 01 00 80 0c 80 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:11.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 34
	Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: 71400000-715fffff
	Prefetchable memory behind bridge: 0000000900200000-00000009003fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #17, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
			Slot #17, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 05 05 00 21 21 00 00
20: 40 71 50 71 21 00 31 00 09 00 00 00 09 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00
70: 10 08 09 00 22 68 79 11 00 00 01 00 df 0c 88 00
80: c0 07 20 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:12.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 35
	Bus: primary=02, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: 71600000-716fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #18, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 06 06 00 31 31 00 00
20: 60 71 60 71 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00
70: 10 08 09 00 22 58 79 12 00 00 12 60 80 0c 00 00
80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:13.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 36
	Bus: primary=02, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 00004000-00004fff
	Memory behind bridge: 71700000-717fffff
	Prefetchable memory behind bridge: 0000000900400000-00000009004fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #19, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 07 07 00 41 41 00 00
20: 70 71 70 71 41 00 41 00 09 00 00 00 09 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00
70: 10 08 09 00 22 68 79 13 00 00 11 60 80 0c 00 00
80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:02:14.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 37
	Bus: primary=02, secondary=08, subordinate=08, sec-latency=0
	Memory behind bridge: 71800000-718fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fffff040  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #20, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 02 08 08 00 f1 01 00 00
20: 80 71 80 71 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00
40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff
50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00
70: 10 08 09 00 22 58 79 14 00 00 12 60 80 0c 00 00
80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00
90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00
a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:06:00.0 SATA controller: Marvell Technology Group Ltd. Device 9170 (rev 12) (prog-if 01 [AHCI 1.0])
	Subsystem: Marvell Technology Group Ltd. Device 9170
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 21
	Region 0: I/O ports at 3010 [size=8]
	Region 1: I/O ports at 3020 [size=4]
	Region 2: I/O ports at 3018 [size=8]
	Region 3: I/O ports at 3024 [size=4]
	Region 4: I/O ports at 3000 [size=16]
	Region 5: Memory at 71610000 (32-bit, non-prefetchable) [size=512]
	Expansion ROM at 71600000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fffff040  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: ahci
00: 4b 1b 70 91 07 04 10 00 12 01 06 01 00 00 00 00
10: 11 30 00 00 21 30 00 00 19 30 00 00 25 30 00 00
20: 01 30 00 00 00 00 61 71 00 00 00 00 4b 1b 70 91
30: 00 00 ff ff 40 00 00 00 00 00 00 00 ff 01 00 00
40: 01 50 03 40 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 01 00 40 f0 ff ff 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 12 00 02 87 64 00 10 20 09 00 12 3c 03 00
80: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
a0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
	Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 22
	Region 0: I/O ports at 4000 [size=256]
	Region 2: Memory at 71700000 (64-bit, non-prefetchable) [size=4K]
	Region 4: Memory at 900400000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 01
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00000800
	Capabilities: [d0] Vital Product Data
		Not readable
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
	Capabilities: [170 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: r8169
00: ec 10 68 81 07 04 10 00 0c 00 00 02 10 00 00 00
10: 01 40 00 00 00 00 00 00 04 00 70 71 00 00 00 00
20: 0c 00 40 00 09 00 00 00 00 00 00 00 ec 10 23 01
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
40: 01 50 c3 ff 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 02 c0 8c 64 00 10 20 19 00 11 7c 47 00
80: 00 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 1f 08 0c 00 00 00 00 00 02 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 d0 03 80 04 00 00 00 04 08 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 03 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:08:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) (prog-if 30 [XHCI])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 24
	Region 0: Memory at 71800000 (64-bit, non-prefetchable) [size=64K]
	Region 2: Memory at 71810000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
		Vector table: BAR=2 offset=00000000
		PBA: BAR=2 offset=00001000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Device Serial Number 08-00-28-00-00-20-00-00
	Kernel driver in use: xhci_hcd
00: 4c 10 41 82 06 04 10 00 02 30 03 0c 10 00 00 00
10: 04 00 80 71 00 00 00 00 04 00 81 71 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
40: 01 48 03 7e 08 00 00 00 05 70 86 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 30 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 c0 02 00 c3 8f 64 00 10 20 09 00 12 5c 07 00
80: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
a0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 08 00 00 0f 00 00 00 00 00 00 00 00 00
c0: 11 00 07 80 02 00 00 00 02 10 00 00 00 00 00 00
d0: 00 00 00 00 ab 0d 00 00 1b 00 00 00 3f 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0001:00:00.0 Host bridge: ARM Device 0100
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 255
	Region 0: Memory at 2900000000 (64-bit, non-prefetchable) [size=4M]
	Bus: primary=00, secondary=01, subordinate=11, sec-latency=0
	I/O behind bridge: 00000000-00000fff
	Prefetchable memory behind bridge: 00000000-000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [274 v1] Transaction Processing Hints
		Interrupt vector mode supported
		Device specific mode supported
		Steering table in TPH capability structure
	Capabilities: [300 v1] #19
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [5c0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 01
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [640 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000001, Page Request Allocation: 00000000
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [910 v1] #25
	Capabilities: [920 v1] #27
	Capabilities: [9c0 v1] #26
00: b5 13 00 01 07 00 10 00 00 00 00 06 00 00 01 00
10: 04 00 00 00 29 00 00 00 00 01 11 00 00 00 00 00
20: f0 ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 1e 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 01 90 03 5a 08 00 00 00 00 00 00 00 00 00 00 00
90: 05 b0 80 01 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 c0 00 00 00 00 00 00 08 00 00 00 00 00 00 00
c0: 10 00 42 00 23 80 00 00 10 29 00 00 04 ad 61 00
d0: 00 00 01 01 00 00 00 00 00 00 40 00 00 00 00 00
e0: 00 00 00 00 32 18 75 00 00 00 00 00 1e 00 80 01
f0: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
Bjorn Helgaas Dec. 11, 2019, 8:17 p.m. UTC | #4
On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote:
> On Tue, 10 Dec 2019 08:41:15 -0600
> Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> > > From: Deepak Pandey <Deepak.Pandey@arm.com>
> > > 
> > > The Arm N1SDP SoC suffers from some PCIe integration issues, most
> > > prominently config space accesses to not existing BDFs being answered
> > > with a bus abort, resulting in an SError.  
> > 
> > Can we tease this apart a little more?  Linux doesn't program all the
> > bits that control error signaling, so even on hardware that works
> > perfectly, much of this behavior is determined by what firmware did.
> > I wonder if Linux could be more careful about this.
> > 
> > "Bus abort" is not a term used in PCIe.
> 
> Yes, sorry, that was my sloppy term, also aiming more at the CPU
> side of the bus, between the cores and the RC.
>
> >  IIUC, a config read to a
> > device that doesn't exist should terminate with an Unsupported Request
> > completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.
> 
> Yes, that's what Lorenzo mentioned as well.
> 
> > The UR should be an uncorrectable non-fatal error (Table 6-5), and
> > Figures 6-2 and 6-3 show how it should be handled and when it should
> > be signaled as a system error.  In case you don't have a copy of the
> > spec, I extracted those two figures and put them at [1].
> 
> Thanks for that.
> So in the last few months we tossed several ideas around how to
> work-around this without kernel intervention, all of them turned out
> to be not working. There are indeed registers in the RC that
> influence error reporting to the CPU side, but even if we could
> suppress (or catch) the SError, we can't recover and fixup the read
> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I
> understood this, there are gates missing which are supposed to
> translate this specific UR into a valid "all-1s" response.

But the commit log says firmware scanned the bus (catching the
SErrors).  Shouldn't Linux be able to catch them the same way?

The "all-1s" response directly from hardware is typical of most
platforms, but I don't think it's strictly required by the PCIe spec
and I don't think it's absolutely essential even to Linux.  If you can
catch the SErrors, isn't there a way for software to fabricate that
all-1s data and continue after the read?

> > Even ECAM compliance is not really minor -- if this controller were
> > fully compliant with the spec, you would need ZERO Linux changes to
> > support it.  Every quirk like this means additional maintenance
> > burden, and it's not just a one-time thing.  It means old kernels that
> > *should* "just work" on your system will not work unless somebody
> > backports the quirk.
> 
> I am well aware of that, and we had quite some discussions
> internally, with quite some opposition.  ...

The main point is that *future* silicon should be designed to avoid
this issue.  I hope at least that part was not controversial.

If we want to take advantage of the generic PCI code supplied by
Linux, we have to expect that the hardware will play by the rules of
PCI.

Bjorn
Andre Przywara Dec. 12, 2019, 11:05 a.m. UTC | #5
On 11/12/2019 20:17, Bjorn Helgaas wrote:

Hi Bjorn,

> On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote:
>> On Tue, 10 Dec 2019 08:41:15 -0600
>> Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
>>>> From: Deepak Pandey <Deepak.Pandey@arm.com>
>>>>
>>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most
>>>> prominently config space accesses to not existing BDFs being answered
>>>> with a bus abort, resulting in an SError.  
>>>
>>> Can we tease this apart a little more?  Linux doesn't program all the
>>> bits that control error signaling, so even on hardware that works
>>> perfectly, much of this behavior is determined by what firmware did.
>>> I wonder if Linux could be more careful about this.
>>>
>>> "Bus abort" is not a term used in PCIe.
>>
>> Yes, sorry, that was my sloppy term, also aiming more at the CPU
>> side of the bus, between the cores and the RC.
>>
>>>  IIUC, a config read to a
>>> device that doesn't exist should terminate with an Unsupported Request
>>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.
>>
>> Yes, that's what Lorenzo mentioned as well.
>>
>>> The UR should be an uncorrectable non-fatal error (Table 6-5), and
>>> Figures 6-2 and 6-3 show how it should be handled and when it should
>>> be signaled as a system error.  In case you don't have a copy of the
>>> spec, I extracted those two figures and put them at [1].
>>
>> Thanks for that.
>> So in the last few months we tossed several ideas around how to
>> work-around this without kernel intervention, all of them turned out
>> to be not working. There are indeed registers in the RC that
>> influence error reporting to the CPU side, but even if we could
>> suppress (or catch) the SError, we can't recover and fixup the read
>> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I
>> understood this, there are gates missing which are supposed to
>> translate this specific UR into a valid "all-1s" response.
> 
> But the commit log says firmware scanned the bus (catching the
> SErrors).  Shouldn't Linux be able to catch them the same way?

Not really. The scanning is done by the SCP management processor, which is a Cortex-M class core on the same bus. So it's a simple, single core running very early after power-on, when the actual AP cores are still off. The SError handler is set to just increase a value, then to return. This value is then checked before and after a config space access for a given
BDF: https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp

On the AP cores that run Linux later on this is quite different: The SError is asynchronous, imprecise (inexact) and has no syndrome information. That means we can't attribute this anymore to the faulting instruction, we don't even know if it happened due to this config space access. The CPU might have executed later instructions already, so the state is broken at this point. SError basically means: the system is screwed up.
Because this is quite common for SErrors, we don't even allow to register SError handlers in arm64 Linux.

So even if we could somehow handle this is in Linux, it would be a much greater and intrusive hack, so I'd rather stick with this version.
 
> The "all-1s" response directly from hardware is typical of most
> platforms, but I don't think it's strictly required by the PCIe spec
> and I don't think it's absolutely essential even to Linux.  If you can
> catch the SErrors, isn't there a way for software to fabricate that
> all-1s data and continue after the read?

That was an idea we had as well, but due to the points mentioned above this is not possible.

>>> Even ECAM compliance is not really minor -- if this controller were
>>> fully compliant with the spec, you would need ZERO Linux changes to
>>> support it.  Every quirk like this means additional maintenance
>>> burden, and it's not just a one-time thing.  It means old kernels that
>>> *should* "just work" on your system will not work unless somebody
>>> backports the quirk.
>>
>> I am well aware of that, and we had quite some discussions
>> internally, with quite some opposition.  ...
> 
> The main point is that *future* silicon should be designed to avoid
> this issue.  I hope at least that part was not controversial.

Yes, the design goal was to be completely standards (SBSA, ACPI, ECAM) compliant, it was just the usual "things happen" that wasn't spotted in time.

> If we want to take advantage of the generic PCI code supplied by
> Linux, we have to expect that the hardware will play by the rules of
> PCI.

You don't need to convince me ;-), but I think the lesson has been learned.

Cheers,
Andre.
Andrew Murray Dec. 12, 2019, 12:37 p.m. UTC | #6
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> From: Deepak Pandey <Deepak.Pandey@arm.com>
> 
> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> prominently config space accesses to not existing BDFs being answered
> with a bus abort, resulting in an SError.

It wouldn't be a surprise if the host controller handled UR completions
differently depending on if they were in response to a Type 0 configuration
request or a Type 1 configuration request. (I think I've seen this before).

Have you verified that you still get a bus abort when you attempt to
perform a config read of a non-existent device downstream of the PCIe switch?
(and thus as a response to a Type 1 request).

I ask because if this is the case, and knowing that the PCIe switch is
fixed, then it would be possible to simplify this quirk (by just making
assumptions of the presence of devices in bus 0 rather than all the busses).


> To mitigate this, the firmware scans the bus before boot (catching the
> SErrors) and creates a table with valid BDFs, which acts as a filter for
> Linux' config space accesses.
> 
> Add code consulting the table as an ACPI PCIe quirk, also register the
> corresponding device tree based description of the host controller.
> Also fix the other two minor issues on the way, namely not being fully
> ECAM compliant and config space accesses being restricted to 32-bit
> accesses only.
> 
> This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> and to access *any* devices (there are no platform devices except UART).

This implies that this quirk has no side-effects and everything will work
as expected - but this is only true for the simple case. For example hot
plug won't work, SR-IOV, and others won't work.

Also what happens for devices that return CRS? - does this also result in an
abort? Does that mean that the firmware will consider these devices as not
present instead of not ready yet? If this is an issue, then FLR of devices
will also create issues (resulting in SErrors for users).

I think it would be helpful to update this commit message to indicate that
this makes it work better, but it may be broken in certain ways.


> 
> Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com>
> [Sudipto: extend to cover the CCIX root port as well]
> Signed-off-by: Sudipto Paul <sudipto.paul@arm.com>
> [Andre: fix coding style issues, rewrite some parts, add DT support]
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/configs/defconfig        |   1 +
>  drivers/acpi/pci_mcfg.c             |   7 +
>  drivers/pci/controller/Kconfig      |  11 ++
>  drivers/pci/controller/Makefile     |   1 +
>  drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++
>  include/linux/pci-ecam.h            |   2 +
>  6 files changed, 218 insertions(+)
>  create mode 100644 drivers/pci/controller/pcie-n1sdp.c
> 
> diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
> index 6a83ba2aea3e..58124ef5070b 100644
> --- a/arch/arm64/configs/defconfig
> +++ b/arch/arm64/configs/defconfig
> @@ -177,6 +177,7 @@ CONFIG_NET_9P=y
>  CONFIG_NET_9P_VIRTIO=y
>  CONFIG_PCI=y
>  CONFIG_PCIEPORTBUS=y
> +CONFIG_PCI_QUIRKS=y
>  CONFIG_PCI_IOV=y
>  CONFIG_HOTPLUG_PCI=y
>  CONFIG_HOTPLUG_PCI_ACPI=y
> diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
> index 6b347d9920cc..7a2b41b9ab57 100644
> --- a/drivers/acpi/pci_mcfg.c
> +++ b/drivers/acpi/pci_mcfg.c
> @@ -142,6 +142,13 @@ static struct mcfg_fixup mcfg_quirks[] = {
>  	XGENE_V2_ECAM_MCFG(4, 0),
>  	XGENE_V2_ECAM_MCFG(4, 1),
>  	XGENE_V2_ECAM_MCFG(4, 2),
> +
> +#define N1SDP_ECAM_MCFG(rev, seg, ops) \
> +	{"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops }
> +
> +	/* N1SDP SoC with v1 PCIe controller */
> +	N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops),
> +	N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops),
>  };
>  
>  static char mcfg_oem_id[ACPI_OEM_ID_SIZE];
> diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
> index c77069c8ee5d..45700d32f02e 100644
> --- a/drivers/pci/controller/Kconfig
> +++ b/drivers/pci/controller/Kconfig
> @@ -37,6 +37,17 @@ config PCI_FTPCI100
>  	depends on OF
>  	default ARCH_GEMINI
>  
> +config PCIE_HOST_N1SDP_ECAM
> +	bool "ARM N1SDP PCIe Controller"
> +	depends on ARM64
> +	depends on OF || (ACPI && PCI_QUIRKS)
> +	select PCI_HOST_COMMON
> +	default y if ARCH_VEXPRESS
> +	help
> +	  Say Y here if you want PCIe support for the Arm N1SDP platform.
> +	  The controller is ECAM compliant, but needs a quirk to workaround
> +	  an integration issue.

Again - please indicate the scope of support provided.

> +
>  config PCI_TEGRA
>  	bool "NVIDIA Tegra PCIe controller"
>  	depends on ARCH_TEGRA || COMPILE_TEST
> diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
> index 3d4f597f15ce..5f47fefbd67d 100644
> --- a/drivers/pci/controller/Makefile
> +++ b/drivers/pci/controller/Makefile
> @@ -28,6 +28,7 @@ obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
>  obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
>  obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
>  obj-$(CONFIG_VMD) += vmd.o
> +obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o
>  # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
>  obj-y				+= dwc/
>  
> diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c
> new file mode 100644
> index 000000000000..620ab221466c
> --- /dev/null
> +++ b/drivers/pci/controller/pcie-n1sdp.c
> @@ -0,0 +1,196 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2018/2019 ARM Ltd.
> + *
> + * This quirk is to mask the following issues:
> + * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus
> + *		  error (signalled as an asynchronous SError)
> + * - MCFG BDF mapping: the root complex is mapped separately from the device
> + *		       config space
> + * - Non 32-bit accesses to config space are not supported.
> + *
> + * At boot time the SCP board firmware creates a discovery table with
> + * the root complex' base address and the valid BDF values, discovered while
> + * scanning the config space and catching the SErrors.
> + * Linux responds only to the EPs listed in this table, returning NULL

NIT: it will respond to the switch devices which aren't EPs.


> + * for the rest.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/ioport.h>
> +#include <linux/sizes.h>
> +#include <linux/of_pci.h>
> +#include <linux/of.h>
> +#include <linux/pci-ecam.h>
> +#include <linux/platform_device.h>
> +#include <linux/module.h>
> +
> +/* Platform specific values as hardcoded in the firmware. */
> +#define AP_NS_SHARED_MEM_BASE	0x06000000
> +#define MAX_SEGMENTS		2		/* Two PCIe root complexes. */
> +#define BDF_TABLE_SIZE		SZ_16K
> +
> +/*
> + * Shared memory layout as written by the SCP upon boot time:
> + *  ----
> + *  Discover data header --> RC base address
> + *                       \-> BDF Count
> + *  Discover data        --> BDF 0...n
> + *  ----
> + */
> +struct pcie_discovery_data {
> +	u32 rc_base_addr;
> +	u32 nr_bdfs;
> +	u32 valid_bdfs[0];
> +} *pcie_discovery_data[MAX_SEGMENTS];
> +
> +void __iomem *rc_remapped_addr[MAX_SEGMENTS];
> +
> +/*
> + * map_bus() is called before we do a config space access for a certain
> + * device. We use this to check whether this device is valid, avoiding
> + * config space accesses which would result in an SError otherwise.
> + */
> +static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn,
> +				       int where)
> +{
> +	struct pci_config_window *cfg = bus->sysdata;
> +	unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> +	unsigned int busn = bus->number;
> +	unsigned int segment = bus->domain_nr;
> +	unsigned int bdf_addr;
> +	unsigned int table_count, i;
> +
> +	if (segment >= MAX_SEGMENTS ||
> +	    busn < cfg->busr.start || busn > cfg->busr.end)
> +		return NULL;
> +
> +	/* The PCIe root complex has a separate config space mapping. */
> +	if (busn == 0 && devfn == 0)
> +		return rc_remapped_addr[segment] + where;
> +
> +	busn -= cfg->busr.start;
> +	bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift);
> +	table_count = pcie_discovery_data[segment]->nr_bdfs;
> +	for (i = 0; i < table_count; i++) {
> +		if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i])
> +			return pci_ecam_map_bus(bus, devfn, where);
> +	}
> +
> +	return NULL;
> +}
> +
> +static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment)
> +{
> +	phys_addr_t table_base;
> +	struct device *dev = cfg->parent;
> +	struct pcie_discovery_data *shared_data;
> +	size_t bdfs_size;
> +
> +	if (segment >= MAX_SEGMENTS)
> +		return -ENODEV;
> +
> +	table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE;

How can you be sure that this table is populated and isn't junk? I.e. using an older
SCP version?

> +
> +	if (!request_mem_region(table_base, BDF_TABLE_SIZE,
> +				"PCIe valid BDFs")) {
> +		dev_err(dev, "PCIe BDF shared region request failed\n");
> +		return -ENOMEM;
> +	}
> +
> +	shared_data = devm_ioremap(dev,
> +				   table_base, BDF_TABLE_SIZE);
> +	if (!shared_data)
> +		return -ENOMEM;
> +
> +	/* Copy the valid BDFs structure to allocated normal memory. */
> +	bdfs_size = sizeof(struct pcie_discovery_data) +
> +		    sizeof(u32) * shared_data->nr_bdfs;
> +	pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL);
> +	if (!pcie_discovery_data[segment])
> +		return -ENOMEM;
> +
> +	memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size);
> +
> +	rc_remapped_addr[segment] = devm_ioremap_nocache(dev,
> +						shared_data->rc_base_addr,
> +						PCI_CFG_SPACE_EXP_SIZE);
> +	if (!rc_remapped_addr[segment]) {
> +		dev_err(dev, "Cannot remap root port base\n");
> +		return -ENOMEM;
> +	}
> +
> +	devm_iounmap(dev, shared_data);
> +
> +	return 0;
> +}
> +
> +static int pci_n1sdp_pcie_init(struct pci_config_window *cfg)
> +{
> +	return pci_n1sdp_init(cfg, 0);
> +}
> +
> +static int pci_n1sdp_ccix_init(struct pci_config_window *cfg)
> +{
> +	return pci_n1sdp_init(cfg, 1);
> +}
> +
> +struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = {
> +	.bus_shift	= 20,
> +	.init		= pci_n1sdp_pcie_init,
> +	.pci_ops	= {
> +		.map_bus        = pci_n1sdp_map_bus,
> +		.read           = pci_generic_config_read32,
> +		.write          = pci_generic_config_write32,
> +	}
> +};
> +
> +struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = {
> +	.bus_shift	= 20,
> +	.init		= pci_n1sdp_ccix_init,
> +	.pci_ops	= {
> +		.map_bus        = pci_n1sdp_map_bus,
> +		.read           = pci_generic_config_read32,
> +		.write          = pci_generic_config_write32,
> +	}
> +};
> +
> +static const struct of_device_id n1sdp_pcie_of_match[] = {
> +	{ .compatible = "arm,n1sdp-pcie" },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match);
> +
> +static int n1sdp_pcie_probe(struct platform_device *pdev)
> +{
> +	const struct device_node *of_node = pdev->dev.of_node;
> +	u32 segment;
> +
> +	if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) {
> +		dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n");
> +		return -EINVAL;
> +	}

Can you use of_get_pci_domain_nr here?

Thanks,

Andrew Murray

> +
> +	switch (segment) {
> +	case 0:
> +		return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops);
> +	case 1:
> +		return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops);
> +	}
> +
> +	dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n",
> +		MAX_SEGMENTS);
> +
> +	return -EINVAL;
> +}
> +
> +static struct platform_driver n1sdp_pcie_driver = {
> +	.driver = {
> +		.name = KBUILD_MODNAME,
> +		.of_match_table = n1sdp_pcie_of_match,
> +		.suppress_bind_attrs = true,
> +	},
> +	.probe = n1sdp_pcie_probe,
> +};
> +builtin_platform_driver(n1sdp_pcie_driver);
> diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h
> index a73164c85e78..03cdea69f4e8 100644
> --- a/include/linux/pci-ecam.h
> +++ b/include/linux/pci-ecam.h
> @@ -57,6 +57,8 @@ extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */
>  extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */
>  extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */
>  extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */
> +extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */
> +extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */
>  #endif
>  
>  #ifdef CONFIG_PCI_HOST_COMMON
> -- 
> 2.17.1
>
Robin Murphy Dec. 12, 2019, 1:44 p.m. UTC | #7
On 12/12/2019 11:05 am, Andre Przywara wrote:
> On 11/12/2019 20:17, Bjorn Helgaas wrote:
> 
> Hi Bjorn,
> 
>> On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote:
>>> On Tue, 10 Dec 2019 08:41:15 -0600
>>> Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
>>>>> From: Deepak Pandey <Deepak.Pandey@arm.com>
>>>>>
>>>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most
>>>>> prominently config space accesses to not existing BDFs being answered
>>>>> with a bus abort, resulting in an SError.
>>>>
>>>> Can we tease this apart a little more?  Linux doesn't program all the
>>>> bits that control error signaling, so even on hardware that works
>>>> perfectly, much of this behavior is determined by what firmware did.
>>>> I wonder if Linux could be more careful about this.
>>>>
>>>> "Bus abort" is not a term used in PCIe.
>>>
>>> Yes, sorry, that was my sloppy term, also aiming more at the CPU
>>> side of the bus, between the cores and the RC.
>>>
>>>>   IIUC, a config read to a
>>>> device that doesn't exist should terminate with an Unsupported Request
>>>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.
>>>
>>> Yes, that's what Lorenzo mentioned as well.
>>>
>>>> The UR should be an uncorrectable non-fatal error (Table 6-5), and
>>>> Figures 6-2 and 6-3 show how it should be handled and when it should
>>>> be signaled as a system error.  In case you don't have a copy of the
>>>> spec, I extracted those two figures and put them at [1].
>>>
>>> Thanks for that.
>>> So in the last few months we tossed several ideas around how to
>>> work-around this without kernel intervention, all of them turned out
>>> to be not working. There are indeed registers in the RC that
>>> influence error reporting to the CPU side, but even if we could
>>> suppress (or catch) the SError, we can't recover and fixup the read
>>> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I
>>> understood this, there are gates missing which are supposed to
>>> translate this specific UR into a valid "all-1s" response.
>>
>> But the commit log says firmware scanned the bus (catching the
>> SErrors).  Shouldn't Linux be able to catch them the same way?
> 
> Not really. The scanning is done by the SCP management processor, which is a Cortex-M class core on the same bus. So it's a simple, single core running very early after power-on, when the actual AP cores are still off. The SError handler is set to just increase a value, then to return. This value is then checked before and after a config space access for a given
> BDF: https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp
> 
> On the AP cores that run Linux later on this is quite different: The SError is asynchronous, imprecise (inexact) and has no syndrome information. That means we can't attribute this anymore to the faulting instruction, we don't even know if it happened due to this config space access. The CPU might have executed later instructions already, so the state is broken at this point. SError basically means: the system is screwed up.
> Because this is quite common for SErrors, we don't even allow to register SError handlers in arm64 Linux.

Furthermore, on the main application processor, SError might be 
delivered to EL3 firmware well beyond the reach of Linux, so we can make 
zero assumptions about how it's handled and whether we'll ever see it, 
or survive the result (EL3 is at liberty to say "oh, something went 
wrong, I'll reset the system immediately").

Robin.
> So even if we could somehow handle this is in Linux, it would be a much greater and intrusive hack, so I'd rather stick with this version.
>   
>> The "all-1s" response directly from hardware is typical of most
>> platforms, but I don't think it's strictly required by the PCIe spec
>> and I don't think it's absolutely essential even to Linux.  If you can
>> catch the SErrors, isn't there a way for software to fabricate that
>> all-1s data and continue after the read?
> 
> That was an idea we had as well, but due to the points mentioned above this is not possible.
> 
>>>> Even ECAM compliance is not really minor -- if this controller were
>>>> fully compliant with the spec, you would need ZERO Linux changes to
>>>> support it.  Every quirk like this means additional maintenance
>>>> burden, and it's not just a one-time thing.  It means old kernels that
>>>> *should* "just work" on your system will not work unless somebody
>>>> backports the quirk.
>>>
>>> I am well aware of that, and we had quite some discussions
>>> internally, with quite some opposition.  ...
>>
>> The main point is that *future* silicon should be designed to avoid
>> this issue.  I hope at least that part was not controversial.
> 
> Yes, the design goal was to be completely standards (SBSA, ACPI, ECAM) compliant, it was just the usual "things happen" that wasn't spotted in time.
> 
>> If we want to take advantage of the generic PCI code supplied by
>> Linux, we have to expect that the hardware will play by the rules of
>> PCI.
> 
> You don't need to convince me ;-), but I think the lesson has been learned.
> 
> Cheers,
> Andre.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
Andrew Murray Dec. 12, 2019, 9:07 p.m. UTC | #8
On Tue, Dec 10, 2019 at 08:41:15AM -0600, Bjorn Helgaas wrote:
> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> > From: Deepak Pandey <Deepak.Pandey@arm.com>
> > 
> > The Arm N1SDP SoC suffers from some PCIe integration issues, most
> > prominently config space accesses to not existing BDFs being answered
> > with a bus abort, resulting in an SError.
> 
> Can we tease this apart a little more?  Linux doesn't program all the
> bits that control error signaling, so even on hardware that works
> perfectly, much of this behavior is determined by what firmware did.
> I wonder if Linux could be more careful about this.
> 
> "Bus abort" is not a term used in PCIe.  IIUC, a config read to a
> device that doesn't exist should terminate with an Unsupported Request
> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.
> 
> The UR should be an uncorrectable non-fatal error (Table 6-5), and
> Figures 6-2 and 6-3 show how it should be handled and when it should
> be signaled as a system error.  In case you don't have a copy of the
> spec, I extracted those two figures and put them at [1].
> 
> Can you collect "lspci -vvxxx" output to see if we can correlate it
> with those figures and the behavior you see?
> 
> [1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing
> 
> > To mitigate this, the firmware scans the bus before boot (catching the
> > SErrors) and creates a table with valid BDFs, which acts as a filter for
> > Linux' config space accesses.
> > 
> > Add code consulting the table as an ACPI PCIe quirk, also register the
> > corresponding device tree based description of the host controller.
> > Also fix the other two minor issues on the way, namely not being fully
> > ECAM compliant and config space accesses being restricted to 32-bit
> > accesses only.
> 
> As I'm sure you've noticed, controllers that support only 32-bit
> config writes are not spec compliant and devices may not work
> correctly.  The comment in pci_generic_config_write32() explains why.
> 
> You may not trip over this problem frequently, but I wouldn't call it
> a "minor" issue because when you *do* trip over it, you have no
> indication that a register was corrupted.
> 
> Even ECAM compliance is not really minor -- if this controller were
> fully compliant with the spec, you would need ZERO Linux changes to
> support it.  Every quirk like this means additional maintenance
> burden, and it's not just a one-time thing.  It means old kernels that
> *should* "just work" on your system will not work unless somebody
> backports the quirk.

With regards to URs resulting in unwanted aborts or similar - this seems
to be a very common theme amongst ARM PCI controller drivers. For example
both ARM32 imx6 and ARM32 keystone have fault handlers to handle an abort
and fabricate a 0xffffffff read value.

The ARM32 rcar driver, whilst it doesn't appear to produce an abort, does
read the PCI_STATUS register after making a config read to determine if
any aborts have happened - in which case it reports
PCIBIOS_DEVICE_NOT_FOUND.

And as recently reported [1], the rockchip driver also appears to produce
aborts.

I suspect that this ARM64 controller driver won't be the last either. Thus
any solution here may form the basis of copy-cat solutions for subsequent
controllers.

From my understanding of the issues, the ARM64 serrors are imprecise and
as a result there isn't a sensible way of using them to determine that a
read is a UR. So where there are no other solutions to suppress the
generation of an abort by the controller, the only solutions that seem to
exist are 1) pre-scan the devices in firmware and only talk to those devices
in Linux - a safe option but limiting - perhaps with side effects for CRS
and 2) the approach rcar takes in using the PCI_STATUS register - though
you'd end up having to mask the serror (PSTATE.A) for a limited period of
time - a risky option (you'll miss real serrors) - but with no side effects.

(I don't know if option 2 is feasible in this case by the way).

[1] https://lore.kernel.org/linux-pci/2a381384-9d47-a7e2-679c-780950cd862d@rock-chips.com/2-0001-WFT-PCI-rockchip-play-game-with-unsupported-request-.patch

Thanks,

Andrew Murray

> 
> > This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> > and to access *any* devices (there are no platform devices except UART).
Andre Przywara Dec. 13, 2019, 2:23 p.m. UTC | #9
On Thu, 12 Dec 2019 12:37:48 +0000
Andrew Murray <andrew.murray@arm.com> wrote:

Hi Andrew,

> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> > From: Deepak Pandey <Deepak.Pandey@arm.com>
> > 
> > The Arm N1SDP SoC suffers from some PCIe integration issues, most
> > prominently config space accesses to not existing BDFs being answered
> > with a bus abort, resulting in an SError.  
> 
> It wouldn't be a surprise if the host controller handled UR completions
> differently depending on if they were in response to a Type 0 configuration
> request or a Type 1 configuration request. (I think I've seen this before).

Yeah, that rings a bell here as well, I remember something with the RK3399 PCIe RC.
 
> Have you verified that you still get a bus abort when you attempt to
> perform a config read of a non-existent device downstream of the PCIe switch?
> (and thus as a response to a Type 1 request).

I think I have checked this (please confirm if my experiment is valid): I get SErrors for both probing non-existent devices on bus 0 and other busses.

> I ask because if this is the case, and knowing that the PCIe switch is
> fixed, then it would be possible to simplify this quirk (by just making
> assumptions of the presence of devices in bus 0 rather than all the busses).

Yeah, thanks for trying, but no luck here ;-)

> > To mitigate this, the firmware scans the bus before boot (catching the
> > SErrors) and creates a table with valid BDFs, which acts as a filter for
> > Linux' config space accesses.
> > 
> > Add code consulting the table as an ACPI PCIe quirk, also register the
> > corresponding device tree based description of the host controller.
> > Also fix the other two minor issues on the way, namely not being fully
> > ECAM compliant and config space accesses being restricted to 32-bit
> > accesses only.
> > 
> > This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> > and to access *any* devices (there are no platform devices except UART).  
> 
> This implies that this quirk has no side-effects and everything will work
> as expected - but this is only true for the simple case. For example hot
> plug won't work, SR-IOV, and others won't work.

Yeah, good point, I should have mentioned this. This is really a best effort hack^Wworkaround to make the system work as best as possible.
Yes, SR-IOV won't work. We are about to evaluate our options here, but for now it's just not supported on this system.

I don't think hot plug is of a particular concern here, since the hardware doesn't support physical hot plug. I am not sure if a Thunderbolt add-in card would trigger this problem, but in the worst case it just wouldn't work.

> Also what happens for devices that return CRS? - does this also result in an
> abort?

I haven't tried, and it looks like it's hard to trigger? But chances are indeed that it could generate SErrors.

> Does that mean that the firmware will consider these devices as not
> present instead of not ready yet? If this is an issue, then FLR of devices
> will also create issues (resulting in SErrors for users).
> 
> I think it would be helpful to update this commit message to indicate that
> this makes it work better, but it may be broken in certain ways.

Good point. This is really a pragmatic patch to make the system usable at all.

> > Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com>
> > [Sudipto: extend to cover the CCIX root port as well]
> > Signed-off-by: Sudipto Paul <sudipto.paul@arm.com>
> > [Andre: fix coding style issues, rewrite some parts, add DT support]
> > Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> > ---
> >  arch/arm64/configs/defconfig        |   1 +
> >  drivers/acpi/pci_mcfg.c             |   7 +
> >  drivers/pci/controller/Kconfig      |  11 ++
> >  drivers/pci/controller/Makefile     |   1 +
> >  drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++
> >  include/linux/pci-ecam.h            |   2 +
> >  6 files changed, 218 insertions(+)
> >  create mode 100644 drivers/pci/controller/pcie-n1sdp.c
> > 
> > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
> > index 6a83ba2aea3e..58124ef5070b 100644
> > --- a/arch/arm64/configs/defconfig
> > +++ b/arch/arm64/configs/defconfig
> > @@ -177,6 +177,7 @@ CONFIG_NET_9P=y
> >  CONFIG_NET_9P_VIRTIO=y
> >  CONFIG_PCI=y
> >  CONFIG_PCIEPORTBUS=y
> > +CONFIG_PCI_QUIRKS=y
> >  CONFIG_PCI_IOV=y
> >  CONFIG_HOTPLUG_PCI=y
> >  CONFIG_HOTPLUG_PCI_ACPI=y
> > diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
> > index 6b347d9920cc..7a2b41b9ab57 100644
> > --- a/drivers/acpi/pci_mcfg.c
> > +++ b/drivers/acpi/pci_mcfg.c
> > @@ -142,6 +142,13 @@ static struct mcfg_fixup mcfg_quirks[] = {
> >  	XGENE_V2_ECAM_MCFG(4, 0),
> >  	XGENE_V2_ECAM_MCFG(4, 1),
> >  	XGENE_V2_ECAM_MCFG(4, 2),
> > +
> > +#define N1SDP_ECAM_MCFG(rev, seg, ops) \
> > +	{"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops }
> > +
> > +	/* N1SDP SoC with v1 PCIe controller */
> > +	N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops),
> > +	N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops),
> >  };
> >  
> >  static char mcfg_oem_id[ACPI_OEM_ID_SIZE];
> > diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
> > index c77069c8ee5d..45700d32f02e 100644
> > --- a/drivers/pci/controller/Kconfig
> > +++ b/drivers/pci/controller/Kconfig
> > @@ -37,6 +37,17 @@ config PCI_FTPCI100
> >  	depends on OF
> >  	default ARCH_GEMINI
> >  
> > +config PCIE_HOST_N1SDP_ECAM
> > +	bool "ARM N1SDP PCIe Controller"
> > +	depends on ARM64
> > +	depends on OF || (ACPI && PCI_QUIRKS)
> > +	select PCI_HOST_COMMON
> > +	default y if ARCH_VEXPRESS
> > +	help
> > +	  Say Y here if you want PCIe support for the Arm N1SDP platform.
> > +	  The controller is ECAM compliant, but needs a quirk to workaround
> > +	  an integration issue.  
> 
> Again - please indicate the scope of support provided.
> 
> > +
> >  config PCI_TEGRA
> >  	bool "NVIDIA Tegra PCIe controller"
> >  	depends on ARCH_TEGRA || COMPILE_TEST
> > diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
> > index 3d4f597f15ce..5f47fefbd67d 100644
> > --- a/drivers/pci/controller/Makefile
> > +++ b/drivers/pci/controller/Makefile
> > @@ -28,6 +28,7 @@ obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
> >  obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> >  obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
> >  obj-$(CONFIG_VMD) += vmd.o
> > +obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o
> >  # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
> >  obj-y				+= dwc/
> >  
> > diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c
> > new file mode 100644
> > index 000000000000..620ab221466c
> > --- /dev/null
> > +++ b/drivers/pci/controller/pcie-n1sdp.c
> > @@ -0,0 +1,196 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Copyright (C) 2018/2019 ARM Ltd.
> > + *
> > + * This quirk is to mask the following issues:
> > + * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus
> > + *		  error (signalled as an asynchronous SError)
> > + * - MCFG BDF mapping: the root complex is mapped separately from the device
> > + *		       config space
> > + * - Non 32-bit accesses to config space are not supported.
> > + *
> > + * At boot time the SCP board firmware creates a discovery table with
> > + * the root complex' base address and the valid BDF values, discovered while
> > + * scanning the config space and catching the SErrors.
> > + * Linux responds only to the EPs listed in this table, returning NULL  
> 
> NIT: it will respond to the switch devices which aren't EPs.
> 
> 
> > + * for the rest.
> > + */
> > +
> > +#include <linux/kernel.h>
> > +#include <linux/init.h>
> > +#include <linux/ioport.h>
> > +#include <linux/sizes.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of.h>
> > +#include <linux/pci-ecam.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/module.h>
> > +
> > +/* Platform specific values as hardcoded in the firmware. */
> > +#define AP_NS_SHARED_MEM_BASE	0x06000000
> > +#define MAX_SEGMENTS		2		/* Two PCIe root complexes. */
> > +#define BDF_TABLE_SIZE		SZ_16K
> > +
> > +/*
> > + * Shared memory layout as written by the SCP upon boot time:
> > + *  ----
> > + *  Discover data header --> RC base address
> > + *                       \-> BDF Count
> > + *  Discover data        --> BDF 0...n
> > + *  ----
> > + */
> > +struct pcie_discovery_data {
> > +	u32 rc_base_addr;
> > +	u32 nr_bdfs;
> > +	u32 valid_bdfs[0];
> > +} *pcie_discovery_data[MAX_SEGMENTS];
> > +
> > +void __iomem *rc_remapped_addr[MAX_SEGMENTS];
> > +
> > +/*
> > + * map_bus() is called before we do a config space access for a certain
> > + * device. We use this to check whether this device is valid, avoiding
> > + * config space accesses which would result in an SError otherwise.
> > + */
> > +static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn,
> > +				       int where)
> > +{
> > +	struct pci_config_window *cfg = bus->sysdata;
> > +	unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> > +	unsigned int busn = bus->number;
> > +	unsigned int segment = bus->domain_nr;
> > +	unsigned int bdf_addr;
> > +	unsigned int table_count, i;
> > +
> > +	if (segment >= MAX_SEGMENTS ||
> > +	    busn < cfg->busr.start || busn > cfg->busr.end)
> > +		return NULL;
> > +
> > +	/* The PCIe root complex has a separate config space mapping. */
> > +	if (busn == 0 && devfn == 0)
> > +		return rc_remapped_addr[segment] + where;
> > +
> > +	busn -= cfg->busr.start;
> > +	bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift);
> > +	table_count = pcie_discovery_data[segment]->nr_bdfs;
> > +	for (i = 0; i < table_count; i++) {
> > +		if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i])
> > +			return pci_ecam_map_bus(bus, devfn, where);
> > +	}
> > +
> > +	return NULL;
> > +}
> > +
> > +static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment)
> > +{
> > +	phys_addr_t table_base;
> > +	struct device *dev = cfg->parent;
> > +	struct pcie_discovery_data *shared_data;
> > +	size_t bdfs_size;
> > +
> > +	if (segment >= MAX_SEGMENTS)
> > +		return -ENODEV;
> > +
> > +	table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE;  
> 
> How can you be sure that this table is populated and isn't junk? I.e. using an older
> SCP version?

This is just (and always was) part of the firmware. If not, it won't work. Technically we tie this to the DT compatible or the ACPI signature.

> > +
> > +	if (!request_mem_region(table_base, BDF_TABLE_SIZE,
> > +				"PCIe valid BDFs")) {
> > +		dev_err(dev, "PCIe BDF shared region request failed\n");
> > +		return -ENOMEM;
> > +	}
> > +
> > +	shared_data = devm_ioremap(dev,
> > +				   table_base, BDF_TABLE_SIZE);
> > +	if (!shared_data)
> > +		return -ENOMEM;
> > +
> > +	/* Copy the valid BDFs structure to allocated normal memory. */
> > +	bdfs_size = sizeof(struct pcie_discovery_data) +
> > +		    sizeof(u32) * shared_data->nr_bdfs;
> > +	pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL);
> > +	if (!pcie_discovery_data[segment])
> > +		return -ENOMEM;
> > +
> > +	memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size);
> > +
> > +	rc_remapped_addr[segment] = devm_ioremap_nocache(dev,
> > +						shared_data->rc_base_addr,
> > +						PCI_CFG_SPACE_EXP_SIZE);
> > +	if (!rc_remapped_addr[segment]) {
> > +		dev_err(dev, "Cannot remap root port base\n");
> > +		return -ENOMEM;
> > +	}
> > +
> > +	devm_iounmap(dev, shared_data);
> > +
> > +	return 0;
> > +}
> > +
> > +static int pci_n1sdp_pcie_init(struct pci_config_window *cfg)
> > +{
> > +	return pci_n1sdp_init(cfg, 0);
> > +}
> > +
> > +static int pci_n1sdp_ccix_init(struct pci_config_window *cfg)
> > +{
> > +	return pci_n1sdp_init(cfg, 1);
> > +}
> > +
> > +struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = {
> > +	.bus_shift	= 20,
> > +	.init		= pci_n1sdp_pcie_init,
> > +	.pci_ops	= {
> > +		.map_bus        = pci_n1sdp_map_bus,
> > +		.read           = pci_generic_config_read32,
> > +		.write          = pci_generic_config_write32,
> > +	}
> > +};
> > +
> > +struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = {
> > +	.bus_shift	= 20,
> > +	.init		= pci_n1sdp_ccix_init,
> > +	.pci_ops	= {
> > +		.map_bus        = pci_n1sdp_map_bus,
> > +		.read           = pci_generic_config_read32,
> > +		.write          = pci_generic_config_write32,
> > +	}
> > +};
> > +
> > +static const struct of_device_id n1sdp_pcie_of_match[] = {
> > +	{ .compatible = "arm,n1sdp-pcie" },
> > +	{ },
> > +};
> > +MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match);
> > +
> > +static int n1sdp_pcie_probe(struct platform_device *pdev)
> > +{
> > +	const struct device_node *of_node = pdev->dev.of_node;
> > +	u32 segment;
> > +
> > +	if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) {
> > +		dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n");
> > +		return -EINVAL;
> > +	}  
> 
> Can you use of_get_pci_domain_nr here?

Ah, indeed. There seems to be an of_... function for everything ;-)

Cheers,
Andre.

> > +
> > +	switch (segment) {
> > +	case 0:
> > +		return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops);
> > +	case 1:
> > +		return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops);
> > +	}
> > +
> > +	dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n",
> > +		MAX_SEGMENTS);
> > +
> > +	return -EINVAL;
> > +}
> > +
> > +static struct platform_driver n1sdp_pcie_driver = {
> > +	.driver = {
> > +		.name = KBUILD_MODNAME,
> > +		.of_match_table = n1sdp_pcie_of_match,
> > +		.suppress_bind_attrs = true,
> > +	},
> > +	.probe = n1sdp_pcie_probe,
> > +};
> > +builtin_platform_driver(n1sdp_pcie_driver);
> > diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h
> > index a73164c85e78..03cdea69f4e8 100644
> > --- a/include/linux/pci-ecam.h
> > +++ b/include/linux/pci-ecam.h
> > @@ -57,6 +57,8 @@ extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */
> >  extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */
> >  extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */
> >  extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */
> > +extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */
> > +extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */
> >  #endif
> >  
> >  #ifdef CONFIG_PCI_HOST_COMMON
> > -- 
> > 2.17.1
> >
Andre Przywara Dec. 13, 2019, 2:39 p.m. UTC | #10
On Thu, 12 Dec 2019 21:07:24 +0000
Andrew Murray <andrew.murray@arm.com> wrote:

Hi,

> On Tue, Dec 10, 2019 at 08:41:15AM -0600, Bjorn Helgaas wrote:
> > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:  

[ ... ]

> > Even ECAM compliance is not really minor -- if this controller were
> > fully compliant with the spec, you would need ZERO Linux changes to
> > support it.  Every quirk like this means additional maintenance
> > burden, and it's not just a one-time thing.  It means old kernels that
> > *should* "just work" on your system will not work unless somebody
> > backports the quirk.  
> 
> With regards to URs resulting in unwanted aborts or similar - this seems
> to be a very common theme amongst ARM PCI controller drivers. For example
> both ARM32 imx6 and ARM32 keystone have fault handlers to handle an abort
> and fabricate a 0xffffffff read value.
> 
> The ARM32 rcar driver, whilst it doesn't appear to produce an abort, does
> read the PCI_STATUS register after making a config read to determine if
> any aborts have happened - in which case it reports
> PCIBIOS_DEVICE_NOT_FOUND.
> 
> And as recently reported [1], the rockchip driver also appears to produce
> aborts.
> 
> I suspect that this ARM64 controller driver won't be the last either. Thus
> any solution here may form the basis of copy-cat solutions for subsequent
> controllers.

Well, I think Bjorn is aware of them, but was actually hoping that those broken controllers would go away at some point ;-)
And just to make this clear: I would categorise this issue as an integration bug, which just can't be fixed in hardware or firmware easily. It was never meant to be this way. So I am not sure we should promote generic solutions here.

> From my understanding of the issues, the ARM64 serrors are imprecise and
> as a result there isn't a sensible way of using them to determine that a
> read is a UR. So where there are no other solutions to suppress the
> generation of an abort by the controller, the only solutions that seem to
> exist are 1) pre-scan the devices in firmware and only talk to those devices
> in Linux - a safe option but limiting - perhaps with side effects for CRS
> and 2) the approach rcar takes in using the PCI_STATUS register - though
> you'd end up having to mask the serror (PSTATE.A) for a limited period of
> time - a risky option (you'll miss real serrors) - but with no side effects.
> 
> (I don't know if option 2 is feasible in this case by the way).

Interesting, we might evaluate this, but mostly out of curiosity or for debugging. I don't think it's really a better option.
If there is a safe way of making this work in the majority of cases, that should be the way to go. Setting PSTATE.A sounds quite wacky to me.

Thanks,
Andre.

> [1] https://lore.kernel.org/linux-pci/2a381384-9d47-a7e2-679c-780950cd862d@rock-chips.com/2-0001-WFT-PCI-rockchip-play-game-with-unsupported-request-.patch
> 
> Thanks,
> 
> Andrew Murray
> 
> >   
> > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing
> > > and to access *any* devices (there are no platform devices except UART).
Bjorn Helgaas Dec. 13, 2019, 9:07 p.m. UTC | #11
On Thu, Dec 12, 2019 at 11:05:31AM +0000, Andre Przywara wrote:
> On 11/12/2019 20:17, Bjorn Helgaas wrote:
> > On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote:
> >> On Tue, 10 Dec 2019 08:41:15 -0600
> >> Bjorn Helgaas <helgaas@kernel.org> wrote:
> >>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
> >>>> From: Deepak Pandey <Deepak.Pandey@arm.com>
> >>>>
> >>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> >>>> prominently config space accesses to not existing BDFs being answered
> >>>> with a bus abort, resulting in an SError.  
> >>>
> >>> Can we tease this apart a little more?  Linux doesn't program all the
> >>> bits that control error signaling, so even on hardware that works
> >>> perfectly, much of this behavior is determined by what firmware did.
> >>> I wonder if Linux could be more careful about this.
> >>>
> >>> "Bus abort" is not a term used in PCIe.
> >>
> >> Yes, sorry, that was my sloppy term, also aiming more at the CPU
> >> side of the bus, between the cores and the RC.
> >>
> >>>  IIUC, a config read to a
> >>> device that doesn't exist should terminate with an Unsupported Request
> >>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.
> >>
> >> Yes, that's what Lorenzo mentioned as well.
> >>
> >>> The UR should be an uncorrectable non-fatal error (Table 6-5), and
> >>> Figures 6-2 and 6-3 show how it should be handled and when it should
> >>> be signaled as a system error.  In case you don't have a copy of the
> >>> spec, I extracted those two figures and put them at [1].
> >>
> >> Thanks for that.
> >> So in the last few months we tossed several ideas around how to
> >> work-around this without kernel intervention, all of them turned out
> >> to be not working. There are indeed registers in the RC that
> >> influence error reporting to the CPU side, but even if we could
> >> suppress (or catch) the SError, we can't recover and fixup the read
> >> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I
> >> understood this, there are gates missing which are supposed to
> >> translate this specific UR into a valid "all-1s" response.
> > 
> > But the commit log says firmware scanned the bus (catching the
> > SErrors).  Shouldn't Linux be able to catch them the same way?
> 
> Not really. The scanning is done by the SCP management processor,
> which is a Cortex-M class core on the same bus. So it's a simple,
> single core running very early after power-on, when the actual AP
> cores are still off. The SError handler is set to just increase a
> value, then to return. This value is then checked before and after a
> config space access for a given BDF:
> https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp
> 
> On the AP cores that run Linux later on this is quite different: The
> SError is asynchronous, imprecise (inexact) and has no syndrome
> information. That means we can't attribute this anymore to the
> faulting instruction, we don't even know if it happened due to this
> config space access. The CPU might have executed later instructions
> already, so the state is broken at this point. SError basically
> means: the system is screwed up.  Because this is quite common for
> SErrors, we don't even allow to register SError handlers in arm64
> Linux.
> 
> So even if we could somehow handle this is in Linux, it would be a
> much greater and intrusive hack, so I'd rather stick with this
> version.

The problem is that from a PCIe point of view, UR is something we
should be able to tolerate.  It happens during enumeration and also
during hotplug.  It definitely does not mean "the system is screwed up
and must be rebooted."

To go back to Figure 6-3, I'm getting the impression that the "System
Error" shown at the top is *not* the "SError" you're referring to.  If
they were the same, the Root Control enable bits should gate it, but
according to your lspci, those enable bits are cleared, yet you still
take SErrors.

SError is asynchronous and imprecise.  Is there no way to do the
config access in a way that makes it precise, by adding some kind of
sync?  There's no reason we can't single-thread config accesses and
maybe even MMIO/IO port accesses as well if necessary.

Bjorn
Jon Masters Dec. 18, 2019, 2:21 a.m. UTC | #12
On 12/9/19 11:26 AM, Will Deacon wrote:
> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
>> From: Deepak Pandey <Deepak.Pandey@arm.com>
>>
>> The Arm N1SDP SoC suffers from some PCIe integration issues, most
>> prominently config space accesses to not existing BDFs being answered
>> with a bus abort, resulting in an SError.
> 
> "Do as I say, not as I do"?

In my former role I asked nicely that these patches not be posted 
upstream, but I see that they ended up being posted anyway. Hacking up 
upstream Linux to cover for the fact that a (reference) platform is 
non-standard is not only not good form but it actively harms the community.

You'll have people consume this platform and not realize that it's 
broken, IP won't get fixed, and generally it'll be a mess. Yes, it's 
unfortunate, but so was taping out that platform without working PCI. We 
all know what should have happened, and what the right move ahead is.

Jon.
Andre Przywara Dec. 18, 2019, 10:22 a.m. UTC | #13
On Tue, 17 Dec 2019 21:21:17 -0500
Jon Masters <jcm@jonmasters.org> wrote:

Hi Jon,

> On 12/9/19 11:26 AM, Will Deacon wrote:
> > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:  
> >> From: Deepak Pandey <Deepak.Pandey@arm.com>
> >>
> >> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> >> prominently config space accesses to not existing BDFs being answered
> >> with a bus abort, resulting in an SError.  
> > 
> > "Do as I say, not as I do"?  
> 
> In my former role I asked nicely that these patches not be posted 
> upstream, but I see that they ended up being posted anyway. Hacking up 
> upstream Linux to cover for the fact that a (reference) platform is 
> non-standard is not only not good form but it actively harms the community.

Please keep in mind that this platform was designed to be standards compliant, it is just due to an integration problem that this is not the case with this silicon. So we end up with the usual hardware errata, which the kernel can fix up. I agree it's not nice, and I also want it fixed in hardware, but I guess that's the usual software guy's pipe dream.

> You'll have people consume this platform and not realize that it's 
> broken, IP won't get fixed, and generally it'll be a mess.

I don't see how yet another ACPI quirk in the ACPI quirk framework(!) will make a mess.
Actually the rest of the system is standards compliant (it even uses ACPI from the very beginning ;-), so it's just this problem that prevents us from using the system in the proper, standards compliant way. Effectively we are back to the embedded times with compiling your own kernel and somehow getting a root filesystem on the hard drive.
If there would be mainline kernel support, all of this would go away and would could use standard distro installers (given they backport the patch).

> Yes, it's 
> unfortunate, but so was taping out that platform without working PCI. We 
> all know what should have happened, and what the right move ahead is.

That may come as a surprise to some, but Arm Ltd. is actually not really in the business of *producing silicon*, so a respin of the chip was and is not an option. I too wish it would be different, but that's how it is.

Cheers,
Andre.
Marc Zyngier Dec. 18, 2019, 1:53 p.m. UTC | #14
On 2019-12-18 10:22, Andre Przywara wrote:
> On Tue, 17 Dec 2019 21:21:17 -0500
> Jon Masters <jcm@jonmasters.org> wrote:
>
> Hi Jon,
>
>> On 12/9/19 11:26 AM, Will Deacon wrote:
>> > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:
>> >> From: Deepak Pandey <Deepak.Pandey@arm.com>
>> >>
>> >> The Arm N1SDP SoC suffers from some PCIe integration issues, most
>> >> prominently config space accesses to not existing BDFs being 
>> answered
>> >> with a bus abort, resulting in an SError.
>> >
>> > "Do as I say, not as I do"?
>>
>> In my former role I asked nicely that these patches not be posted
>> upstream, but I see that they ended up being posted anyway. Hacking 
>> up
>> upstream Linux to cover for the fact that a (reference) platform is
>> non-standard is not only not good form but it actively harms the 
>> community.
>
> Please keep in mind that this platform was designed to be standards
> compliant, it is just due to an integration problem that this is not
> the case with this silicon. So we end up with the usual hardware
> errata, which the kernel can fix up. I agree it's not nice, and I 
> also
> want it fixed in hardware, but I guess that's the usual software 
> guy's
> pipe dream.
>
>> You'll have people consume this platform and not realize that it's
>> broken, IP won't get fixed, and generally it'll be a mess.
>
> I don't see how yet another ACPI quirk in the ACPI quirk framework(!)
> will make a mess.
> Actually the rest of the system is standards compliant (it even uses
> ACPI from the very beginning ;-), so it's just this problem that
> prevents us from using the system in the proper, standards compliant
> way. Effectively we are back to the embedded times with compiling 
> your
> own kernel and somehow getting a root filesystem on the hard drive.
> If there would be mainline kernel support, all of this would go away
> and would could use standard distro installers (given they backport
> the patch).
>
>> Yes, it's
>> unfortunate, but so was taping out that platform without working 
>> PCI. We
>> all know what should have happened, and what the right move ahead 
>> is.
>
> That may come as a surprise to some, but Arm Ltd. is actually not
> really in the business of *producing silicon*, so a respin of the 
> chip
> was and is not an option. I too wish it would be different, but 
> that's
> how it is.

In all honesty, I really wonder why we are even having this discussion:

- The HW is unavailable to the mere mortal. And even most superheroes
   cannot get their hands on it

- Even with this terrible son of a hack, essential PCIe features cannot
   work (and yes, I do consider SR-IOV as an essential feature)

- If we take this hack and somehow get this thing to run with mainline,
   we will never be able to say "no" to this kind of unfinished,
   *prototype* implementations ever again

To sum it up: a hack that doesn't really work for a prototype that you
can't really buy? Why should we even care?

         M.
Andre Przywara Dec. 18, 2019, 3:07 p.m. UTC | #15
On Fri, 13 Dec 2019 15:07:07 -0600
Bjorn Helgaas <helgaas@kernel.org> wrote:

Hi Bjorn,

> On Thu, Dec 12, 2019 at 11:05:31AM +0000, Andre Przywara wrote:
> > On 11/12/2019 20:17, Bjorn Helgaas wrote:  
> > > On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote:  
> > >> On Tue, 10 Dec 2019 08:41:15 -0600
> > >> Bjorn Helgaas <helgaas@kernel.org> wrote:  
> > >>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote:  
> > >>>> From: Deepak Pandey <Deepak.Pandey@arm.com>
> > >>>>
> > >>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most
> > >>>> prominently config space accesses to not existing BDFs being answered
> > >>>> with a bus abort, resulting in an SError.    
> > >>>
> > >>> Can we tease this apart a little more?  Linux doesn't program all the
> > >>> bits that control error signaling, so even on hardware that works
> > >>> perfectly, much of this behavior is determined by what firmware did.
> > >>> I wonder if Linux could be more careful about this.
> > >>>
> > >>> "Bus abort" is not a term used in PCIe.  
> > >>
> > >> Yes, sorry, that was my sloppy term, also aiming more at the CPU
> > >> side of the bus, between the cores and the RC.
> > >>  
> > >>>  IIUC, a config read to a
> > >>> device that doesn't exist should terminate with an Unsupported Request
> > >>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1.  
> > >>
> > >> Yes, that's what Lorenzo mentioned as well.
> > >>  
> > >>> The UR should be an uncorrectable non-fatal error (Table 6-5), and
> > >>> Figures 6-2 and 6-3 show how it should be handled and when it should
> > >>> be signaled as a system error.  In case you don't have a copy of the
> > >>> spec, I extracted those two figures and put them at [1].  
> > >>
> > >> Thanks for that.
> > >> So in the last few months we tossed several ideas around how to
> > >> work-around this without kernel intervention, all of them turned out
> > >> to be not working. There are indeed registers in the RC that
> > >> influence error reporting to the CPU side, but even if we could
> > >> suppress (or catch) the SError, we can't recover and fixup the read
> > >> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I
> > >> understood this, there are gates missing which are supposed to
> > >> translate this specific UR into a valid "all-1s" response.  
> > > 
> > > But the commit log says firmware scanned the bus (catching the
> > > SErrors).  Shouldn't Linux be able to catch them the same way?  
> > 
> > Not really. The scanning is done by the SCP management processor,
> > which is a Cortex-M class core on the same bus. So it's a simple,
> > single core running very early after power-on, when the actual AP
> > cores are still off. The SError handler is set to just increase a
> > value, then to return. This value is then checked before and after a
> > config space access for a given BDF:
> > https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp
> > 
> > On the AP cores that run Linux later on this is quite different: The
> > SError is asynchronous, imprecise (inexact) and has no syndrome
> > information. That means we can't attribute this anymore to the
> > faulting instruction, we don't even know if it happened due to this
> > config space access. The CPU might have executed later instructions
> > already, so the state is broken at this point. SError basically
> > means: the system is screwed up.  Because this is quite common for
> > SErrors, we don't even allow to register SError handlers in arm64
> > Linux.
> > 
> > So even if we could somehow handle this is in Linux, it would be a
> > much greater and intrusive hack, so I'd rather stick with this
> > version.  
> 
> The problem is that from a PCIe point of view, UR is something we
> should be able to tolerate.  It happens during enumeration and also
> during hotplug.  It definitely does not mean "the system is screwed up
> and must be rebooted."

I agree, I am also wondering why an (ARM) SError was considered an appropriate answer in the first place. Maybe it's due to some confusion between the ARM architecture term "SError" and the PCI #SERR signal?

> To go back to Figure 6-3, I'm getting the impression that the "System
> Error" shown at the top is *not* the "SError" you're referring to.  If
> they were the same, the Root Control enable bits should gate it, but
> according to your lspci, those enable bits are cleared, yet you still
> take SErrors.

Yes, "System Error" in PCI lingo is definitely something different and unrelated to the ARM architecture SError.
I also checked the (legacy) PCI command register, the PCIe baseline error handling and AER, all of them seem to not propagate the SERR signal. So I reckon it's something which does this on the root complex integration level, outside of generic PCIe. So far I also couldn't find a masking bit in the RC data sheet as well.
 
> SError is asynchronous and imprecise.  Is there no way to do the
> config access in a way that makes it precise, by adding some kind of
> sync?  There's no reason we can't single-thread config accesses and
> maybe even MMIO/IO port accesses as well if necessary.

There is, on this core, but it becomes even more dodgy: While you can *block* delivery of an SError (by setting PSTATE.A), you couldn't clear it so far. ARMv8.2 introduced the "esb" instruction, which allows us to actually consume the SError, so clearing the A bit afterwards would continue operation as normal. This particular core supports v8.2, but the Rockchip SoC does not. So it would not help in the general case, also has that bitter taste of possibly masking an unrelated SError.

So I managed to hack some proof of concept up where I can indeed fence in the SError around the config space access, within the kernel, so I can *almost* certainly say whether this readl/readw triggered this. But that would require aarch64 specific code (even instructions that require an ARMv8.2 compatible core) in the PCI driver, and so is probably more ugly than the existing solution. Plus we would need to iron out the remaining uncertainty, not sure that's actually possible.

Cheers,
Andre.
diff mbox series

Patch

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 6a83ba2aea3e..58124ef5070b 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -177,6 +177,7 @@  CONFIG_NET_9P=y
 CONFIG_NET_9P_VIRTIO=y
 CONFIG_PCI=y
 CONFIG_PCIEPORTBUS=y
+CONFIG_PCI_QUIRKS=y
 CONFIG_PCI_IOV=y
 CONFIG_HOTPLUG_PCI=y
 CONFIG_HOTPLUG_PCI_ACPI=y
diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
index 6b347d9920cc..7a2b41b9ab57 100644
--- a/drivers/acpi/pci_mcfg.c
+++ b/drivers/acpi/pci_mcfg.c
@@ -142,6 +142,13 @@  static struct mcfg_fixup mcfg_quirks[] = {
 	XGENE_V2_ECAM_MCFG(4, 0),
 	XGENE_V2_ECAM_MCFG(4, 1),
 	XGENE_V2_ECAM_MCFG(4, 2),
+
+#define N1SDP_ECAM_MCFG(rev, seg, ops) \
+	{"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops }
+
+	/* N1SDP SoC with v1 PCIe controller */
+	N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops),
+	N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops),
 };
 
 static char mcfg_oem_id[ACPI_OEM_ID_SIZE];
diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
index c77069c8ee5d..45700d32f02e 100644
--- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfig
@@ -37,6 +37,17 @@  config PCI_FTPCI100
 	depends on OF
 	default ARCH_GEMINI
 
+config PCIE_HOST_N1SDP_ECAM
+	bool "ARM N1SDP PCIe Controller"
+	depends on ARM64
+	depends on OF || (ACPI && PCI_QUIRKS)
+	select PCI_HOST_COMMON
+	default y if ARCH_VEXPRESS
+	help
+	  Say Y here if you want PCIe support for the Arm N1SDP platform.
+	  The controller is ECAM compliant, but needs a quirk to workaround
+	  an integration issue.
+
 config PCI_TEGRA
 	bool "NVIDIA Tegra PCIe controller"
 	depends on ARCH_TEGRA || COMPILE_TEST
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
index 3d4f597f15ce..5f47fefbd67d 100644
--- a/drivers/pci/controller/Makefile
+++ b/drivers/pci/controller/Makefile
@@ -28,6 +28,7 @@  obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
 obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
 obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
 obj-$(CONFIG_VMD) += vmd.o
+obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o
 # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
 obj-y				+= dwc/
 
diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c
new file mode 100644
index 000000000000..620ab221466c
--- /dev/null
+++ b/drivers/pci/controller/pcie-n1sdp.c
@@ -0,0 +1,196 @@ 
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018/2019 ARM Ltd.
+ *
+ * This quirk is to mask the following issues:
+ * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus
+ *		  error (signalled as an asynchronous SError)
+ * - MCFG BDF mapping: the root complex is mapped separately from the device
+ *		       config space
+ * - Non 32-bit accesses to config space are not supported.
+ *
+ * At boot time the SCP board firmware creates a discovery table with
+ * the root complex' base address and the valid BDF values, discovered while
+ * scanning the config space and catching the SErrors.
+ * Linux responds only to the EPs listed in this table, returning NULL
+ * for the rest.
+ */
+
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/ioport.h>
+#include <linux/sizes.h>
+#include <linux/of_pci.h>
+#include <linux/of.h>
+#include <linux/pci-ecam.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+
+/* Platform specific values as hardcoded in the firmware. */
+#define AP_NS_SHARED_MEM_BASE	0x06000000
+#define MAX_SEGMENTS		2		/* Two PCIe root complexes. */
+#define BDF_TABLE_SIZE		SZ_16K
+
+/*
+ * Shared memory layout as written by the SCP upon boot time:
+ *  ----
+ *  Discover data header --> RC base address
+ *                       \-> BDF Count
+ *  Discover data        --> BDF 0...n
+ *  ----
+ */
+struct pcie_discovery_data {
+	u32 rc_base_addr;
+	u32 nr_bdfs;
+	u32 valid_bdfs[0];
+} *pcie_discovery_data[MAX_SEGMENTS];
+
+void __iomem *rc_remapped_addr[MAX_SEGMENTS];
+
+/*
+ * map_bus() is called before we do a config space access for a certain
+ * device. We use this to check whether this device is valid, avoiding
+ * config space accesses which would result in an SError otherwise.
+ */
+static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn,
+				       int where)
+{
+	struct pci_config_window *cfg = bus->sysdata;
+	unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+	unsigned int busn = bus->number;
+	unsigned int segment = bus->domain_nr;
+	unsigned int bdf_addr;
+	unsigned int table_count, i;
+
+	if (segment >= MAX_SEGMENTS ||
+	    busn < cfg->busr.start || busn > cfg->busr.end)
+		return NULL;
+
+	/* The PCIe root complex has a separate config space mapping. */
+	if (busn == 0 && devfn == 0)
+		return rc_remapped_addr[segment] + where;
+
+	busn -= cfg->busr.start;
+	bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift);
+	table_count = pcie_discovery_data[segment]->nr_bdfs;
+	for (i = 0; i < table_count; i++) {
+		if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i])
+			return pci_ecam_map_bus(bus, devfn, where);
+	}
+
+	return NULL;
+}
+
+static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment)
+{
+	phys_addr_t table_base;
+	struct device *dev = cfg->parent;
+	struct pcie_discovery_data *shared_data;
+	size_t bdfs_size;
+
+	if (segment >= MAX_SEGMENTS)
+		return -ENODEV;
+
+	table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE;
+
+	if (!request_mem_region(table_base, BDF_TABLE_SIZE,
+				"PCIe valid BDFs")) {
+		dev_err(dev, "PCIe BDF shared region request failed\n");
+		return -ENOMEM;
+	}
+
+	shared_data = devm_ioremap(dev,
+				   table_base, BDF_TABLE_SIZE);
+	if (!shared_data)
+		return -ENOMEM;
+
+	/* Copy the valid BDFs structure to allocated normal memory. */
+	bdfs_size = sizeof(struct pcie_discovery_data) +
+		    sizeof(u32) * shared_data->nr_bdfs;
+	pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL);
+	if (!pcie_discovery_data[segment])
+		return -ENOMEM;
+
+	memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size);
+
+	rc_remapped_addr[segment] = devm_ioremap_nocache(dev,
+						shared_data->rc_base_addr,
+						PCI_CFG_SPACE_EXP_SIZE);
+	if (!rc_remapped_addr[segment]) {
+		dev_err(dev, "Cannot remap root port base\n");
+		return -ENOMEM;
+	}
+
+	devm_iounmap(dev, shared_data);
+
+	return 0;
+}
+
+static int pci_n1sdp_pcie_init(struct pci_config_window *cfg)
+{
+	return pci_n1sdp_init(cfg, 0);
+}
+
+static int pci_n1sdp_ccix_init(struct pci_config_window *cfg)
+{
+	return pci_n1sdp_init(cfg, 1);
+}
+
+struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = {
+	.bus_shift	= 20,
+	.init		= pci_n1sdp_pcie_init,
+	.pci_ops	= {
+		.map_bus        = pci_n1sdp_map_bus,
+		.read           = pci_generic_config_read32,
+		.write          = pci_generic_config_write32,
+	}
+};
+
+struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = {
+	.bus_shift	= 20,
+	.init		= pci_n1sdp_ccix_init,
+	.pci_ops	= {
+		.map_bus        = pci_n1sdp_map_bus,
+		.read           = pci_generic_config_read32,
+		.write          = pci_generic_config_write32,
+	}
+};
+
+static const struct of_device_id n1sdp_pcie_of_match[] = {
+	{ .compatible = "arm,n1sdp-pcie" },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match);
+
+static int n1sdp_pcie_probe(struct platform_device *pdev)
+{
+	const struct device_node *of_node = pdev->dev.of_node;
+	u32 segment;
+
+	if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) {
+		dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n");
+		return -EINVAL;
+	}
+
+	switch (segment) {
+	case 0:
+		return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops);
+	case 1:
+		return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops);
+	}
+
+	dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n",
+		MAX_SEGMENTS);
+
+	return -EINVAL;
+}
+
+static struct platform_driver n1sdp_pcie_driver = {
+	.driver = {
+		.name = KBUILD_MODNAME,
+		.of_match_table = n1sdp_pcie_of_match,
+		.suppress_bind_attrs = true,
+	},
+	.probe = n1sdp_pcie_probe,
+};
+builtin_platform_driver(n1sdp_pcie_driver);
diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h
index a73164c85e78..03cdea69f4e8 100644
--- a/include/linux/pci-ecam.h
+++ b/include/linux/pci-ecam.h
@@ -57,6 +57,8 @@  extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */
 extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */
 extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */
 extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */
+extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */
+extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */
 #endif
 
 #ifdef CONFIG_PCI_HOST_COMMON