[BUG] Bisected Problem with LSI PCI FC Adapter

From: Dirk Gouders <dirk@gouders.net>

Dirk Gouders <dirk@gouders.net> writes:

> Dirk Gouders <dirk@gouders.net> writes:
>
>> Bjorn Helgaas <bhelgaas@google.com> writes:
>>
>>> On Thu, Sep 11, 2014 at 3:24 PM, Dirk Gouders <dirk@gouders.net> wrote:
>>>> Bjorn Helgaas <bhelgaas@google.com> writes:
>>>>
>>>>> On Thu, Sep 11, 2014 at 2:33 PM, Dirk Gouders <dirk@gouders.net> wrote:
>>>>>> What I was currently trying was to construct a test-environment so that
>>>>>> I do not need to do tests and diagnosis on a busy machine.
>>>>>>
>>>>>> I noticed that this problem seems to start with the narrow Root
>>>>>> Bridge window (00-07) but every other machine that I had a look at,
>>>>>> starts with (00-ff), so those will not trigger my problem.
>>>>>>
>>>>>> I thought I could perhaps try to shrink the window in
>>>>>> acpi_pci_root_add() to trigger the problem and that kind of works: it
>>>>>> triggers it but not exactly the same way, because it basically ends at
>>>>>> this code in pci_scan_bridge():
>>>>>>
>>>>>>         if (max >= bus->busn_res.end) {
>>>>>>                 dev_warn(&dev->dev, "can't allocate child bus %02x from %pR (pass %d)\n",
>>>>>>                          max, &bus->busn_res, pass);
>>>>>>                 goto out;
>>>>>>         }
>>>>>>
>>>>>> If this could work but I am just missing a small detail, I would be
>>>>>> glad to hear about it and do the first tests this way.  If it is
>>>>>> complete nonsense, I will just use the machine that triggers the problem
>>>>>> for the tests.
>>>>>
>>>>> I was about to suggest the same thing.  If the problem is related to
>>>>> the bus number change, we should be able to force that to happen on a
>>>>> different machine.  Your approach sounds good, so I'm guessing we just
>>>>> need a tweak.
>>>>>
>>>>> I would first double-check that the PCI adapters are identical,
>>>>> including the firmware on the card.  Can you also include your patch
>>>>> and the resulting dmesg (with debug enabled as before)?
>>>>
>>>> Currently I am at home doing just tests for understanding and that I can
>>>> hopefully use when I am back in the office.
>>>>
>>>> I already noticed the the backup FC Adapter on the test machine is not
>>>> exactly the same: it is Rev. 1 whereas the one on the failing machine is
>>>> Rev. 2.
>>>>
>>>> So, here at home my tests let a NIC disappear.  Different from the
>>>> original problem but I was just trying to reconstruct the szenario of a
>>>> misconfigured bridge causing a reconfiguration.
>>>>
>>>> What I was trying is:
>>>>
>>>> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
>>>> index e6ae603..fd146b3 100644
>>>> --- a/drivers/acpi/pci_root.c
>>>> +++ b/drivers/acpi/pci_root.c
>>>> @@ -556,6 +556,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
>>>>         strcpy(acpi_device_name(device), ACPI_PCI_ROOT_DEVICE_NAME);
>>>>         strcpy(acpi_device_class(device), ACPI_PCI_ROOT_CLASS);
>>>>         device->driver_data = root;
>>>> +       root->secondary.end = 0x02;
>>>>
>>>>         pr_info(PREFIX "%s [%s] (domain %04x %pR)\n",
>>>>                acpi_device_name(device), acpi_device_bid(device),
>>>>
>>>> The device that disappears is a NIC:
>>>>
>>>> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
>>>> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
>>>> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
>>>> 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
>>>> 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
>>>> 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
>>>> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
>>>> 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
>>>> 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4)
>>>> 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
>>>> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
>>>> 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
>>>> 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
>>>> 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
>>>> 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>>>>
>>>> This is the one that is missing with the above change:
>>>> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>>>
>>> This situation is a little different, so I don't think you're
>>> reproducing the situation we want to test.  On this box, you have:
>>>
>>>     pci_bus 0000:00: root bus resource [bus 00-02]
>>>     pci 0000:00:1c.0: PCI bridge to [bus 01]
>>>     pci 0000:00:1c.4: PCI bridge to [bus 02]
>>>
>>> so we find all the devices on bus 00 and bus 02 (there's nothing on
>>> bus 01).  My guess is the 03:00.0 device is normally behind the
>>> 00:1c.5 bridge, but we don't even scan behind that bridge because we
>>> can't allocate a secondary bus number for it (we're not smart enough
>>> to take advantage of the empty bus 01).
>>>
>>> On the failing box, it's different because we *do* have unused bus
>>> number space, and we do actually reconfigure the bridge to use it.
>>> It's just that the FC adapter doesn't respond when we use the new bus
>>> number for it.
>>>
>>> You might be able to do something similar on the test box by:
>>>
>>>   - Keeping your root->secondary.end = 02 patch, so you still have [bus 00-02].
>>>   - Ignoring bridges 00:1c.0 and 00:1c.4.  I would just test for those
>>> devfns in pci_scan_device() and when you see them, return NULL instead
>>> of trying to read the vendor ID.
>>>
>>> Then 00:1c.5 is probably configured by the BIOS for [bus 03], but
>>> that's outside the root bridge range, so we should reconfigure it to
>>> use [bus 01].  Then we should scan behind it, and we'll probably
>>> discover the NIC that was previously at 03:00.0.  The device *should*
>>> just work at the new bus number, since it probably doesn't have the
>>> same bug the FC adapter does.
>>
>> Thanks for the explanation.  I tried to ignore the two bridges but the
>> machine stopped with the "reconfiguring" message.
>>
>> Anyway, if I understood you correctly with the backup FC adapter I have
>> good chances, because there is the needed unused bus number space and I
>> don't have to ignore bridges.  I will test in a few hours and report.
>
> It turned out that there is no unused bus number space on the test
> machine:
>
> -[0000:00]-+-01.0-[01-02]--+-0d.0-[02]--
>            |               +-0e.0  Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
>            |               \-0e.1  Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
>            +-02.0  Broadcom BCM5785 [HT1000] Legacy South Bridge
>            +-02.1  Broadcom BCM5785 [HT1000] IDE
>            +-02.2  Broadcom BCM5785 [HT1000] LPC
>            +-03.0  Broadcom BCM5785 [HT1000] USB
>            +-03.1  Broadcom BCM5785 [HT1000] USB
>            +-03.2  Broadcom BCM5785 [HT1000] USB
>            +-06.0-[03]--
>            +-07.0-[04]--+-04.0  Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet
>            |            \-04.1  Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet
>            +-08.0-[05]--
>            +-09.0-[06]--
>            +-0a.0-[07]----00.0  LSI Logic / Symbios Logic FC949ES Fibre Channel Adapter
>            +-0b.0-[08]--
>            +-0c.0  Advanced Micro Devices, Inc. [AMD/ATI] ES1000
>            +-18.0  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
>            +-18.1  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Address Map
>            +-18.2  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] DRAM Controller
>            +-18.3  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
>            +-19.0  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
>            +-19.1  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Address Map
>            +-19.2  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] DRAM Controller
>            \-19.3  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
>
> So, I used the following changes to create the space and limit the root
> bridge window:
>
> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
> index e6ae603..a41066b 100644
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -556,6 +556,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
>  	strcpy(acpi_device_name(device), ACPI_PCI_ROOT_DEVICE_NAME);
>  	strcpy(acpi_device_class(device), ACPI_PCI_ROOT_CLASS);
>  	device->driver_data = root;
> +	root->secondary.end = 0x06;
>  
>  	pr_info(PREFIX "%s [%s] (domain %04x %pR)\n",
>  	       acpi_device_name(device), acpi_device_bid(device),
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index e3cf8a2..d8dc02d 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -1323,6 +1323,9 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
>  	struct pci_dev *dev;
>  	u32 l;
>  
> +	if (devfn == PCI_DEVFN(0x08, 0) || devfn == PCI_DEVFN(0x09,0))
> +		return NULL;
> +
>  	if (!pci_bus_read_dev_vendor_id(bus, devfn, &l, 60*1000))
>  		return NULL;
>  
> And to me it looks as if that is the situation, I need for testing
> Yinghai's patches; the FC Adapter disappeared similar to that on the
> other machine (find the dmesg output at the end):
>
> -[0000:00]-+-01.0-[01-02]--+-0d.0-[02]--
>            |               +-0e.0  Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
>            |               \-0e.1  Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
>            +-02.0  Broadcom BCM5785 [HT1000] Legacy South Bridge
>            +-02.1  Broadcom BCM5785 [HT1000] IDE
>            +-02.2  Broadcom BCM5785 [HT1000] LPC
>            +-03.0  Broadcom BCM5785 [HT1000] USB
>            +-03.1  Broadcom BCM5785 [HT1000] USB
>            +-03.2  Broadcom BCM5785 [HT1000] USB
>            +-06.0-[03]--
>            +-07.0-[04]--+-04.0  Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet
>            |            \-04.1  Broadcom Corporation NetXtreme BCM5780 Gigabit Ethernet
>            +-0a.0-[05]--
>            +-0b.0-[06]--
>            +-0c.0  Advanced Micro Devices, Inc. [AMD/ATI] ES1000
>            +-18.0  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
>            +-18.1  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Address Map
>            +-18.2  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] DRAM Controller
>            +-18.3  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
>            +-19.0  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
>            +-19.1  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Address Map
>            +-19.2  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] DRAM Controller
>            \-19.3  Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
>
>
> So, I wanted to test the patches but unfortunately, the patches do not
> apply cleanly; git am refuses to take the files (unknown format) and git
> apply gives errors. I tried to apply them against Linus' current, v3.16,
> v3.15 and v3.14.  What am I probably doing wrong, Yinghai?

Yinghai,

I now applied your patches and tested them:

echo 1 > /sys/bus/pci/devices/0000\:00\:0a.0/pcie_link_disable
echo 0 > /sys/bus/pci/devices/0000\:00\:0a.0/pcie_link_disable

The FC Adapter did not show up, this is the dmesg output (I also tested
with 0b.0):

[  143.294168] pcieport 0000:00:0a.0: pcie_link_disable_set: lnk_ctrl = 18
[  148.284456] pcieport 0000:00:0a.0: pcie_link_disable_set: lnk_ctrl = 8
[  304.065942] pcieport 0000:00:0b.0: pcie_link_disable_set: lnk_ctrl = 18
[  309.035278] pcieport 0000:00:0b.0: pcie_link_disable_set: lnk_ctrl = 8

So, probably my test environment does not work as expected and I have to
test it on the failing machine.

I hope I applied the patches correctly, I attach them to this mail.

Dirk
From b57b339bf10237e234c491c58f6dd0a1f575c34e Mon Sep 17 00:00:00 2001
From: Dirk Gouders <dirk@gouders.net>
Date: Fri, 12 Sep 2014 21:16:06 +0200
Subject: [PATCH 1/4] PCI: Add generic pcie_link_disable

Remove not needed return value checking that Linus pointed out before.

Will use it from /sys/.../pcie/link_disable

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/Makefile    |  2 +-
 drivers/pci/pcie-link.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h     |  2 ++
 3 files changed, 45 insertions(+), 1 deletion(-)
 create mode 100644 drivers/pci/pcie-link.c

[BUG] Bisected Problem with LSI PCI FC Adapter

Commit Message

Comments

Patch