| Message ID | 20180529160155.1738-1-mika.westerberg@linux.intel.com |
|---|---|
| State | Accepted |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | PCI: Fixes and cleanups for native PCIe, SHPC and ACPI hotplug | expand |
On Tue, 2018-05-29 at 19:01 +0300, Mika Westerberg wrote: > When a system is using native PCIe hotplug for Thunderbolt it will be > only present in the system when there is a device connected. This > pretty > much follows the BIOS assisted hotplug behaviour. > > Thunderbolt host router integrated PCIe switch has two additional PCIe > downstream bridges that lead to NHI (Thunderbolt host controller) and > xHCI > (USB 3 host controller) respectively. These downstream bridges are not > marked being hotplug capable. Reason for that is to preserve > resources. > Otherwise the OS would distribute remaining resources between all > downstream bridges making these two bridges consume precious resources > of the actual hotplug bridges. > > Now, because these two bridges are not marked being hotplug capable > the OS > will not enable hotplug interrupt for them either and will not receive > interrupt when devices behind them are hot-added. Solution to this is > that the BIOS sends ACPI Notify() to the root port let the OS know it > needs to rescan for added and/or removed devices. > > Here is how the mechanism is supposed to work when a Thunderbolt > endpoint is connected to one of the ports. In case of a standard USB-C > device only the xHCI is hot-added otherwise steps are the same. > > 1. Initially there is only the PCIe root port that is controlled by > the pciehp driver > > 00:1b.0 (Hotplug+) -- > > 2. Then we get native PCIe hotplug interrupt and once it is handled > the > topology looks as following > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > +- 02:01.0 (HotPlug+) > \- 02:02.0 -- > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > they don't have anything behind them currently. Bridge 02:01.0 is > hotplug capable and used for extending the topology. At this point > the required PCIe devices are enabled and ACPI Notify() is sent to > the root port. The resulting topology is expected to look like > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host > controller > +- 02:01.0 (HotPlug+) > \- 02:02.0 -- xHCI host controller > > However, the current ACPI hotplug implementation scans the whole > 00:1b.0 > hotplug slot and everything behind it regardless whether native PCIe > is > used or not, and it expects that the BIOS has configured bridge > resources upfront. If that's not the case it assigns resources using > minimal allocation (everything currently found just barely fit) > preventing future extension. In addition to that, if there is another > native PCIe hotplug going on we may find the new PCIe switch only > partially ready (all links are not fully trained yet) confusing pciehp > when it finally starts to enumerate for new devices. > > To make this work better with the native PCIe (pciehp) and standard > PCI > (shpchp) hotplug drivers, we let them handle all slot management and > resource allocation for hotplug bridges and restrict ACPI hotplug to > non-hotplug bridges. > Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com> > Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > --- > drivers/pci/hotplug/acpiphp_glue.c | 75 +++++++++++++++++++++++---- > --- > 1 file changed, 58 insertions(+), 17 deletions(-) > > diff --git a/drivers/pci/hotplug/acpiphp_glue.c > b/drivers/pci/hotplug/acpiphp_glue.c > index 318b6a6f6341..e2bcd9fc3fd2 100644 > --- a/drivers/pci/hotplug/acpiphp_glue.c > +++ b/drivers/pci/hotplug/acpiphp_glue.c > @@ -287,11 +287,12 @@ static acpi_status > acpiphp_add_context(acpi_handle handle, u32 lvl, void *data, > /* > * Expose slots to user space for functions that have _EJ0 or > _RMV or > * are located in dock stations. Do not expose them for > devices handled > - * by the native PCIe hotplug (PCIeHP), becuase that code is > supposed to > - * expose slots to user space in those cases. > + * by the native PCIe hotplug (PCIeHP) or standard PCI > hotplug > + * (SHPCHP), because that code is supposed to expose slots to > user > + * space in those cases. > */ > if ((acpi_pci_check_ejectable(pbus, handle) || > is_dock_device(adev)) > - && !(pdev && pdev->is_hotplug_bridge && > pciehp_is_native(pdev))) { > + && !(pdev && hotplug_is_native(pdev))) { > unsigned long long sun; > int retval; > > @@ -430,6 +431,29 @@ static int acpiphp_rescan_slot(struct > acpiphp_slot *slot) > return pci_scan_slot(slot->bus, PCI_DEVFN(slot->device, 0)); > } > > +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) > +{ > + struct pci_bus *bus = bridge->subordinate; > + struct pci_dev *dev; > + int max; > + > + if (!bus) > + return; > + > + max = bus->busn_res.start; > + /* Scan already configured non-hotplug bridges */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) > + max = pci_scan_bridge(bus, dev, max, 0); > + } > + > + /* Scan non-hotplug bridges that need to be reconfigured */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) > + max = pci_scan_bridge(bus, dev, max, 1); > + } > +} > + > /** > * enable_slot - enable, configure a slot > * @slot: slot to be enabled > @@ -442,25 +466,42 @@ static void enable_slot(struct acpiphp_slot > *slot) > struct pci_dev *dev; > struct pci_bus *bus = slot->bus; > struct acpiphp_func *func; > - int max, pass; > - LIST_HEAD(add_list); > > - acpiphp_rescan_slot(slot); > - max = acpiphp_max_busnr(bus); > - for (pass = 0; pass < 2; pass++) { > + if (bus->self && hotplug_is_native(bus->self)) { > + /* > + * If native hotplug is used, it will take care of > hotplug > + * slot management and resource allocation for > hotplug > + * bridges. However, ACPI hotplug may still be used > for > + * non-hotplug bridges to bring in additional devices > such > + * as Thunderbolt host controller. > + */ > for_each_pci_bridge(dev, bus) { > - if (PCI_SLOT(dev->devfn) != slot->device) > - continue; > - > - max = pci_scan_bridge(bus, dev, max, pass); > - if (pass && dev->subordinate) { > - check_hotplug_bridge(slot, dev); > - pcibios_resource_survey_bus(dev- > >subordinate); > - __pci_bus_size_bridges(dev- > >subordinate, &add_list); > + if (PCI_SLOT(dev->devfn) == slot->device) > + acpiphp_native_scan_bridge(dev); > + } > + pci_assign_unassigned_bridge_resources(bus->self); > + } else { > + LIST_HEAD(add_list); > + int max, pass; > + > + acpiphp_rescan_slot(slot); > + max = acpiphp_max_busnr(bus); > + for (pass = 0; pass < 2; pass++) { > + for_each_pci_bridge(dev, bus) { > + if (PCI_SLOT(dev->devfn) != slot- > >device) > + continue; > + > + max = pci_scan_bridge(bus, dev, max, > pass); > + if (pass && dev->subordinate) { > + check_hotplug_bridge(slot, > dev); > + pcibios_resource_survey_bus(d > ev->subordinate); > + __pci_bus_size_bridges(dev- > >subordinate, > + &add_l > ist); > + } > } > } > + __pci_bus_assign_resources(bus, &add_list, NULL); > } > - __pci_bus_assign_resources(bus, &add_list, NULL); > > acpiphp_sanitize_bus(bus); > pcie_bus_configure_settings(bus);
On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > When a system is using native PCIe hotplug for Thunderbolt it will be > only present in the system when there is a device connected. This pretty > much follows the BIOS assisted hotplug behaviour. > > Thunderbolt host router integrated PCIe switch has two additional PCIe > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > (USB 3 host controller) respectively. These downstream bridges are not > marked being hotplug capable. Reason for that is to preserve resources. > Otherwise the OS would distribute remaining resources between all > downstream bridges making these two bridges consume precious resources > of the actual hotplug bridges. > > Now, because these two bridges are not marked being hotplug capable the OS > will not enable hotplug interrupt for them either and will not receive > interrupt when devices behind them are hot-added. Solution to this is > that the BIOS sends ACPI Notify() to the root port let the OS know it > needs to rescan for added and/or removed devices. > > Here is how the mechanism is supposed to work when a Thunderbolt > endpoint is connected to one of the ports. In case of a standard USB-C > device only the xHCI is hot-added otherwise steps are the same. > > 1. Initially there is only the PCIe root port that is controlled by > the pciehp driver > > 00:1b.0 (Hotplug+) -- > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > topology looks as following > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > +- 02:01.0 (HotPlug+) > \- 02:02.0 -- Help me out here. In PCIe terms, I assume we basically hot-added this switch: 01:00.0 Switch Upstream port 02:00.0 Switch Downstream Port 02:01.0 Switch Downstream Port 02:02.0 Switch Downstream Port Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus number space to all the downstream ports, but there are currently no devices below any of them. Well, duh, that's exactly what you said below: > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > they don't have anything behind them currently. Bridge 02:01.0 is > hotplug capable and used for extending the topology. At this point > the required PCIe devices are enabled and ACPI Notify() is sent to > the root port. The resulting topology is expected to look like > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > +- 02:01.0 (HotPlug+) > \- 02:02.0 -- xHCI host controller > I guess this means we should ultimately end up with these new devices: 03:00.0 Thunderbolt host controller 39:00.0 xHCI host controller (Can you send "lspci -vv" output so I can see the names, device types, etc? I'm still trying to map the Thunderbolt "host router", NHI, etc terminology into PCIe concepts.) > However, the current ACPI hotplug implementation scans the whole 00:1b.0 > hotplug slot and everything behind it regardless whether native PCIe is > used or not, and it expects that the BIOS has configured bridge > resources upfront. If that's not the case it assigns resources using > minimal allocation (everything currently found just barely fit) > preventing future extension. I assume we got a Bus Check notification to the root port. The spec says OSPM should re-enumerate starting from the root port (I'm looking at ACPI 6.2, sec 5.6.6). It would sure be nice if the spec somehow indicated that this re-enumeration should skip parts of the tree. I'm not really sure how we were supposed to infer this coordination requirement from the existing specs. It does suggest that the Notify should be sent as close as possible to the point where it's required, which would be 02:00.0 and 02:02.0 here, but since that whole switch was hot-added by pciehp, the firmware doesn't necessarily know anything about it. > In addition to that, if there is another > native PCIe hotplug going on we may find the new PCIe switch only > partially ready (all links are not fully trained yet) confusing pciehp > when it finally starts to enumerate for new devices. > > To make this work better with the native PCIe (pciehp) and standard PCI > (shpchp) hotplug drivers, we let them handle all slot management and > resource allocation for hotplug bridges and restrict ACPI hotplug to > non-hotplug bridges. > Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com> > Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > --- > drivers/pci/hotplug/acpiphp_glue.c | 75 +++++++++++++++++++++++------- > 1 file changed, 58 insertions(+), 17 deletions(-) > > diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c > index 318b6a6f6341..e2bcd9fc3fd2 100644 > --- a/drivers/pci/hotplug/acpiphp_glue.c > +++ b/drivers/pci/hotplug/acpiphp_glue.c > @@ -287,11 +287,12 @@ static acpi_status acpiphp_add_context(acpi_handle handle, u32 lvl, void *data, > /* > * Expose slots to user space for functions that have _EJ0 or _RMV or > * are located in dock stations. Do not expose them for devices handled > - * by the native PCIe hotplug (PCIeHP), becuase that code is supposed to > - * expose slots to user space in those cases. > + * by the native PCIe hotplug (PCIeHP) or standard PCI hotplug > + * (SHPCHP), because that code is supposed to expose slots to user > + * space in those cases. > */ > if ((acpi_pci_check_ejectable(pbus, handle) || is_dock_device(adev)) > - && !(pdev && pdev->is_hotplug_bridge && pciehp_is_native(pdev))) { > + && !(pdev && hotplug_is_native(pdev))) { > unsigned long long sun; > int retval; > > @@ -430,6 +431,29 @@ static int acpiphp_rescan_slot(struct acpiphp_slot *slot) > return pci_scan_slot(slot->bus, PCI_DEVFN(slot->device, 0)); > } > > +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) > +{ > + struct pci_bus *bus = bridge->subordinate; > + struct pci_dev *dev; > + int max; > + > + if (!bus) > + return; > + > + max = bus->busn_res.start; > + /* Scan already configured non-hotplug bridges */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) > + max = pci_scan_bridge(bus, dev, max, 0); > + } > + > + /* Scan non-hotplug bridges that need to be reconfigured */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) > + max = pci_scan_bridge(bus, dev, max, 1); > + } > +} > + > /** > * enable_slot - enable, configure a slot > * @slot: slot to be enabled > @@ -442,25 +466,42 @@ static void enable_slot(struct acpiphp_slot *slot) > struct pci_dev *dev; > struct pci_bus *bus = slot->bus; > struct acpiphp_func *func; > - int max, pass; > - LIST_HEAD(add_list); > > - acpiphp_rescan_slot(slot); > - max = acpiphp_max_busnr(bus); > - for (pass = 0; pass < 2; pass++) { > + if (bus->self && hotplug_is_native(bus->self)) { > + /* > + * If native hotplug is used, it will take care of hotplug > + * slot management and resource allocation for hotplug > + * bridges. However, ACPI hotplug may still be used for > + * non-hotplug bridges to bring in additional devices such > + * as Thunderbolt host controller. > + */ > for_each_pci_bridge(dev, bus) { > - if (PCI_SLOT(dev->devfn) != slot->device) > - continue; > - > - max = pci_scan_bridge(bus, dev, max, pass); > - if (pass && dev->subordinate) { > - check_hotplug_bridge(slot, dev); > - pcibios_resource_survey_bus(dev->subordinate); > - __pci_bus_size_bridges(dev->subordinate, &add_list); > + if (PCI_SLOT(dev->devfn) == slot->device) > + acpiphp_native_scan_bridge(dev); > + } > + pci_assign_unassigned_bridge_resources(bus->self); > + } else { > + LIST_HEAD(add_list); > + int max, pass; > + > + acpiphp_rescan_slot(slot); > + max = acpiphp_max_busnr(bus); > + for (pass = 0; pass < 2; pass++) { > + for_each_pci_bridge(dev, bus) { > + if (PCI_SLOT(dev->devfn) != slot->device) > + continue; > + > + max = pci_scan_bridge(bus, dev, max, pass); > + if (pass && dev->subordinate) { > + check_hotplug_bridge(slot, dev); > + pcibios_resource_survey_bus(dev->subordinate); > + __pci_bus_size_bridges(dev->subordinate, > + &add_list); > + } > } > } > + __pci_bus_assign_resources(bus, &add_list, NULL); > } > - __pci_bus_assign_resources(bus, &add_list, NULL); > > acpiphp_sanitize_bus(bus); > pcie_bus_configure_settings(bus); > -- > 2.17.0 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-acpi" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jun 01, 2018 at 09:11:18AM -0500, Bjorn Helgaas wrote: > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > When a system is using native PCIe hotplug for Thunderbolt it will be > > only present in the system when there is a device connected. This pretty > > much follows the BIOS assisted hotplug behaviour. > > > > Thunderbolt host router integrated PCIe switch has two additional PCIe > > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > > (USB 3 host controller) respectively. These downstream bridges are not > > marked being hotplug capable. Reason for that is to preserve resources. > > Otherwise the OS would distribute remaining resources between all > > downstream bridges making these two bridges consume precious resources > > of the actual hotplug bridges. > > > > Now, because these two bridges are not marked being hotplug capable the OS > > will not enable hotplug interrupt for them either and will not receive > > interrupt when devices behind them are hot-added. Solution to this is > > that the BIOS sends ACPI Notify() to the root port let the OS know it > > needs to rescan for added and/or removed devices. > > > > Here is how the mechanism is supposed to work when a Thunderbolt > > endpoint is connected to one of the ports. In case of a standard USB-C > > device only the xHCI is hot-added otherwise steps are the same. > > > > 1. Initially there is only the PCIe root port that is controlled by > > the pciehp driver > > > > 00:1b.0 (Hotplug+) -- > > > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > > topology looks as following > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > > +- 02:01.0 (HotPlug+) > > \- 02:02.0 -- > > Help me out here. In PCIe terms, I assume we basically hot-added this > switch: > > 01:00.0 Switch Upstream port > 02:00.0 Switch Downstream Port > 02:01.0 Switch Downstream Port > 02:02.0 Switch Downstream Port > > Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus > number space to all the downstream ports, but there are currently no > devices below any of them. Well, duh, that's exactly what you said > below: > > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > > they don't have anything behind them currently. Bridge 02:01.0 is > > hotplug capable and used for extending the topology. At this point > > the required PCIe devices are enabled and ACPI Notify() is sent to > > the root port. The resulting topology is expected to look like > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > > +- 02:01.0 (HotPlug+) > > \- 02:02.0 -- xHCI host controller > > > > I guess this means we should ultimately end up with these new devices: > > 03:00.0 Thunderbolt host controller > 39:00.0 xHCI host controller That's right. > (Can you send "lspci -vv" output so I can see the names, device types, > etc? I'm still trying to map the Thunderbolt "host router", NHI, etc > terminology into PCIe concepts.) The full lspci -vv is here: https://bugzilla.kernel.org/attachment.cgi?id=275703 Just to clarify: Thunderbolt host router = The whole Thunderbolt add-in-card, including PCIe switch, Thunderbolt host controller (NHI) and USB 3.0 host controller (xHCI). > > However, the current ACPI hotplug implementation scans the whole 00:1b.0 > > hotplug slot and everything behind it regardless whether native PCIe is > > used or not, and it expects that the BIOS has configured bridge > > resources upfront. If that's not the case it assigns resources using > > minimal allocation (everything currently found just barely fit) > > preventing future extension. > > I assume we got a Bus Check notification to the root port. The spec > says OSPM should re-enumerate starting from the root port (I'm looking > at ACPI 6.2, sec 5.6.6). It would sure be nice if the spec somehow > indicated that this re-enumeration should skip parts of the tree. > > I'm not really sure how we were supposed to infer this coordination > requirement from the existing specs. It does suggest that the Notify > should be sent as close as possible to the point where it's required, > which would be 02:00.0 and 02:02.0 here, but since that whole switch > was hot-added by pciehp, the firmware doesn't necessarily know > anything about it. Right.
On Fri, Jun 01, 2018 at 05:24:04PM +0300, Mika Westerberg wrote: > On Fri, Jun 01, 2018 at 09:11:18AM -0500, Bjorn Helgaas wrote: > > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > > When a system is using native PCIe hotplug for Thunderbolt it will be > > > only present in the system when there is a device connected. This pretty > > > much follows the BIOS assisted hotplug behaviour. > > > > > > Thunderbolt host router integrated PCIe switch has two additional PCIe > > > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > > > (USB 3 host controller) respectively. These downstream bridges are not > > > marked being hotplug capable. Reason for that is to preserve resources. > > > Otherwise the OS would distribute remaining resources between all > > > downstream bridges making these two bridges consume precious resources > > > of the actual hotplug bridges. > > > > > > Now, because these two bridges are not marked being hotplug capable the OS > > > will not enable hotplug interrupt for them either and will not receive > > > interrupt when devices behind them are hot-added. Solution to this is > > > that the BIOS sends ACPI Notify() to the root port let the OS know it > > > needs to rescan for added and/or removed devices. > > > > > > Here is how the mechanism is supposed to work when a Thunderbolt > > > endpoint is connected to one of the ports. In case of a standard USB-C > > > device only the xHCI is hot-added otherwise steps are the same. > > > > > > 1. Initially there is only the PCIe root port that is controlled by > > > the pciehp driver > > > > > > 00:1b.0 (Hotplug+) -- > > > > > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > > > topology looks as following > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > > > +- 02:01.0 (HotPlug+) > > > \- 02:02.0 -- > > > > Help me out here. In PCIe terms, I assume we basically hot-added this > > switch: > > > > 01:00.0 Switch Upstream port > > 02:00.0 Switch Downstream Port > > 02:01.0 Switch Downstream Port > > 02:02.0 Switch Downstream Port > > > > Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus > > number space to all the downstream ports, but there are currently no > > devices below any of them. Well, duh, that's exactly what you said > > below: > > > > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > > > they don't have anything behind them currently. Bridge 02:01.0 is > > > hotplug capable and used for extending the topology. At this point > > > the required PCIe devices are enabled and ACPI Notify() is sent to > > > the root port. The resulting topology is expected to look like > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > > > +- 02:01.0 (HotPlug+) > > > \- 02:02.0 -- xHCI host controller > > > > > > > I guess this means we should ultimately end up with these new devices: > > > > 03:00.0 Thunderbolt host controller > > 39:00.0 xHCI host controller > > That's right. > > > (Can you send "lspci -vv" output so I can see the names, device types, > > etc? I'm still trying to map the Thunderbolt "host router", NHI, etc > > terminology into PCIe concepts.) > > The full lspci -vv is here: > > https://bugzilla.kernel.org/attachment.cgi?id=275703 Thanks, that's quite an intimidating PCIe tree with several levels of Thunderbolt stuff. If you disconnect/reconnect the cable (or I guess the add-in card at the top level) closest to the root port, does this all work correctly? I assume the pciehp hotplug adds just the top-level switch (01:00.0), then an ACPI Notify() adds the NHI and xHCI and configures the tunnels, then another pciehp event adds the next-level switch, and another Notify() sets up more tunnels, etc, etc? > Just to clarify: > > Thunderbolt host router = The whole Thunderbolt add-in-card, including > PCIe switch, Thunderbolt host controller > (NHI) and USB 3.0 host controller (xHCI). I assume the main reason for using ACPI hotplug here is because Linux doesn't know how to set up the Thunderbolt tunnels, so some sort of firmware has to do it? How does the BIOS figure out when to send the Notify()? If the host router is built into the motherboard, I can see how there might be some path for BIOS to notice a device being connected to the Thunderbolt host router, and then it could power up the host router (causing a pciehp hot-add), and then send the Notify(). But if this is actually a separate add-in card, does that mean the tunnel setup has to be done via the option ROM somehow? Or does the add-in card only work on systems that already have Thunderbolt support in their BIOS? If so, how does this work if the card is hot-added? Do we add the switch via pciehp, and something else in Linux tells ACPI to issue the Notify()? Bjorn
On Fri, Jun 01, 2018 at 01:41:09PM -0500, Bjorn Helgaas wrote: > On Fri, Jun 01, 2018 at 05:24:04PM +0300, Mika Westerberg wrote: > > On Fri, Jun 01, 2018 at 09:11:18AM -0500, Bjorn Helgaas wrote: > > > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > > > When a system is using native PCIe hotplug for Thunderbolt it will be > > > > only present in the system when there is a device connected. This pretty > > > > much follows the BIOS assisted hotplug behaviour. > > > > > > > > Thunderbolt host router integrated PCIe switch has two additional PCIe > > > > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > > > > (USB 3 host controller) respectively. These downstream bridges are not > > > > marked being hotplug capable. Reason for that is to preserve resources. > > > > Otherwise the OS would distribute remaining resources between all > > > > downstream bridges making these two bridges consume precious resources > > > > of the actual hotplug bridges. > > > > > > > > Now, because these two bridges are not marked being hotplug capable the OS > > > > will not enable hotplug interrupt for them either and will not receive > > > > interrupt when devices behind them are hot-added. Solution to this is > > > > that the BIOS sends ACPI Notify() to the root port let the OS know it > > > > needs to rescan for added and/or removed devices. > > > > > > > > Here is how the mechanism is supposed to work when a Thunderbolt > > > > endpoint is connected to one of the ports. In case of a standard USB-C > > > > device only the xHCI is hot-added otherwise steps are the same. > > > > > > > > 1. Initially there is only the PCIe root port that is controlled by > > > > the pciehp driver > > > > > > > > 00:1b.0 (Hotplug+) -- > > > > > > > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > > > > topology looks as following > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > > > > +- 02:01.0 (HotPlug+) > > > > \- 02:02.0 -- > > > > > > Help me out here. In PCIe terms, I assume we basically hot-added this > > > switch: > > > > > > 01:00.0 Switch Upstream port > > > 02:00.0 Switch Downstream Port > > > 02:01.0 Switch Downstream Port > > > 02:02.0 Switch Downstream Port > > > > > > Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus > > > number space to all the downstream ports, but there are currently no > > > devices below any of them. Well, duh, that's exactly what you said > > > below: > > > > > > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > > > > they don't have anything behind them currently. Bridge 02:01.0 is > > > > hotplug capable and used for extending the topology. At this point > > > > the required PCIe devices are enabled and ACPI Notify() is sent to > > > > the root port. The resulting topology is expected to look like > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > > > > +- 02:01.0 (HotPlug+) > > > > \- 02:02.0 -- xHCI host controller > > > > > > > > > > I guess this means we should ultimately end up with these new devices: > > > > > > 03:00.0 Thunderbolt host controller > > > 39:00.0 xHCI host controller > > > > That's right. > > > > > (Can you send "lspci -vv" output so I can see the names, device types, > > > etc? I'm still trying to map the Thunderbolt "host router", NHI, etc > > > terminology into PCIe concepts.) > > > > The full lspci -vv is here: > > > > https://bugzilla.kernel.org/attachment.cgi?id=275703 > > Thanks, that's quite an intimidating PCIe tree with several levels of > Thunderbolt stuff. > > If you disconnect/reconnect the cable (or I guess the add-in card at > the top level) closest to the root port, does this all work correctly? Yes it does. > I assume the pciehp hotplug adds just the top-level switch (01:00.0), > then an ACPI Notify() adds the NHI and xHCI and configures the > tunnels, then another pciehp event adds the next-level switch, and > another Notify() sets up more tunnels, etc, etc? It is the firmware running on the Thunderbolt host router that sets up the tunnels and triggers standard PCIe hotplug once it is done. Notify() is only used to bring in those two controllers to the first PCIe switch. Reason for using Notify() here is that then we don't need to mark the two downstream ports leading to xHCI and NHI to be hotplug ports and thus the OS does not spread the available bus space/resources to those ports. If you keep connecting more devices then standard PCIe hotplug is used and there will be no Notify(). > > Just to clarify: > > > > Thunderbolt host router = The whole Thunderbolt add-in-card, including > > PCIe switch, Thunderbolt host controller > > (NHI) and USB 3.0 host controller (xHCI). > > I assume the main reason for using ACPI hotplug here is because Linux > doesn't know how to set up the Thunderbolt tunnels, so some sort of > firmware has to do it? Firmware does it regardless of what OS is running (with the exception of Apple hardware, of course). Once it establishes a tunnel a standard PCIe hotplug event is triggered. ACPI hotplug is only used to bring in those two devices of the host router. You might wonder why they aren't all handled by the pciehp and the reason is that if you only connect USB-C device (not TBT) ACPI hotplug only finds xHCI since you don't need the NHI for that. > How does the BIOS figure out when to send the Notify()? If the host > router is built into the motherboard, I can see how there might be > some path for BIOS to notice a device being connected to the > Thunderbolt host router, and then it could power up the host router > (causing a pciehp hot-add), and then send the Notify(). There is a GPIO on the AIC that is wired to trigger ACPI GPE and the GPE handler does the Notify(). > But if this is actually a separate add-in card, does that mean the > tunnel setup has to be done via the option ROM somehow? It is done in the firmware running on the host router (AIC). > Or does the add-in card only work on systems that already have > Thunderbolt support in their BIOS? If so, how does this work if the > card is hot-added? Do we add the switch via pciehp, and something > else in Linux tells ACPI to issue the Notify()? The BIOS needs to have Thunderbolt support built in but I think that is pretty "generic" and that is one of the reasons the Notify() is send to the root port and not to the exact downstream ports where those two controllers (xHCI, NHI) are connected to. I don't know all the details but I think it works like that.
On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > When a system is using native PCIe hotplug for Thunderbolt it will be > only present in the system when there is a device connected. This pretty > much follows the BIOS assisted hotplug behaviour. > ... > +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) > +{ > + struct pci_bus *bus = bridge->subordinate; > + struct pci_dev *dev; > + int max; > + > + if (!bus) > + return; > + > + max = bus->busn_res.start; > + /* Scan already configured non-hotplug bridges */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) Here we test "dev->is_hotplug_bridge" and below we use "hotplug_is_native(bus->self)". Is the difference significant, or should we use hotplug_is_native() here as well? If we do need to use is_hotplug_bridge here, maybe a comment about the difference would be helpful. > + max = pci_scan_bridge(bus, dev, max, 0); > + } > + > + /* Scan non-hotplug bridges that need to be reconfigured */ > + for_each_pci_bridge(dev, bus) { > + if (!dev->is_hotplug_bridge) > + max = pci_scan_bridge(bus, dev, max, 1); > + } > +} > + > /** > * enable_slot - enable, configure a slot > * @slot: slot to be enabled > @@ -442,25 +466,42 @@ static void enable_slot(struct acpiphp_slot *slot) > struct pci_dev *dev; > struct pci_bus *bus = slot->bus; > struct acpiphp_func *func; > - int max, pass; > - LIST_HEAD(add_list); > > - acpiphp_rescan_slot(slot); > - max = acpiphp_max_busnr(bus); > - for (pass = 0; pass < 2; pass++) { > + if (bus->self && hotplug_is_native(bus->self)) { > + /* > + * If native hotplug is used, it will take care of hotplug > + * slot management and resource allocation for hotplug > + * bridges. However, ACPI hotplug may still be used for > + * non-hotplug bridges to bring in additional devices such > + * as Thunderbolt host controller. > + */ > for_each_pci_bridge(dev, bus) { > - if (PCI_SLOT(dev->devfn) != slot->device) > - continue; > - > - max = pci_scan_bridge(bus, dev, max, pass); > - if (pass && dev->subordinate) { > - check_hotplug_bridge(slot, dev); > - pcibios_resource_survey_bus(dev->subordinate); > - __pci_bus_size_bridges(dev->subordinate, &add_list); > + if (PCI_SLOT(dev->devfn) == slot->device) > + acpiphp_native_scan_bridge(dev); > + } > + pci_assign_unassigned_bridge_resources(bus->self); > + } else {
On Fri, Jun 01, 2018 at 04:35:05PM -0500, Bjorn Helgaas wrote: > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > When a system is using native PCIe hotplug for Thunderbolt it will be > > only present in the system when there is a device connected. This pretty > > much follows the BIOS assisted hotplug behaviour. > > ... > > > +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) > > +{ > > + struct pci_bus *bus = bridge->subordinate; > > + struct pci_dev *dev; > > + int max; > > + > > + if (!bus) > > + return; > > + > > + max = bus->busn_res.start; > > + /* Scan already configured non-hotplug bridges */ > > + for_each_pci_bridge(dev, bus) { > > + if (!dev->is_hotplug_bridge) > > Here we test "dev->is_hotplug_bridge" and below we use > "hotplug_is_native(bus->self)". Is the difference significant, or > should we use hotplug_is_native() here as well? > > If we do need to use is_hotplug_bridge here, maybe a comment about the > difference would be helpful. I think here hotplug_is_native() would work as well. The idea is that we need to avoid scanning bridges that are handled by pciehp (or shpchp).
On Fri, Jun 01, 2018 at 10:19:05PM +0300, Mika Westerberg wrote: > On Fri, Jun 01, 2018 at 01:41:09PM -0500, Bjorn Helgaas wrote: > > On Fri, Jun 01, 2018 at 05:24:04PM +0300, Mika Westerberg wrote: > > > On Fri, Jun 01, 2018 at 09:11:18AM -0500, Bjorn Helgaas wrote: > > > > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > > > > When a system is using native PCIe hotplug for Thunderbolt it will be > > > > > only present in the system when there is a device connected. This pretty > > > > > much follows the BIOS assisted hotplug behaviour. > > > > > > > > > > Thunderbolt host router integrated PCIe switch has two additional PCIe > > > > > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > > > > > (USB 3 host controller) respectively. These downstream bridges are not > > > > > marked being hotplug capable. Reason for that is to preserve resources. > > > > > Otherwise the OS would distribute remaining resources between all > > > > > downstream bridges making these two bridges consume precious resources > > > > > of the actual hotplug bridges. > > > > > > > > > > Now, because these two bridges are not marked being hotplug capable the OS > > > > > will not enable hotplug interrupt for them either and will not receive > > > > > interrupt when devices behind them are hot-added. Solution to this is > > > > > that the BIOS sends ACPI Notify() to the root port let the OS know it > > > > > needs to rescan for added and/or removed devices. > > > > > > > > > > Here is how the mechanism is supposed to work when a Thunderbolt > > > > > endpoint is connected to one of the ports. In case of a standard USB-C > > > > > device only the xHCI is hot-added otherwise steps are the same. > > > > > > > > > > 1. Initially there is only the PCIe root port that is controlled by > > > > > the pciehp driver > > > > > > > > > > 00:1b.0 (Hotplug+) -- > > > > > > > > > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > > > > > topology looks as following > > > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > > > > > +- 02:01.0 (HotPlug+) > > > > > \- 02:02.0 -- > > > > > > > > Help me out here. In PCIe terms, I assume we basically hot-added this > > > > switch: > > > > > > > > 01:00.0 Switch Upstream port > > > > 02:00.0 Switch Downstream Port > > > > 02:01.0 Switch Downstream Port > > > > 02:02.0 Switch Downstream Port > > > > > > > > Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus > > > > number space to all the downstream ports, but there are currently no > > > > devices below any of them. Well, duh, that's exactly what you said > > > > below: > > > > > > > > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > > > > > they don't have anything behind them currently. Bridge 02:01.0 is > > > > > hotplug capable and used for extending the topology. At this point > > > > > the required PCIe devices are enabled and ACPI Notify() is sent to > > > > > the root port. The resulting topology is expected to look like > > > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > > > > > +- 02:01.0 (HotPlug+) > > > > > \- 02:02.0 -- xHCI host controller > > > > > > > > > > > > > I guess this means we should ultimately end up with these new devices: > > > > > > > > 03:00.0 Thunderbolt host controller > > > > 39:00.0 xHCI host controller > > > > > > That's right. And after the host router firmware sets up the tunnels, is there a step 4 where we get another pciehp event from 02:01.0 and we enumerate the Thunderbolt switch (which I assume looks like a regular PCIe switch to the PCI core)? Does the add-in card actually contain all the following devices (from the lspci you pointed me to)? 01:00.0 Switch Upstream Port to [bus 02-39] 02:00.0 Switch Downstream Port to [bus 03] (to NHI) 02:01.0 Switch Downstream Port to [bus 04-38] (to Thunderbolt switch) 02:02.0 Switch Downstream Port to [bus 39] (to xHCI) 03:00.0 Thunderbolt Host Controller (NHI) Endpoint 39:00.0 xHCI Endpoint 04:00.0 Switch Upstream Port to [bus 05-38] \ 05:01.0 Switch Downstream Port to [bus 06-09] | Thunderbolt Switch 05:04.0 Switch Downstream Port to [bus 0a-38] / That would correspond to Figure 1-1 here: https://developer.apple.com/library/content/documentation/HardwareDrivers/Conceptual/ThunderboltDevGuide/Basics/Basics.html except that the figure doesn't show the xHCI controller. > > > > (Can you send "lspci -vv" output so I can see the names, device types, > > > > etc? I'm still trying to map the Thunderbolt "host router", NHI, etc > > > > terminology into PCIe concepts.) > > > > > > The full lspci -vv is here: > > > > > > https://bugzilla.kernel.org/attachment.cgi?id=275703 > > > > Thanks, that's quite an intimidating PCIe tree with several levels of > > Thunderbolt stuff. > > > > If you disconnect/reconnect the cable (or I guess the add-in card at > > the top level) closest to the root port, does this all work correctly? > > Yes it does. I'm honestly amazed :) > > I assume the pciehp hotplug adds just the top-level switch (01:00.0), > > then an ACPI Notify() adds the NHI and xHCI and configures the > > tunnels, then another pciehp event adds the next-level switch, and > > another Notify() sets up more tunnels, etc, etc? > > It is the firmware running on the Thunderbolt host router that sets up > the tunnels and triggers standard PCIe hotplug once it is done. Notify() > is only used to bring in those two controllers to the first PCIe switch. > Reason for using Notify() here is that then we don't need to mark the > two downstream ports leading to xHCI and NHI to be hotplug ports and > thus the OS does not spread the available bus space/resources to those > ports. > > If you keep connecting more devices then standard PCIe hotplug is used > and there will be no Notify(). > > > > Just to clarify: > > > > > > Thunderbolt host router = The whole Thunderbolt add-in-card, including > > > PCIe switch, Thunderbolt host controller > > > (NHI) and USB 3.0 host controller (xHCI). > > > > I assume the main reason for using ACPI hotplug here is because Linux > > doesn't know how to set up the Thunderbolt tunnels, so some sort of > > firmware has to do it? > > Firmware does it regardless of what OS is running (with the exception of > Apple hardware, of course). Once it establishes a tunnel a standard PCIe > hotplug event is triggered. > > ACPI hotplug is only used to bring in those two devices of the host > router. You might wonder why they aren't all handled by the pciehp and > the reason is that if you only connect USB-C device (not TBT) ACPI > hotplug only finds xHCI since you don't need the NHI for that. > > > How does the BIOS figure out when to send the Notify()? If the host > > router is built into the motherboard, I can see how there might be > > some path for BIOS to notice a device being connected to the > > Thunderbolt host router, and then it could power up the host router > > (causing a pciehp hot-add), and then send the Notify(). > > There is a GPIO on the AIC that is wired to trigger ACPI GPE and the GPE > handler does the Notify(). So this GPIO must be part of the reason for the mysterious Thunderbolt header, i.e., https://superuser.com/questions/1024865/for-what-is-the-thunderbolt-aic-connector-used You mentioned two reasons for using the ACPI Notify(): 1) To avoid having the OS assign more resources than necessary to the bridges leading to NHI and xHCI. 2) To avoid adding NHI at all if we only need USB-C. They both seem sort of minor. If those devices were brought in via the original pciehp hot-add, we would only allocate the resources they need since they're both endpoints and we'd know exactly what they needed. An unused NHI would only consume 260K of MMIO space and one bus number (which I think we will always assign anyway because of "PCI: Take all bridges into account when calculating bus numbers for extension"). The requirement for the GPIO header and a separate cable to it is a huge hassle so it seems like there must be more to it than just those two things. But that's not really germane to this patch anyway because we have to support the hardware/firmware as it is, not as we might imagine things could be. > > But if this is actually a separate add-in card, does that mean the > > tunnel setup has to be done via the option ROM somehow? > > It is done in the firmware running on the host router (AIC). > > > Or does the add-in card only work on systems that already have > > Thunderbolt support in their BIOS? If so, how does this work if the > > card is hot-added? Do we add the switch via pciehp, and something > > else in Linux tells ACPI to issue the Notify()? > > The BIOS needs to have Thunderbolt support built in but I think that is > pretty "generic" and that is one of the reasons the Notify() is send to > the root port and not to the exact downstream ports where those two > controllers (xHCI, NHI) are connected to. I don't know all the details > but I think it works like that. Thanks for all this background. It really helps me put things together. Bjorn
On Fri, Jun 01, 2018 at 04:50:27PM -0500, Bjorn Helgaas wrote: > On Fri, Jun 01, 2018 at 10:19:05PM +0300, Mika Westerberg wrote: > > On Fri, Jun 01, 2018 at 01:41:09PM -0500, Bjorn Helgaas wrote: > > > On Fri, Jun 01, 2018 at 05:24:04PM +0300, Mika Westerberg wrote: > > > > On Fri, Jun 01, 2018 at 09:11:18AM -0500, Bjorn Helgaas wrote: > > > > > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > > > > > When a system is using native PCIe hotplug for Thunderbolt it will be > > > > > > only present in the system when there is a device connected. This pretty > > > > > > much follows the BIOS assisted hotplug behaviour. > > > > > > > > > > > > Thunderbolt host router integrated PCIe switch has two additional PCIe > > > > > > downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI > > > > > > (USB 3 host controller) respectively. These downstream bridges are not > > > > > > marked being hotplug capable. Reason for that is to preserve resources. > > > > > > Otherwise the OS would distribute remaining resources between all > > > > > > downstream bridges making these two bridges consume precious resources > > > > > > of the actual hotplug bridges. > > > > > > > > > > > > Now, because these two bridges are not marked being hotplug capable the OS > > > > > > will not enable hotplug interrupt for them either and will not receive > > > > > > interrupt when devices behind them are hot-added. Solution to this is > > > > > > that the BIOS sends ACPI Notify() to the root port let the OS know it > > > > > > needs to rescan for added and/or removed devices. > > > > > > > > > > > > Here is how the mechanism is supposed to work when a Thunderbolt > > > > > > endpoint is connected to one of the ports. In case of a standard USB-C > > > > > > device only the xHCI is hot-added otherwise steps are the same. > > > > > > > > > > > > 1. Initially there is only the PCIe root port that is controlled by > > > > > > the pciehp driver > > > > > > > > > > > > 00:1b.0 (Hotplug+) -- > > > > > > > > > > > > 2. Then we get native PCIe hotplug interrupt and once it is handled the > > > > > > topology looks as following > > > > > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- > > > > > > +- 02:01.0 (HotPlug+) > > > > > > \- 02:02.0 -- > > > > > > > > > > Help me out here. In PCIe terms, I assume we basically hot-added this > > > > > switch: > > > > > > > > > > 01:00.0 Switch Upstream port > > > > > 02:00.0 Switch Downstream Port > > > > > 02:01.0 Switch Downstream Port > > > > > 02:02.0 Switch Downstream Port > > > > > > > > > > Only 02:01.0 has PCI_EXP_SLTCAP_HPC set. We can assign secondary bus > > > > > number space to all the downstream ports, but there are currently no > > > > > devices below any of them. Well, duh, that's exactly what you said > > > > > below: > > > > > > > > > > > 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and > > > > > > they don't have anything behind them currently. Bridge 02:01.0 is > > > > > > hotplug capable and used for extending the topology. At this point > > > > > > the required PCIe devices are enabled and ACPI Notify() is sent to > > > > > > the root port. The resulting topology is expected to look like > > > > > > > > > > > > 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller > > > > > > +- 02:01.0 (HotPlug+) > > > > > > \- 02:02.0 -- xHCI host controller > > > > > > > > > > > > > > > > I guess this means we should ultimately end up with these new devices: > > > > > > > > > > 03:00.0 Thunderbolt host controller > > > > > 39:00.0 xHCI host controller > > > > > > > > That's right. > > And after the host router firmware sets up the tunnels, is there a > step 4 where we get another pciehp event from 02:01.0 and we enumerate > the Thunderbolt switch (which I assume looks like a regular PCIe > switch to the PCI core)? Yes there is. > Does the add-in card actually contain all > the following devices (from the lspci you pointed me to)? > > 01:00.0 Switch Upstream Port to [bus 02-39] > 02:00.0 Switch Downstream Port to [bus 03] (to NHI) > 02:01.0 Switch Downstream Port to [bus 04-38] (to Thunderbolt switch) > 02:02.0 Switch Downstream Port to [bus 39] (to xHCI) > > 03:00.0 Thunderbolt Host Controller (NHI) Endpoint > 39:00.0 xHCI Endpoint This belongs to the Thunderbolt host router (AIC, add-in-card). > 04:00.0 Switch Upstream Port to [bus 05-38] \ > 05:01.0 Switch Downstream Port to [bus 06-09] | Thunderbolt Switch > 05:04.0 Switch Downstream Port to [bus 0a-38] / > > That would correspond to Figure 1-1 here: https://developer.apple.com/library/content/documentation/HardwareDrivers/Conceptual/ThunderboltDevGuide/Basics/Basics.html > except that the figure doesn't show the xHCI controller. This one shows another Thunderbolt device connected to the host router via Thunderbolt cable (it also includes PCIe switch and PCIe endpoint and one PCIe downstream port where the topology is extended, 05:04). In case of Titan Ridge (the next Thunderbolt host router) there will be also xHCI. > > > > > (Can you send "lspci -vv" output so I can see the names, device types, > > > > > etc? I'm still trying to map the Thunderbolt "host router", NHI, etc > > > > > terminology into PCIe concepts.) > > > > > > > > The full lspci -vv is here: > > > > > > > > https://bugzilla.kernel.org/attachment.cgi?id=275703 > > > > > > Thanks, that's quite an intimidating PCIe tree with several levels of > > > Thunderbolt stuff. > > > > > > If you disconnect/reconnect the cable (or I guess the add-in card at > > > the top level) closest to the root port, does this all work correctly? > > > > Yes it does. > > I'm honestly amazed :) I've spent good part of January sitting in the lab testing this with various systems and it was the best solution so far :) > > > I assume the pciehp hotplug adds just the top-level switch (01:00.0), > > > then an ACPI Notify() adds the NHI and xHCI and configures the > > > tunnels, then another pciehp event adds the next-level switch, and > > > another Notify() sets up more tunnels, etc, etc? > > > > It is the firmware running on the Thunderbolt host router that sets up > > the tunnels and triggers standard PCIe hotplug once it is done. Notify() > > is only used to bring in those two controllers to the first PCIe switch. > > Reason for using Notify() here is that then we don't need to mark the > > two downstream ports leading to xHCI and NHI to be hotplug ports and > > thus the OS does not spread the available bus space/resources to those > > ports. > > > > If you keep connecting more devices then standard PCIe hotplug is used > > and there will be no Notify(). > > > > > > Just to clarify: > > > > > > > > Thunderbolt host router = The whole Thunderbolt add-in-card, including > > > > PCIe switch, Thunderbolt host controller > > > > (NHI) and USB 3.0 host controller (xHCI). > > > > > > I assume the main reason for using ACPI hotplug here is because Linux > > > doesn't know how to set up the Thunderbolt tunnels, so some sort of > > > firmware has to do it? > > > > Firmware does it regardless of what OS is running (with the exception of > > Apple hardware, of course). Once it establishes a tunnel a standard PCIe > > hotplug event is triggered. > > > > ACPI hotplug is only used to bring in those two devices of the host > > router. You might wonder why they aren't all handled by the pciehp and > > the reason is that if you only connect USB-C device (not TBT) ACPI > > hotplug only finds xHCI since you don't need the NHI for that. > > > > > How does the BIOS figure out when to send the Notify()? If the host > > > router is built into the motherboard, I can see how there might be > > > some path for BIOS to notice a device being connected to the > > > Thunderbolt host router, and then it could power up the host router > > > (causing a pciehp hot-add), and then send the Notify(). > > > > There is a GPIO on the AIC that is wired to trigger ACPI GPE and the GPE > > handler does the Notify(). > > So this GPIO must be part of the reason for the mysterious Thunderbolt > header, i.e., https://superuser.com/questions/1024865/for-what-is-the-thunderbolt-aic-connector-used > > You mentioned two reasons for using the ACPI Notify(): > > 1) To avoid having the OS assign more resources than necessary to the > bridges leading to NHI and xHCI. > > 2) To avoid adding NHI at all if we only need USB-C. > > They both seem sort of minor. If those devices were brought in via > the original pciehp hot-add, we would only allocate the resources they > need since they're both endpoints and we'd know exactly what they > needed. An unused NHI would only consume 260K of MMIO space and one > bus number (which I think we will always assign anyway because of > "PCI: Take all bridges into account when calculating bus numbers for > extension"). > > The requirement for the GPIO header and a separate cable to it is a > huge hassle so it seems like there must be more to it than just those > two things. But that's not really germane to this patch anyway > because we have to support the hardware/firmware as it is, not as we > might imagine things could be. Indeed. The matter of fact is that these systems are currently shipping and we need to deal with them. In case of laptops such as Dell, the Alpine Ridge chip (Thunderbolt host router) is soldered on the motherboard, though. GPIO connection is still required. > > > But if this is actually a separate add-in card, does that mean the > > > tunnel setup has to be done via the option ROM somehow? > > > > It is done in the firmware running on the host router (AIC). > > > > > Or does the add-in card only work on systems that already have > > > Thunderbolt support in their BIOS? If so, how does this work if the > > > card is hot-added? Do we add the switch via pciehp, and something > > > else in Linux tells ACPI to issue the Notify()? > > > > The BIOS needs to have Thunderbolt support built in but I think that is > > pretty "generic" and that is one of the reasons the Notify() is send to > > the root port and not to the exact downstream ports where those two > > controllers (xHCI, NHI) are connected to. I don't know all the details > > but I think it works like that. > > Thanks for all this background. It really helps me put things > together. No problem :)
On Sat, Jun 02, 2018 at 12:48:18AM +0300, Mika Westerberg wrote: > On Fri, Jun 01, 2018 at 04:35:05PM -0500, Bjorn Helgaas wrote: > > On Tue, May 29, 2018 at 07:01:55PM +0300, Mika Westerberg wrote: > > > When a system is using native PCIe hotplug for Thunderbolt it will be > > > only present in the system when there is a device connected. This pretty > > > much follows the BIOS assisted hotplug behaviour. > > > ... > > > > > +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) > > > +{ > > > + struct pci_bus *bus = bridge->subordinate; > > > + struct pci_dev *dev; > > > + int max; > > > + > > > + if (!bus) > > > + return; > > > + > > > + max = bus->busn_res.start; > > > + /* Scan already configured non-hotplug bridges */ > > > + for_each_pci_bridge(dev, bus) { > > > + if (!dev->is_hotplug_bridge) > > > > Here we test "dev->is_hotplug_bridge" and below we use > > "hotplug_is_native(bus->self)". Is the difference significant, or > > should we use hotplug_is_native() here as well? > > > > If we do need to use is_hotplug_bridge here, maybe a comment about the > > difference would be helpful. > > I think here hotplug_is_native() would work as well. The idea is > that we need to avoid scanning bridges that are handled by pciehp > (or shpchp). I made that change on my branch. That way we won't have to worry about the cases where we set is_hotplug_bridge based on some ACPI methods. I think those would be bugs because I think we *do* want acpiphp to scan them, and if we tested "!dev->is_hotplug_bridge", we would not scan them. Here's what's on my branch: commit eb4c82c49180cf46dd7f9ed15ea96d1da190fea6 Author: Mika Westerberg <mika.westerberg@linux.intel.com> Date: Tue May 29 19:01:55 2018 +0300 ACPI / hotplug / PCI: Don't scan bridges managed by native hotplug When acpiphp re-enumerates a PCI hierarchy because of an ACPI Notify() event, we should skip bridges managed by native hotplug (pciehp or shpchp). We don't want to scan below a native hotplug bridge until the hotplug controller generates a hot-add event. A typical scenario is a Root Port leading to a Thunderbolt host router that remains powered off until something is connected to it. See [1] for the lspci details. 1. Before something is connected, only the Root Port exists. It has PCI_EXP_SLTCAP_HPC set and pciehp is responsible for hotplug: 00:1b.0 Root Port (HotPlug+) 2. When a USB-C or Thunderbolt device is connected, the Switch in the Thunderbolt host router is powered up, the Root Port signals a hotplug add event and pciehp enumerates the Switch: 01:00.0 Switch Upstream Port to [bus 02-39] 02:00.0 Switch Downstream Port to [bus 03] (HotPlug-, to NHI) 02:01.0 Switch Downstream Port to [bus 04-38] (HotPlug+, to Thunderbolt connector) 02:02.0 Switch Downstream Port to [bus 39] (HotPlug-, to xHCI) The 02:00.0 and 02:02.0 Ports lead to Endpoints that are not powered up yet. The Ports have PCI_EXP_SLTCAP_HPC cleared, so pciehp doesn't handle hotplug for them and we assign minimal resources to them. The 02:01.0 Port has PCI_EXP_SLTCAP_HPC set, so pciehp handles native hotplug events for it. 3. The BIOS powers up the xHCI controller. If a Thunderbolt device was connected (not just a USB-C device), it also powers up the NHI. Then it sends an ACPI Notify() to the Root Port, and acpiphp enumerates the new device(s): 03:00.0 Thunderbolt Host Controller (NHI) Endpoint 39:00.0 xHCI Endpoint 4. If a Thunderbolt device was connected, the host router firmware uses the NHI to set up Thunderbolt tunnels and triggers a native hotplug event (via 02:01.0 in this example). Then pciehp enumerates the new Thunderbolt devices: 04:00.0 Switch Upstream Port to [bus 05-38] 05:01.0 Switch Downstream Port to [bus 06-09] (HotPlug-) 05:04.0 Switch Downstream Port to [bus 0a-38] (HotPlug+) In this example, 05:01.0 leads to another Switch and some NICs. This subtree is static, so 05:01.0 doesn't support hotplug and has PCI_EXP_SLTCAP_HPC cleared. In step 3, acpiphp previously enumerated everything below the Root Port, including things below the 02:01.0 Port. We don't want that because pciehp expects to manage hotplug below that Port, and firmware on the host router may be in the middle of configuring its Link so it may not be ready yet. To make this work better with the native PCIe (pciehp) and standard PCI (shpchp) hotplug drivers, we let them handle all slot management and resource allocation for hotplug bridges and restrict ACPI hotplug to non-hotplug bridges. [1] https://bugzilla.kernel.org/show_bug.cgi?id=199581#c5 Link: https://lkml.kernel.org/r/20180529160155.1738-1-mika.westerberg@linux.intel.com Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com> [bhelgaas: changelog, use hotplug_is_native() instead of dev->is_hotplug_bridge] Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c index b45b375c0e6c..bc9e19642567 100644 --- a/drivers/pci/hotplug/acpiphp_glue.c +++ b/drivers/pci/hotplug/acpiphp_glue.c @@ -287,11 +287,12 @@ static acpi_status acpiphp_add_context(acpi_handle handle, u32 lvl, void *data, /* * Expose slots to user space for functions that have _EJ0 or _RMV or * are located in dock stations. Do not expose them for devices handled - * by the native PCIe hotplug (PCIeHP), becuase that code is supposed to - * expose slots to user space in those cases. + * by the native PCIe hotplug (PCIeHP) or standard PCI hotplug + * (SHPCHP), because that code is supposed to expose slots to user + * space in those cases. */ if ((acpi_pci_check_ejectable(pbus, handle) || is_dock_device(adev)) - && !(pdev && pdev->is_hotplug_bridge && pciehp_is_native(pdev))) { + && !(pdev && hotplug_is_native(pdev))) { unsigned long long sun; int retval; @@ -430,6 +431,29 @@ static int acpiphp_rescan_slot(struct acpiphp_slot *slot) return pci_scan_slot(slot->bus, PCI_DEVFN(slot->device, 0)); } +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) +{ + struct pci_bus *bus = bridge->subordinate; + struct pci_dev *dev; + int max; + + if (!bus) + return; + + max = bus->busn_res.start; + /* Scan already configured non-hotplug bridges */ + for_each_pci_bridge(dev, bus) { + if (!hotplug_is_native(dev)) + max = pci_scan_bridge(bus, dev, max, 0); + } + + /* Scan non-hotplug bridges that need to be reconfigured */ + for_each_pci_bridge(dev, bus) { + if (!hotplug_is_native(dev)) + max = pci_scan_bridge(bus, dev, max, 1); + } +} + /** * enable_slot - enable, configure a slot * @slot: slot to be enabled @@ -442,25 +466,42 @@ static void enable_slot(struct acpiphp_slot *slot) struct pci_dev *dev; struct pci_bus *bus = slot->bus; struct acpiphp_func *func; - int max, pass; - LIST_HEAD(add_list); - acpiphp_rescan_slot(slot); - max = acpiphp_max_busnr(bus); - for (pass = 0; pass < 2; pass++) { + if (bus->self && hotplug_is_native(bus->self)) { + /* + * If native hotplug is used, it will take care of hotplug + * slot management and resource allocation for hotplug + * bridges. However, ACPI hotplug may still be used for + * non-hotplug bridges to bring in additional devices such + * as a Thunderbolt host controller. + */ for_each_pci_bridge(dev, bus) { - if (PCI_SLOT(dev->devfn) != slot->device) - continue; - - max = pci_scan_bridge(bus, dev, max, pass); - if (pass && dev->subordinate) { - check_hotplug_bridge(slot, dev); - pcibios_resource_survey_bus(dev->subordinate); - __pci_bus_size_bridges(dev->subordinate, &add_list); + if (PCI_SLOT(dev->devfn) == slot->device) + acpiphp_native_scan_bridge(dev); + } + pci_assign_unassigned_bridge_resources(bus->self); + } else { + LIST_HEAD(add_list); + int max, pass; + + acpiphp_rescan_slot(slot); + max = acpiphp_max_busnr(bus); + for (pass = 0; pass < 2; pass++) { + for_each_pci_bridge(dev, bus) { + if (PCI_SLOT(dev->devfn) != slot->device) + continue; + + max = pci_scan_bridge(bus, dev, max, pass); + if (pass && dev->subordinate) { + check_hotplug_bridge(slot, dev); + pcibios_resource_survey_bus(dev->subordinate); + __pci_bus_size_bridges(dev->subordinate, + &add_list); + } } } + __pci_bus_assign_resources(bus, &add_list, NULL); } - __pci_bus_assign_resources(bus, &add_list, NULL); acpiphp_sanitize_bus(bus); pcie_bus_configure_settings(bus);
On Sat, Jun 02, 2018 at 12:46:59AM -0500, Bjorn Helgaas wrote: > > I think here hotplug_is_native() would work as well. The idea is > > that we need to avoid scanning bridges that are handled by pciehp > > (or shpchp). > > I made that change on my branch. That way we won't have to worry > about the cases where we set is_hotplug_bridge based on some ACPI > methods. I think those would be bugs because I think we *do* want > acpiphp to scan them, and if we tested "!dev->is_hotplug_bridge", we > would not scan them. I agree.
diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c index 318b6a6f6341..e2bcd9fc3fd2 100644 --- a/drivers/pci/hotplug/acpiphp_glue.c +++ b/drivers/pci/hotplug/acpiphp_glue.c @@ -287,11 +287,12 @@ static acpi_status acpiphp_add_context(acpi_handle handle, u32 lvl, void *data, /* * Expose slots to user space for functions that have _EJ0 or _RMV or * are located in dock stations. Do not expose them for devices handled - * by the native PCIe hotplug (PCIeHP), becuase that code is supposed to - * expose slots to user space in those cases. + * by the native PCIe hotplug (PCIeHP) or standard PCI hotplug + * (SHPCHP), because that code is supposed to expose slots to user + * space in those cases. */ if ((acpi_pci_check_ejectable(pbus, handle) || is_dock_device(adev)) - && !(pdev && pdev->is_hotplug_bridge && pciehp_is_native(pdev))) { + && !(pdev && hotplug_is_native(pdev))) { unsigned long long sun; int retval; @@ -430,6 +431,29 @@ static int acpiphp_rescan_slot(struct acpiphp_slot *slot) return pci_scan_slot(slot->bus, PCI_DEVFN(slot->device, 0)); } +static void acpiphp_native_scan_bridge(struct pci_dev *bridge) +{ + struct pci_bus *bus = bridge->subordinate; + struct pci_dev *dev; + int max; + + if (!bus) + return; + + max = bus->busn_res.start; + /* Scan already configured non-hotplug bridges */ + for_each_pci_bridge(dev, bus) { + if (!dev->is_hotplug_bridge) + max = pci_scan_bridge(bus, dev, max, 0); + } + + /* Scan non-hotplug bridges that need to be reconfigured */ + for_each_pci_bridge(dev, bus) { + if (!dev->is_hotplug_bridge) + max = pci_scan_bridge(bus, dev, max, 1); + } +} + /** * enable_slot - enable, configure a slot * @slot: slot to be enabled @@ -442,25 +466,42 @@ static void enable_slot(struct acpiphp_slot *slot) struct pci_dev *dev; struct pci_bus *bus = slot->bus; struct acpiphp_func *func; - int max, pass; - LIST_HEAD(add_list); - acpiphp_rescan_slot(slot); - max = acpiphp_max_busnr(bus); - for (pass = 0; pass < 2; pass++) { + if (bus->self && hotplug_is_native(bus->self)) { + /* + * If native hotplug is used, it will take care of hotplug + * slot management and resource allocation for hotplug + * bridges. However, ACPI hotplug may still be used for + * non-hotplug bridges to bring in additional devices such + * as Thunderbolt host controller. + */ for_each_pci_bridge(dev, bus) { - if (PCI_SLOT(dev->devfn) != slot->device) - continue; - - max = pci_scan_bridge(bus, dev, max, pass); - if (pass && dev->subordinate) { - check_hotplug_bridge(slot, dev); - pcibios_resource_survey_bus(dev->subordinate); - __pci_bus_size_bridges(dev->subordinate, &add_list); + if (PCI_SLOT(dev->devfn) == slot->device) + acpiphp_native_scan_bridge(dev); + } + pci_assign_unassigned_bridge_resources(bus->self); + } else { + LIST_HEAD(add_list); + int max, pass; + + acpiphp_rescan_slot(slot); + max = acpiphp_max_busnr(bus); + for (pass = 0; pass < 2; pass++) { + for_each_pci_bridge(dev, bus) { + if (PCI_SLOT(dev->devfn) != slot->device) + continue; + + max = pci_scan_bridge(bus, dev, max, pass); + if (pass && dev->subordinate) { + check_hotplug_bridge(slot, dev); + pcibios_resource_survey_bus(dev->subordinate); + __pci_bus_size_bridges(dev->subordinate, + &add_list); + } } } + __pci_bus_assign_resources(bus, &add_list, NULL); } - __pci_bus_assign_resources(bus, &add_list, NULL); acpiphp_sanitize_bus(bus); pcie_bus_configure_settings(bus);
When a system is using native PCIe hotplug for Thunderbolt it will be only present in the system when there is a device connected. This pretty much follows the BIOS assisted hotplug behaviour. Thunderbolt host router integrated PCIe switch has two additional PCIe downstream bridges that lead to NHI (Thunderbolt host controller) and xHCI (USB 3 host controller) respectively. These downstream bridges are not marked being hotplug capable. Reason for that is to preserve resources. Otherwise the OS would distribute remaining resources between all downstream bridges making these two bridges consume precious resources of the actual hotplug bridges. Now, because these two bridges are not marked being hotplug capable the OS will not enable hotplug interrupt for them either and will not receive interrupt when devices behind them are hot-added. Solution to this is that the BIOS sends ACPI Notify() to the root port let the OS know it needs to rescan for added and/or removed devices. Here is how the mechanism is supposed to work when a Thunderbolt endpoint is connected to one of the ports. In case of a standard USB-C device only the xHCI is hot-added otherwise steps are the same. 1. Initially there is only the PCIe root port that is controlled by the pciehp driver 00:1b.0 (Hotplug+) -- 2. Then we get native PCIe hotplug interrupt and once it is handled the topology looks as following 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- +- 02:01.0 (HotPlug+) \- 02:02.0 -- 3. Bridges 02:00.0 and 02:02.0 are not marked as hotplug capable and they don't have anything behind them currently. Bridge 02:01.0 is hotplug capable and used for extending the topology. At this point the required PCIe devices are enabled and ACPI Notify() is sent to the root port. The resulting topology is expected to look like 00:1b.0 (Hotplug+) -- 01:00.0 --+- 02:00.0 -- Thunderbolt host controller +- 02:01.0 (HotPlug+) \- 02:02.0 -- xHCI host controller However, the current ACPI hotplug implementation scans the whole 00:1b.0 hotplug slot and everything behind it regardless whether native PCIe is used or not, and it expects that the BIOS has configured bridge resources upfront. If that's not the case it assigns resources using minimal allocation (everything currently found just barely fit) preventing future extension. In addition to that, if there is another native PCIe hotplug going on we may find the new PCIe switch only partially ready (all links are not fully trained yet) confusing pciehp when it finally starts to enumerate for new devices. To make this work better with the native PCIe (pciehp) and standard PCI (shpchp) hotplug drivers, we let them handle all slot management and resource allocation for hotplug bridges and restrict ACPI hotplug to non-hotplug bridges. Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com> Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> --- drivers/pci/hotplug/acpiphp_glue.c | 75 +++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 17 deletions(-)