| Message ID | 20220330140823.28826-2-shlomop@pliops.com |
|---|---|
| State | New |
| Headers | show |
| Series | Intel Sky Lake-E host root ports check. | expand |
On 2022-03-30 08:08, Shlomo Pongratz wrote: > In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to > the whitelist") > Andrew Maier added the Sky Lake-E additional devices > 2031, 2032 and 2033 root ports to the already existing 2030 device. > > The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C and D, > respectively and if all exist they will occupy slots 0 till 3 in that order. > > The original code handled only the case where the devices in the whiltlist are s/whiltlist/whitelist/ > host bridges and assumed that they will be found on slot 0. > > This assumption doesn't hold for root ports so an explicit test was added to > cover this case. > > Signed-off-by: Shlomo Pongratz <shlomop@pliops.com> This patch looks good to me. Much cleaner and easier to understand than the previous ones. Reviewed-by: Logan Gunthorpe <logang@deltatee.com> Thanks, Logan
On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote: > @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host) > > if (!root) > return NULL; > - if (root->devfn != PCI_DEVFN(0, 0)) > + > + /* Is it a host bridge or a root port? */ This is a better comment: /* host bridges must have a 0 devfn, but some of the entries in the whilelist are root ports that can have any devfn */ > + if (root->devfn != PCI_DEVFN(0, 0) && > + pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT) > return NULL; Indentation is wrong Jason
On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote: > On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote: > > @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host) > > > > if (!root) > > return NULL; > > - if (root->devfn != PCI_DEVFN(0, 0)) > > + > > + /* Is it a host bridge or a root port? */ > > This is a better comment: > > /* host bridges must have a 0 devfn, but some of the entries in the > whilelist are root ports that can have any devfn */ Is this something in the spec or is it just common practice? The PCIe spec says very little about "host bridges" and I don't remember anything about them having to be devfn 0 or even that they have to be materialized as PCI devices. > > + if (root->devfn != PCI_DEVFN(0, 0) && > > + pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT) > > return NULL; > > Indentation is wrong > > Jason
On Wed, Mar 30, 2022 at 02:10:17PM -0500, Bjorn Helgaas wrote: > On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote: > > On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote: > > > @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host) > > > > > > if (!root) > > > return NULL; > > > - if (root->devfn != PCI_DEVFN(0, 0)) > > > + > > > + /* Is it a host bridge or a root port? */ > > > > This is a better comment: > > > > /* host bridges must have a 0 devfn, but some of the entries in the > > whilelist are root ports that can have any devfn */ > > Is this something in the spec or is it just common practice? The PCIe > spec says very little about "host bridges" and I don't remember > anything about them having to be devfn 0 or even that they have to be > materialized as PCI devices. I think we are relying on common practice here, but I don't know why this check was added in the first place? Logan? Jason
On 2022-03-30 13:11, Jason Gunthorpe wrote: > On Wed, Mar 30, 2022 at 02:10:17PM -0500, Bjorn Helgaas wrote: >> On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote: >>> On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote: >>>> @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host) >>>> >>>> if (!root) >>>> return NULL; >>>> - if (root->devfn != PCI_DEVFN(0, 0)) >>>> + >>>> + /* Is it a host bridge or a root port? */ >>> >>> This is a better comment: >>> >>> /* host bridges must have a 0 devfn, but some of the entries in the >>> whilelist are root ports that can have any devfn */ >> >> Is this something in the spec or is it just common practice? The PCIe >> spec says very little about "host bridges" and I don't remember >> anything about them having to be devfn 0 or even that they have to be >> materialized as PCI devices. > > I think we are relying on common practice here, but I don't know why > this check was added in the first place? Logan? Yeah, this was more about common practice than spec. Seeing we only support a subset of devices it was expected that those added to our support list would meet this criteria. The main purpose here is to just get identifying information about the host bridge. I wouldn't say that the host bridge is dev-fn 0 in any sense of the word. What a host bridge is vs a root port is also a bit murky . I'm really not sure what the tree looks like on this SkyLake system, but on my test system, the two host bridges are also root ports. 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00 The point was really just finding the VID/PID of one of the PCI devices belonging to the bridge for identifying whether it was safe to use. Originally this function was implemented as simply pci_get_slot(host->bus, PCI_DEVFN(0, 0)); That was replaced with the current function solely to avoid needing to take pci_bus_sem in this path. The check for dev-fn 0 was kept to ensure it matched the old method. Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is probably a good idea, assuming it works for all existing systems. I'd expect it would be set for all the devices currently allowed. Logan
On Wed, Mar 30, 2022 at 01:36:25PM -0600, Logan Gunthorpe wrote: > Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is > probably a good idea, assuming it works for all existing systems. I'd > expect it would be set for all the devices currently allowed. I think if we find a PCI ID in the white list as we go up the PCI hierarchy we should just declare success? Does it matter if it is the top of the tree or if it is a root port or host bridge? The IDs in this list only exist as part of SOCs, so seeing them at all should confirm the whole SOC.. Jason
On 2022-03-30 13:38, Jason Gunthorpe wrote: > On Wed, Mar 30, 2022 at 01:36:25PM -0600, Logan Gunthorpe wrote: > >> Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is >> probably a good idea, assuming it works for all existing systems. I'd >> expect it would be set for all the devices currently allowed. > > I think if we find a PCI ID in the white list as we go up the PCI > hierarchy we should just declare success? Well, in older hardware we also have to ensure that the devices are on the same host bridge. (See REQ_SAME_HOST_BRIDGE). So it's a bit more complicated than that... > Does it matter if it is the top of the tree or if it is a root port or > host bridge? The IDs in this list only exist as part of SOCs, so > seeing them at all should confirm the whole SOC.. Not sure if that will always hold with multi-socket systems. The code currently uses pci_find_host_bridge() for each device, then uses pci_host_bridge_dev() simply to find identifying information about the host bridge. It doesn't really matter where in the tree things are, but it would be add for the host bridge not to be at the top, I would have thought. Logan
diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 30b1df3c9d2f..107905dfa229 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -327,15 +327,18 @@ static const struct pci_p2pdma_whitelist_entry { /* * This lookup function tries to find the PCI device corresponding to a given - * host bridge. + * host bridge or a root port. * * It assumes the host bridge device is the first PCI device in the - * bus->devices list and that the devfn is 00.0. These assumptions should hold - * for all the devices in the whitelist above. + * bus->devices list and that the devfn is 00.0. The first assumption should + * hold for all the devices in the whitelist above, however the second one + * doesn't always holds for root ports, for example Intel SkyLake-E devices + * 2030, 2031, 2032 and 2033 which are root ports (A, B, C and D respectively). + * So being a root port is checked explicitly. * - * This function is equivalent to pci_get_slot(host->bus, 0), however it does - * not take the pci_bus_sem lock seeing __host_bridge_whitelist() must not - * sleep. + * This function is equivalent to pci_get_slot(host->bus, 0) (except for + * the root port test), however it does not take the pci_bus_sem lock seeing + * __host_bridge_whitelist() must not sleep. * * For this to be safe, the caller should hold a reference to a device on the * bridge, which should ensure the host_bridge device will not be freed @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host) if (!root) return NULL; - if (root->devfn != PCI_DEVFN(0, 0)) + + /* Is it a host bridge or a root port? */ + if (root->devfn != PCI_DEVFN(0, 0) && + pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT) return NULL; return root;
In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to the whitelist") Andrew Maier added the Sky Lake-E additional devices 2031, 2032 and 2033 root ports to the already existing 2030 device. The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C and D, respectively and if all exist they will occupy slots 0 till 3 in that order. The original code handled only the case where the devices in the whiltlist are host bridges and assumed that they will be found on slot 0. This assumption doesn't hold for root ports so an explicit test was added to cover this case. Signed-off-by: Shlomo Pongratz <shlomop@pliops.com> --- drivers/pci/p2pdma.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)