[V7,1/1] Intel Sky Lake-E host root ports check.

In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C,
 D to the whitelist")
Andrew Maier added the Sky Lake-E additional devices
2031, 2032 and 2033 root ports to the already existing 2030 device.

The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C
and D, respectively and if all exist they will occupy slots 0 till 3 in
that order.

The original code handled only the case where the devices in the whitelist
are host bridges and assumed that they will be found on slot 0.

Since this assumption doesn't hold for root ports, add a test to cover this
case.

Signed-off-by: Shlomo Pongratz <shlomop@pliops.com>
---
 drivers/pci/p2pdma.c | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)

> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index 30b1df3c9d2f..187047be83a0 100644
> --- a/drivers/pci/p2pdma.c
> +++ b/drivers/pci/p2pdma.c
> @@ -327,15 +327,19 @@ static const struct pci_p2pdma_whitelist_entry {
>   
>   /*
>    * This lookup function tries to find the PCI device corresponding to a given
> - * host bridge.
> + * host bridge or a root port.
>    *
>    * It assumes the host bridge device is the first PCI device in the
> - * bus->devices list and that the devfn is 00.0. These assumptions should hold
> - * for all the devices in the whitelist above.
> + * bus->devices list and that the devfn is 00.0. The first assumption should
> + * hold for all the devices in the whitelist above, however the second
> + * assumption doesn't always hold for root ports.
> + * For example for Intel Skylake devices 2030, 2031, 2032 and 2033,
> + * which are root ports (A, B, C and D respectively).
> + * So the function checks explicitly that the device is a root port.
>    *
> - * This function is equivalent to pci_get_slot(host->bus, 0), however it does
> - * not take the pci_bus_sem lock seeing __host_bridge_whitelist() must not
> - * sleep.
> + * This function is equivalent to pci_get_slot(host->bus, 0) (except for
> + * the root port test), however it does not take the pci_bus_sem lock seeing
> + * __host_bridge_whitelist() must not sleep.
>    *
>    * For this to be safe, the caller should hold a reference to a device on the
>    * bridge, which should ensure the host_bridge device will not be freed
> @@ -350,10 +354,19 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
>   
>   	if (!root)
>   		return NULL;
> -	if (root->devfn != PCI_DEVFN(0, 0))
> -		return NULL;
>   
> -	return root;
> +	/* Verify that the device is a host bridge or a root port
> +	 * It is assumed that host bridges have a 0 devfn, (common practice)
> +	 * but some of the entries in the whitelist are root ports that can
> +	 * have any devfn
> +	 */
> +	if (root->devfn == PCI_DEVFN(0, 0))
> +		return root;
> +
> +	if (pci_pcie_type(root) == PCI_EXP_TYPE_ROOT_PORT)
> +		return root;
> +
> +	return NULL;
>   }
>   
>   static bool __host_bridge_whitelist(struct pci_host_bridge *host,

Thanks, it also helped our system.

Tested-by: Maor Gottlieb <maorg@nvidia.com>

On Sun, Apr 10, 2022 at 01:52:13PM +0300, Shlomo Pongratz wrote:
> In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C,
>  D to the whitelist")
> Andrew Maier added the Sky Lake-E additional devices
> 2031, 2032 and 2033 root ports to the already existing 2030 device.
> 
> The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C
> and D, respectively and if all exist they will occupy slots 0 till 3 in
> that order.
> 
> The original code handled only the case where the devices in the whitelist
> are host bridges and assumed that they will be found on slot 0.
> 
> Since this assumption doesn't hold for root ports, add a test to cover this
> case.
> 
> Signed-off-by: Shlomo Pongratz <shlomop@pliops.com>

Applied to pci/p2pdma as below, thanks!

> ---
>  drivers/pci/p2pdma.c | 31 ++++++++++++++++++++++---------
>  1 file changed, 22 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index 30b1df3c9d2f..187047be83a0 100644
> --- a/drivers/pci/p2pdma.c
> +++ b/drivers/pci/p2pdma.c
> @@ -327,15 +327,19 @@ static const struct pci_p2pdma_whitelist_entry {
>  
>  /*
>   * This lookup function tries to find the PCI device corresponding to a given
> - * host bridge.
> + * host bridge or a root port.
>   *
>   * It assumes the host bridge device is the first PCI device in the
> - * bus->devices list and that the devfn is 00.0. These assumptions should hold
> - * for all the devices in the whitelist above.
> + * bus->devices list and that the devfn is 00.0. The first assumption should
> + * hold for all the devices in the whitelist above, however the second
> + * assumption doesn't always hold for root ports.
> + * For example for Intel Skylake devices 2030, 2031, 2032 and 2033,
> + * which are root ports (A, B, C and D respectively).
> + * So the function checks explicitly that the device is a root port.
>   *
> - * This function is equivalent to pci_get_slot(host->bus, 0), however it does
> - * not take the pci_bus_sem lock seeing __host_bridge_whitelist() must not
> - * sleep.
> + * This function is equivalent to pci_get_slot(host->bus, 0) (except for
> + * the root port test), however it does not take the pci_bus_sem lock seeing
> + * __host_bridge_whitelist() must not sleep.
>   *
>   * For this to be safe, the caller should hold a reference to a device on the
>   * bridge, which should ensure the host_bridge device will not be freed
> @@ -350,10 +354,19 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
>  
>  	if (!root)
>  		return NULL;
> -	if (root->devfn != PCI_DEVFN(0, 0))
> -		return NULL;
>  
> -	return root;
> +	/* Verify that the device is a host bridge or a root port
> +	 * It is assumed that host bridges have a 0 devfn, (common practice)
> +	 * but some of the entries in the whitelist are root ports that can
> +	 * have any devfn
> +	 */
> +	if (root->devfn == PCI_DEVFN(0, 0))
> +		return root;
> +
> +	if (pci_pcie_type(root) == PCI_EXP_TYPE_ROOT_PORT)
> +		return root;
> +
> +	return NULL;
>  }
>  
>  static bool __host_bridge_whitelist(struct pci_host_bridge *host,

commit 1af7c26c59eb ("PCI/P2PDMA: Whitelist Intel Skylake-E Root Ports at any devfn")
Author: Shlomo Pongratz <shlomopongratz@gmail.com>
Date:   Sun Apr 10 13:52:13 2022 +0300

    PCI/P2PDMA: Whitelist Intel Skylake-E Root Ports at any devfn
    
    In 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to
    the whitelist"), Andrew Maier added Skylake-E 2031, 2032, and 2033 Root
    Ports to the pci_p2pdma_whitelist[], so we assume P2PDMA between devices
    below these ports works.
    
    Previously we only checked the whitelist for a device at devfn 00.0 on the
    root bus, which is often a "host bridge".  But these Skylake Root Ports may
    be at any devfn and there may be no "host bridge" device.
    
    Generalize pci_host_bridge_dev() so we check the first device on the root
    bus, whether it is devfn 00.0 or a PCIe Root Port, against the whitelist.
    
    [bhelgaas: commit log, comment]
    Link: https://lore.kernel.org/r/20220410105213.690-2-shlomop@pliops.com
    Tested-by: Maor Gottlieb <maorg@nvidia.com>
    Signed-off-by: Shlomo Pongratz <shlomop@pliops.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Cc: Andrew Maier <andrew.maier@eideticom.com>

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
index 30b1df3c9d2f..462b429ad243 100644
--- a/drivers/pci/p2pdma.c
+++ b/drivers/pci/p2pdma.c
@@ -326,15 +326,16 @@ static const struct pci_p2pdma_whitelist_entry {
 };
 
 /*
- * This lookup function tries to find the PCI device corresponding to a given
- * host bridge.
+ * If the first device on host's root bus is either devfn 00.0 or a PCIe
+ * Root Port, return it.  Otherwise return NULL.
  *
- * It assumes the host bridge device is the first PCI device in the
- * bus->devices list and that the devfn is 00.0. These assumptions should hold
- * for all the devices in the whitelist above.
+ * We often use a devfn 00.0 "host bridge" in the pci_p2pdma_whitelist[]
+ * (though there is no PCI/PCIe requirement for such a device).  On some
+ * platforms, e.g., Intel Skylake, there is no such host bridge device, and
+ * pci_p2pdma_whitelist[] may contain a Root Port at any devfn.
  *
- * This function is equivalent to pci_get_slot(host->bus, 0), however it does
- * not take the pci_bus_sem lock seeing __host_bridge_whitelist() must not
+ * This function is similar to pci_get_slot(host->bus, 0), but it does
+ * not take the pci_bus_sem lock since __host_bridge_whitelist() must not
  * sleep.
  *
  * For this to be safe, the caller should hold a reference to a device on the
@@ -350,10 +351,14 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
 
 	if (!root)
 		return NULL;
-	if (root->devfn != PCI_DEVFN(0, 0))
-		return NULL;
 
-	return root;
+	if (root->devfn == PCI_DEVFN(0, 0))
+		return root;
+
+	if (pci_pcie_type(root) == PCI_EXP_TYPE_ROOT_PORT)
+		return root;
+
+	return NULL;
 }
 
 static bool __host_bridge_whitelist(struct pci_host_bridge *host,