| Message ID | 426f75e09ac1a6879a6d51f592bf683c698b4bda.1580959044.git.sbobroff@linux.ibm.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | [1/1] vfio-pci/nvlink2: Allow fallback to ibm,mmio-atsd[0] | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | Successfully applied on branch powerpc/merge (530a1cfd52af0aba1af4b1c9a7bc66a202a459b1) |
| snowpatch_ozlabs/build-ppc64le | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64be | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64e | success | Build succeeded |
| snowpatch_ozlabs/build-pmac32 | success | Build succeeded |
| snowpatch_ozlabs/checkpatch | warning | total: 0 errors, 0 warnings, 1 checks, 19 lines checked |
| snowpatch_ozlabs/needsstable | success | Patch has no Fixes tags |
On 06/02/2020 14:17, Sam Bobroff wrote: > Older versions of skiboot only provide a single value in the device > tree property "ibm,mmio-atsd", even when multiple Address Translation > Shoot Down (ATSD) registers are present. This prevents NVLink2 devices > (other than the first) from being used with vfio-pci because vfio-pci > expects to be able to assign a dedicated ATSD register to each NVLink2 > device. > > However, ATSD registers can be shared among devices. This change > allows vfio-pci to fall back to sharing the register at index 0 if > necessary. > > Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com> > --- > drivers/vfio/pci/vfio_pci_nvlink2.c | 13 +++++++++++-- > 1 file changed, 11 insertions(+), 2 deletions(-) > > diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c > index f2983f0f84be..851ba673882b 100644 > --- a/drivers/vfio/pci/vfio_pci_nvlink2.c > +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c > @@ -420,8 +420,17 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) > > if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index, > &mmio_atsd)) { > - dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > - mmio_atsd = 0; > + dev_warn(&vdev->pdev->dev, > + "No ibm,mmio-atsd[%d] found: trying ibm,mmio-atsd[0]\n", > + nvlink_index); We do not really need this warning (nvlink_index doesn't matter that much, we can work out from the device tree what happened), warnings below are enough (if you really want, you can print nvlink_index there). Either way, Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru> > + if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", 0, > + &mmio_atsd)) { > + dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > + mmio_atsd = 0; > + } else { > + dev_warn(&vdev->pdev->dev, > + "Using fallback ibm,mmio-atsd[0] for ATSD.\n"); > + } > } > > if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) { >
On Thu, Feb 06, 2020 at 03:23:03PM +1100, Alexey Kardashevskiy wrote: > > > On 06/02/2020 14:17, Sam Bobroff wrote: > > Older versions of skiboot only provide a single value in the device > > tree property "ibm,mmio-atsd", even when multiple Address Translation > > Shoot Down (ATSD) registers are present. This prevents NVLink2 devices > > (other than the first) from being used with vfio-pci because vfio-pci > > expects to be able to assign a dedicated ATSD register to each NVLink2 > > device. > > > > However, ATSD registers can be shared among devices. This change > > allows vfio-pci to fall back to sharing the register at index 0 if > > necessary. > > > > Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com> > > --- > > drivers/vfio/pci/vfio_pci_nvlink2.c | 13 +++++++++++-- > > 1 file changed, 11 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c > > index f2983f0f84be..851ba673882b 100644 > > --- a/drivers/vfio/pci/vfio_pci_nvlink2.c > > +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c > > @@ -420,8 +420,17 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) > > > > if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index, > > &mmio_atsd)) { > > - dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > > - mmio_atsd = 0; > > + dev_warn(&vdev->pdev->dev, > > + "No ibm,mmio-atsd[%d] found: trying ibm,mmio-atsd[0]\n", > > + nvlink_index); > > > We do not really need this warning (nvlink_index doesn't matter that > much, we can work out from the device tree what happened), warnings > below are enough (if you really want, you can print nvlink_index there). > > Either way, > > Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru> > Thanks, I'll change it if there's some reason to do another version but otherwise leave it as is. Sam. > > > > > + if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", 0, > > + &mmio_atsd)) { > > + dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > > + mmio_atsd = 0; > > + } else { > > + dev_warn(&vdev->pdev->dev, > > + "Using fallback ibm,mmio-atsd[0] for ATSD.\n"); > > + } > > } > > > > if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) { > > > > -- > Alexey
On Fri, Feb 07, 2020 at 01:39:14PM +1100, Sam Bobroff wrote: > On Thu, Feb 06, 2020 at 03:23:03PM +1100, Alexey Kardashevskiy wrote: > > > > > > On 06/02/2020 14:17, Sam Bobroff wrote: > > > Older versions of skiboot only provide a single value in the device > > > tree property "ibm,mmio-atsd", even when multiple Address Translation > > > Shoot Down (ATSD) registers are present. This prevents NVLink2 devices > > > (other than the first) from being used with vfio-pci because vfio-pci > > > expects to be able to assign a dedicated ATSD register to each NVLink2 > > > device. > > > > > > However, ATSD registers can be shared among devices. This change > > > allows vfio-pci to fall back to sharing the register at index 0 if > > > necessary. > > > > > > Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com> > > > --- > > > drivers/vfio/pci/vfio_pci_nvlink2.c | 13 +++++++++++-- > > > 1 file changed, 11 insertions(+), 2 deletions(-) > > > > > > diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c > > > index f2983f0f84be..851ba673882b 100644 > > > --- a/drivers/vfio/pci/vfio_pci_nvlink2.c > > > +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c > > > @@ -420,8 +420,17 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) > > > > > > if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index, > > > &mmio_atsd)) { > > > - dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > > > - mmio_atsd = 0; > > > + dev_warn(&vdev->pdev->dev, > > > + "No ibm,mmio-atsd[%d] found: trying ibm,mmio-atsd[0]\n", > > > + nvlink_index); > > > > > > We do not really need this warning (nvlink_index doesn't matter that > > much, we can work out from the device tree what happened), warnings > > below are enough (if you really want, you can print nvlink_index there). > > > > Either way, > > > > Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru> > > > > Thanks, > > I'll change it if there's some reason to do another version but > otherwise leave it as is. > > Sam. Oh! I almost forgot: maybe this should carry a fixes tag? Fixes: 7f92891778df ("vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver") > > > > > > > > > + if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", 0, > > > + &mmio_atsd)) { > > > + dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); > > > + mmio_atsd = 0; > > > + } else { > > > + dev_warn(&vdev->pdev->dev, > > > + "Using fallback ibm,mmio-atsd[0] for ATSD.\n"); > > > + } > > > } > > > > > > if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) { > > > > > > > -- > > Alexey
diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c index f2983f0f84be..851ba673882b 100644 --- a/drivers/vfio/pci/vfio_pci_nvlink2.c +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c @@ -420,8 +420,17 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index, &mmio_atsd)) { - dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); - mmio_atsd = 0; + dev_warn(&vdev->pdev->dev, + "No ibm,mmio-atsd[%d] found: trying ibm,mmio-atsd[0]\n", + nvlink_index); + if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", 0, + &mmio_atsd)) { + dev_warn(&vdev->pdev->dev, "No available ATSD found\n"); + mmio_atsd = 0; + } else { + dev_warn(&vdev->pdev->dev, + "Using fallback ibm,mmio-atsd[0] for ATSD.\n"); + } } if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) {
Older versions of skiboot only provide a single value in the device tree property "ibm,mmio-atsd", even when multiple Address Translation Shoot Down (ATSD) registers are present. This prevents NVLink2 devices (other than the first) from being used with vfio-pci because vfio-pci expects to be able to assign a dedicated ATSD register to each NVLink2 device. However, ATSD registers can be shared among devices. This change allows vfio-pci to fall back to sharing the register at index 0 if necessary. Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com> --- drivers/vfio/pci/vfio_pci_nvlink2.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)