| Message ID | 20210405052404.213889-3-leon@kernel.org |
|---|---|
| State | New |
| Headers | show |
| Series | Enable relaxed ordering for ULPs | expand |
On 4/5/2021 1:23 AM, Leon Romanovsky wrote: > From: Avihai Horon <avihaih@nvidia.com> > > Enable Relaxed Ordering in __ib_alloc_pd() allocation of the > local_dma_lkey. > > This will take effect only for devices that don't pre-allocate the lkey > but allocate it per PD allocation. > > Signed-off-by: Avihai Horon <avihaih@nvidia.com> > Reviewed-by: Michael Guralnik <michaelgur@nvidia.com> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > --- > drivers/infiniband/core/verbs.c | 3 ++- > drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c | 1 + > 2 files changed, 3 insertions(+), 1 deletion(-) > > diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > index a1782f8a6ca0..9b719f7d6fd5 100644 > --- a/drivers/infiniband/core/verbs.c > +++ b/drivers/infiniband/core/verbs.c > @@ -287,7 +287,8 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags, > if (device->attrs.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) > pd->local_dma_lkey = device->local_dma_lkey; > else > - mr_access_flags |= IB_ACCESS_LOCAL_WRITE; > + mr_access_flags |= > + IB_ACCESS_LOCAL_WRITE | IB_ACCESS_RELAXED_ORDERING; So, do local_dma_lkey's get relaxed ordering unconditionally? > if (flags & IB_PD_UNSAFE_GLOBAL_RKEY) { > pr_warn("%s: enabling unsafe global rkey\n", caller); > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c > index b3fa783698a0..d74827694f92 100644 > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c > @@ -66,6 +66,7 @@ struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc) > int ret; > > /* Support only LOCAL_WRITE flag for DMA MRs */ > + acc &= ~IB_ACCESS_RELAXED_ORDERING; > if (acc & ~IB_ACCESS_LOCAL_WRITE) { > dev_warn(&dev->pdev->dev, > "unsupported dma mr access flags %#x\n", acc); Why does the pvrdma driver require relaxed ordering to be off? Tom.
On 4/5/21 11:01 AM, Tom Talpey wrote: > On 4/5/2021 1:23 AM, Leon Romanovsky wrote: >> From: Avihai Horon <avihaih@nvidia.com> >> >> Enable Relaxed Ordering in __ib_alloc_pd() allocation of the >> local_dma_lkey. >> >> This will take effect only for devices that don't pre-allocate the lkey >> but allocate it per PD allocation. >> >> Signed-off-by: Avihai Horon <avihaih@nvidia.com> >> Reviewed-by: Michael Guralnik <michaelgur@nvidia.com> >> Signed-off-by: Leon Romanovsky <leonro@nvidia.com> >> --- >> drivers/infiniband/core/verbs.c | 3 ++- >> drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c | 1 + >> 2 files changed, 3 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c >> index a1782f8a6ca0..9b719f7d6fd5 100644 >> --- a/drivers/infiniband/core/verbs.c >> +++ b/drivers/infiniband/core/verbs.c >> @@ -287,7 +287,8 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags, >> if (device->attrs.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) >> pd->local_dma_lkey = device->local_dma_lkey; >> else >> - mr_access_flags |= IB_ACCESS_LOCAL_WRITE; >> + mr_access_flags |= >> + IB_ACCESS_LOCAL_WRITE | IB_ACCESS_RELAXED_ORDERING; > > So, do local_dma_lkey's get relaxed ordering unconditionally? > >> if (flags & IB_PD_UNSAFE_GLOBAL_RKEY) { >> pr_warn("%s: enabling unsafe global rkey\n", caller); >> diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c >> index b3fa783698a0..d74827694f92 100644 >> --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c >> +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c >> @@ -66,6 +66,7 @@ struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc) >> int ret; >> /* Support only LOCAL_WRITE flag for DMA MRs */ >> + acc &= ~IB_ACCESS_RELAXED_ORDERING; >> if (acc & ~IB_ACCESS_LOCAL_WRITE) { >> dev_warn(&dev->pdev->dev, >> "unsupported dma mr access flags %#x\n", acc); > > Why does the pvrdma driver require relaxed ordering to be off? PVRDMA doesn't support any other flags other than LOCAL_WRITE for DMA MRs so the MR creation will fail if any new unconditionally added flag isn't cleared.
On Mon, Apr 05, 2021 at 02:01:16PM -0400, Tom Talpey wrote: > On 4/5/2021 1:23 AM, Leon Romanovsky wrote: > > From: Avihai Horon <avihaih@nvidia.com> > > > > Enable Relaxed Ordering in __ib_alloc_pd() allocation of the > > local_dma_lkey. > > > > This will take effect only for devices that don't pre-allocate the lkey > > but allocate it per PD allocation. > > > > Signed-off-by: Avihai Horon <avihaih@nvidia.com> > > Reviewed-by: Michael Guralnik <michaelgur@nvidia.com> > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > > --- > > drivers/infiniband/core/verbs.c | 3 ++- > > drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c | 1 + > > 2 files changed, 3 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > > index a1782f8a6ca0..9b719f7d6fd5 100644 > > --- a/drivers/infiniband/core/verbs.c > > +++ b/drivers/infiniband/core/verbs.c > > @@ -287,7 +287,8 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags, > > if (device->attrs.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) > > pd->local_dma_lkey = device->local_dma_lkey; > > else > > - mr_access_flags |= IB_ACCESS_LOCAL_WRITE; > > + mr_access_flags |= > > + IB_ACCESS_LOCAL_WRITE | IB_ACCESS_RELAXED_ORDERING; > > So, do local_dma_lkey's get relaxed ordering unconditionally? Yes, in mlx5, this lkey is created on the fly. Thanks
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index a1782f8a6ca0..9b719f7d6fd5 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -287,7 +287,8 @@ struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags, if (device->attrs.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) pd->local_dma_lkey = device->local_dma_lkey; else - mr_access_flags |= IB_ACCESS_LOCAL_WRITE; + mr_access_flags |= + IB_ACCESS_LOCAL_WRITE | IB_ACCESS_RELAXED_ORDERING; if (flags & IB_PD_UNSAFE_GLOBAL_RKEY) { pr_warn("%s: enabling unsafe global rkey\n", caller); diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c index b3fa783698a0..d74827694f92 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c @@ -66,6 +66,7 @@ struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc) int ret; /* Support only LOCAL_WRITE flag for DMA MRs */ + acc &= ~IB_ACCESS_RELAXED_ORDERING; if (acc & ~IB_ACCESS_LOCAL_WRITE) { dev_warn(&dev->pdev->dev, "unsupported dma mr access flags %#x\n", acc);