| Message ID | 20261001-fix-p2p-acs-v4-0-v9-8-1a8e0f50ddd9@nvidia.com |
|---|---|
| State | New |
| Headers | show |
| Series | PCI/P2PDMA: Route peer-to-peer DMA by TLP class | expand |
> From: Leon Romanovsky <leonro@nvidia.com> > > ACS P2P Completion Redirect leaves Completions carrying the Relaxed > Ordering attribute alone. PCIe r7.0 sec 6.12.1.1 redirects only those "that > do not have the Relaxed Ordering Attribute bit set", and sec 7.7.12.5 > describes the enable bit as "applicable only to Completions whose Relaxed > Ordering Attribute is clear". P2PDMA reports one answer for every kind of > TLP, so a client whose provider returns such Completions is sent through > the host bridge for a redirect that never happens to it. > > Add enum pci_p2pdma_tlp_flags and let a caller state that property. > > Reviewed-by: Logan Gunthorpe <logang@deltatee.com> > Tested-by: Tushar Dave <tdave@nvidia.com> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> Sashiko has reviewed this patch and found no issues. It looks great!
On Thu, Oct 01, 2026 at 02:55:16PM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky <leonro@nvidia.com> > > ACS P2P Completion Redirect leaves Completions carrying the Relaxed > Ordering attribute alone. PCIe r7.0 sec 6.12.1.1 redirects only those "that > do not have the Relaxed Ordering Attribute bit set", and sec 7.7.12.5 > describes the enable bit as "applicable only to Completions whose Relaxed > Ordering Attribute is clear". P2PDMA reports one answer for every kind of > TLP, so a client whose provider returns such Completions is sent through > the host bridge for a redirect that never happens to it. I guess pci_acs_p2pdma_completion() returned PCI_ACS_P2PDMA_REDIRECT even for RO Completions? But that didn't change the hardware behavior -- maybe the caller *thought* Completions were routed through the host bridge, but they actually weren't. I kind of lost the plot here. What does the caller do with this information? I don't think a device is *required* to set RO even when it is enabled, and it may set RO on some transactions but not others. > Add enum pci_p2pdma_tlp_flags and let a caller state that property. > > Reviewed-by: Logan Gunthorpe <logang@deltatee.com> > Tested-by: Tushar Dave <tdave@nvidia.com> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > --- > drivers/pci/p2pdma.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c > index 43e225cc5735..1fadef6d0609 100644 > --- a/drivers/pci/p2pdma.c > +++ b/drivers/pci/p2pdma.c > @@ -544,11 +544,15 @@ pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags) > /* > * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes. > * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect > - * affects a Completion. > + * affects a Completion, and that one leaves Completions carrying the Relaxed > + * Ordering attribute alone. > */ > static enum pci_acs_p2pdma_state > pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags) > { > + if (tlp_flags & PCI_P2PDMA_TLP_RELAXED_CPL) > + return PCI_ACS_P2PDMA_DIRECT; > + > return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT : > PCI_ACS_P2PDMA_DIRECT; > } > > -- > 2.55.0 >
On Tue, Oct 06, 2026 at 05:21:09PM -0500, Bjorn Helgaas wrote: > On Thu, Oct 01, 2026 at 02:55:16PM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky <leonro@nvidia.com> > > > > ACS P2P Completion Redirect leaves Completions carrying the Relaxed > > Ordering attribute alone. PCIe r7.0 sec 6.12.1.1 redirects only those "that > > do not have the Relaxed Ordering Attribute bit set", and sec 7.7.12.5 > > describes the enable bit as "applicable only to Completions whose Relaxed > > Ordering Attribute is clear". P2PDMA reports one answer for every kind of > > TLP, so a client whose provider returns such Completions is sent through > > the host bridge for a redirect that never happens to it. > > I guess pci_acs_p2pdma_completion() returned PCI_ACS_P2PDMA_REDIRECT > even for RO Completions? But that didn't change the hardware > behavior -- maybe the caller *thought* Completions were routed through > the host bridge, but they actually weren't. > > I kind of lost the plot here. What does the caller do with this > information? I don't think a device is *required* to set RO even when > it is enabled, and it may set RO on some transactions but not others. The mlx5 patch, which is currently on hold, uses the PCI_P2PDMA_TLP_RELAXED_CPL TLP flag to dynamically enable or disable ATS. https://lore.kernel.org/all/20260920-fix-p2p-acs-v4-0-v7-19-ca0828ab697c@nvidia.com/ Thanks > > > Add enum pci_p2pdma_tlp_flags and let a caller state that property. > > > > Reviewed-by: Logan Gunthorpe <logang@deltatee.com> > > Tested-by: Tushar Dave <tdave@nvidia.com> > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > > --- > > drivers/pci/p2pdma.c | 6 +++++- > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c > > index 43e225cc5735..1fadef6d0609 100644 > > --- a/drivers/pci/p2pdma.c > > +++ b/drivers/pci/p2pdma.c > > @@ -544,11 +544,15 @@ pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags) > > /* > > * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes. > > * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect > > - * affects a Completion. > > + * affects a Completion, and that one leaves Completions carrying the Relaxed > > + * Ordering attribute alone. > > */ > > static enum pci_acs_p2pdma_state > > pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags) > > { > > + if (tlp_flags & PCI_P2PDMA_TLP_RELAXED_CPL) > > + return PCI_ACS_P2PDMA_DIRECT; > > + > > return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT : > > PCI_ACS_P2PDMA_DIRECT; > > } > > > > -- > > 2.55.0 > >
diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 43e225cc5735..1fadef6d0609 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -544,11 +544,15 @@ pci_acs_p2pdma_request(u16 ctrl, unsigned int tlp_flags) /* * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes. * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect - * affects a Completion. + * affects a Completion, and that one leaves Completions carrying the Relaxed + * Ordering attribute alone. */ static enum pci_acs_p2pdma_state pci_acs_p2pdma_completion(u16 ctrl, unsigned int tlp_flags) { + if (tlp_flags & PCI_P2PDMA_TLP_RELAXED_CPL) + return PCI_ACS_P2PDMA_DIRECT; + return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT; }