[v9,02/18] PCI/P2PDMA: Derive routing from directional ACS controls

From: Leon Romanovsky <leonro@nvidia.com>

pci_bridge_has_acs_redir() treats Request and Completion Redirect as
interchangeable. On asymmetric fabrics, a control for only the reverse TLP
direction can unnecessarily force P2PDMA through the host bridge.

Evaluate Request Redirect for client Requests and Completion Redirect for
provider read Completions. Continue treating enabled Egress Control
conservatively as a Request redirect.

Fixes: 52916982af48 ("PCI/P2PDMA: Support peer-to-peer memory")
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Tested-by: Tushar Dave <tdave@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/pci/p2pdma.c | 75 ++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 58 insertions(+), 17 deletions(-)

> From: Leon Romanovsky <leonro@nvidia.com>
> 
> pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> direction can unnecessarily force P2PDMA through the host bridge.
> 
> Evaluate Request Redirect for client Requests and Completion Redirect for
> provider read Completions. Continue treating enabled Egress Control
> conservatively as a Request redirect.
> 
> Fixes: 52916982af48 ("PCI/P2PDMA: Support peer-to-peer memory")
> Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> Tested-by: Tushar Dave <tdave@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Sashiko has reviewed this patch and found no issues. It looks great!

On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> direction can unnecessarily force P2PDMA through the host bridge.

Does "the reverse TLP direction" refer to Completions?

> Evaluate Request Redirect for client Requests and Completion Redirect for
> provider read Completions. Continue treating enabled Egress Control
> conservatively as a Request redirect.

Completion Redirect is intended to avoid ordering rule violations
between Completions and Requests when Requests are redirected (PCIe
r7.0, sec 6.12.1.1).  I assume this patch preserves the ordering rule,
but does the commit log need to say something about that?  I don't
know enough about P2P DMA for it to be obvious to me.

Not really a question for this series, but p2pdma.c and p2pdma.rst
refer to "clients" and "providers", neither of which are mentioned in
the PCIe spec.  In this case it sounds like a client is a Requester
and a provider is a Completer in spec terms.  Is that always the case?
If so, "client" and "provider" in this paragraph are not adding any
information.

If "client" is not the same concept as "Requester" and "provider" not
the same as "Completer", maybe p2pdma.rst could explain the
difference?

> Fixes: 52916982af48 ("PCI/P2PDMA: Support peer-to-peer memory")
> Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> Tested-by: Tushar Dave <tdave@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> ---
>  drivers/pci/p2pdma.c | 75 ++++++++++++++++++++++++++++++++++++++++------------
>  1 file changed, 58 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index 4e4d2df17a45..12612b82d80d 100644
> --- a/drivers/pci/p2pdma.c
> +++ b/drivers/pci/p2pdma.c
> @@ -21,6 +21,8 @@
>  #include <linux/seq_buf.h>
>  #include <linux/xarray.h>
>  
> +#include "pci.h"
> +
>  struct pci_p2pdma {
>  	struct gen_pool *pool;
>  	bool p2pmem_published;
> @@ -490,26 +492,56 @@ static struct pci_dev *find_parent_pci_dev(struct device *dev)
>  	return NULL;
>  }
>  
> +enum pci_acs_p2pdma_state {
> +	PCI_ACS_P2PDMA_DIRECT,
> +	PCI_ACS_P2PDMA_REDIRECT,
> +};
> +
>  /*
> - * Check if a PCI bridge has its ACS redirection bits set to redirect P2P
> - * TLPs upstream via ACS. Returns 1 if the packets will be redirected
> - * upstream, 0 otherwise.
> + * Decide how a peer-to-peer Request at an ACS-capable ingress port routes,
> + * from that port's ACS Control register.
> + *
> + * Linux does not read the Egress Control Vector, so Egress Control is treated
> + * conservatively as a redirect. Per PCIe r7.0 Table 6-11 the outcomes it
> + * selects are a direct route and an ACS Violation, and neither one lets peer
> + * bus addressing be assumed.
>   */
> -static int pci_bridge_has_acs_redir(struct pci_dev *pdev)
> +static enum pci_acs_p2pdma_state
> +pci_acs_p2pdma_request(u16 ctrl)
>  {
> -	int pos;
> -	u16 ctrl;
> +	return ctrl & (PCI_ACS_RR | PCI_ACS_EC) ?
> +		PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT;
> +}
>  
> -	pos = pdev->acs_cap;
> -	if (!pos)
> -		return 0;
> +/*
> + * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes.
> + * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect
> + * affects a Completion.
> + */
> +static enum pci_acs_p2pdma_state
> +pci_acs_p2pdma_completion(u16 ctrl)
> +{
> +	return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT :
> +				   PCI_ACS_P2PDMA_DIRECT;
> +}
>  
> -	pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl);
> +/*
> + * Read @pdev's ACS Control register. A device without an ACS capability has
> + * no peer-to-peer controls at all, which routes the same as having them all
> + * clear. Returns false when the register is present but cannot be read; @ctrl
> + * is then meaningless.
> + */
> +static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, u16 *ctrl)
> +{
> +	int pos;
>  
> -	if (ctrl & (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC))
> -		return 1;
> +	pos = pdev->acs_cap;
> +	if (!pos) {
> +		*ctrl = 0;
> +		return true;
> +	}
>  
> -	return 0;
> +	return !pci_read_config_word(pdev, pos + PCI_ACS_CTRL, ctrl);
>  }
>  
>  static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *pdev)
> @@ -698,6 +730,10 @@ static unsigned long map_types_idx(struct pci_dev *client)
>   * then to Device B. The mapping type returned depends on the ACS
>   * redirection setting of the ports along the path.
>   *
> + * The client initiates Requests to provider memory. Check Request Redirect
> + * on the client path and Completion Redirect for read Completions on the
> + * provider path.
> + *
>   * If ACS redirect is set on any port in the path, traffic between the
>   * devices will go through the host bridge, so return
>   * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; otherwise return
> @@ -721,6 +757,7 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
>  	int dist_a = 0;
>  	int dist_b = 0;
>  	char buf[128];
> +	u16 ctrl;
>  
>  	seq_buf_init(&acs_list, buf, sizeof(buf));
>  
> @@ -732,7 +769,9 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
>  	while (a) {
>  		dist_b = 0;
>  
> -		if (pci_bridge_has_acs_redir(a)) {
> +		if (!pci_acs_p2pdma_ctrl(a, &ctrl) ||
> +		    pci_acs_p2pdma_completion(ctrl) ==
> +			    PCI_ACS_P2PDMA_REDIRECT) {
>  			seq_buf_print_bus_devfn(&acs_list, a);
>  			acs_cnt++;
>  		}
> @@ -761,7 +800,9 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
>  		if (a == bb)
>  			break;
>  
> -		if (pci_bridge_has_acs_redir(bb)) {
> +		if (!pci_acs_p2pdma_ctrl(bb, &ctrl) ||
> +		    pci_acs_p2pdma_request(ctrl) ==
> +			    PCI_ACS_P2PDMA_REDIRECT) {
>  			seq_buf_print_bus_devfn(&acs_list, bb);
>  			acs_cnt++;
>  		}
> @@ -1109,10 +1150,10 @@ EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
>  /**
>   * pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers
>   * @provider: P2PDMA provider structure
> - * @dev: Target device for the transfer
> + * @dev: Client device that initiates the transfer
>   *
>   * Determines how peer-to-peer DMA transfers should be mapped between
> - * the provider and the target device. The mapping type indicates whether
> + * the provider and the client device. The mapping type indicates whether
>   * the transfer can be done directly through PCI switches or must go
>   * through the host bridge.
>   */
> 
> -- 
> 2.55.0
>

On Tue, Oct 06, 2026 at 03:49:47PM -0500, Bjorn Helgaas wrote:
> On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@nvidia.com>
> > 
> > pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> > interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> > direction can unnecessarily force P2PDMA through the host bridge.
> 
> Does "the reverse TLP direction" refer to Completions?

In general, the P2P code treats TLPs flowing from device A to device B the
same as TLPs flowing from device B to device A.

However, in the context of this commit message, yes: completions flow in the
opposite direction from the device's perspective.

> 
> > Evaluate Request Redirect for client Requests and Completion Redirect for
> > provider read Completions. Continue treating enabled Egress Control
> > conservatively as a Request redirect.
> 
> Completion Redirect is intended to avoid ordering rule violations
> between Completions and Requests when Requests are redirected (PCIe
> r7.0, sec 6.12.1.1).  I assume this patch preserves the ordering rule,
> but does the commit log need to say something about that?  I don't
> know enough about P2P DMA for it to be obvious to me.

I don't think so, i didn't change anything related to ordering.

> 
> Not really a question for this series, but p2pdma.c and p2pdma.rst
> refer to "clients" and "providers", neither of which are mentioned in
> the PCIe spec.  In this case it sounds like a client is a Requester
> and a provider is a Completer in spec terms.  Is that always the case?
> If so, "client" and "provider" in this paragraph are not adding any
> information.
> 
> If "client" is not the same concept as "Requester" and "provider" not
> the same as "Completer", maybe p2pdma.rst could explain the
> difference?

Client vs. provider are actual target vs. initiator. They express the
device role in the flow.

Thanks

> 
> > Fixes: 52916982af48 ("PCI/P2PDMA: Support peer-to-peer memory")
> > Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> > Tested-by: Tushar Dave <tdave@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> >  drivers/pci/p2pdma.c | 75 ++++++++++++++++++++++++++++++++++++++++------------
> >  1 file changed, 58 insertions(+), 17 deletions(-)
> > 
> > diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> > index 4e4d2df17a45..12612b82d80d 100644
> > --- a/drivers/pci/p2pdma.c
> > +++ b/drivers/pci/p2pdma.c
> > @@ -21,6 +21,8 @@
> >  #include <linux/seq_buf.h>
> >  #include <linux/xarray.h>
> >  
> > +#include "pci.h"
> > +
> >  struct pci_p2pdma {
> >  	struct gen_pool *pool;
> >  	bool p2pmem_published;
> > @@ -490,26 +492,56 @@ static struct pci_dev *find_parent_pci_dev(struct device *dev)
> >  	return NULL;
> >  }
> >  
> > +enum pci_acs_p2pdma_state {
> > +	PCI_ACS_P2PDMA_DIRECT,
> > +	PCI_ACS_P2PDMA_REDIRECT,
> > +};
> > +
> >  /*
> > - * Check if a PCI bridge has its ACS redirection bits set to redirect P2P
> > - * TLPs upstream via ACS. Returns 1 if the packets will be redirected
> > - * upstream, 0 otherwise.
> > + * Decide how a peer-to-peer Request at an ACS-capable ingress port routes,
> > + * from that port's ACS Control register.
> > + *
> > + * Linux does not read the Egress Control Vector, so Egress Control is treated
> > + * conservatively as a redirect. Per PCIe r7.0 Table 6-11 the outcomes it
> > + * selects are a direct route and an ACS Violation, and neither one lets peer
> > + * bus addressing be assumed.
> >   */
> > -static int pci_bridge_has_acs_redir(struct pci_dev *pdev)
> > +static enum pci_acs_p2pdma_state
> > +pci_acs_p2pdma_request(u16 ctrl)
> >  {
> > -	int pos;
> > -	u16 ctrl;
> > +	return ctrl & (PCI_ACS_RR | PCI_ACS_EC) ?
> > +		PCI_ACS_P2PDMA_REDIRECT : PCI_ACS_P2PDMA_DIRECT;
> > +}
> >  
> > -	pos = pdev->acs_cap;
> > -	if (!pos)
> > -		return 0;
> > +/*
> > + * Decide how a peer-to-peer Completion at an ACS-capable ingress port routes.
> > + * PCIe r7.0 sec 6.12.1.1: no ACS control other than P2P Completion Redirect
> > + * affects a Completion.
> > + */
> > +static enum pci_acs_p2pdma_state
> > +pci_acs_p2pdma_completion(u16 ctrl)
> > +{
> > +	return ctrl & PCI_ACS_CR ? PCI_ACS_P2PDMA_REDIRECT :
> > +				   PCI_ACS_P2PDMA_DIRECT;
> > +}
> >  
> > -	pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl);
> > +/*
> > + * Read @pdev's ACS Control register. A device without an ACS capability has
> > + * no peer-to-peer controls at all, which routes the same as having them all
> > + * clear. Returns false when the register is present but cannot be read; @ctrl
> > + * is then meaningless.
> > + */
> > +static bool pci_acs_p2pdma_ctrl(struct pci_dev *pdev, u16 *ctrl)
> > +{
> > +	int pos;
> >  
> > -	if (ctrl & (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC))
> > -		return 1;
> > +	pos = pdev->acs_cap;
> > +	if (!pos) {
> > +		*ctrl = 0;
> > +		return true;
> > +	}
> >  
> > -	return 0;
> > +	return !pci_read_config_word(pdev, pos + PCI_ACS_CTRL, ctrl);
> >  }
> >  
> >  static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *pdev)
> > @@ -698,6 +730,10 @@ static unsigned long map_types_idx(struct pci_dev *client)
> >   * then to Device B. The mapping type returned depends on the ACS
> >   * redirection setting of the ports along the path.
> >   *
> > + * The client initiates Requests to provider memory. Check Request Redirect
> > + * on the client path and Completion Redirect for read Completions on the
> > + * provider path.
> > + *
> >   * If ACS redirect is set on any port in the path, traffic between the
> >   * devices will go through the host bridge, so return
> >   * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE; otherwise return
> > @@ -721,6 +757,7 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
> >  	int dist_a = 0;
> >  	int dist_b = 0;
> >  	char buf[128];
> > +	u16 ctrl;
> >  
> >  	seq_buf_init(&acs_list, buf, sizeof(buf));
> >  
> > @@ -732,7 +769,9 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
> >  	while (a) {
> >  		dist_b = 0;
> >  
> > -		if (pci_bridge_has_acs_redir(a)) {
> > +		if (!pci_acs_p2pdma_ctrl(a, &ctrl) ||
> > +		    pci_acs_p2pdma_completion(ctrl) ==
> > +			    PCI_ACS_P2PDMA_REDIRECT) {
> >  			seq_buf_print_bus_devfn(&acs_list, a);
> >  			acs_cnt++;
> >  		}
> > @@ -761,7 +800,9 @@ calc_map_type_and_dist(struct pci_dev *provider, struct pci_dev *client,
> >  		if (a == bb)
> >  			break;
> >  
> > -		if (pci_bridge_has_acs_redir(bb)) {
> > +		if (!pci_acs_p2pdma_ctrl(bb, &ctrl) ||
> > +		    pci_acs_p2pdma_request(ctrl) ==
> > +			    PCI_ACS_P2PDMA_REDIRECT) {
> >  			seq_buf_print_bus_devfn(&acs_list, bb);
> >  			acs_cnt++;
> >  		}
> > @@ -1109,10 +1150,10 @@ EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
> >  /**
> >   * pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers
> >   * @provider: P2PDMA provider structure
> > - * @dev: Target device for the transfer
> > + * @dev: Client device that initiates the transfer
> >   *
> >   * Determines how peer-to-peer DMA transfers should be mapped between
> > - * the provider and the target device. The mapping type indicates whether
> > + * the provider and the client device. The mapping type indicates whether
> >   * the transfer can be done directly through PCI switches or must go
> >   * through the host bridge.
> >   */
> > 
> > -- 
> > 2.55.0
> >

On Wed, Oct 07, 2026 at 09:32:27AM +0300, Leon Romanovsky wrote:
> On Tue, Oct 06, 2026 at 03:49:47PM -0500, Bjorn Helgaas wrote:
> > On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> > > From: Leon Romanovsky <leonro@nvidia.com>
> > > 
> > > pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> > > interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> > > direction can unnecessarily force P2PDMA through the host bridge.
> > 
> > Does "the reverse TLP direction" refer to Completions?
> 
> In general, the P2P code treats TLPs flowing from device A to device B the
> same as TLPs flowing from device B to device A.
> 
> However, in the context of this commit message, yes: completions flow in the
> opposite direction from the device's perspective.

And I guess asymmetric fabrics must mean fabrics where Request
Redirect and Completion Redirect are not set the same way?

> > > Evaluate Request Redirect for client Requests and Completion Redirect for
> > > provider read Completions. Continue treating enabled Egress Control
> > > conservatively as a Request redirect.
> > 
> > Completion Redirect is intended to avoid ordering rule violations
> > between Completions and Requests when Requests are redirected (PCIe
> > r7.0, sec 6.12.1.1).  I assume this patch preserves the ordering rule,
> > but does the commit log need to say something about that?  I don't
> > know enough about P2P DMA for it to be obvious to me.
> 
> I don't think so, i didn't change anything related to ordering.

I don't think there's anything in this whole series that changes any
ACS settings, so I shouldn't have wondered about *preserving* the
ordering rule.

But I asked about ordering because it sounds like this patch expects
to encounter asymmetric fabrics where Request Redirect and Completion
Redirect may not be set the same way, and the spec implies that
asymmetry may result in ordering violations.

> > Not really a question for this series, but p2pdma.c and p2pdma.rst
> > refer to "clients" and "providers", neither of which are mentioned in
> > the PCIe spec.  In this case it sounds like a client is a Requester
> > and a provider is a Completer in spec terms.  Is that always the case?
> > If so, "client" and "provider" in this paragraph are not adding any
> > information.
> > 
> > If "client" is not the same concept as "Requester" and "provider" not
> > the same as "Completer", maybe p2pdma.rst could explain the
> > difference?
> 
> Client vs. provider are actual target vs. initiator. They express the
> device role in the flow.

I'd rather use "Requester" and "Completer" when possible because they
have specific meanings in the PCI spec and they correspond to the ACS
control bits.  Client, provider, target, initiator are all from the
outer world that makes use of PCIe constructs, but they don't mean
anything inside the PCIe world.

Bjorn

On Wed, Oct 07, 2026 at 03:40:45PM -0500, Bjorn Helgaas wrote:
> On Wed, Oct 07, 2026 at 09:32:27AM +0300, Leon Romanovsky wrote:
> > On Tue, Oct 06, 2026 at 03:49:47PM -0500, Bjorn Helgaas wrote:
> > > On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@nvidia.com>
> > > > 
> > > > pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> > > > interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> > > > direction can unnecessarily force P2PDMA through the host bridge.
> > > 
> > > Does "the reverse TLP direction" refer to Completions?
> > 
> > In general, the P2P code treats TLPs flowing from device A to device B the
> > same as TLPs flowing from device B to device A.
> > 
> > However, in the context of this commit message, yes: completions flow in the
> > opposite direction from the device's perspective.
> 
> And I guess asymmetric fabrics must mean fabrics where Request
> Redirect and Completion Redirect are not set the same way?

Yes, in some of our systems, deviceA to deviceB is going through
different route than deviceB to deviceA.

> 
> > > > Evaluate Request Redirect for client Requests and Completion Redirect for
> > > > provider read Completions. Continue treating enabled Egress Control
> > > > conservatively as a Request redirect.
> > > 
> > > Completion Redirect is intended to avoid ordering rule violations
> > > between Completions and Requests when Requests are redirected (PCIe
> > > r7.0, sec 6.12.1.1).  I assume this patch preserves the ordering rule,
> > > but does the commit log need to say something about that?  I don't
> > > know enough about P2P DMA for it to be obvious to me.
> > 
> > I don't think so, i didn't change anything related to ordering.
> 
> I don't think there's anything in this whole series that changes any
> ACS settings, so I shouldn't have wondered about *preserving* the
> ordering rule.
> 
> But I asked about ordering because it sounds like this patch expects
> to encounter asymmetric fabrics where Request Redirect and Completion
> Redirect may not be set the same way, and the spec implies that
> asymmetry may result in ordering violations.

My guess is that the hardware and system architects have ensured this
doesn't happen. At least, I haven't received any complaints about this
series during internal testing.

> 
> > > Not really a question for this series, but p2pdma.c and p2pdma.rst
> > > refer to "clients" and "providers", neither of which are mentioned in
> > > the PCIe spec.  In this case it sounds like a client is a Requester
> > > and a provider is a Completer in spec terms.  Is that always the case?
> > > If so, "client" and "provider" in this paragraph are not adding any
> > > information.
> > > 
> > > If "client" is not the same concept as "Requester" and "provider" not
> > > the same as "Completer", maybe p2pdma.rst could explain the
> > > difference?
> > 
> > Client vs. provider are actual target vs. initiator. They express the
> > device role in the flow.
> 
> I'd rather use "Requester" and "Completer" when possible because they
> have specific meanings in the PCI spec and they correspond to the ACS
> control bits.  Client, provider, target, initiator are all from the
> outer world that makes use of PCIe constructs, but they don't mean
> anything inside the PCIe world.

These names come from p2p users such as dma-buf and NVMe. I didn't want
to rename them as they already exists and is in use inside p2p code.

Thanks

> 
> Bjorn