[1/3] PCI: vmd: Reduce VMD vectors using NVMe calculation
diff mbox series

Message ID 1573040408-3831-2-git-send-email-jonathan.derrick@intel.com
State Changes Requested
Delegated to: Lorenzo Pieralisi
Headers show
Series
  • PCI: vmd: Reducing tail latency by affining to the storage stack
Related show

Commit Message

Derrick, Jonathan Nov. 6, 2019, 11:40 a.m. UTC
In order to better affine VMD IRQs, VMD IRQ lists, and child NVMe
devices, reduce the number of VMD vectors exposed to the MSI domain
using the same calculation as NVMe. VMD will still retain one vector for
pciehp and non-NVMe vectors. The remaining will match the maximum number
of NVMe child device IO vectors.

Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
---
 drivers/pci/controller/vmd.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

Comments

Keith Busch Nov. 6, 2019, 6:02 p.m. UTC | #1
On Wed, Nov 06, 2019 at 04:40:06AM -0700, Jon Derrick wrote:
> +	max_vectors = min_t(int, vmd->msix_count, num_possible_cpus() + 1);
> +	if (nvec > max_vectors)
> +		return max_vectors;

If vmd's msix vectors beyond num_possible_cpus() are inaccessible, why
not just limit vmd's msix_count the same way?
Derrick, Jonathan Nov. 6, 2019, 7:51 p.m. UTC | #2
On Thu, 2019-11-07 at 03:02 +0900, Keith Busch wrote:
> On Wed, Nov 06, 2019 at 04:40:06AM -0700, Jon Derrick wrote:
> > +	max_vectors = min_t(int, vmd->msix_count, num_possible_cpus() + 1);
> > +	if (nvec > max_vectors)
> > +		return max_vectors;
> 
> If vmd's msix vectors beyond num_possible_cpus() are inaccessible, why
> not just limit vmd's msix_count the same way?

Yes I could probably do that instead

Patch
diff mbox series

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 8bce647..ebe7ff6 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -260,9 +260,20 @@  static int vmd_msi_prepare(struct irq_domain *domain, struct device *dev,
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct vmd_dev *vmd = vmd_from_bus(pdev->bus);
+	int max_vectors;
 
-	if (nvec > vmd->msix_count)
-		return vmd->msix_count;
+	/*
+	 * VMD exists primarily as an NVMe storage domain. It thus makes sense
+	 * to reduce the number of VMD vectors exposed to child devices using
+	 * the same calculation as the NVMe driver. This allows better affinity
+	 * matching along the entire stack when multiple device vectors share
+	 * VMD IRQ lists. One additional VMD vector is reserved for pciehp and
+	 * non-NVMe interrupts, and NVMe Admin Queue interrupts can also be
+	 * placed on this slow interrupt.
+	 */
+	max_vectors = min_t(int, vmd->msix_count, num_possible_cpus() + 1);
+	if (nvec > max_vectors)
+		return max_vectors;
 
 	memset(arg, 0, sizeof(*arg));
 	return 0;