diff mbox

[6/6] NVMe: Always use MSI/MSI-x interrupts

Message ID 1468854521-24019-7-git-send-email-tim.gardner@canonical.com
State New
Headers show

Commit Message

Tim Gardner July 18, 2016, 3:08 p.m. UTC
From: Keith Busch <keith.busch@intel.com>

BugLink: http://bugs.launchpad.net/bugs/1602724

Multiple users have reported device initialization failure due the driver
not receiving legacy PCI interrupts. This is not unique to any particular
controller, but has been observed on multiple platforms.

There have been no issues reported or observed when with message signaled
interrupts, so this patch attempts to use MSI-x during initialization,
falling back to MSI. If that fails, legacy would become the default.

The setup_io_queues error handling had to change as a result: the admin
queue's msix_entry used to be initialized to the legacy IRQ. The case
where nr_io_queues is 0 would fail request_irq when setting up the admin
queue's interrupt since re-enabling MSI-x fails with 0 vectors, leaving
the admin queue's msix_entry invalid. Instead, return success immediately.

Reported-by: Tim Muhlemmer <muhlemmer@gmail.com>
Reported-by: Jon Derrick <jonathan.derrick@intel.com>
Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
(back ported from commit a5229050b69cfffb690b546c357ca5a60434c0c8)
Signed-off-by: Tim Gardner <tim.gardner@canonical.com>

 Conflicts:
	drivers/nvme/host/pci.c
---
 drivers/nvme/host/pci.c         | 28 +++++++++-------
 spl/rpm/redhat/spl-dkms.spec.in | 72 +----------------------------------------
 spl/rpm/redhat/spl.spec.in      | 72 +----------------------------------------
 3 files changed, 19 insertions(+), 153 deletions(-)
 mode change 100644 => 120000 spl/rpm/redhat/spl-dkms.spec.in
 mode change 100644 => 120000 spl/rpm/redhat/spl.spec.in

Comments

Kamal Mostafa July 19, 2016, 6:08 p.m. UTC | #1
On Mon, Jul 18, 2016 at 09:08:41AM -0600, Tim Gardner wrote:
> From: Keith Busch <keith.busch@intel.com>
> 
> BugLink: http://bugs.launchpad.net/bugs/1602724
> 
> Multiple users have reported device initialization failure due the driver
> not receiving legacy PCI interrupts. This is not unique to any particular
> controller, but has been observed on multiple platforms.
> 
> There have been no issues reported or observed when with message signaled
> interrupts, so this patch attempts to use MSI-x during initialization,
> falling back to MSI. If that fails, legacy would become the default.
> 
> The setup_io_queues error handling had to change as a result: the admin
> queue's msix_entry used to be initialized to the legacy IRQ. The case
> where nr_io_queues is 0 would fail request_irq when setting up the admin
> queue's interrupt since re-enabling MSI-x fails with 0 vectors, leaving
> the admin queue's msix_entry invalid. Instead, return success immediately.
> 
> Reported-by: Tim Muhlemmer <muhlemmer@gmail.com>
> Reported-by: Jon Derrick <jonathan.derrick@intel.com>
> Signed-off-by: Keith Busch <keith.busch@intel.com>
> Signed-off-by: Jens Axboe <axboe@fb.com>
> (back ported from commit a5229050b69cfffb690b546c357ca5a60434c0c8)
> Signed-off-by: Tim Gardner <tim.gardner@canonical.com>
> 
>  Conflicts:
> 	drivers/nvme/host/pci.c
> ---
>  drivers/nvme/host/pci.c         | 28 +++++++++-------


There's some unrelated cruft in this patch:

>  spl/rpm/redhat/spl-dkms.spec.in | 72 +----------------------------------------
>  spl/rpm/redhat/spl.spec.in      | 72 +----------------------------------------
>  3 files changed, 19 insertions(+), 153 deletions(-)
>  mode change 100644 => 120000 spl/rpm/redhat/spl-dkms.spec.in
>  mode change 100644 => 120000 spl/rpm/redhat/spl.spec.in

 -Kamal
Tim Gardner July 19, 2016, 9:36 p.m. UTC | #2
V2 - Cleaned up the unintended cruft in patch #6.

rtg

[PATCH 1/6 Xenial SRU V2] nvme: use a work item to submit async event requests
[PATCH 2/6 Xenial SRU V2] nvme: don't poll the CQ from the kthread
[PATCH 3/6 Xenial SRU V2] nvme: replace the kthread with a per-device watchdog timer
[PATCH 4/6 Xenial SRU V2] NVMe: Fix reset/remove race
[PATCH 5/6 Xenial SRU V2] nvme: Avoid reset work on watchdog timer function during error recovery
[PATCH 6/6 Xenial SRU V2] NVMe: Always use MSI/MSI-x interrupts
Kamal Mostafa July 19, 2016, 10:09 p.m. UTC | #3

Brad Figg July 22, 2016, 10:51 a.m. UTC | #4

Kamal Mostafa July 25, 2016, 4:05 p.m. UTC | #5

diff mbox

Patch

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 9c587b6..b7eb037 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1513,9 +1513,9 @@  static int nvme_setup_io_queues(struct nvme_dev *dev)
 	 * access to the admin queue, as that might be only way to fix them up.
 	 */
 	if (result > 0) {
-		dev_err(dev->dev, "Could not set queue count (%d)\n", result);
-		nr_io_queues = 0;
-		result = 0;
+		dev_err(dev->ctrl.device,
+			"Could not set queue count (%d)\n", result);
+		return 0;
 	}
 
 	if (dev->cmb && NVME_CMB_SQS(dev->cmbsz)) {
@@ -1549,7 +1549,9 @@  static int nvme_setup_io_queues(struct nvme_dev *dev)
 	 * If we enable msix early due to not intx, disable it again before
 	 * setting up the full range we need.
 	 */
-	if (!pdev->irq)
+	if (pdev->msi_enabled)
+		pci_disable_msi(pdev);
+	else if (pdev->msix_enabled)
 		pci_disable_msix(pdev);
 
 	for (i = 0; i < nr_io_queues; i++)
@@ -1729,7 +1731,6 @@  static int nvme_dev_map(struct nvme_dev *dev)
 	if (pci_enable_device_mem(pdev))
 		return result;
 
-	dev->entry[0].vector = pdev->irq;
 	pci_set_master(pdev);
 	bars = pci_select_bars(pdev, IORESOURCE_MEM);
 	if (!bars)
@@ -1752,13 +1753,18 @@  static int nvme_dev_map(struct nvme_dev *dev)
 	}
 
 	/*
-	 * Some devices don't advertse INTx interrupts, pre-enable a single
-	 * MSIX vec for setup. We'll adjust this later.
+	 * Some devices and/or platforms don't advertise or work with INTx
+	 * interrupts. Pre-enable a single MSIX or MSI vec for setup. We'll
+	 * adjust this later.
 	 */
-	if (!pdev->irq) {
-		result = pci_enable_msix(pdev, dev->entry, 1);
-		if (result < 0)
-			goto unmap;
+	if (pci_enable_msix(pdev, dev->entry, 1)) {
+		pci_enable_msi(pdev);
+		dev->entry[0].vector = pdev->irq;
+	}
+
+	if (!dev->entry[0].vector) {
+		result = -ENODEV;
+		goto unmap;
 	}
 
 	cap = lo_hi_readq(dev->bar + NVME_REG_CAP);
diff --git a/spl/rpm/redhat/spl-dkms.spec.in b/spl/rpm/redhat/spl-dkms.spec.in
deleted file mode 100644
index 949660e..0000000
--- a/spl/rpm/redhat/spl-dkms.spec.in
+++ /dev/null
@@ -1,71 +0,0 @@ 
-%{?!packager: %define packager Brian Behlendorf <behlendorf1@llnl.gov>}
-
-%define module  @PACKAGE@
-%define mkconf  scripts/dkms.mkconf
-
-Name:           %{module}-dkms
-
-Version:        @VERSION@
-Release:        @RELEASE@%{?dist}
-Summary:        Kernel module(s) (dkms)
-
-Group:          System Environment/Kernel
-License:        GPLv2+
-URL:            http://zfsonlinux.org/
-Source0:        %{module}-%{version}.tar.gz
-BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-BuildArch:      noarch
-
-Requires:       dkms >= 2.2.0.2
-Requires:       gcc, make, perl
-Requires:       kernel-devel
-Provides:       %{module}-kmod = %{version}
-
-%description
-This package contains the dkms kernel modules required to emulate
-several interfaces provided by the Solaris kernel.
-
-%prep
-%setup -q -n %{module}-%{version}
-
-%build
-%{mkconf} -n %{module} -v %{version} -f dkms.conf
-
-%install
-if [ "$RPM_BUILD_ROOT" != "/" ]; then
-    rm -rf $RPM_BUILD_ROOT
-fi
-mkdir -p $RPM_BUILD_ROOT/usr/src/
-cp -rf ${RPM_BUILD_DIR}/%{module}-%{version} $RPM_BUILD_ROOT/usr/src/
-
-%clean
-if [ "$RPM_BUILD_ROOT" != "/" ]; then
-    rm -rf $RPM_BUILD_ROOT
-fi
-
-%files
-%defattr(-,root,root)
-/usr/src/%{module}-%{version}
-
-%post
-for POSTINST in /usr/lib/dkms/common.postinst; do
-    if [ -f $POSTINST ]; then
-        $POSTINST %{module} %{version}
-        exit $?
-    fi
-    echo "WARNING: $POSTINST does not exist."
-done
-echo -e "ERROR: DKMS version is too old and %{module} was not"
-echo -e "built with legacy DKMS support."
-echo -e "You must either rebuild %{module} with legacy postinst"
-echo -e "support or upgrade DKMS to a more current version."
-exit 1
-
-%preun
-echo -e "Uninstall of %{module} module (version %{version}) beginning:"
-dkms remove -m %{module} -v %{version} --all --rpm_safe_upgrade
-exit 0
-
-%changelog
-* %(date "+%a %b %d %Y") %packager %{version}-%{release}
-- Automatic build by DKMS
diff --git a/spl/rpm/redhat/spl-dkms.spec.in b/spl/rpm/redhat/spl-dkms.spec.in
new file mode 120000
index 0000000..900f524
--- /dev/null
+++ b/spl/rpm/redhat/spl-dkms.spec.in
@@ -0,0 +1 @@ 
+../generic/spl-dkms.spec.in
\ No newline at end of file
diff --git a/spl/rpm/redhat/spl.spec.in b/spl/rpm/redhat/spl.spec.in
deleted file mode 100644
index ac8b8d2..0000000
--- a/spl/rpm/redhat/spl.spec.in
+++ /dev/null
@@ -1,71 +0,0 @@ 
-Name:           @PACKAGE@
-Version:        @VERSION@
-Release:        @RELEASE@%{?dist}
-Summary:        Commands to control the kernel modules
-
-Group:          System Environment/Kernel
-License:        GPLv2+
-URL:            http://zfsonlinux.org/
-Source0:        %{name}-%{version}.tar.gz
-BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-Requires:       %{name}-kmod = %{version}
-Provides:       %{name}-kmod-common = %{version}
-
-%description
-This package contains the commands to verify the SPL
-kernel modules are functioning properly.
-
-%prep
-%setup -q
-
-%build
-%configure --with-config=user
-make %{?_smp_mflags}
-
-%install
-%{__rm} -rf $RPM_BUILD_ROOT
-make install DESTDIR=%{?buildroot}
-
-%files
-%doc AUTHORS COPYING DISCLAIMER
-%{_sbindir}/*
-%{_mandir}/man1/*
-%{_mandir}/man5/*
-
-%changelog
-* Tue Mar 22 2016 Ned Bass <bass6@llnl.gov> - 0.6.5.6-1
-- Remove artificial architecture restrictions in packaging
-- Add support for s390[x] zfsonlinux/spl#537
-* Wed Mar 9 2016 Ned Bass <bass6@llnl.gov> - 0.6.5.5-1
-- Linux 4.5 compatibility zfsonlinux/spl#524
-- Create working debuginfo packages on Red Hat zfsonlinux/zfs#4224
-- Allow copy-builtin to run multiple times zfsonlinux/spl#526
-- Use safer flags for in-kernel memory allocations zfsonlinux/spl#523
-- Fix potential deadlock in cv_wait() zfsonlinux/zfs#4106
-- Fix livelock in shrinker zfsonlinux/zfs#3936
-* Fri Jan  8 2016 Ned Bass <bass6@llnl.gov> - 0.6.5.4-1
-- Build fixes on SPARC and some kernels
-- Fix taskq dynamic spawning deadlock
-- Fix builtin kernel builds
-- Fix crash due to overflow in P2ROUNDUP macro
-- Fix deadlock during direct memory reclaim
-* Tue Oct 13 2015 Ned Bass <bass6@llnl.gov> - 0.6.5.3-1
-- Fix CPU hotplug zfsonlinux/spl#482
-- Disable dynamic taskqs by default to avoid deadlock zfsonlinux/spl#484
-* Tue Sep 29 2015 Ned Bass <bass6@llnl.gov> - 0.6.5.2-1
-- Released 0.6.5.2-1
-- Fix PAX Patch/Grsec SLAB_USERCOPY panic zfsonlinux/zfs#3796
-- Always remove during dkms uninstall/update zfsonlinux/spl#476
-* Thu Sep 19 2015 Ned Bass <bass6@llnl.gov> - 0.6.5.1-1
-- Released 0.6.5.1-1, no changes from spl-0.6.5
-* Thu Sep 10 2015 Brian Behlendorf <behlendorf1@llnl.gov> - 0.6.5-1
-- Released 0.6.5-1, detailed release notes are available at:
-- https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.5
-* Wed Apr  8 2015 Brian Behlendorf <behlendorf1@llnl.gov> - 0.6.4-1
-- Released 0.6.4-1
-* Thu Jun 12 2014 Brian Behlendorf <behlendorf1@llnl.gov> - 0.6.3-1
-- Released 0.6.3-1
-* Wed Aug 21 2013 Brian Behlendorf <behlendorf1@llnl.gov> - 0.6.2-1
-- Released 0.6.2-1
-* Fri Mar 22 2013 Brian Behlendorf <behlendorf1@llnl.gov> - 0.6.1-1
-- First official stable release.
diff --git a/spl/rpm/redhat/spl.spec.in b/spl/rpm/redhat/spl.spec.in
new file mode 120000
index 0000000..d3276f0
--- /dev/null
+++ b/spl/rpm/redhat/spl.spec.in
@@ -0,0 +1 @@ 
+../generic/spl.spec.in
\ No newline at end of file