From patchwork Fri Jan 13 20:24:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Streetman X-Patchwork-Id: 715243 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) by ozlabs.org (Postfix) with ESMTP id 3v0YyF6QB6z9vDk; Sat, 14 Jan 2017 07:24:37 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1cS8Oo-0002PI-S0; Fri, 13 Jan 2017 20:24:34 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1cS8Oi-0002Ou-TH for kernel-team@lists.ubuntu.com; Fri, 13 Jan 2017 20:24:28 +0000 Received: from 45-27-90-188.lightspeed.rlghnc.sbcglobal.net ([45.27.90.188] helo=toughbook.lan) by youngberry.canonical.com with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1cS8Oi-00039R-Eh; Fri, 13 Jan 2017 20:24:28 +0000 From: Dan Streetman To: kernel-team@lists.ubuntu.com Subject: [Yakkety][PATCH] nvme: revert NVMe: only setup MSIX once Date: Fri, 13 Jan 2017 15:24:19 -0500 Message-Id: <20170113202419.21184-1-dan.streetman@canonical.com> X-Mailer: git-send-email 2.9.3 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: kernel-team-bounces@lists.ubuntu.com BugLink: http://bugs.launchpad.net/bugs/1626894 Revert the following commit from Yakkety: 62ed27a2ff91 ("UBUNTU: SAUCE: (no-up) NVMe: only setup MSIX once") This commit was added to work around a problem that is actually a bug in the Xen kernel code, which prevented multiple NVMe controllers from initializing in a Xen guest. However, it appears to be causing NVMe controller initialization failures in some (but not all) non-Xen environments. Since it is not needed once the actual Xen bug is fixed, it should be reverted so the non-Xen NVMe initialization works again, and the real Xen bugfix applied. Signed-off-by: Dan Streetman --- drivers/nvme/host/pci.c | 70 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 21 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 9206ff1..3a4078a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1490,7 +1490,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) struct pci_dev *pdev = to_pci_dev(dev->dev); int result, i, vecs, nr_io_queues, size; - nr_io_queues = dev->max_qid; + nr_io_queues = num_online_cpus(); result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues); if (result < 0) return result; @@ -1522,8 +1522,45 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) adminq->q_db = dev->dbs; } + /* Deregister the admin queue's interrupt */ + free_irq(dev->entry[0].vector, adminq); + + /* + * If we enable msix early due to not intx, disable it again before + * setting up the full range we need. + */ + if (pdev->msi_enabled) + pci_disable_msi(pdev); + else if (pdev->msix_enabled) + pci_disable_msix(pdev); + + for (i = 0; i < nr_io_queues; i++) + dev->entry[i].entry = i; + vecs = pci_enable_msix_range(pdev, dev->entry, 1, nr_io_queues); + if (vecs < 0) { + vecs = pci_enable_msi_range(pdev, 1, min(nr_io_queues, 32)); + if (vecs < 0) { + vecs = 1; + } else { + for (i = 0; i < vecs; i++) + dev->entry[i].vector = i + pdev->irq; + } + } + + /* + * Should investigate if there's a performance win from allocating + * more queues than interrupt vectors; it might allow the submission + * path to scale better, even if the receive path is limited by the + * number of interrupts. + */ + nr_io_queues = vecs; dev->max_qid = nr_io_queues; + result = queue_request_irq(dev, adminq, adminq->irqname); + if (result) { + adminq->cq_vector = -1; + goto free_queues; + } return nvme_create_io_queues(dev); free_queues: @@ -1675,7 +1712,7 @@ static int nvme_dev_add(struct nvme_dev *dev) static int nvme_pci_enable(struct nvme_dev *dev) { u64 cap; - int result = -ENOMEM, nr_io_queues, i, vecs; + int result = -ENOMEM; struct pci_dev *pdev = to_pci_dev(dev->dev); if (pci_enable_device_mem(pdev)) @@ -1692,30 +1729,21 @@ static int nvme_pci_enable(struct nvme_dev *dev) goto disable; } - nr_io_queues = num_possible_cpus(); - - for (i = 0; i < nr_io_queues; i++) - dev->entry[i].entry = i; - vecs = pci_enable_msix_range(pdev, dev->entry, 1, nr_io_queues); - if (vecs < 0) { - vecs = pci_enable_msi_range(pdev, 1, min(nr_io_queues, 32)); - if (vecs < 0) { - result = vecs; - goto disable; - } else { - for (i = 0; i < vecs; i++) - dev->entry[i].vector = i + pdev->irq; - } + /* + * Some devices and/or platforms don't advertise or work with INTx + * interrupts. Pre-enable a single MSIX or MSI vec for setup. We'll + * adjust this later. + */ + if (pci_enable_msix(pdev, dev->entry, 1)) { + pci_enable_msi(pdev); + dev->entry[0].vector = pdev->irq; } - if (vecs < 1) { - dev_err(dev->ctrl.device, "Failed to get any MSI/MSIX interrupts\n"); - result = -ENOSPC; + if (!dev->entry[0].vector) { + result = -ENODEV; goto disable; } - dev->max_qid = vecs; - cap = lo_hi_readq(dev->bar + NVME_REG_CAP); dev->q_depth = min_t(int, NVME_CAP_MQES(cap) + 1, NVME_Q_DEPTH);