From patchwork Mon Aug 6 23:25:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Alex G." X-Patchwork-Id: 954309 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="KShT0vAS"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kv5H1qvdz9s4v for ; Tue, 7 Aug 2018 09:30:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6D3AE84EF0; Mon, 6 Aug 2018 23:30:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eLnpBD-nWYLe; Mon, 6 Aug 2018 23:30:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4334A84E94; Mon, 6 Aug 2018 23:30:08 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 0D1D11C3F86 for ; Mon, 6 Aug 2018 23:26:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 0A118861E0 for ; Mon, 6 Aug 2018 23:26:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JBUIz-ZXOALs for ; Mon, 6 Aug 2018 23:26:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-oi0-f66.google.com (mail-oi0-f66.google.com [209.85.218.66]) by hemlock.osuosl.org (Postfix) with ESMTPS id D9D09861D6 for ; Mon, 6 Aug 2018 23:26:21 +0000 (UTC) Received: by mail-oi0-f66.google.com with SMTP id b15-v6so25244746oib.10 for ; Mon, 06 Aug 2018 16:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2WXbpsYMTz/QG1vzL3x5MJRxezLdU4OG6c9SB+TpFWA=; b=KShT0vAS5hdq8cmK7p8xAxvFu3GTtkHqyiyOhMk3kF3ZqC81EslEflDrsAFnSN/YI/ BGa6s6HkJiekjU/j4+xd26gWrMfB2h0X1LB1J7dWFb/2aOs1MvDY88W4nn7ve0YW1pFW 8iJLbjZEVZzvg9Mz+Msi1vS5h5xZmdar9mebyCKKQKtCsHvhRegcKWVgPsWqrrFQfB+e J21rrtssA+i2kQQTZ9s54nt67p4nVvPs7PLLjo4TIsgb110om0+O2bdd4Rm5LMoy0bs/ jyRx9nCf94AG5R8OxEVd0bfFMBa8NUAfNnhhTTEsSEIVRDNGXyMmfxlF/OsyisXMh88p VCrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2WXbpsYMTz/QG1vzL3x5MJRxezLdU4OG6c9SB+TpFWA=; b=NtjQ2/fV99CIj8MkNmYA+N+1EEItVFIPBsZ0XUSjAPC9d2ARIXhQlx7KQMykbff8Pw XyPC0WXYrq7zebfd9d+od4CQvgGiWv+ke+4cvtYCVgST8e3zH/risQw1oKt+CHbf+G7x cFiSQS3xVIaFHPnj6a/ApSnQQMBIPBIZj3rtWgnIt9C7hCN29FxEMzEc4CeOK+UH157V Qte+U9ASplXMp3CNyBC1vSeJsuLy7r8qXXIsIWh9Me4dw7nGeUFQIfXDxoH2hu5FR5ix qsKYsUxbQd4Dr2WxulpDxa7Y1a08zINeTui2ODR/E4cHNLiNqWwr561g/ZzKoHDNqWwq 1buQ== X-Gm-Message-State: AOUpUlF+DEK3DTRyMS5rj66VBmuwmfYjs54J5+0rvRalR0UsObyj+Ri8 RtLmMtbYWlnnltrgV8B2iQo= X-Google-Smtp-Source: AAOMgpcqOUA/kNzkVMbDdq5HkfCHfTJrwB4RWKrpc/P5ByRsBSjCYn4QtKuK5YdKRDODlfCDhs4ckQ== X-Received: by 2002:aca:a641:: with SMTP id p62-v6mr17010358oie.151.1533597981033; Mon, 06 Aug 2018 16:26:21 -0700 (PDT) Received: from nuclearis2-1.lan (c-98-195-139-126.hsd1.tx.comcast.net. [98.195.139.126]) by smtp.gmail.com with ESMTPSA id w13-v6sm10613328oiw.51.2018.08.06.16.26.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 06 Aug 2018 16:26:19 -0700 (PDT) From: Alexandru Gagniuc To: linux-pci@vger.kernel.org, bhelgaas@google.com, jakub.kicinski@netronome.com Date: Mon, 6 Aug 2018 18:25:35 -0500 Message-Id: <20180806232600.25694-1-mr.nuke.me@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: X-Mailman-Approved-At: Mon, 06 Aug 2018 23:30:06 +0000 Subject: [Intel-wired-lan] [PATCH v6 1/9] PCI: Check for PCIe downtraining conditions X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alex_gagniuc@dellteam.com, Dirk van der Merwe , Tariq Toukan , Leon Romanovsky , linux-rdma@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, shyam_iyer@dell.com, Michael Chan , linux-kernel@vger.kernel.org, keith.busch@intel.com, Saeed Mahameed , Ariel Elior , intel-wired-lan@lists.osuosl.org, Alexandru Gagniuc , austin_bolen@dell.com, everest-linux-l2@cavium.com, "David S. Miller" , Ganesh Goudar MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" PCIe downtraining happens when both the device and PCIe port are capable of a larger bus width or higher speed than negotiated. Downtraining might be indicative of other problems in the system, and identifying this from userspace is neither intuitive, nor straightforward. The easiest way to detect this is with pcie_print_link_status(), since the bottleneck is usually the link that is downtrained. It's not a perfect solution, but it works extremely well in most cases. Signed-off-by: Alexandru Gagniuc Acked-by: David S. Miller --- drivers/pci/pci.c | 22 ++++++++++++++++++---- drivers/pci/probe.c | 21 +++++++++++++++++++++ include/linux/pci.h | 1 + 3 files changed, 40 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 316496e99da9..414ad7b3abdb 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -5302,14 +5302,15 @@ u32 pcie_bandwidth_capable(struct pci_dev *dev, enum pci_bus_speed *speed, } /** - * pcie_print_link_status - Report the PCI device's link speed and width + * __pcie_print_link_status - Report the PCI device's link speed and width * @dev: PCI device to query + * @verbose: Be verbose -- print info even when enough bandwidth is available. * * Report the available bandwidth at the device. If this is less than the * device is capable of, report the device's maximum possible bandwidth and * the upstream link that limits its performance to less than that. */ -void pcie_print_link_status(struct pci_dev *dev) +void __pcie_print_link_status(struct pci_dev *dev, bool verbose) { enum pcie_link_width width, width_cap; enum pci_bus_speed speed, speed_cap; @@ -5319,11 +5320,11 @@ void pcie_print_link_status(struct pci_dev *dev) bw_cap = pcie_bandwidth_capable(dev, &speed_cap, &width_cap); bw_avail = pcie_bandwidth_available(dev, &limiting_dev, &speed, &width); - if (bw_avail >= bw_cap) + if (bw_avail >= bw_cap && verbose) pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth (%s x%d link)\n", bw_cap / 1000, bw_cap % 1000, PCIE_SPEED2STR(speed_cap), width_cap); - else + else if (bw_avail < bw_cap) pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth, limited by %s x%d link at %s (capable of %u.%03u Gb/s with %s x%d link)\n", bw_avail / 1000, bw_avail % 1000, PCIE_SPEED2STR(speed), width, @@ -5331,6 +5332,19 @@ void pcie_print_link_status(struct pci_dev *dev) bw_cap / 1000, bw_cap % 1000, PCIE_SPEED2STR(speed_cap), width_cap); } + +/** + * pcie_print_link_status - Report the PCI device's link speed and width + * @dev: PCI device to query + * + * Report the available bandwidth at the device. If this is less than the + * device is capable of, report the device's maximum possible bandwidth and + * the upstream link that limits its performance to less than that. + */ +void pcie_print_link_status(struct pci_dev *dev) +{ + __pcie_print_link_status(dev, true); +} EXPORT_SYMBOL(pcie_print_link_status); /** diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 611adcd9c169..1c8c26dd2cb2 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2205,6 +2205,24 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) return dev; } +static void pcie_check_upstream_link(struct pci_dev *dev) +{ + if (!pci_is_pcie(dev)) + return; + + /* Look from the device up to avoid downstream ports with no devices. */ + if ((pci_pcie_type(dev) != PCI_EXP_TYPE_ENDPOINT) && + (pci_pcie_type(dev) != PCI_EXP_TYPE_LEG_END) && + (pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM)) + return; + + /* Multi-function PCIe share the same link/status. */ + if (PCI_FUNC(dev->devfn) != 0 || dev->is_virtfn) + return; + + __pcie_print_link_status(dev, false); +} + static void pci_init_capabilities(struct pci_dev *dev) { /* Enhanced Allocation */ @@ -2240,6 +2258,9 @@ static void pci_init_capabilities(struct pci_dev *dev) /* Advanced Error Reporting */ pci_aer_init(dev); + /* Check link and detect downtrain errors */ + pcie_check_upstream_link(dev); + if (pci_probe_reset_function(dev) == 0) dev->reset_fn = 1; } diff --git a/include/linux/pci.h b/include/linux/pci.h index c133ccfa002e..d212de231259 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1087,6 +1087,7 @@ int pcie_set_mps(struct pci_dev *dev, int mps); u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev, enum pci_bus_speed *speed, enum pcie_link_width *width); +void __pcie_print_link_status(struct pci_dev *dev, bool verbose); void pcie_print_link_status(struct pci_dev *dev); int pcie_flr(struct pci_dev *dev); int __pci_reset_function_locked(struct pci_dev *dev);