From patchwork Wed Jan 29 15:29:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230968 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=jv1IbUf9; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qT5gHKz9sRh for ; Thu, 30 Jan 2020 02:29:57 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726931AbgA2P3y (ORCPT ); Wed, 29 Jan 2020 10:29:54 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55748 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726893AbgA2P3y (ORCPT ); Wed, 29 Jan 2020 10:29:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id C4CB54760B; Wed, 29 Jan 2020 15:29:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311791; x=1582126192; bh=LX6qQyIwNV4gpow30no2wsV7vfvZuzzeqvb hWFOQ6PE=; b=jv1IbUf9LJXY5E5pjKJvWStCkeUWME2HNSEH29x2k3/E6L76DaB iOhJAG00+aLCaw+AbFJrgAUl0hVj2H8Zu35S3avojc5iI8Jzz/dSpptAgl5ynUMX QXNHjgWlZc5PqI/4LEvf2T6gXXNy1AmhCbRNDATn26rSxxiorZ5i4zVE= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7oGd0TvFo5uE; Wed, 29 Jan 2020 18:29:51 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id BC33247603; Wed, 29 Jan 2020 18:29:50 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:50 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko , Srinath Mannam , Marta Rybczynska Subject: [PATCH v7 01/26] PCI: Fix race condition in pci_enable/disable_device() Date: Wed, 29 Jan 2020 18:29:12 +0300 Message-ID: <20200129152937.311162-2-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This is a yet another approach to fix an old [1-2] concurrency issue, when: - two or more devices are being hot-added into a bridge which was initially empty; - a bridge with two or more devices is being hot-added; - during boot, if BIOS/bootloader/firmware doesn't pre-enable bridges. The problem is that a bridge is reported as enabled before the MEM/IO bits are actually written to the PCI_COMMAND register, so another driver thread starts memory requests through the not-yet-enabled bridge: CPU0 CPU1 pci_enable_device_mem() pci_enable_device_mem() pci_enable_bridge() pci_enable_bridge() pci_is_enabled() return false; atomic_inc_return(enable_cnt) Start actual enabling the bridge ... pci_is_enabled() ... return true; ... Start memory requests <-- FAIL ... Set the PCI_COMMAND_MEMORY bit <-- Must wait for this Protect the pci_enable/disable_device() and pci_enable_bridge(), which is similar to the previous solution from commit 40f11adc7cd9 ("PCI: Avoid race while enabling upstream bridges"), but adding a per-device mutexes and preventing the dev->enable_cnt from from incrementing early. CC: Srinath Mannam CC: Marta Rybczynska Signed-off-by: Sergei Miroshnichenko [1] https://lore.kernel.org/linux-pci/1501858648-22228-1-git-send-email-srinath.mannam@broadcom.com/T/#u [RFC PATCH v3] pci: Concurrency issue during pci enable bridge [2] https://lore.kernel.org/linux-pci/744877924.5841545.1521630049567.JavaMail.zimbra@kalray.eu/T/#u [RFC PATCH] nvme: avoid race-conditions when enabling devices --- drivers/pci/pci.c | 26 ++++++++++++++++++++++---- drivers/pci/probe.c | 1 + include/linux/pci.h | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index df21e3227b57..0366289c75e9 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1672,6 +1672,8 @@ static void pci_enable_bridge(struct pci_dev *dev) struct pci_dev *bridge; int retval; + mutex_lock(&dev->enable_mutex); + bridge = pci_upstream_bridge(dev); if (bridge) pci_enable_bridge(bridge); @@ -1679,6 +1681,7 @@ static void pci_enable_bridge(struct pci_dev *dev) if (pci_is_enabled(dev)) { if (!dev->is_busmaster) pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); return; } @@ -1687,11 +1690,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_err(dev, "Error enabling bridge (%d), continuing\n", retval); pci_set_master(dev); + mutex_unlock(&dev->enable_mutex); } static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) { struct pci_dev *bridge; + /* Enable-locking of bridges is performed within the pci_enable_bridge() */ + bool need_lock = !dev->subordinate; int err; int i, bars = 0; @@ -1707,8 +1713,13 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) dev->current_state = (pmcsr & PCI_PM_CTRL_STATE_MASK); } - if (atomic_inc_return(&dev->enable_cnt) > 1) + if (need_lock) + mutex_lock(&dev->enable_mutex); + if (pci_is_enabled(dev)) { + if (need_lock) + mutex_unlock(&dev->enable_mutex); return 0; /* already enabled */ + } bridge = pci_upstream_bridge(dev); if (bridge) @@ -1723,8 +1734,10 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) bars |= (1 << i); err = do_pci_enable_device(dev, bars); - if (err < 0) - atomic_dec(&dev->enable_cnt); + if (err >= 0) + atomic_inc(&dev->enable_cnt); + if (need_lock) + mutex_unlock(&dev->enable_mutex); return err; } @@ -1968,15 +1981,20 @@ void pci_disable_device(struct pci_dev *dev) if (dr) dr->enabled = 0; + mutex_lock(&dev->enable_mutex); dev_WARN_ONCE(&dev->dev, atomic_read(&dev->enable_cnt) <= 0, "disabling already-disabled device"); - if (atomic_dec_return(&dev->enable_cnt) != 0) + if (atomic_dec_return(&dev->enable_cnt) != 0) { + mutex_unlock(&dev->enable_mutex); return; + } do_pci_disable_device(dev); dev->is_busmaster = 0; + + mutex_unlock(&dev->enable_mutex); } EXPORT_SYMBOL(pci_disable_device); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 512cb4312ddd..2a7e81f146e4 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2169,6 +2169,7 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) INIT_LIST_HEAD(&dev->bus_list); dev->dev.type = &pci_dev_type; dev->bus = pci_bus_get(bus); + mutex_init(&dev->enable_mutex); return dev; } diff --git a/include/linux/pci.h b/include/linux/pci.h index c393dff2d66f..e473e70834b5 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -424,6 +424,7 @@ struct pci_dev { unsigned int no_vf_scan:1; /* Don't scan for VFs after IOV enablement */ pci_dev_flags_t dev_flags; atomic_t enable_cnt; /* pci_enable_device has been called */ + struct mutex enable_mutex; u32 saved_config_space[16]; /* Config space saved at suspend time */ struct hlist_head saved_cap_space; From patchwork Wed Jan 29 15:29:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230966 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=iSy8qORm; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qT1kflz9sRY for ; Thu, 30 Jan 2020 02:29:57 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726842AbgA2P3y (ORCPT ); Wed, 29 Jan 2020 10:29:54 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55756 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726913AbgA2P3y (ORCPT ); Wed, 29 Jan 2020 10:29:54 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1F45F47610; Wed, 29 Jan 2020 15:29:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311792; x=1582126193; bh=FCclUi7mbseaPmr15QdjS7XIaf3Bj0PVVyq fUm8eskI=; b=iSy8qORmXuVcs8qaQx2G7RaTDb3L29si/zpnTwdnmSohGwnasjo yoAkzjPi+Xc3B5+SiKau8YrhhAyTsPmQ2YE9I4hwztfm501ebL05IHYUi1C1kJJC 6OeWTGl9rD4hc/CMQwS18wu5BYvZ7nbA+BZ98RUBTw6HMEU6ovjIGDK4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jMJXsBPtGwUX; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 1D47147604; Wed, 29 Jan 2020 18:29:51 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:50 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 02/26] PCI: Enable bridge's I/O and MEM access for hotplugged devices Date: Wed, 29 Jan 2020 18:29:13 +0300 Message-ID: <20200129152937.311162-3-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The PCI_COMMAND_IO and PCI_COMMAND_MEMORY bits of the bridge must be updated not only when enabling the bridge for the first time, but also if a hotplugged device requests these types of resources. For example, if a bridge had all its slots empty, the IO and MEM bits will not be set, and a hot hotplugged device will fail. Originally these bits were set by the pci_enable_device_flags() only, which exits early if the bridge is already pci_is_enabled(). So let's check them again when requested. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 0366289c75e9..cb0042f28e6a 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1679,6 +1679,14 @@ static void pci_enable_bridge(struct pci_dev *dev) pci_enable_bridge(bridge); if (pci_is_enabled(dev)) { + int i, bars = 0; + + for (i = PCI_BRIDGE_RESOURCES; i < DEVICE_COUNT_RESOURCE; i++) { + if (dev->resource[i].flags & (IORESOURCE_MEM | IORESOURCE_IO)) + bars |= (1 << i); + } + do_pci_enable_device(dev, bars); + if (!dev->is_busmaster) pci_set_master(dev); mutex_unlock(&dev->enable_mutex); From patchwork Wed Jan 29 15:29:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230971 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=a73zkjO0; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qW4rpxz9sPK for ; Thu, 30 Jan 2020 02:29:59 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726964AbgA2P34 (ORCPT ); Wed, 29 Jan 2020 10:29:56 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55780 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726893AbgA2P34 (ORCPT ); Wed, 29 Jan 2020 10:29:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 86DFE47617; Wed, 29 Jan 2020 15:29:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311792; x=1582126193; bh=VQoCGMDO2i0LihBl3EKKUplVZDpvhoPezO9 +fTxjTdg=; b=a73zkjO0Y1P4lADJt2/tEPr1GzM2ZsefB15GmaWHzF8KsrQyZI0 9GvdgUg433EuMnzGzt/9wSaVccUmiTTSeLvB+UiV7dPGKMTMtiUGCsv2ba7gZt9+ rW4ko65RhlUP4WnuPIBOgfFe0BycBdB1zbHE4BSWr0JyqMTtx2EQGJYw= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hEAT51PHg9XD; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 3A60F47605; Wed, 29 Jan 2020 18:29:51 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:50 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko , Sam Bobroff , Rajat Jain , Lukas Wunner , Oliver O'Halloran , David Laight Subject: [PATCH v7 03/26] PCI: hotplug: Initial support of the movable BARs feature Date: Wed, 29 Jan 2020 18:29:14 +0300 Message-ID: <20200129152937.311162-4-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When hot-adding a device, the involved bridge may have windows not big enough (or fragmented too much) for newly requested BARs to fit in. But expanding these bridge windows may be impossible if they are wedged between "neighboring" BARs. Still, it may be possible to allocate a memory region for new BARs, if at least some utilized BARs can be moved, with the following procedure: 1) notify all the drivers which support movable BARs to pause and release the BARs; the rest of the drivers are guaranteed that their devices will not get BARs moved; 2) release all the bridge windows and movable BARs; 3) try to recalculate new bridge windows that will fit all the BAR types: - immovable (including those marked with PCI_FIXED); - movable; - newly requested by hot-added devices; 4) if the previous step fails, disable BARs for one of the hot-added devices and retry from step 3; 5) notify the drivers, so they remap BARs and resume. If bridge calculation and BAR assignment fail with hot-added devices, this patchset disables their BARs, falling back to the same amount and size of BARs as they were before the hotplug event. The kernel succeeded then - so the same BAR layout will be reproduced again. This makes the prior reservation of memory by BIOS/bootloader/firmware not required anymore for the PCI hotplug. Drivers indicate their support of movable BARs by implementing the new .rescan_prepare() and .rescan_done() hooks in the struct pci_driver. All device's activity must be paused during a rescan, and iounmap()+ioremap() must be applied to every used BAR. If a device is not bound to a driver, its BARs are considered movable. For a higher probability of the successful BAR reassignment, all the BARs and bridge windows should be released during a rescan, not only those with higher addresses. One example when it is needed, BAR(I) is moved to free a gap for the new BAR(II): Before: ==================== parent bridge window =============== ---- hotplug bridge window ---- | BAR(I) | fixed BAR | fixed BAR | fixed BAR | ^^^^^^ ^ | new BAR(II) After: ==================== parent bridge window ========================= ----------- hotplug bridge window ----------- | new BAR(II) | fixed BAR | fixed BAR | fixed BAR | BAR(I) | ^^^^^^^^^^^ ^^^^^^ Another example is a fragmented bridge window jammed between fixed BARs: Before: ===================== parent bridge window ======================== ---------- hotplug bridge window ---------- | fixed BAR | | BAR(I) | | BAR(II) | | BAR(III) | fixed BAR | ^^^^^^ ^ ^^^^^^^ ^^^^^^^^ | new BAR(IV) After: ==================== parent bridge window ========================= ---------- hotplug bridge window ---------- | fixed BAR | BAR(I) | BAR(II) | BAR(III) | new BAR(IV) | fixed BAR | ^^^^^^ ^^^^^^^ ^^^^^^^^ ^^^^^^^^^^^ This patch is a preparation for future patches with actual implementation, and for now it just does the following: - declares the feature; - defines bool pci_can_move_bars and bool pci_dev_bar_movable(dev, bar); - invokes the new .rescan_prepare() and .rescan_done() driver notifiers; - disables the feature for the powerpc (support will be added later, in another series). This is disabled by default in this first patch of the series, until a patch in the middle, which finalizes the actual implementation. Then it can be overridden per-arch using the pci_can_move_bars=false flag or by the following command line option: pci=no_movable_bars CC: Sam Bobroff CC: Rajat Jain CC: Lukas Wunner CC: Oliver O'Halloran CC: David Laight Signed-off-by: Sergei Miroshnichenko --- .../admin-guide/kernel-parameters.txt | 1 + arch/powerpc/platforms/powernv/pci.c | 2 + arch/powerpc/platforms/pseries/setup.c | 2 + drivers/pci/pci.c | 4 + drivers/pci/pci.h | 2 + drivers/pci/probe.c | 81 ++++++++++++++++++- include/linux/pci.h | 6 ++ 7 files changed, 96 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index ec92120a7952..9f42cf43bf08 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3610,6 +3610,7 @@ may put more devices in an IOMMU group. force_floating [S390] Force usage of floating interrupts. nomio [S390] Do not use MIO instructions. + no_movable_bars Don't allow BARs to be moved during hotplug pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management. diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c index c0bea75ac27b..ec772aa31ccf 100644 --- a/arch/powerpc/platforms/powernv/pci.c +++ b/arch/powerpc/platforms/powernv/pci.c @@ -939,6 +939,8 @@ void __init pnv_pci_init(void) { struct device_node *np; + pci_can_move_bars = false; + pci_add_flags(PCI_CAN_SKIP_ISA_ALIGN); /* If we don't have OPAL, eg. in sim, just skip PCI probe */ diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index 0c8421dd01ab..2edb41f55237 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -927,6 +927,8 @@ static void __init pseries_init(void) { pr_debug(" -> pseries_init()\n"); + pci_can_move_bars = false; + #ifdef CONFIG_HVC_CONSOLE if (firmware_has_feature(FW_FEATURE_LPAR)) hvc_vio_init_early(); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index cb0042f28e6a..6741c7f111ac 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -79,6 +79,8 @@ static void pci_dev_d3_sleep(struct pci_dev *dev) int pci_domains_supported = 1; #endif +bool pci_can_move_bars; + #define DEFAULT_CARDBUS_IO_SIZE (256) #define DEFAULT_CARDBUS_MEM_SIZE (64*1024*1024) /* pci=cbmemsize=nnM,cbiosize=nn can override this */ @@ -6477,6 +6479,8 @@ static int __init pci_setup(char *str) pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS); } else if (!strncmp(str, "disable_acs_redir=", 18)) { disable_acs_redir_param = str + 18; + } else if (!strncmp(str, "no_movable_bars", 15)) { + pci_can_move_bars = false; } else { pr_err("PCI: Unknown option `%s'\n", str); } diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index a0a53bd05a0b..ce8c53ca7a1e 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -289,6 +289,8 @@ void pci_disable_bridge_window(struct pci_dev *dev); struct pci_bus *pci_bus_get(struct pci_bus *bus); void pci_bus_put(struct pci_bus *bus); +bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res); + /* PCIe link information */ #define PCIE_SPEED2STR(speed) \ ((speed) == PCIE_SPEED_16_0GT ? "16 GT/s" : \ diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 2a7e81f146e4..9b6d15587f5d 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2935,6 +2935,7 @@ int pci_host_probe(struct pci_host_bridge *bridge) * or pci_bus_assign_resources(). */ if (pci_has_flag(PCI_PROBE_ONLY)) { + pci_can_move_bars = false; pci_bus_claim_resources(bus); } else { pci_bus_size_bridges(bus); @@ -3127,6 +3128,68 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) return max; } +bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res) +{ + struct pci_bus_region region; + + if (!pci_can_move_bars) + return false; + + if (res->flags & IORESOURCE_PCI_FIXED) + return false; + + if (dev->driver && dev->driver->rescan_prepare) + return true; + + /* Workaround for the legacy VGA memory 0xa0000-0xbffff */ + pcibios_resource_to_bus(dev->bus, ®ion, res); + if (region.start == 0xa0000) + return false; + + if (!dev->driver && !res->child) + return true; + + return false; +} + +static void pci_bus_rescan_prepare(struct pci_bus *bus) +{ + struct pci_dev *dev; + + if (bus->self) + pci_config_pm_runtime_get(bus->self); + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (child) + pci_bus_rescan_prepare(child); + + if (dev->driver && + dev->driver->rescan_prepare) + dev->driver->rescan_prepare(dev); + } +} + +static void pci_bus_rescan_done(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + + if (dev->driver && + dev->driver->rescan_done) + dev->driver->rescan_done(dev); + + if (child) + pci_bus_rescan_done(child); + } + + if (bus->self) + pci_config_pm_runtime_put(bus->self); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3139,9 +3202,23 @@ unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge) unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; + struct pci_bus *root = bus; + + while (!pci_is_root_bus(root)) + root = root->parent; + + if (pci_can_move_bars) { + pci_bus_rescan_prepare(root); + + max = pci_scan_child_bus(root); + pci_assign_unassigned_root_bus_resources(root); + + pci_bus_rescan_done(root); + } else { + max = pci_scan_child_bus(bus); + pci_assign_unassigned_bus_resources(bus); + } - max = pci_scan_child_bus(bus); - pci_assign_unassigned_bus_resources(bus); pci_bus_add_devices(bus); return max; diff --git a/include/linux/pci.h b/include/linux/pci.h index e473e70834b5..a543f647d337 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -817,6 +817,8 @@ struct module; * e.g. drivers/net/e100.c. * @sriov_configure: Optional driver callback to allow configuration of * number of VFs to enable via sysfs "sriov_numvfs" file. + * @rescan_prepare: Prepare to BAR movement - called before PCI rescan. + * @rescan_done: Remap BARs and restore after PCI rescan. * @err_handler: See Documentation/PCI/pci-error-recovery.rst * @groups: Sysfs attribute groups. * @driver: Driver model structure. @@ -832,6 +834,8 @@ struct pci_driver { int (*resume)(struct pci_dev *dev); /* Device woken up */ void (*shutdown)(struct pci_dev *dev); int (*sriov_configure)(struct pci_dev *dev, int num_vfs); /* On PF */ + void (*rescan_prepare)(struct pci_dev *dev); + void (*rescan_done)(struct pci_dev *dev); const struct pci_error_handlers *err_handler; const struct attribute_group **groups; struct device_driver driver; @@ -1392,6 +1396,8 @@ void pci_setup_bridge(struct pci_bus *bus); resource_size_t pcibios_window_alignment(struct pci_bus *bus, unsigned long type); +extern bool pci_can_move_bars; + #define PCI_VGA_STATE_CHANGE_BRIDGE (1 << 0) #define PCI_VGA_STATE_CHANGE_DECODES (1 << 1) From patchwork Wed Jan 29 15:29:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230969 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=O7tFY6Ea; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qV2xq8z9sRk for ; Thu, 30 Jan 2020 02:29:58 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726939AbgA2P3z (ORCPT ); Wed, 29 Jan 2020 10:29:55 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55764 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726913AbgA2P3z (ORCPT ); Wed, 29 Jan 2020 10:29:55 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 5849947616; Wed, 29 Jan 2020 15:29:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311793; x=1582126194; bh=UV+sluQbKW5VDEjIsRiyd0/sF1OCV3U61py PBRhBXus=; b=O7tFY6EaOxSd0Je8jvpeZbfzGL0y/dVdBwKRArDJbyqSeD4FnYe z4ykBfFKV0w4yiKurbY3n7TXGiXG02F5XST3ivTimS/6B+Ad/2RbblbsFI/MyuFU dr3bXAn8iT4EnrGJ2MLoPS8dPZvfvBHlNTnnrKfcH5i7d1gA06MX9tsY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ocaUymSUS9Xf; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0BEE547606; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:50 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 04/26] PCI: Add version of release_child_resources() aware of immovable BARs Date: Wed, 29 Jan 2020 18:29:15 +0300 Message-ID: <20200129152937.311162-5-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When using the release_child_resources() to release a bridge window, it drops the .start field of child's BARs to zero, but with the STARTALIGN flag remaining set, which makes this resource invalid for re-assignment. Immovable BARs must preserve their offset and size: those marked with the PCI_FIXED or which are bound by drivers without support of the movable BARs feature. Add the pci_release_child_resources() to replace release_child_resources() in handling the described PCI-specific cases. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 54 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index f279826204eb..4c464430478f 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1500,6 +1500,54 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge, (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\ IORESOURCE_MEM_64) +/* + * Similar to generic release_child_resources(), but aware of immovable BARs and + * PCI_FIXED and STARTALIGN flags + */ +static void pci_release_child_resources(struct pci_bus *bus, struct resource *r) +{ + struct pci_dev *dev; + + if (!bus || !r) + return; + + if (r->flags & IORESOURCE_PCI_FIXED) + return; + + r->child = NULL; + + list_for_each_entry(dev, &bus->devices, bus_list) { + int i; + + for (i = 0; i < PCI_NUM_RESOURCES; i++) { + struct resource *tmp = &dev->resource[i]; + resource_size_t size = resource_size(tmp); + + if (!tmp->flags || tmp->parent != r) + continue; + + tmp->parent = NULL; + tmp->sibling = NULL; + + pci_release_child_resources(dev->subordinate, tmp); + + tmp->flags &= ~IORESOURCE_STARTALIGN; + tmp->flags |= IORESOURCE_SIZEALIGN; + + if (!pci_dev_bar_movable(dev, tmp)) { + pci_dbg(dev, "release immovable BAR %d %pR (%s), keep its flags, base and size\n", + i, tmp, tmp->name); + continue; + } + + pci_dbg(dev, "release BAR %d %pR (%s)\n", i, tmp, tmp->name); + + tmp->start = 0; + tmp->end = size - 1; + } + } +} + static void pci_bridge_release_resources(struct pci_bus *bus, unsigned long type) { @@ -1540,7 +1588,11 @@ static void pci_bridge_release_resources(struct pci_bus *bus, return; /* If there are children, release them all */ - release_child_resources(r); + if (pci_can_move_bars) + pci_release_child_resources(bus, r); + else + release_child_resources(r); + if (!release_resource(r)) { type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_info(dev, "resource %d %pR released\n", From patchwork Wed Jan 29 15:29:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230970 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=F7USgEFb; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qV6ThHz9sRm for ; Thu, 30 Jan 2020 02:29:58 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726959AbgA2P34 (ORCPT ); Wed, 29 Jan 2020 10:29:56 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55788 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726913AbgA2P34 (ORCPT ); Wed, 29 Jan 2020 10:29:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 6AE634761A; Wed, 29 Jan 2020 15:29:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311794; x=1582126195; bh=qCJ684UfztdQ8GmZ4DdMqFhe7FDmUKT2og3 ldW7el80=; b=F7USgEFbzOvRsiFy/9se1UbLySbQBL1F7WcucRFPqeBx1eqY6Vi 9XndO0f0ED3soefu1ErZT1qE1wYL0Mbo0qNxiWEAsz9tk94HA5DVd7DusV1/Bsno qdFNlCW6CBJL03VUpmZMzvhQ7kSymYEe30CC8ZbbGgZPPobBDEF9qfMY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kZVzFQ9YJURe; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0FD1747607; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:51 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 05/26] PCI: hotplug: Fix reassigning the released BARs Date: Wed, 29 Jan 2020 18:29:16 +0300 Message-ID: <20200129152937.311162-6-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Bridge windows are temporarily released during a PCI rescan, and their old size is not relevant anymore - it will be recreated in pbus_size_*() from scratch. To make these functions work correctly, set zero size of released windows. If BAR assignment fails after a PCI hotplug event, the kernel will retry with a fall-back reduced configuration, so don't apply reset_resource() for non-window BARs to keep them valid for the next attempt. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 4c464430478f..2cf07003d5fc 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -295,7 +295,9 @@ static void assign_requested_resources_sorted(struct list_head *head, 0 /* don't care */, 0 /* don't care */); } - reset_resource(res); + if (!pci_can_move_bars || + idx >= PCI_BRIDGE_RESOURCES) + reset_resource(res); } } } @@ -1597,8 +1599,8 @@ static void pci_bridge_release_resources(struct pci_bus *bus, type = old_flags = r->flags & PCI_RES_TYPE_MASK; pci_info(dev, "resource %d %pR released\n", PCI_BRIDGE_RESOURCES + idx, r); - /* Keep the old size */ - r->end = resource_size(r) - 1; + /* Don't keep the old size if the bridge will be recalculated */ + r->end = pci_can_move_bars ? 0 : (resource_size(r) - 1); r->start = 0; r->flags = 0; From patchwork Wed Jan 29 15:29:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230977 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=TQJMJBsR; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qb532Rz9sRm for ; Thu, 30 Jan 2020 02:30:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726977AbgA2P35 (ORCPT ); Wed, 29 Jan 2020 10:29:57 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726940AbgA2P34 (ORCPT ); Wed, 29 Jan 2020 10:29:56 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B019A47622; Wed, 29 Jan 2020 15:29:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311794; x=1582126195; bh=a8caSKjaHcX9suJCtM2242IUOCtyhj9Wz3o 39yZG1+A=; b=TQJMJBsRQd5XgIpbAAmgnA6V4BvhHFlKSQ5IDecmTUq0V8zE63H NfmP/j2PSM8L/xBANq59JGY4RTI/wHcjeKGW1DDW2hIMtnYG46EoYoqhYqSUXG2r Nr1NCNbXVtKndxqwNU/8JWozVrZhhZhbmr/CKeyHZlK61vm6+CZsMv7M= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YTdlROFsgnqe; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 13BC047608; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:51 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 06/26] PCI: hotplug: Recalculate every bridge window during rescan Date: Wed, 29 Jan 2020 18:29:17 +0300 Message-ID: <20200129152937.311162-7-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the movable BARs feature is enabled and a rescan has been requested, release all the bridge windows and recalculate them from scratch, taking into account all kinds for BARs: immovable+fixed, movable, new. Comparing to simply trying to expand bridge windows, this also employs the PCI ability to shuffle BARs within a bridge, increasing the chances to find a memory space to fit BARs for newly hotplugged devices, especially if no (not enough) gaps were reserved by the BIOS/bootloader/firmware. The last step of writing the recalculated windows to the bridges is done by the new pci_setup_bridges() function. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.h | 1 + drivers/pci/probe.c | 22 ++++++++++++++++++++++ drivers/pci/setup-bus.c | 16 ++++++++++++++++ 3 files changed, 39 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index ce8c53ca7a1e..fe995993b481 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -283,6 +283,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, struct list_head *realloc_head, struct list_head *fail_head); bool pci_bus_clip_resource(struct pci_dev *dev, int idx); +void pci_bus_release_root_bridge_resources(struct pci_bus *bus); void pci_reassigndev_resource_alignment(struct pci_dev *dev); void pci_disable_bridge_window(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 9b6d15587f5d..daaaebdfd4c4 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3190,6 +3190,25 @@ static void pci_bus_rescan_done(struct pci_bus *bus) pci_config_pm_runtime_put(bus->self); } +static void pci_setup_bridges(struct pci_bus *bus) +{ + struct pci_dev *dev; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child; + + if (!pci_dev_is_added(dev)) + continue; + + child = dev->subordinate; + if (child) + pci_setup_bridges(child); + } + + if (bus->self) + pci_setup_bridge(bus); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3211,8 +3230,11 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_bus_rescan_prepare(root); max = pci_scan_child_bus(root); + + pci_bus_release_root_bridge_resources(root); pci_assign_unassigned_root_bus_resources(root); + pci_setup_bridges(root); pci_bus_rescan_done(root); } else { max = pci_scan_child_bus(bus); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 2cf07003d5fc..7a9bf979908d 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1654,6 +1654,22 @@ static void pci_bus_release_bridge_resources(struct pci_bus *bus, pci_bridge_release_resources(bus, type); } +void pci_bus_release_root_bridge_resources(struct pci_bus *root_bus) +{ + int i; + struct resource *r; + + pci_bus_release_bridge_resources(root_bus, IORESOURCE_IO, whole_subtree); + pci_bus_release_bridge_resources(root_bus, IORESOURCE_MEM, whole_subtree); + pci_bus_release_bridge_resources(root_bus, + IORESOURCE_MEM_64 | IORESOURCE_PREFETCH, + whole_subtree); + + pci_bus_for_each_resource(root_bus, r, i) { + pci_release_child_resources(root_bus, r); + } +} + static void pci_bus_dump_res(struct pci_bus *bus) { struct resource *res; From patchwork Wed Jan 29 15:29:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230973 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=cqoigw+G; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qY2PCBz9sRh for ; Thu, 30 Jan 2020 02:30:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727080AbgA2P37 (ORCPT ); Wed, 29 Jan 2020 10:29:59 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727069AbgA2P36 (ORCPT ); Wed, 29 Jan 2020 10:29:58 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id E073F475F3; Wed, 29 Jan 2020 15:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311795; x=1582126196; bh=DAqmEsLnVAqSAcoFGQUTwvRuQLh427NtVrL oUiLmN6E=; b=cqoigw+GJ2dT0rLaXnIFxCgCNjVfituGNfPfD+Ko2rxWSZHF87f nXiTu9lVyJhvjnIf32UHP3skwbMrAJbBFo7qW9lwSGvhW4RDkmyhY1oahXF94N/y mnMyjRu9TGLOifSLcpBrOJjjpi6YFsvWK4Zmrxe9wbdeiIh+o3edgAnY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JHhvAxiRfWc7; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 17E7847609; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:51 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 07/26] PCI: hotplug: Don't allow hot-added devices to steal resources Date: Wed, 29 Jan 2020 18:29:18 +0300 Message-ID: <20200129152937.311162-8-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When movable BARs are enabled, the PCI subsystem at first releases all the bridge windows and then attempts to assign resources both to previously working devices and to the newly hotplugged ones, with the same priority. If a hotplugged device gets its BARs first, this may lead to lack of space for already working devices, which is unacceptable. If that happens, mark one of the new devices with the newly introduced flag PCI_DEV_DISABLED_BARS (if it is not yet marked) and retry the BAR recalculation. The worst case would be no BARs for hotplugged devices, while all the rest just continue working. The algorithm is simple and it doesn't retry different subsets of hot-added devices in case of a failure, e.g. if there are no space to allocate BARs for both hotplugged devices A and B, but is enough for just A, the A will be marked with PCI_DEV_DISABLED_BARS first, then (after the next failure) - B. As a result, A will not get BARs while it could. This issue is only relevant when hotplugging two and more devices simultaneously. Add a new res_mask bitmask to the struct pci_dev for storing the indices of assigned BARs. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.h | 11 +++++ drivers/pci/probe.c | 102 ++++++++++++++++++++++++++++++++++++++-- drivers/pci/setup-bus.c | 15 ++++++ drivers/pci/setup-res.c | 3 ++ include/linux/pci.h | 1 + 5 files changed, 129 insertions(+), 3 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index fe995993b481..3b4c982772d3 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -406,6 +406,7 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 +#define PCI_DEV_DISABLED_BARS 1 static inline void pci_dev_assign_added(struct pci_dev *dev, bool added) { @@ -417,6 +418,16 @@ static inline bool pci_dev_is_added(const struct pci_dev *dev) return test_bit(PCI_DEV_ADDED, &dev->priv_flags); } +static inline void pci_dev_disable_bars(struct pci_dev *dev) +{ + assign_bit(PCI_DEV_DISABLED_BARS, &dev->priv_flags, true); +} + +static inline bool pci_dev_bars_enabled(const struct pci_dev *dev) +{ + return !test_bit(PCI_DEV_DISABLED_BARS, &dev->priv_flags); +} + #ifdef CONFIG_PCIEAER #include diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index daaaebdfd4c4..1e8bbf5c99f6 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3152,6 +3152,23 @@ bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res) return false; } +static unsigned int pci_dev_count_res_mask(struct pci_dev *dev) +{ + unsigned int res_mask = 0; + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + res_mask |= (1 << i); + } + + return res_mask; +} + static void pci_bus_rescan_prepare(struct pci_bus *bus) { struct pci_dev *dev; @@ -3162,6 +3179,8 @@ static void pci_bus_rescan_prepare(struct pci_bus *bus) list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child = dev->subordinate; + dev->res_mask = pci_dev_count_res_mask(dev); + if (child) pci_bus_rescan_prepare(child); @@ -3197,7 +3216,7 @@ static void pci_setup_bridges(struct pci_bus *bus) list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *child; - if (!pci_dev_is_added(dev)) + if (!pci_dev_is_added(dev) || !pci_dev_bars_enabled(dev)) continue; child = dev->subordinate; @@ -3209,6 +3228,83 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) +{ + struct pci_dev *dev; + + if (!bus) + return NULL; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child_bus = dev->subordinate; + + if (child_bus) { + struct pci_dev *next_new_dev; + + next_new_dev = pci_find_next_new_device(child_bus); + if (next_new_dev) + return next_new_dev; + } + + if (!pci_dev_is_added(dev) && pci_dev_bars_enabled(dev)) + return dev; + } + + return NULL; +} + +static bool pci_bus_check_all_bars_reassigned(struct pci_bus *bus) +{ + struct pci_dev *dev; + bool ret = true; + + if (!bus) + return false; + + list_for_each_entry(dev, &bus->devices, bus_list) { + struct pci_bus *child = dev->subordinate; + unsigned int res_mask = pci_dev_count_res_mask(dev); + + if (!pci_dev_bars_enabled(dev)) + continue; + + if (dev->res_mask & ~res_mask) { + pci_err(dev, "Non-re-enabled resources found: 0x%x -> 0x%x\n", + dev->res_mask, res_mask); + ret = false; + } + + if (child && !pci_bus_check_all_bars_reassigned(child)) + ret = false; + } + + return ret; +} + +static void pci_reassign_root_bus_resources(struct pci_bus *root) +{ + do { + struct pci_dev *next_new_dev; + + pci_assign_unassigned_root_bus_resources(root); + + if (pci_bus_check_all_bars_reassigned(root)) + break; + + next_new_dev = pci_find_next_new_device(root); + if (!next_new_dev) { + dev_err(&root->dev, "failed to re-assign resources even after ignoring all the hotplugged devices\n"); + break; + } + + dev_warn(&root->dev, "failed to re-assign resources, disable the next hotplugged device %s and retry\n", + dev_name(&next_new_dev->dev)); + + pci_dev_disable_bars(next_new_dev); + pci_bus_release_root_bridge_resources(root); + } while (true); +} + /** * pci_rescan_bus - Scan a PCI bus for devices * @bus: PCI bus to scan @@ -3228,11 +3324,11 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) if (pci_can_move_bars) { pci_bus_rescan_prepare(root); + pci_bus_release_root_bridge_resources(root); max = pci_scan_child_bus(root); - pci_bus_release_root_bridge_resources(root); - pci_assign_unassigned_root_bus_resources(root); + pci_reassign_root_bus_resources(root); pci_setup_bridges(root); pci_bus_rescan_done(root); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 7a9bf979908d..b9521ca8b24c 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -128,6 +128,9 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) { int i; + if (!pci_dev_bars_enabled(dev)) + return; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r; struct pci_dev_resource *dev_res, *tmp; @@ -177,6 +180,9 @@ static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) { u16 class = dev->class >> 8; + if (!pci_dev_bars_enabled(dev)) + return; + /* Don't touch classless devices or host bridges or IOAPICs */ if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) return; @@ -278,6 +284,9 @@ static void assign_requested_resources_sorted(struct list_head *head, int idx; list_for_each_entry(dev_res, head, list) { + if (!pci_dev_bars_enabled(dev_res->dev)) + continue; + res = dev_res->res; idx = res - &dev_res->dev->resource[0]; if (resource_size(res) && @@ -1012,6 +1021,9 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, list_for_each_entry(dev, &bus->devices, bus_list) { int i; + if (!pci_dev_bars_enabled(dev)) + continue; + for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct resource *r = &dev->resource[i]; resource_size_t r_size; @@ -1368,6 +1380,9 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, pbus_assign_resources_sorted(bus, realloc_head, fail_head); list_for_each_entry(dev, &bus->devices, bus_list) { + if (!pci_dev_bars_enabled(dev)) + continue; + pdev_assign_fixed_resources(dev); b = dev->subordinate; diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index d8ca40a97693..17c79057f517 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -458,6 +458,9 @@ int pci_enable_resources(struct pci_dev *dev, int mask) int i; struct resource *r; + if (!pci_dev_bars_enabled(dev)) + return -ENOSPC; + pci_read_config_word(dev, PCI_COMMAND, &cmd); old_cmd = cmd; diff --git a/include/linux/pci.h b/include/linux/pci.h index a543f647d337..a8f2c26757fe 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -375,6 +375,7 @@ struct pci_dev { */ unsigned int irq; struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */ + unsigned int res_mask; /* Bitmask of assigned resources */ bool match_driver; /* Skip attaching driver */ From patchwork Wed Jan 29 15:29:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230972 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=nxYSUhCt; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qX4tXwz9sRY for ; Thu, 30 Jan 2020 02:30:00 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727078AbgA2P36 (ORCPT ); Wed, 29 Jan 2020 10:29:58 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55788 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726913AbgA2P36 (ORCPT ); Wed, 29 Jan 2020 10:29:58 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B8F4647624; Wed, 29 Jan 2020 15:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311795; x=1582126196; bh=Tgz54u7diwIvb8dpIP5Ngql5sr+OK2jAKnY mPkeaoNM=; b=nxYSUhCtAVLUme49e0lXcJlNhlXtj1daLyaPmPeJ30kSanSqRZQ KH1cUfUO1VBKlLw0MPlGolX2ea5AMQMuV3uciC1a0xPbaFk3e+gx3PTIqOvCNKKG 399ck0r2A/gmrF9arLjsL/d1pHFx7NZHIG611JcVIUQO7Ce+l5khNg+M= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 863QO87jZGyP; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 2F831475F3; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:51 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 08/26] PCI: hotplug: Try to reassign movable BARs only once Date: Wed, 29 Jan 2020 18:29:19 +0300 Message-ID: <20200129152937.311162-9-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With enabled BAR movement, BARs and bridge windows can only be assigned to their direct parents, so there can be only one variant of resource tree, thus every retry within the pci_assign_unassigned_root_bus_resources() will result in the same tree, and it is enough to try just once. In case of failures the pci_reassign_root_bus_resources() disables BARs for one of the hotplugged devices and tries the assignment again. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index b9521ca8b24c..e87fefa12f62 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1827,6 +1827,13 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus) int pci_try_num = 1; enum enable_type enable_local; + if (pci_can_move_bars) { + __pci_bus_size_bridges(bus, NULL); + __pci_bus_assign_resources(bus, NULL, NULL); + + goto dump; + } + /* Don't realloc if asked to do so */ enable_local = pci_realloc_detect(bus, pci_realloc_enable); if (pci_realloc_enabled(enable_local)) { From patchwork Wed Jan 29 15:29:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230975 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=rpI444NS; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qZ4XTGz9sPK for ; Thu, 30 Jan 2020 02:30:02 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726913AbgA2PaB (ORCPT ); Wed, 29 Jan 2020 10:30:01 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55828 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727077AbgA2PaA (ORCPT ); Wed, 29 Jan 2020 10:30:00 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1779C4760F; Wed, 29 Jan 2020 15:29:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311796; x=1582126197; bh=FxfooPUZdX4osd6o3fhcnc5BoEyMI6zqcFl ZXQHVZwc=; b=rpI444NS4Dk/che48Z50MnWdrk4y/+xrwx2Uiym8EX7gOEDUrFD sKSMMsKElthCVsyEl/L36ve7v9HfZ9iF94QKSL/DoVEAuQwLZCku0TnnJBSzXTRa Ql+ubY0ysph1a/KCRGDO1ptmADHEdtQzCSz4G1TH00dhMPZuY/ESOrCU= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b6h_P5owj5dW; Wed, 29 Jan 2020 18:29:56 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 6ED5C4760D; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:51 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 09/26] PCI: hotplug: Calculate immovable parts of bridge windows Date: Wed, 29 Jan 2020 18:29:20 +0300 Message-ID: <20200129152937.311162-10-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When movable BARs are enabled, and if a bridge contains a device with fixed (IORESOURCE_PCI_FIXED) or immovable BARs, the corresponding windows can't be moved too far away from their original positions - they must still contain all the fixed/immovable BARs, like that: 1) Window position before a bus rescan: | <-- root bridge window --> | | | | | <-- bridge window --> | | | | movable BARs | **fixed BAR** | | ^^^^^^^^^^^^ 2) Possible valid outcome after rescan and move: | <-- root bridge window --> | | | | | <-- bridge window --> | | | | **fixed BAR** | Movable BARs | | ^^^^^^^^^^^^ An immovable area of a bridge window is a range that covers all the fixed and immovable BARs of direct children, and all the fixed area of children bridges: | <-- root bridge window --> | | | | | <-- bridge window level 1 --> | | | | ******************** immovable area ******************* | | | | | | | | **fixed BAR** | <-- bridge window level 2 --> | BARs | | | | | *********** immovable area *********** | | | | | | | | | | | | **fixed BAR** | BARs | **fixed BAR** | | | ^^^^ To store these areas, the .immovable_range field has been added to struct pci_bus for every bridge window type: IO, MEM and PREFETCH. It is filled recursively from leaves to the root before a rescan. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.h | 14 ++++++++ drivers/pci/probe.c | 88 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 6 ++++ 3 files changed, 108 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 3b4c982772d3..5f2051c8531c 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -404,6 +404,20 @@ static inline bool pci_dev_is_disconnected(const struct pci_dev *dev) return dev->error_state == pci_channel_io_perm_failure; } +static inline int pci_get_bridge_resource_idx(struct resource *r) +{ + int idx = 1; + + if (r->flags & IORESOURCE_IO) + idx = 0; + else if (!(r->flags & IORESOURCE_PREFETCH)) + idx = 1; + else if (r->flags & IORESOURCE_MEM_64) + idx = 2; + + return idx; +} + /* pci_dev priv_flags */ #define PCI_DEV_ADDED 0 #define PCI_DEV_DISABLED_BARS 1 diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 1e8bbf5c99f6..bb584038d5b4 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -546,6 +546,7 @@ void pci_read_bridge_bases(struct pci_bus *child) static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) { struct pci_bus *b; + int idx; b = kzalloc(sizeof(*b), GFP_KERNEL); if (!b) @@ -562,6 +563,11 @@ static struct pci_bus *pci_alloc_bus(struct pci_bus *parent) if (parent) b->domain_nr = parent->domain_nr; #endif + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + b->immovable_range[idx].start = 0; + b->immovable_range[idx].end = 0; + } + return b; } @@ -3228,6 +3234,87 @@ static void pci_setup_bridges(struct pci_bus *bus) pci_setup_bridge(bus); } +static void pci_bus_update_immovable_range(struct pci_bus *bus) +{ + struct pci_dev *dev; + int idx; + resource_size_t start, end; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + bus->immovable_range[idx].start = 0; + bus->immovable_range[idx].end = 0; + } + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_immovable_range(dev->subordinate); + + list_for_each_entry(dev, &bus->devices, bus_list) { + int i; + struct pci_bus *child = dev->subordinate; + + for (i = 0; i < PCI_BRIDGE_RESOURCES; ++i) { + struct resource *r = &dev->resource[i]; + + if (!r->flags || (r->flags & IORESOURCE_UNSET) || !r->parent) + continue; + + if (!pci_dev_bar_movable(dev, r)) { + idx = pci_get_bridge_resource_idx(r); + start = bus->immovable_range[idx].start; + end = bus->immovable_range[idx].end; + + if (!start || start > r->start) + start = r->start; + if (end < r->end) + end = r->end; + + if (bus->immovable_range[idx].start != start || + bus->immovable_range[idx].end != end) { + dev_dbg(&bus->dev, "Found fixed BAR%d 0x%llx-0x%llx in %s, expand the fixed bridge window %d to 0x%llx-0x%llx\n", + i, + (unsigned long long)r->start, + (unsigned long long)r->end, + dev_name(&dev->dev), idx, + (unsigned long long)start, + (unsigned long long)end); + bus->immovable_range[idx].start = start; + bus->immovable_range[idx].end = end; + } + } + } + + if (child) { + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + struct resource *child_immovable_range = + &child->immovable_range[idx]; + + if (child_immovable_range->start >= + child_immovable_range->end) + continue; + + start = bus->immovable_range[idx].start; + end = bus->immovable_range[idx].end; + + if (!start || start > child_immovable_range->start) + start = child_immovable_range->start; + if (end < child_immovable_range->end) + end = child_immovable_range->end; + + if (start < bus->immovable_range[idx].start || + end > bus->immovable_range[idx].end) { + dev_dbg(&bus->dev, "Expand the fixed bridge window %d from %s to 0x%llx-0x%llx\n", + idx, dev_name(&child->dev), + (unsigned long long)start, + (unsigned long long)end); + bus->immovable_range[idx].start = start; + bus->immovable_range[idx].end = end; + } + } + } + } +} + static struct pci_dev *pci_find_next_new_device(struct pci_bus *bus) { struct pci_dev *dev; @@ -3324,6 +3411,7 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) if (pci_can_move_bars) { pci_bus_rescan_prepare(root); + pci_bus_update_immovable_range(root); pci_bus_release_root_bridge_resources(root); max = pci_scan_child_bus(root); diff --git a/include/linux/pci.h b/include/linux/pci.h index a8f2c26757fe..d6eb498f7997 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -582,6 +582,12 @@ struct pci_bus { struct list_head resources; /* Address space routed to this bus */ struct resource busn_res; /* Bus numbers routed to this bus */ + /* + * If there are fixed or immovable resources in the bridge window, this range + * contains the lowest start address and highest end address of them. + */ + struct resource immovable_range[PCI_BRIDGE_RESOURCE_NUM]; + struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ void *sysdata; /* Hook for sys-specific extension */ From patchwork Wed Jan 29 15:29:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230976 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=BFcO7pGt; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qb1kW4z9sRY for ; Thu, 30 Jan 2020 02:30:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727088AbgA2PaB (ORCPT ); Wed, 29 Jan 2020 10:30:01 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726952AbgA2PaA (ORCPT ); Wed, 29 Jan 2020 10:30:00 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 00CC947625; Wed, 29 Jan 2020 15:29:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311796; x=1582126197; bh=bWnPaZUGtbi755uNVGa6pv8QekKJ1F945uz Qa4YKads=; b=BFcO7pGtHhDhpnYj12iWmNGEnNJgLpsLtC8ou+5xfIqQh5BUAM8 xL4lPmbIEpPJwM1GEfSPu0hUk+snWgaza/l2PshXKlK/0nU9lrDlg9+yBur91GA6 Sxvrcz3K7EqEQjKQwJpp3r+llVWSHaeimyZmyjVzZZDkrxf0GLCmvu3E= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kjgEAHxNDe3v; Wed, 29 Jan 2020 18:29:56 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 945E44760F; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:52 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 10/26] PCI: Include fixed and immovable BARs into the bus size calculating Date: Wed, 29 Jan 2020 18:29:21 +0300 Message-ID: <20200129152937.311162-11-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The only difference between the fixed/immovable and movable BARs is a size and offset preservation after they are released (the corresponding struct resource* detached from a bridge window for a while during a bus rescan). Include fixed/immovable BARs into result of pbus_size_mem(). Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index e87fefa12f62..5874c352b41e 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -891,6 +891,11 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size, resource_size_t children_add_size = 0; resource_size_t min_align, align; + resource_size_t fixed_start = bus->immovable_range[0].start; + resource_size_t fixed_end = bus->immovable_range[0].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + if (!b_res) return; @@ -898,6 +903,9 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size, if (b_res->parent) return; + if (min_size < fixed_size) + min_size = fixed_size; + min_align = window_alignment(bus, IORESOURCE_IO); list_for_each_entry(dev, &bus->devices, bus_list) { int i; @@ -1006,6 +1014,15 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, resource_size_t children_add_size = 0; resource_size_t children_add_align = 0; resource_size_t add_align = 0; + int idx = pci_get_bridge_resource_idx(b_res); + + resource_size_t fixed_start = bus->immovable_range[idx].start; + resource_size_t fixed_end = bus->immovable_range[idx].end; + resource_size_t fixed_size = (fixed_start < fixed_end) ? + (fixed_end - fixed_start + 1) : 0; + + if (min_size < fixed_size) + min_size = fixed_size; if (!b_res) return -ENOSPC; @@ -1028,12 +1045,22 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, struct resource *r = &dev->resource[i]; resource_size_t r_size; - if (r->parent || (r->flags & IORESOURCE_PCI_FIXED) || + if (r->parent || ((r->flags & mask) != type && (r->flags & mask) != type2 && (r->flags & mask) != type3)) continue; r_size = resource_size(r); + + if (pci_can_move_bars && !pci_dev_bar_movable(dev, r)) { + size += r_size; + + continue; + } else if (!pci_can_move_bars && + (r->flags & IORESOURCE_PCI_FIXED)) { + continue; + } + #ifdef CONFIG_PCI_IOV /* Put SRIOV requested res to the optional list */ if (realloc_head && i >= PCI_IOV_RESOURCES && From patchwork Wed Jan 29 15:29:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230974 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=SooZxLki; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qY721yz9sRk for ; Thu, 30 Jan 2020 02:30:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727082AbgA2PaA (ORCPT ); Wed, 29 Jan 2020 10:30:00 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55788 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726913AbgA2PaA (ORCPT ); Wed, 29 Jan 2020 10:30:00 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 3BCF34762A; Wed, 29 Jan 2020 15:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311798; x=1582126199; bh=hfrGP7ZuV7gZotJUWNFmK1pWrC8q21FdDJn WkhmZCp4=; b=SooZxLkirNPJsmv/J/eaBXlKuMPIRKkxD1/DGAxyKFdBoWYHlNr w4fxUkeiOMmHHNn2ax/vm/LRj8/WdTh8/KObqhQLnOJANdeGc1N15xjFvsQSPZ/D uTbSgSG6JJv5mNtJxFJ0AgLUDbGbcePqOqSBJPh/k/wCSqQAEuPMzBdM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xfoJSsSlqrZr; Wed, 29 Jan 2020 18:29:58 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id D037447603; Wed, 29 Jan 2020 18:29:52 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:52 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 11/26] PCI: hotplug: movable BARs: Compute limits for relocated bridge windows Date: Wed, 29 Jan 2020 18:29:22 +0300 Message-ID: <20200129152937.311162-12-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With enabled movable BARs, bridge windows are recalculated during each PCI rescan. Some of the BARs below a bridge may be immovable: these areas are represented by the .immovable_range field in struct pci_bus. If a bridge window size is equal to its immovable range, it can only be assigned to the start of this range. But if a bridge window size is larger, and this difference in size is denoted as "delta", the window can start from (immovable_range.start - delta) to (immovable_range.start), and it can end from (immovable_range.end) to (immovable_range.end + delta). This range (the new .realloc_range field in struct pci_bus) must then be compared with immovable ranges of neighbouring bridges to guarantee absence of intersections. This patch only calculates valid ranges for reallocated bridges during pci rescan, and the next one will make use of these values during allocation. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.h | 1 + drivers/pci/setup-bus.c | 86 +++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 6 +++ 3 files changed, 93 insertions(+) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 5f2051c8531c..ddd6fd33f9c3 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -291,6 +291,7 @@ struct pci_bus *pci_bus_get(struct pci_bus *bus); void pci_bus_put(struct pci_bus *bus); bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res); +void pci_bus_update_realloc_range(struct pci_bus *bus); /* PCIe link information */ #define PCIE_SPEED2STR(speed) \ diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 5874c352b41e..71babb968830 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1837,6 +1837,91 @@ static enum enable_type pci_realloc_detect(struct pci_bus *bus, } #endif +/* + * Calculate the address margins where the bridge windows may be allocated to fit all + * the fixed and immovable BARs beneath. + */ +void pci_bus_update_realloc_range(struct pci_bus *bus) +{ + struct pci_dev *dev; + struct pci_bus *parent = bus->parent; + int idx; + + list_for_each_entry(dev, &bus->devices, bus_list) + if (dev->subordinate) + pci_bus_update_realloc_range(dev->subordinate); + + if (!parent || !bus->self) + return; + + for (idx = 0; idx < PCI_BRIDGE_RESOURCE_NUM; ++idx) { + struct resource *immovable_range = &bus->immovable_range[idx]; + resource_size_t window_size = resource_size(bus->resource[idx]); + resource_size_t realloc_start, realloc_end; + + bus->realloc_range[idx].start = 0; + bus->realloc_range[idx].end = 0; + + /* Check if there any immovable BARs under the bridge */ + if (immovable_range->start >= immovable_range->end) + continue; + + /* The lowest possible address where the bridge window can start */ + realloc_start = immovable_range->end - window_size + 1; + /* The highest possible address where the bridge window can end */ + realloc_end = immovable_range->start + window_size - 1; + + if (realloc_start > immovable_range->start) + realloc_start = immovable_range->start; + + if (realloc_end < immovable_range->end) + realloc_end = immovable_range->end; + + /* + * Check that realloc range doesn't intersect with hard fixed ranges + * of neighboring bridges + */ + list_for_each_entry(dev, &parent->devices, bus_list) { + struct pci_bus *neighbor = dev->subordinate; + struct resource *n_imm_range; + + if (!neighbor) { + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; ++i) { + struct resource *nr = &dev->resource[i]; + + if (!nr->flags || + !nr->start || + pci_dev_bar_movable(dev, nr)) + continue; + + if (nr->end < immovable_range->start && + nr->end > realloc_start) + realloc_start = nr->end; + } + + continue; + } + + if (neighbor == bus) + continue; + + n_imm_range = &neighbor->immovable_range[idx]; + + if (n_imm_range->start >= n_imm_range->end) + continue; + + if (n_imm_range->end < immovable_range->start && + n_imm_range->end > realloc_start) + realloc_start = n_imm_range->end; + } + + bus->realloc_range[idx].start = realloc_start; + bus->realloc_range[idx].end = realloc_end; + } +} + /* * First try will not touch PCI bridge res. * Second and later try will clear small leaf bridge res. @@ -1856,6 +1941,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus) if (pci_can_move_bars) { __pci_bus_size_bridges(bus, NULL); + pci_bus_update_realloc_range(bus); __pci_bus_assign_resources(bus, NULL, NULL); goto dump; diff --git a/include/linux/pci.h b/include/linux/pci.h index d6eb498f7997..8830a254405f 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -588,6 +588,12 @@ struct pci_bus { */ struct resource immovable_range[PCI_BRIDGE_RESOURCE_NUM]; + /* + * Acceptable address range, where the bridge window may reside, considering its + * size, so it will cover all the fixed and immovable BARs below. + */ + struct resource realloc_range[PCI_BRIDGE_RESOURCE_NUM]; + struct pci_ops *ops; /* Configuration access functions */ struct msi_controller *msi; /* MSI controller */ void *sysdata; /* Hook for sys-specific extension */ From patchwork Wed Jan 29 15:29:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230978 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=lzAeexn5; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qc3Fqhz9sRh for ; Thu, 30 Jan 2020 02:30:04 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726952AbgA2PaC (ORCPT ); Wed, 29 Jan 2020 10:30:02 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727069AbgA2PaC (ORCPT ); Wed, 29 Jan 2020 10:30:02 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 5C86A47603; Wed, 29 Jan 2020 15:30:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311799; x=1582126200; bh=bmwN9fucNgOjss6M+KYp9Z9Ghaj3gIyZRpb zz/h1cq8=; b=lzAeexn5OMRtBgQdk1JNN+8Cli2iiUVucHTm73Z5HgYKCTWsJ1c zVPp95HdfMukTIxiTfFIQsVrZvS1h1gf3HnVmXBbzxVTWGHIkPF2SpmOBkj+/z90 V3cdxbqmb3WeXparFOrE6++c1hkZtjln7IjgWd5KJOk/mM4vseo3eNZs= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pdDiEaZEbkfT; Wed, 29 Jan 2020 18:29:59 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0EAFA4760C; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:52 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 12/26] PCI: Make sure bridge windows include their fixed BARs Date: Wed, 29 Jan 2020 18:29:23 +0300 Message-ID: <20200129152937.311162-13-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When choosing a start address for a bridge window, it should be not just a lowest possible address: this window must cover every underlying immovable BAR. The lowest address that satisfies this requirement is the .realloc_range field of struct pci_bus. After allocating a bridge window, validate that it covers all its immovable BARs: this range is put to the .immovable_range field of struct pci_bus. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/bus.c | 2 +- drivers/pci/setup-res.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 8e40b3e6da77..a1efa87e31b9 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -192,7 +192,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, * this is an already-configured bridge window, its start * overrides "min". */ - if (avail.start) + if (min_used < avail.start) min_used = avail.start; max = avail.end; diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 17c79057f517..3582ae68840b 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -248,9 +248,23 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, struct resource *res = dev->resource + resno; resource_size_t min; int ret; + resource_size_t start = (resource_size_t)-1; + resource_size_t end = 0; min = (res->flags & IORESOURCE_IO) ? PCIBIOS_MIN_IO : PCIBIOS_MIN_MEM; + if (pci_can_move_bars && dev->subordinate && resno >= PCI_BRIDGE_RESOURCES) { + struct pci_bus *child_bus = dev->subordinate; + int b_resno = resno - PCI_BRIDGE_RESOURCES; + struct resource *immovable_range = &child_bus->immovable_range[b_resno]; + + if (immovable_range->start < immovable_range->end) { + start = immovable_range->start; + end = immovable_range->end; + min = child_bus->realloc_range[b_resno].start; + } + } + /* * First, try exact prefetching match. Even if a 64-bit * prefetchable bridge window is below 4GB, we can't put a 32-bit @@ -262,7 +276,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH | IORESOURCE_MEM_64, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; /* * If the prefetchable window is only 32 bits wide, we can put @@ -274,7 +288,7 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, IORESOURCE_PREFETCH, pcibios_align_resource, dev); if (ret == 0) - return 0; + goto check_fixed; } /* @@ -287,6 +301,19 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, ret = pci_bus_alloc_resource(bus, res, size, align, min, 0, pcibios_align_resource, dev); +check_fixed: + if (pci_can_move_bars && ret == 0 && start < end) { + if (res->start > start || res->end < end) { + dev_err(&bus->dev, "fixed area 0x%llx-0x%llx for %s doesn't fit in the allocated %pR (0x%llx-0x%llx)", + (unsigned long long)start, (unsigned long long)end, + dev_name(&dev->dev), + res, (unsigned long long)res->start, + (unsigned long long)res->end); + release_resource(res); + return -1; + } + } + return ret; } From patchwork Wed Jan 29 15:29:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230979 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=g7WOw6jJ; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qd0ZWrz9sRk for ; Thu, 30 Jan 2020 02:30:05 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727091AbgA2PaD (ORCPT ); Wed, 29 Jan 2020 10:30:03 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727077AbgA2PaC (ORCPT ); Wed, 29 Jan 2020 10:30:02 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1E5B54762B; Wed, 29 Jan 2020 15:30:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311799; x=1582126200; bh=C47aqFnigDF5PDyzO6/Lpy6uwW+MgjBh1/M SfttIRBc=; b=g7WOw6jJIZtp+aSUnL011b4sCehSvbBah5AppzuJ5CnKmID6VuB tWzWaq1PVe8XN9ay/N/XDaO9+//l3S5vk4q1iDFbXoqWvBfAypOsI5lTg2/dUWkC YccmwyUxXDco4Rwr/D7nCtHHurMmeB/VlbWTX1e2dTDN4WGF2ZNUbx6o= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gtXl5HZZe5JH; Wed, 29 Jan 2020 18:29:59 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 3A39A47604; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:52 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 13/26] PCI: hotplug: Add support of immovable BARs to pci_assign_resource() Date: Wed, 29 Jan 2020 18:29:24 +0300 Message-ID: <20200129152937.311162-14-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The PCI_FIXED and other immovable BARs must be assigned after the bridge windows of parent bridge and before "usual" BARs, but currently they are assigned the last by the pdev_assign_fixed_resources(). Let the immovable BARs be handled by pci_assign_resource() in the same way as it does for movable ones, assigning them in correct order, unifying the code. Allow matching IORESOURCE_PCI_FIXED prefetchable BARs to non-prefetchable windows, so they follow the same rules as immovable BARs. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.h | 2 ++ drivers/pci/setup-bus.c | 25 ++++++++++++++++++------- drivers/pci/setup-res.c | 5 ++++- 3 files changed, 24 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index ddd6fd33f9c3..3ffbee2c3243 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -293,6 +293,8 @@ void pci_bus_put(struct pci_bus *bus); bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res); void pci_bus_update_realloc_range(struct pci_bus *bus); +int assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r); + /* PCIe link information */ #define PCIE_SPEED2STR(speed) \ ((speed) == PCIE_SPEED_16_0GT ? "16 GT/s" : \ diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 71babb968830..cdbf4afe1334 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1356,21 +1356,31 @@ void pci_bus_size_bridges(struct pci_bus *bus) } EXPORT_SYMBOL(pci_bus_size_bridges); -static void assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) +int assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r) { int i; struct resource *parent_r; - unsigned long mask = IORESOURCE_IO | IORESOURCE_MEM | - IORESOURCE_PREFETCH; + unsigned long mask = IORESOURCE_TYPE_BITS; pci_bus_for_each_resource(b, parent_r, i) { if (!parent_r) continue; - if ((r->flags & mask) == (parent_r->flags & mask) && - resource_contains(parent_r, r)) - request_resource(parent_r, r); + if ((r->flags & mask) != (parent_r->flags & mask)) + continue; + + if (parent_r->flags & IORESOURCE_PREFETCH && + !(r->flags & IORESOURCE_PREFETCH)) + continue; + + if (resource_contains(parent_r, r)) { + if (!request_resource(parent_r, r)) + return 0; + } } + + dev_err(&b->dev, "failed to assign immovable %pR\n", r); + return -EBUSY; } /* @@ -1410,7 +1420,8 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, if (!pci_dev_bars_enabled(dev)) continue; - pdev_assign_fixed_resources(dev); + if (!pci_can_move_bars) + pdev_assign_fixed_resources(dev); b = dev->subordinate; if (!b) diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 3582ae68840b..a7d81816d1ea 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -339,7 +339,10 @@ int pci_assign_resource(struct pci_dev *dev, int resno) resource_size_t align, size; int ret; - if (res->flags & IORESOURCE_PCI_FIXED) + if (pci_can_move_bars && !pci_dev_bar_movable(dev, res) && + resno < PCI_ROM_RESOURCE && res->start) + return assign_fixed_resource_on_bus(dev->bus, res); + else if (!pci_can_move_bars && res->flags & IORESOURCE_PCI_FIXED) return 0; res->flags |= IORESOURCE_UNSET; From patchwork Wed Jan 29 15:29:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230980 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=pVRfNt4y; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qd4qh0z9sPK for ; Thu, 30 Jan 2020 02:30:05 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727077AbgA2PaD (ORCPT ); Wed, 29 Jan 2020 10:30:03 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55828 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727089AbgA2PaC (ORCPT ); Wed, 29 Jan 2020 10:30:02 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 8F2B54760E; Wed, 29 Jan 2020 15:30:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311800; x=1582126201; bh=2yTzKg+TDE6N0Xk7XBl/HHmcLf+ovCl90d3 fnsUpweM=; b=pVRfNt4ytmK4hw21P9wgws4GrHMaJpGc7mkPLOu/PXx51/StZtW Eq2PVRzIGFaHVOU7VRZfzE7O1n1Edfgcq9RYYidtUF0qFAgWi46AmebZPO3Ig4TL y8H+ryUmWTgPLpEY4tA2xUDiwB5i6UvlVAYRbr1IYuk5wbmbbxuePtsU= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1v0Et-47D0TQ; Wed, 29 Jan 2020 18:30:00 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 6F96447611; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:52 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 14/26] PCI: hotplug: Sort immovable BARs before assignment Date: Wed, 29 Jan 2020 18:29:25 +0300 Message-ID: <20200129152937.311162-15-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Immovable BARs and bridge windows containing immovable BARs must be assigned before the "usual" ones. When assigning an immovable BAR/bridge window, its start address is chosen to be the lowest possible. To prevent conflicts, sort such resources based on the start address of their immovable areas. Add support of immovable BARs and bridge windows to pdev_sort_resources(). Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 66 +++++++++++++++++++++++++++++++++++------ 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index cdbf4afe1334..573d5c67c136 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -122,9 +122,9 @@ static resource_size_t get_res_add_align(struct list_head *head, return dev_res ? dev_res->min_align : 0; } - /* Sort resources by alignment */ -static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) +static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head, + struct list_head *head_immovables) { int i; @@ -136,17 +136,31 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) struct pci_dev_resource *dev_res, *tmp; resource_size_t r_align; struct list_head *n; + struct resource *fixed_res = NULL; r = &dev->resource[i]; - if (r->flags & IORESOURCE_PCI_FIXED) + if (!(r->flags) || r->parent) continue; - if (!(r->flags) || r->parent) + if (pci_can_move_bars) { + if (i >= PCI_BRIDGE_RESOURCES && + dev->subordinate) { + int idx = i - PCI_BRIDGE_RESOURCES; + + fixed_res = &dev->subordinate->immovable_range[idx]; + } else if (!pci_dev_bar_movable(dev, r)) { + fixed_res = r; + } + + if (fixed_res && fixed_res->start >= fixed_res->end) + fixed_res = NULL; + } else if (r->flags & IORESOURCE_PCI_FIXED) { continue; + } r_align = pci_resource_alignment(dev, r); - if (!r_align) { + if (!r_align && !fixed_res) { pci_warn(dev, "BAR %d: %pR has bogus alignment\n", i, r); continue; @@ -158,6 +172,30 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) tmp->res = r; tmp->dev = dev; + if (fixed_res) { + n = head_immovables; + list_for_each_entry(dev_res, head_immovables, list) { + struct resource *c_fixed_res = NULL; + int c_resno = dev_res->res - dev_res->dev->resource; + int br_idx = c_resno - PCI_BRIDGE_RESOURCES; + struct pci_bus *cbus = dev_res->dev->subordinate; + + if (br_idx >= 0) + c_fixed_res = &cbus->immovable_range[br_idx]; + else + c_fixed_res = dev_res->res; + + if (fixed_res->start < c_fixed_res->start) { + n = &dev_res->list; + break; + } + } + /* Insert it just before n */ + list_add_tail(&tmp->list, n); + + continue; + } + /* Fallback is smallest one or list is empty */ n = head; list_for_each_entry(dev_res, head, list) { @@ -176,7 +214,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) } } -static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) +static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head, + struct list_head *head_immovables) { u16 class = dev->class >> 8; @@ -195,7 +234,7 @@ static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) return; } - pdev_sort_resources(dev, head); + pdev_sort_resources(dev, head, head_immovables); } static inline void reset_resource(struct resource *res) @@ -493,8 +532,13 @@ static void pdev_assign_resources_sorted(struct pci_dev *dev, struct list_head *fail_head) { LIST_HEAD(head); + LIST_HEAD(head_immovables); + + __dev_sort_resources(dev, &head, &head_immovables); + + if (!list_empty(&head_immovables)) + __assign_resources_sorted(&head_immovables, NULL, NULL); - __dev_sort_resources(dev, &head); __assign_resources_sorted(&head, add_head, fail_head); } @@ -505,9 +549,13 @@ static void pbus_assign_resources_sorted(const struct pci_bus *bus, { struct pci_dev *dev; LIST_HEAD(head); + LIST_HEAD(head_immovables); list_for_each_entry(dev, &bus->devices, bus_list) - __dev_sort_resources(dev, &head); + __dev_sort_resources(dev, &head, &head_immovables); + + if (!list_empty(&head_immovables)) + __assign_resources_sorted(&head_immovables, NULL, NULL); __assign_resources_sorted(&head, realloc_head, fail_head); } From patchwork Wed Jan 29 15:29:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230981 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=jlYE6faL; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qf3dkMz9sRY for ; Thu, 30 Jan 2020 02:30:06 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727089AbgA2PaD (ORCPT ); Wed, 29 Jan 2020 10:30:03 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55864 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727090AbgA2PaD (ORCPT ); Wed, 29 Jan 2020 10:30:03 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 14DA847631; Wed, 29 Jan 2020 15:30:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311801; x=1582126202; bh=Dxswjjywe+xbcUcUpQxrsAnQF3ejBIGGFAC bdqzbCRM=; b=jlYE6faLTfcawA0GciYGayGwjbzkaEp8OjVHOMWtA7EYXkY0M1k e1qqCkpnNWZP55XcixG2gKcHFFXFkW/S3FQ16eeePPG8/JZmwE6lhQdUaoMWYSvk 2YCBFSsiBojtklxUtfhL26Xn8HCWh0GZ4wiaKzhg6OekGPEHwtzg83cM= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RgKd-x6DstPL; Wed, 29 Jan 2020 18:30:01 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id A34FF47614; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:53 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 15/26] PCI: hotplug: Enable the movable BARs feature by default Date: Wed, 29 Jan 2020 18:29:26 +0300 Message-ID: <20200129152937.311162-16-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This is the last patch in the series which implements the essentials of the movable BARs feature, so it is turned on by default now. Tested on Intel and AMD machines with "pci=pcie_bus_peer2peer" command line argument. In case of problems it is still can be overridden by the following command line option: pcie_movable_bars=off Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 6741c7f111ac..db8cfec2baff 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -79,7 +79,7 @@ static void pci_dev_d3_sleep(struct pci_dev *dev) int pci_domains_supported = 1; #endif -bool pci_can_move_bars; +bool pci_can_move_bars = true; #define DEFAULT_CARDBUS_IO_SIZE (256) #define DEFAULT_CARDBUS_MEM_SIZE (64*1024*1024) From patchwork Wed Jan 29 15:29:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230982 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=nJD6VJXD; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qg1R6mz9sRp for ; Thu, 30 Jan 2020 02:30:07 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727096AbgA2PaE (ORCPT ); Wed, 29 Jan 2020 10:30:04 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727069AbgA2PaE (ORCPT ); Wed, 29 Jan 2020 10:30:04 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 9C8B147634; Wed, 29 Jan 2020 15:30:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311801; x=1582126202; bh=pruCkVN12XWGTbusYnkiSrsH6okdpvdCkfg wa2pRilk=; b=nJD6VJXDwvk98z/j4ouf2ntlXcUYrSIQqxZqPHcJHWwvqXbNFOb +xr5SdfEDJDPIMQVu5tUlf2ufbUuMyqcznOOR7U306Ah0NRnluQDWIjI4u2fX5qX kzhLBJVDurI+0QHi7dL3B4wmZMyzyocWhiATEOUZfpfve5ElQfdDD85w= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2dT5f8D_y4fg; Wed, 29 Jan 2020 18:30:01 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id D663D47612; Wed, 29 Jan 2020 18:29:53 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:53 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 16/26] PCI: Ignore PCIBIOS_MIN_MEM Date: Wed, 29 Jan 2020 18:29:27 +0300 Message-ID: <20200129152937.311162-17-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org BARs and bridge windows are only allowed to be assigned to their parent bus's bridge windows, going up to the root complex's resources. So additional limitations on BAR address are not needed, and the PCIBIOS_MIN_MEM can be ignored. Besides, the value of PCIBIOS_MIN_MEM reported by the BIOS 1.3 on Supermicro H11SSL-i via e820__setup_pci_gap(): [mem 0xebff1000-0xfe9fffff] available for PCI devices is only suitable for a single RC out of four: pci_bus 0000:00: root bus resource [mem 0xec000000-0xefffffff window] pci_bus 0000:20: root bus resource [mem 0xeb800000-0xebefffff window] pci_bus 0000:40: root bus resource [mem 0xeb200000-0xeb5fffff window] pci_bus 0000:60: root bus resource [mem 0xe8b00000-0xeaffffff window] , which makes the AMD EPYC 7251 unable to boot with this movable BARs patchset. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-res.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index a7d81816d1ea..4043aab021dd 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -246,12 +246,13 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, int resno, resource_size_t size, resource_size_t align) { struct resource *res = dev->resource + resno; - resource_size_t min; + resource_size_t min = 0; int ret; resource_size_t start = (resource_size_t)-1; resource_size_t end = 0; - min = (res->flags & IORESOURCE_IO) ? PCIBIOS_MIN_IO : PCIBIOS_MIN_MEM; + if (!pci_can_move_bars) + min = (res->flags & IORESOURCE_IO) ? PCIBIOS_MIN_IO : PCIBIOS_MIN_MEM; if (pci_can_move_bars && dev->subordinate && resno >= PCI_BRIDGE_RESOURCES) { struct pci_bus *child_bus = dev->subordinate; From patchwork Wed Jan 29 15:29:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230983 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=Ai42t2NQ; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qg5Xsvz9sRh for ; Thu, 30 Jan 2020 02:30:07 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727099AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727093AbgA2PaE (ORCPT ); Wed, 29 Jan 2020 10:30:04 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1979147636; Wed, 29 Jan 2020 15:30:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311802; x=1582126203; bh=LePmfUy5jBiOSlUl6/64rkHu27lOHQ/BxZ3 oJp3VpYI=; b=Ai42t2NQbFXgWEnyuoUuiU87m9IF19qVcRcT1YdXN379jkNR5YQ ecMwg2ZTGVp5RurqHbe7+k2/e6E88XC/74xnn+9RtpnUL/PDXSQDyzTLdp3SNHpS VzY8HXYekC+qxSq5TWeZreUc1S6FY10mH4AmaNII7HKfJYKT8Y8Jcbok= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gX9YbeXGe1iu; Wed, 29 Jan 2020 18:30:02 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 123CE47613; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:53 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 17/26] PCI: hotplug: Ignore the MEM BAR offsets from BIOS/bootloader Date: Wed, 29 Jan 2020 18:29:28 +0300 Message-ID: <20200129152937.311162-18-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org BAR allocation by BIOS/UEFI/bootloader/firmware may be non-optimal and it may even clash with the kernel's BAR assignment algorithm. For example, sometimes BIOS doesn't reserve space for SR-IOV BARs, and this bridge window can neither extend (blocked by immovable BARs) nor move (the device itself is immovable). With this patch the kernel will use its own methods of BAR allocating when possible, increasing the chances of successful boot and hotplug. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/probe.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index bb584038d5b4..f8f643dac6d1 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -306,6 +306,14 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, pos, (unsigned long long)region.start); } + if (pci_can_move_bars && res->start && !(res->flags & IORESOURCE_IO)) { + pci_warn(dev, "ignore the current offset of BAR %llx-%llx\n", + l64, l64 + sz64 - 1); + res->start = 0; + res->end = sz64 - 1; + res->flags |= IORESOURCE_SIZEALIGN; + } + goto out; @@ -528,8 +536,10 @@ void pci_read_bridge_bases(struct pci_bus *child) child->resource[i] = &dev->resource[PCI_BRIDGE_RESOURCES+i]; pci_read_bridge_io(child); - pci_read_bridge_mmio(child); - pci_read_bridge_mmio_pref(child); + if (!pci_can_move_bars) { + pci_read_bridge_mmio(child); + pci_read_bridge_mmio_pref(child); + } if (dev->transparent) { pci_bus_for_each_resource(child->parent, res, i) { @@ -2945,6 +2955,8 @@ int pci_host_probe(struct pci_host_bridge *bridge) pci_bus_claim_resources(bus); } else { pci_bus_size_bridges(bus); + if (pci_can_move_bars) + pci_bus_update_realloc_range(bus); pci_bus_assign_resources(bus); list_for_each_entry(child, &bus->children, node) From patchwork Wed Jan 29 15:29:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230985 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=s4OZWPdW; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qh6nrbz9sRk for ; Thu, 30 Jan 2020 02:30:08 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727105AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55828 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727090AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B57D247614; Wed, 29 Jan 2020 15:30:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311802; x=1582126203; bh=cbU3IietN3svDQF6So1LqazSgtFc4F5LGyV nl7Okf+Q=; b=s4OZWPdWCISm6VC6fsLv/XGZ24jwsJBqKgwZ7k7UOVbqfBzZ8Wp jx3Aau3QwP7E0dfwgrU6kaaZMQtBNbynxGnO6wSS6+vnDtB+hPQbq1mXrXXwiff/ iLYqPufDTYr24jlCF984F/PWQsJVp9ckUflHdYlDjIkkOWlExJbQiADo= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wA2O9P8Tx_Zu; Wed, 29 Jan 2020 18:30:02 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 4025247615; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:53 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 18/26] PCI: Treat VGA BARs as immovable Date: Wed, 29 Jan 2020 18:29:29 +0300 Message-ID: <20200129152937.311162-19-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Some framebuffer drivers (efifb) don't act as a PCI driver (like nouveau), but take information about BARs indirectly - from BIOS for example. This makes them vulnerable to BAR movement during boot, when setting reported by BIOS/bootloader differs from new addresses assigned by the kernel. Until every such driver is aware of movable BARs, mark every VGA BAR as immovable. Perhaps this is also useful for splash screens, so they don't flicker. This makes some BARs and bridge windows immovable during boot, so update the parent's struct pci_bus->immovable_range when encountered. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/probe.c | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index f8f643dac6d1..b810d28ebf96 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -164,6 +164,20 @@ static inline unsigned long decode_bar(struct pci_dev *dev, u32 bar) #define PCI_COMMAND_DECODE_ENABLE (PCI_COMMAND_MEMORY | PCI_COMMAND_IO) +static void expand_immovable_range(struct pci_bus *bus, struct resource *res, int idx) +{ + struct resource *immovable_range = &bus->immovable_range[idx]; + + if (!immovable_range->start || immovable_range->start > res->start) + immovable_range->start = res->start; + + if (immovable_range->end < res->end) + immovable_range->end = res->end; + + if (bus->parent) + expand_immovable_range(bus->parent, immovable_range, idx); +} + /** * pci_read_base - Read a PCI BAR * @dev: the PCI device @@ -307,11 +321,16 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, } if (pci_can_move_bars && res->start && !(res->flags & IORESOURCE_IO)) { - pci_warn(dev, "ignore the current offset of BAR %llx-%llx\n", - l64, l64 + sz64 - 1); - res->start = 0; - res->end = sz64 - 1; - res->flags |= IORESOURCE_SIZEALIGN; + if ((dev->class >> 8) == PCI_CLASS_DISPLAY_VGA) { + expand_immovable_range(dev->bus, res, + pci_get_bridge_resource_idx(res)); + } else { + pci_warn(dev, "ignore the current offset of BAR %llx-%llx\n", + l64, l64 + sz64 - 1); + res->start = 0; + res->end = sz64 - 1; + res->flags |= IORESOURCE_SIZEALIGN; + } } goto out; @@ -3164,6 +3183,11 @@ bool pci_dev_bar_movable(struct pci_dev *dev, struct resource *res) if (region.start == 0xa0000) return false; + if (res->start && + !(res->flags & IORESOURCE_IO) && + (dev->class >> 8) == PCI_CLASS_DISPLAY_VGA) + return false; + if (!dev->driver && !res->child) return true; From patchwork Wed Jan 29 15:29:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230984 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=OdfbiGPp; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qh1tTkz9sRm for ; Thu, 30 Jan 2020 02:30:08 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727093AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55864 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727095AbgA2PaE (ORCPT ); Wed, 29 Jan 2020 10:30:04 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 181DA47638; Wed, 29 Jan 2020 15:30:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311803; x=1582126204; bh=OICdl3GBMIlvtb7uIInZqvKfC5xFfg9XVrz HEXhee6U=; b=OdfbiGPpH3LR1/Mgzz67hM4AwJavC/H2QWrZ03YiQYhFPjUzINI 8FYyqsR6GmhBEuFIm2GMmmkXQXzdKnhMxAa3CwknHmUDx4NvL0qDh50ghS749Tqg JFC3X1F/PLxt7H4YlQQf5oK/ebww45XpZ2d0FSLBxaLcwwivam4rPpLY= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bJqwttwmifqX; Wed, 29 Jan 2020 18:30:03 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 6CF1747606; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:53 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 19/26] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Date: Wed, 29 Jan 2020 18:29:30 +0300 Message-ID: <20200129152937.311162-20-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Assure that MPS settings are set up for bridges which are discovered during manually triggered rescan via sysfs. This sequence of bridge init (using pci_rescan_bus()) will be used for pciehp hot-add events when BARs are movable. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/probe.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index b810d28ebf96..5f809ddd72b4 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -3440,7 +3440,7 @@ static void pci_reassign_root_bus_resources(struct pci_bus *root) unsigned int pci_rescan_bus(struct pci_bus *bus) { unsigned int max; - struct pci_bus *root = bus; + struct pci_bus *root = bus, *child; while (!pci_is_root_bus(root)) root = root->parent; @@ -3461,6 +3461,9 @@ unsigned int pci_rescan_bus(struct pci_bus *bus) pci_assign_unassigned_bus_resources(bus); } + list_for_each_entry(child, &root->children, node) + pcie_bus_configure_settings(child); + pci_bus_add_devices(bus); return max; From patchwork Wed Jan 29 15:29:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230986 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=Wl5fEsm3; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qj4HrYz9sPK for ; Thu, 30 Jan 2020 02:30:09 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727090AbgA2PaG (ORCPT ); Wed, 29 Jan 2020 10:30:06 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727069AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id B997F47615; Wed, 29 Jan 2020 15:30:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311803; x=1582126204; bh=KIR0MC8rCtR1zPZojKGWoiMuTlUAufIezN8 /SU4Jg5Q=; b=Wl5fEsm3fMTOSCoZfQa4kn6YpNUakWRTG9okSYRfwGJ9uJz72Z5 xk7hC1LAYvhqHj2yWVgKj7bATi3v1dGzAb/ItBCCH7T6LEwbxcJ6PoH9kbXsKokj vz4Wq72HQatsMYCamaB/vMfztBPkJi+m44OojS46CjOeCLx6XtDxqBLA= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 93xQ11qWup73; Wed, 29 Jan 2020 18:30:03 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id A21EA47619; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:54 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko , "Rafael J . Wysocki" Subject: [PATCH v7 20/26] PNP: Don't reserve BARs for PCI when enabled movable BARs Date: Wed, 29 Jan 2020 18:29:31 +0300 Message-ID: <20200129152937.311162-21-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the Movable BARs feature is supported, the PCI subsystem is able to distribute existing BARs and allocate the new ones itself, without need to reserve gaps by BIOS. CC: Rafael J. Wysocki Signed-off-by: Sergei Miroshnichenko --- drivers/pnp/system.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pnp/system.c b/drivers/pnp/system.c index 6950503741eb..16cd260a609d 100644 --- a/drivers/pnp/system.c +++ b/drivers/pnp/system.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -58,6 +59,11 @@ static void reserve_resources_of_dev(struct pnp_dev *dev) struct resource *res; int i; +#ifdef CONFIG_PCI + if (pci_can_move_bars) + return; +#endif + for (i = 0; (res = pnp_get_resource(dev, IORESOURCE_IO, i)); i++) { if (res->flags & IORESOURCE_DISABLED) continue; From patchwork Wed Jan 29 15:29:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230987 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=vE43uwmk; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qk16qsz9sRY for ; Thu, 30 Jan 2020 02:30:10 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727069AbgA2PaG (ORCPT ); Wed, 29 Jan 2020 10:30:06 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727097AbgA2PaF (ORCPT ); Wed, 29 Jan 2020 10:30:05 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 1C9FB4763A; Wed, 29 Jan 2020 15:30:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311804; x=1582126205; bh=KmV3hNPn7rKXFMzBEaQhlsXjxbcPu4B8Qcn sWgcAriE=; b=vE43uwmkmDO1eg2FoQh6i2DvpOlmKI8sgGRa4xEYfbwTu9MACBr ZI5zV+i2eDC/84XFjdE98T1eqJbbppEEnw6v20ZWe8dfL5T7MqC3J6VL+feTfbro 1SSlNgppQ9ZPkDTc6vAcuXC1bRdeiIh+cmXoTR9PuUtf+sC/quqfPJlk= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iQ7Yf_K5I6yF; Wed, 29 Jan 2020 18:30:04 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id CE1124761D; Wed, 29 Jan 2020 18:29:54 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:54 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 21/26] PCI: hotplug: Don't disable the released bridge windows immediately Date: Wed, 29 Jan 2020 18:29:32 +0300 Message-ID: <20200129152937.311162-22-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On a hotplug event with enabled BAR movement, calculating the new bridge windows takes some time. During this procedure, the structures that represent these windows are "released" - means marked for recalculation by setting to zero. When the new bridge windows are ready, they are written to the registers of every bridge via pci_setup_bridges(). Currently, bridge's registers are updated immediately after releasing a window to disable it. But if a driver doesn't yet support movable BARs, it doesn't stop MEM transactions during the hotplug, so disabled bridge windows will break them. Let the bridge windows remain operating after releasing, as they will be updated to the new values in the end of a hotplug event. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 573d5c67c136..dd94b4ed7358 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1708,7 +1708,8 @@ static void pci_bridge_release_resources(struct pci_bus *bus, /* Avoiding touch the one without PREF */ if (type & IORESOURCE_PREFETCH) type = IORESOURCE_PREFETCH; - __pci_setup_bridge(bus, type); + if (!pci_can_move_bars) + __pci_setup_bridge(bus, type); /* For next child res under same bridge */ r->flags = old_flags; } From patchwork Wed Jan 29 15:29:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230988 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=KE+088j0; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qk5LLfz9sRp for ; Thu, 30 Jan 2020 02:30:10 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727097AbgA2PaG (ORCPT ); Wed, 29 Jan 2020 10:30:06 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55864 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727095AbgA2PaG (ORCPT ); Wed, 29 Jan 2020 10:30:06 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id BA11D47619; Wed, 29 Jan 2020 15:30:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311804; x=1582126205; bh=gDv30ZBnDuev6B5U+AY0bSdQ5WW45y7Ff6t zLvcRW4A=; b=KE+088j06DgckyoYv+3J+kue46iYFvCOmFxz/VtxzUXFR/yja0z EBvcsZQDht4P8eePoUX3SQdn+RrPn1r4JlOZCu2ideDvCd3B2cxU7Q1LEFQMz7h6 2cKwDmlEnbVl4BOAk58Upc7K9wGJZwbiZcz1Y08kHS6j3R5Shp3b2aMI= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VRy_W7xXAqdh; Wed, 29 Jan 2020 18:30:04 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 0F2EE4761E; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:54 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko , Lukas Wunner Subject: [PATCH v7 22/26] PCI: pciehp: Trigger a domain rescan on hp events when enabled movable BARs Date: Wed, 29 Jan 2020 18:29:33 +0300 Message-ID: <20200129152937.311162-23-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org With movable BARs, adding a hotplugged device is not local to its bridge anymore, but it affects the whole RC: BARs, bridge windows and bus numbers can be substantially rearranged. So instead of trying to fit the new devices into preallocated reserved gaps, initiate a full domain rescan. The pci_rescan_bus() covers all the operations of the replaced functions: - assigning new bus numbers, as the pci_hp_add_bridge() does it; - allocating BARs (pci_assign_unassigned_bridge_resources()); - cofiguring MPS settings (pcie_bus_configure_settings()); - binding devices to their drivers (pci_bus_add_devices()). CC: Lukas Wunner Signed-off-by: Sergei Miroshnichenko --- drivers/pci/hotplug/pciehp_pci.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/pci/hotplug/pciehp_pci.c b/drivers/pci/hotplug/pciehp_pci.c index d17f3bf36f70..6d4c1ef38210 100644 --- a/drivers/pci/hotplug/pciehp_pci.c +++ b/drivers/pci/hotplug/pciehp_pci.c @@ -58,6 +58,11 @@ int pciehp_configure_device(struct controller *ctrl) goto out; } + if (pci_can_move_bars) { + pci_rescan_bus(parent); + goto out; + } + for_each_pci_bridge(dev, parent) pci_hp_add_bridge(dev); From patchwork Wed Jan 29 15:29:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230989 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=srKso9Nj; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876ql2hpGz9sRh for ; Thu, 30 Jan 2020 02:30:11 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727095AbgA2PaH (ORCPT ); Wed, 29 Jan 2020 10:30:07 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727110AbgA2PaG (ORCPT ); Wed, 29 Jan 2020 10:30:06 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id 2712E4763E; Wed, 29 Jan 2020 15:30:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311805; x=1582126206; bh=riMEpjx5d73zNdTdzcKU9rJjNdZY1fOOG9n sS7WeBME=; b=srKso9NjyiRhKcKyDPp8NCS83gZ5XUzg39xqdjeZ0U1bkdtM+bb P98pLU/3+Fz24MZlUL0UFGQJi+Y5580UV958HoMTG0VraEOnGuGypup0seO1F4BM uH/Z/qCWEBEWkLMxZkkVDSAqAVMNYWsIYVH4v9lpQodyzBwy/MwqTXl4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZpwwAmECWCKF; Wed, 29 Jan 2020 18:30:05 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 417C547620; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:54 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 23/26] PCI: Don't claim immovable BARs Date: Wed, 29 Jan 2020 18:29:34 +0300 Message-ID: <20200129152937.311162-24-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Immovable BAR always has an address, but its parent bridge window can be not yet calculated (during boot) or temporarily released for re- calculation (during PCI rescan) when pci_claim_resource() is called. Apart from that, immovable BARs now have separate guaranteed mechanism of assigning comparing to usual BARs, so claiming them is not needed. Return immediately from pci_claim_resource() to prevent misleading "can't claim BAR ... no compatible bridge window" error messages Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-res.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 4043aab021dd..dc9b52f11922 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -138,6 +138,9 @@ int pci_claim_resource(struct pci_dev *dev, int resource) return -EINVAL; } + if (pci_can_move_bars && !pci_dev_bar_movable(dev, res)) + return 0; + /* * If we have a shadow copy in RAM, the PCI device doesn't respond * to the shadow range, so we don't need to claim it, and upstream From patchwork Wed Jan 29 15:29:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230990 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=Ohj1xCAg; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qm191kz9sRm for ; Thu, 30 Jan 2020 02:30:12 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727119AbgA2PaI (ORCPT ); Wed, 29 Jan 2020 10:30:08 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55828 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727112AbgA2PaI (ORCPT ); Wed, 29 Jan 2020 10:30:08 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id C0EA947640; Wed, 29 Jan 2020 15:30:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311805; x=1582126206; bh=diYC8NtFprW2FtiGXYlFHAb2ZsYOyKJFZPw rMvocfIA=; b=Ohj1xCAgK5XXZ0KMnpH6MMms9SfsaOLxuGdQam9QzNa5T9Gke+Z fNKD4750wqHYfjcduZ1YwiaNUxg7nu6BN/Bg0wOerdj/7tRO6uoneyEQkAmPEld0 Mq1iZddeZBQVs8E48IVix2moFK6dmJGUgs74xmJPDVdW0DrT+V3ZBtGg= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jGy6FVIm6Gxh; Wed, 29 Jan 2020 18:30:05 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id 723D047607; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:54 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 24/26] PCI: hotplug: Don't reserve bus space when enabled movable BARs Date: Wed, 29 Jan 2020 18:29:35 +0300 Message-ID: <20200129152937.311162-25-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org A hotplugged bridge with many hotplug-capable ports may request reserving more IO space than the machine has. This could be overridden with the "hpiosize=" kernel argument though. But when BARs are movable, no need to reserve space anymore: new BARs are allocated not from reserved gaps, but via rearranging the existing BARs. Requesting a precise amount of space for bridge windows increases the chances of adding the new bridge successfully. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/setup-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index dd94b4ed7358..3ec01a74b7b6 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1316,7 +1316,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) case PCI_HEADER_TYPE_BRIDGE: pci_bridge_check_ranges(bus); - if (bus->self->is_hotplug_bridge) { + if (bus->self->is_hotplug_bridge && !pci_can_move_bars) { additional_io_size = pci_hotplug_io_size; additional_mmio_size = pci_hotplug_mmio_size; additional_mmio_pref_size = pci_hotplug_mmio_pref_size; From patchwork Wed Jan 29 15:29:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230991 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=ivOD8oT0; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qm68Hqz9sRk for ; Thu, 30 Jan 2020 02:30:12 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726647AbgA2PaJ (ORCPT ); Wed, 29 Jan 2020 10:30:09 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55796 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727110AbgA2PaJ (ORCPT ); Wed, 29 Jan 2020 10:30:09 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id EF00547645; Wed, 29 Jan 2020 15:30:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311806; x=1582126207; bh=xeKBFhwWSTTwej2sxwygwuCy4WQRim+mRqk 7q7AVbHg=; b=ivOD8oT0wqkQfyf7JCqW7gE7dYEncKIpuN+hu/DT/2lENcSfQYc hGZVLStWZWiw2pKagoh1VdLmC9/KeVa4dk+f3/X7jsLpxHua3bGcng920eJG9bZH JyM3pDXD3WT4el8xczW7NcSy2DSTJOmtpQEP0u5bz9Ewy83p/SJt+GVo= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UfTyEeKG9cAF; Wed, 29 Jan 2020 18:30:06 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id A730F47621; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:55 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko , , Christoph Hellwig Subject: [PATCH v7 25/26] nvme-pci: Handle movable BARs Date: Wed, 29 Jan 2020 18:29:36 +0300 Message-ID: <20200129152937.311162-26-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hotplugged devices can affect the existing ones by moving their BARs. The PCI subsystem will inform the NVME driver about this by invoking the .rescan_prepare() and .rescan_done() hooks, so the BARs can by re-mapped. Tested under the "randrw" mode of the fio tool, and when using an NVME drive as a root filesystem storage. Before the hotplugging: % sudo cat /proc/iomem ... 3fe800000000-3fe8007fffff : PCI Bus 0020:0b 3fe800000000-3fe8007fffff : PCI Bus 0020:18 3fe800000000-3fe8000fffff : 0020:18:00.0 3fe800000000-3fe8000fffff : nvme ^^^^^^^^^^^^^^^^^^^^^^^^^ 3fe800100000-3fe80017ffff : 0020:18:00.0 ... Then another NVME drive was hot-added, so BARs of the 0020:18:00.0 are moved: % sudo cat /proc/iomem ... 3fe800000000-3fe800ffffff : PCI Bus 0020:0b 3fe800000000-3fe8007fffff : PCI Bus 0020:10 3fe800000000-3fe800003fff : 0020:10:00.0 3fe800000000-3fe800003fff : nvme 3fe800010000-3fe80001ffff : 0020:10:00.0 3fe800800000-3fe800ffffff : PCI Bus 0020:18 3fe800800000-3fe8008fffff : 0020:18:00.0 3fe800800000-3fe8008fffff : nvme ^^^^^^^^^^^^^^^^^^^^^^^^^ 3fe800900000-3fe80097ffff : 0020:18:00.0 ... During the rescanning, both READ and WRITE speeds drop to zero for a while due to driver's pause, then restore. Cc: linux-nvme@lists.infradead.org Cc: Christoph Hellwig Signed-off-by: Sergei Miroshnichenko --- drivers/nvme/host/pci.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 365a2ddbeaa7..42976c13d3f7 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1644,7 +1644,7 @@ static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size) { struct pci_dev *pdev = to_pci_dev(dev->dev); - if (size <= dev->bar_mapped_size) + if (dev->bar && size <= dev->bar_mapped_size) return 0; if (size > pci_resource_len(pdev, 0)) return -ENOMEM; @@ -3049,6 +3049,23 @@ static void nvme_error_resume(struct pci_dev *pdev) flush_work(&dev->ctrl.reset_work); } +static void nvme_rescan_prepare(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_disable(dev, false); + nvme_dev_unmap(dev); + dev->bar = NULL; +} + +static void nvme_rescan_done(struct pci_dev *pdev) +{ + struct nvme_dev *dev = pci_get_drvdata(pdev); + + nvme_dev_map(dev); + nvme_reset_ctrl_sync(&dev->ctrl); +} + static const struct pci_error_handlers nvme_err_handler = { .error_detected = nvme_error_detected, .slot_reset = nvme_slot_reset, @@ -3126,6 +3143,8 @@ static struct pci_driver nvme_driver = { #endif .sriov_configure = pci_sriov_configure_simple, .err_handler = &nvme_err_handler, + .rescan_prepare = nvme_rescan_prepare, + .rescan_done = nvme_rescan_done, }; static int __init nvme_init(void) From patchwork Wed Jan 29 15:29:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergei Miroshnichenko X-Patchwork-Id: 1230992 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yadro.com header.i=@yadro.com header.a=rsa-sha256 header.s=mta-01 header.b=jubS1uzI; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 4876qn3K9Cz9sPK for ; Thu, 30 Jan 2020 02:30:13 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726801AbgA2PaK (ORCPT ); Wed, 29 Jan 2020 10:30:10 -0500 Received: from mta-02.yadro.com ([89.207.88.252]:55810 "EHLO mta-01.yadro.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727112AbgA2PaK (ORCPT ); Wed, 29 Jan 2020 10:30:10 -0500 Received: from localhost (unknown [127.0.0.1]) by mta-01.yadro.com (Postfix) with ESMTP id CA9C24764A; Wed, 29 Jan 2020 15:30:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yadro.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from:received:received:received; s=mta-01; t= 1580311807; x=1582126208; bh=nTwQVNqKc1oZ/BG+7BiUIzTNNF70SzPZsJQ u1kL4W+I=; b=jubS1uzIZkFvGtcwhmM8/Y1bvbiTy5/dm25FTfOqI8R3Ego1Qlv 0BCSqLgeGOtAUgkMHxbNWvDQAl+jNL/XpT3YbFJmkna/B0Z4hrCRWWyAo+gdIBPX 9uVgJtOrjWymI3wy/z4vQcc0RH+Ss4/pL8FPBpWpdeEiz1AUDSuBscW4= X-Virus-Scanned: amavisd-new at yadro.com Received: from mta-01.yadro.com ([127.0.0.1]) by localhost (mta-01.yadro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LhPz2JvsOZeb; Wed, 29 Jan 2020 18:30:07 +0300 (MSK) Received: from T-EXCH-02.corp.yadro.com (t-exch-02.corp.yadro.com [172.17.10.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mta-01.yadro.com (Postfix) with ESMTPS id DE2BC47608; Wed, 29 Jan 2020 18:29:55 +0300 (MSK) Received: from NB-148.yadro.com (172.17.15.136) by T-EXCH-02.corp.yadro.com (172.17.10.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Wed, 29 Jan 2020 18:29:55 +0300 From: Sergei Miroshnichenko To: CC: Bjorn Helgaas , Stefan Roese , , Sergei Miroshnichenko Subject: [PATCH v7 26/26] PCI/portdrv: Declare support of movable BARs Date: Wed, 29 Jan 2020 18:29:37 +0300 Message-ID: <20200129152937.311162-27-s.miroshnichenko@yadro.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com> References: <20200129152937.311162-1-s.miroshnichenko@yadro.com> MIME-Version: 1.0 X-Originating-IP: [172.17.15.136] X-ClientProxiedBy: T-EXCH-01.corp.yadro.com (172.17.10.101) To T-EXCH-02.corp.yadro.com (172.17.10.102) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Currently there are no reliable way to determine if a driver uses BARs of its devices (their struct resource don't always have a child), so if it doesn't explicitly support movable BARs, they are considered immovable. The portdrv driver for PCI switches don't use BARs, so add empty hooks .rescan_prepare() and .rescan_done() to increase chances to allocate new BARs for new devices. Signed-off-by: Sergei Miroshnichenko --- drivers/pci/pcie/portdrv_pci.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c index 160d67c59310..df1faf2fed86 100644 --- a/drivers/pci/pcie/portdrv_pci.c +++ b/drivers/pci/pcie/portdrv_pci.c @@ -205,6 +205,14 @@ static const struct pci_error_handlers pcie_portdrv_err_handler = { .resume = pcie_portdrv_err_resume, }; +static void pcie_portdrv_rescan_prepare(struct pci_dev *pdev) +{ +} + +static void pcie_portdrv_rescan_done(struct pci_dev *pdev) +{ +} + static struct pci_driver pcie_portdriver = { .name = "pcieport", .id_table = &port_pci_ids[0], @@ -215,6 +223,9 @@ static struct pci_driver pcie_portdriver = { .err_handler = &pcie_portdrv_err_handler, + .rescan_prepare = pcie_portdrv_rescan_prepare, + .rescan_done = pcie_portdrv_rescan_done, + .driver.pm = PCIE_PORTDRV_PM_OPS, };