From patchwork Wed Aug 26 12:22:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Heimes X-Patchwork-Id: 1351912 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Bc4ky05Drz9sTR; Wed, 26 Aug 2020 22:23:06 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1kAuSE-0003Uu-T6; Wed, 26 Aug 2020 12:23:02 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSD-0003Uf-Lp for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:01 +0000 Received: from 2.general.fheimes.uk.vpn ([10.172.194.67] helo=T570.fritz.box) by youngberry.canonical.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSD-0008UU-CE for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:01 +0000 From: frank.heimes@canonical.com To: kernel-team@lists.ubuntu.com Subject: [SRU][F][G][PATCH 1/3] s390/pci: fix zpci_bus_link_virtfn() Date: Wed, 26 Aug 2020 14:22:28 +0200 Message-Id: <20200826122230.887710-2-frank.heimes@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200826122230.887710-1-frank.heimes@canonical.com> References: <20200826122230.887710-1-frank.heimes@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Niklas Schnelle BugLink: https://bugs.launchpad.net/bugs/1892849 We were missing the pci_dev_put() for candidate PFs. Furhtermore in discussion with upstream it turns out that somewhat counterintuitively some common code, in particular the vfio-pci driver, assumes that pdev->is_virtfn always implies that pdev->physfn is set, i.e. that VFs are always linked. While POWER does seem to set pdev->is_virtfn even for unlinked functions (see comments in arch/powerpc/kernel/eeh.c:eeh_debugfs_break_device()) for now just be safe and only set pdev->is_virtfn on linking. Also make sure that we only search for parent PFs if the zbus is multifunction and we thus know the devfn values supplied by firmware come from the RID. Fixes: e5794cf1a270 ("s390/pci: create links between PFs and VFs") Cc: # 5.8 Reviewed-by: Pierre Morel Signed-off-by: Niklas Schnelle Signed-off-by: Heiko Carstens (cherry picked from commit 3cddb79afc60bcdb5fd9dd7a1c64a8d03bdd460f) Signed-off-by: Frank Heimes --- arch/s390/pci/pci_bus.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c index 642a99384688..423af8b1da7b 100644 --- a/arch/s390/pci/pci_bus.c +++ b/arch/s390/pci/pci_bus.c @@ -132,13 +132,14 @@ static int zpci_bus_link_virtfn(struct pci_dev *pdev, { int rc; - virtfn->physfn = pci_dev_get(pdev); rc = pci_iov_sysfs_link(pdev, virtfn, vfid); - if (rc) { - pci_dev_put(pdev); - virtfn->physfn = NULL; + if (rc) return rc; - } + + virtfn->is_virtfn = 1; + virtfn->multifunction = 0; + virtfn->physfn = pci_dev_get(pdev); + return 0; } @@ -151,9 +152,9 @@ static int zpci_bus_setup_virtfn(struct zpci_bus *zbus, int vfid = vfn - 1; /* Linux' vfid's start at 0 vfn at 1*/ int rc = 0; - virtfn->is_virtfn = 1; - virtfn->multifunction = 0; - WARN_ON(vfid < 0); + if (!zbus->multifunction) + return 0; + /* If the parent PF for the given VF is also configured in the * instance, it must be on the same zbus. * We can then identify the parent PF by checking what @@ -165,11 +166,17 @@ static int zpci_bus_setup_virtfn(struct zpci_bus *zbus, zdev = zbus->function[i]; if (zdev && zdev->is_physfn) { pdev = pci_get_slot(zbus->bus, zdev->devfn); + if (!pdev) + continue; cand_devfn = pci_iov_virtfn_devfn(pdev, vfid); if (cand_devfn == virtfn->devfn) { rc = zpci_bus_link_virtfn(pdev, virtfn, vfid); + /* balance pci_get_slot() */ + pci_dev_put(pdev); break; } + /* balance pci_get_slot() */ + pci_dev_put(pdev); } } return rc; @@ -178,8 +185,6 @@ static int zpci_bus_setup_virtfn(struct zpci_bus *zbus, static inline int zpci_bus_setup_virtfn(struct zpci_bus *zbus, struct pci_dev *virtfn, int vfn) { - virtfn->is_virtfn = 1; - virtfn->multifunction = 0; return 0; } #endif From patchwork Wed Aug 26 12:22:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Heimes X-Patchwork-Id: 1351914 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Bc4l22H38z9sTX; Wed, 26 Aug 2020 22:23:10 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1kAuSJ-0003W7-2i; Wed, 26 Aug 2020 12:23:07 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSH-0003Vj-MG for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:05 +0000 Received: from 2.general.fheimes.uk.vpn ([10.172.194.67] helo=T570.fritz.box) by youngberry.canonical.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSH-0008UU-CY for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:05 +0000 From: frank.heimes@canonical.com To: kernel-team@lists.ubuntu.com Subject: [SRU][F][G][PATCH 2/3] s390/pci: re-introduce zpci_remove_device() Date: Wed, 26 Aug 2020 14:22:29 +0200 Message-Id: <20200826122230.887710-3-frank.heimes@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200826122230.887710-1-frank.heimes@canonical.com> References: <20200826122230.887710-1-frank.heimes@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Niklas Schnelle BugLink: https://bugs.launchpad.net/bugs/1892849 For fixing the PF to VF link removal we need to perform some action on every removal of a zdev from the common PCI subsystem. So in preparation re-introduce zpci_remove_device() and use that instead of directly calling the common code functions. This was actually still declared from earlier code but no longer implemented. Reviewed-by: Pierre Morel Signed-off-by: Niklas Schnelle Signed-off-by: Heiko Carstens (cherry picked from commit 2f0230b2f2d5fd287a85583eefb5aed35b6fe510) Signed-off-by: Frank Heimes --- arch/s390/pci/pci.c | 19 ++++++++++++------- arch/s390/pci/pci_event.c | 4 ++-- drivers/pci/hotplug/s390_pci_hpc.c | 12 +++++------- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index 3902c9f6f2d6..6e57ff885f8a 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -672,6 +672,16 @@ int zpci_disable_device(struct zpci_dev *zdev) } EXPORT_SYMBOL_GPL(zpci_disable_device); +void zpci_remove_device(struct zpci_dev *zdev) +{ + struct zpci_bus *zbus = zdev->zbus; + struct pci_dev *pdev; + + pdev = pci_get_slot(zbus->bus, zdev->devfn); + if (pdev) + pci_stop_and_remove_bus_device_locked(pdev); +} + int zpci_create_device(struct zpci_dev *zdev) { int rc; @@ -716,13 +726,8 @@ void zpci_release_device(struct kref *kref) { struct zpci_dev *zdev = container_of(kref, struct zpci_dev, kref); - if (zdev->zbus->bus) { - struct pci_dev *pdev; - - pdev = pci_get_slot(zdev->zbus->bus, zdev->devfn); - if (pdev) - pci_stop_and_remove_bus_device_locked(pdev); - } + if (zdev->zbus->bus) + zpci_remove_device(zdev); switch (zdev->state) { case ZPCI_FN_STATE_ONLINE: diff --git a/arch/s390/pci/pci_event.c b/arch/s390/pci/pci_event.c index fdebd286f402..538da3faa048 100644 --- a/arch/s390/pci/pci_event.c +++ b/arch/s390/pci/pci_event.c @@ -118,7 +118,7 @@ static void __zpci_event_availability(struct zpci_ccdf_avail *ccdf) if (!zdev) break; if (pdev) - pci_stop_and_remove_bus_device_locked(pdev); + zpci_remove_device(zdev); ret = zpci_disable_device(zdev); if (ret) @@ -137,7 +137,7 @@ static void __zpci_event_availability(struct zpci_ccdf_avail *ccdf) /* Give the driver a hint that the function is * already unusable. */ pdev->error_state = pci_channel_io_perm_failure; - pci_stop_and_remove_bus_device_locked(pdev); + zpci_remove_device(zdev); } zdev->state = ZPCI_FN_STATE_STANDBY; diff --git a/drivers/pci/hotplug/s390_pci_hpc.c b/drivers/pci/hotplug/s390_pci_hpc.c index b59f84918fe0..c9e790c74051 100644 --- a/drivers/pci/hotplug/s390_pci_hpc.c +++ b/drivers/pci/hotplug/s390_pci_hpc.c @@ -83,21 +83,19 @@ static int disable_slot(struct hotplug_slot *hotplug_slot) struct zpci_dev *zdev = container_of(hotplug_slot, struct zpci_dev, hotplug_slot); struct pci_dev *pdev; - struct zpci_bus *zbus = zdev->zbus; int rc; if (!zpci_fn_configured(zdev->state)) return -EIO; - pdev = pci_get_slot(zbus->bus, zdev->devfn); - if (pdev) { - if (pci_num_vf(pdev)) - return -EBUSY; - - pci_stop_and_remove_bus_device_locked(pdev); + pdev = pci_get_slot(zdev->zbus->bus, zdev->devfn); + if (pdev && pci_num_vf(pdev)) { pci_dev_put(pdev); + return -EBUSY; } + zpci_remove_device(zdev); + rc = zpci_disable_device(zdev); if (rc) return rc; From patchwork Wed Aug 26 12:22:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Heimes X-Patchwork-Id: 1351915 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Bc4l53yHtz9sTX; Wed, 26 Aug 2020 22:23:13 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1kAuSM-0003Xq-7R; Wed, 26 Aug 2020 12:23:10 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSK-0003X7-Py for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:08 +0000 Received: from 2.general.fheimes.uk.vpn ([10.172.194.67] helo=T570.fritz.box) by youngberry.canonical.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kAuSK-0008UU-HM for kernel-team@lists.ubuntu.com; Wed, 26 Aug 2020 12:23:08 +0000 From: frank.heimes@canonical.com To: kernel-team@lists.ubuntu.com Subject: [SRU][F][G][PATCH 3/3] s390/pci: fix PF/VF linking on hot plug Date: Wed, 26 Aug 2020 14:22:30 +0200 Message-Id: <20200826122230.887710-4-frank.heimes@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200826122230.887710-1-frank.heimes@canonical.com> References: <20200826122230.887710-1-frank.heimes@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Niklas Schnelle BugLink: https://bugs.launchpad.net/bugs/1892849 Currently there are four places in which a PCI function is scanned and made available to drivers: 1. In pci_scan_root_bus() as part of the initial zbus creation. 2. In zpci_bus_add_devices() when registering a device in configured state on a zbus that has already been scanned. 3. When a function is already known to zPCI (in reserved/standby state) and configuration is triggered through firmware by PEC 0x301. 4. When a device is already known to zPCI (in standby/reserved state) and configuration is triggered from within Linux using enable_slot(). The PF/VF linking step and setting of pdev->is_virtfn introduced with commit e5794cf1a270 ("s390/pci: create links between PFs and VFs") was only triggered for the second case, which is where VFs created through sriov_numvfs usually land. However unlike some other platforms but like POWER VFs can be individually enabled/disabled through /sys/bus/pci/slots. Fix this by doing VF setup as part of pcibios_bus_add_device() which is called in all of the above cases. Finally to remove the PF/VF links call the common code pci_iov_remove_virtfn() function to remove linked VFs. This takes care of the necessary sysfs cleanup. Fixes: e5794cf1a270 ("s390/pci: create links between PFs and VFs") Cc: # 5.8: 2f0230b2f2d5: s390/pci: re-introduce zpci_remove_device() Cc: # 5.8 Acked-by: Pierre Morel Signed-off-by: Niklas Schnelle Signed-off-by: Heiko Carstens (cherry picked from commit b97bf44f99155e57088e16974afb1f2d7b5287aa) Signed-off-by: Frank Heimes --- arch/s390/pci/pci.c | 5 ++++- arch/s390/pci/pci_bus.c | 27 +++++++++++++++------------ arch/s390/pci/pci_bus.h | 13 +++++++++++++ 3 files changed, 32 insertions(+), 13 deletions(-) diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index 6e57ff885f8a..4b62d6b55024 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -678,8 +678,11 @@ void zpci_remove_device(struct zpci_dev *zdev) struct pci_dev *pdev; pdev = pci_get_slot(zbus->bus, zdev->devfn); - if (pdev) + if (pdev) { + if (pdev->is_virtfn) + return zpci_remove_virtfn(pdev, zdev->vfn); pci_stop_and_remove_bus_device_locked(pdev); + } } int zpci_create_device(struct zpci_dev *zdev) diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c index 423af8b1da7b..5967f3014156 100644 --- a/arch/s390/pci/pci_bus.c +++ b/arch/s390/pci/pci_bus.c @@ -189,6 +189,19 @@ static inline int zpci_bus_setup_virtfn(struct zpci_bus *zbus, } #endif +void pcibios_bus_add_device(struct pci_dev *pdev) +{ + struct zpci_dev *zdev = to_zpci(pdev); + + /* + * With pdev->no_vf_scan the common PCI probing code does not + * perform PF/VF linking. + */ + if (zdev->vfn) + zpci_bus_setup_virtfn(zdev->zbus, pdev, zdev->vfn); + +} + static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev) { struct pci_bus *bus; @@ -219,20 +232,10 @@ static int zpci_bus_add_device(struct zpci_bus *zbus, struct zpci_dev *zdev) } pdev = pci_scan_single_device(bus, zdev->devfn); - if (pdev) { - if (!zdev->is_physfn) { - rc = zpci_bus_setup_virtfn(zbus, pdev, zdev->vfn); - if (rc) - goto failed_with_pdev; - } + if (pdev) pci_bus_add_device(pdev); - } - return 0; -failed_with_pdev: - pci_stop_and_remove_bus_device(pdev); - pci_dev_put(pdev); - return rc; + return 0; } static void zpci_bus_add_devices(struct zpci_bus *zbus) diff --git a/arch/s390/pci/pci_bus.h b/arch/s390/pci/pci_bus.h index 89be3c354b7b..4972433df458 100644 --- a/arch/s390/pci/pci_bus.h +++ b/arch/s390/pci/pci_bus.h @@ -29,3 +29,16 @@ static inline struct zpci_dev *get_zdev_by_bus(struct pci_bus *bus, return (devfn >= ZPCI_FUNCTIONS_PER_BUS) ? NULL : zbus->function[devfn]; } + +#ifdef CONFIG_PCI_IOV +static inline void zpci_remove_virtfn(struct pci_dev *pdev, int vfn) +{ + + pci_lock_rescan_remove(); + /* Linux' vfid's start at 0 vfn at 1 */ + pci_iov_remove_virtfn(pdev->physfn, vfn - 1); + pci_unlock_rescan_remove(); +} +#else /* CONFIG_PCI_IOV */ +static inline void zpci_remove_virtfn(struct pci_dev *pdev, int vfn) {} +#endif /* CONFIG_PCI_IOV */