From patchwork Fri Nov 20 22:51:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 1404189 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CdBdB6dS1z9sVH for ; Sat, 21 Nov 2020 09:52:14 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729072AbgKTWvx (ORCPT ); Fri, 20 Nov 2020 17:51:53 -0500 Received: from mga04.intel.com ([192.55.52.120]:15001 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728905AbgKTWvx (ORCPT ); Fri, 20 Nov 2020 17:51:53 -0500 IronPort-SDR: j7HHFGPrzctiqlqAwMBZBoWSQegDhz3FV5EbGgwuKnjfWVZCRwWgJqizAlvf+ERrvUXtszbthH raj9mO3R81SA== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="168985724" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="168985724" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:53 -0800 IronPort-SDR: KJWkTdg/0ZxrSljrgwAKFzta6h6KhK6aYrsu8/4RD+eV6zhTy2YBvLOtSoSpUIr/LqI85VCrVe LYU9RD/80nGA== X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="357852064" Received: from sabakhle-mobl1.amr.corp.intel.com (HELO jderrick-mobl.amr.corp.intel.com) ([10.213.165.80]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:52 -0800 From: Jon Derrick To: Lorenzo Pieralisi Cc: , Bjorn Helgaas , Nirmal Patel , Sushma Kalakota , Jon Derrick Subject: [PATCH 1/5] PCI: vmd: Reset the VMD subdevice domain on probe Date: Fri, 20 Nov 2020 15:51:40 -0700 Message-Id: <20201120225144.15138-2-jonathan.derrick@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201120225144.15138-1-jonathan.derrick@intel.com> References: <20201120225144.15138-1-jonathan.derrick@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The VMD subdevice domain resource requirements may have changed in-between module loads. Generic PCI resource assignment code may rely on existing resource configuration rather than the VMD preference of re-examining the domain. Add a Secondary Bus Reset to the VMD subdevice domain during driver attachment to clear the PCI config space of the subdevices. Signed-off-by: Jon Derrick --- drivers/pci/controller/vmd.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index c31e4d5..c7b5614 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -424,6 +425,36 @@ static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, .write = vmd_pci_write, }; +static void vmd_domain_reset_sbr(struct vmd_dev *vmd) +{ + char __iomem *base; + int rp; + u16 ctl; + + /* + * Subdevice config space is mapped linearly using 4k config space + * increments. Use increments of 0x8000 to locate root port devices. + */ + for (rp = 0; rp < 4; rp++) { + base = vmd->cfgbar + rp * 0x8000; + if (readl(base + PCI_COMMAND) == 0xFFFFFFFF) + continue; + + /* pci_reset_secondary_bus() */ + ctl = readw(base + PCI_BRIDGE_CONTROL); + ctl |= PCI_BRIDGE_CTL_BUS_RESET; + writew(ctl, base + PCI_BRIDGE_CONTROL); + readw(base + PCI_BRIDGE_CONTROL); + msleep(2); + + ctl &= ~PCI_BRIDGE_CTL_BUS_RESET; + writew(ctl, base + PCI_BRIDGE_CONTROL); + readw(base + PCI_BRIDGE_CONTROL); + } + + ssleep(1); +} + static void vmd_attach_resources(struct vmd_dev *vmd) { vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1]; @@ -707,6 +738,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) if (vmd->irq_domain) dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); + vmd_domain_reset_sbr(vmd); pci_scan_child_bus(vmd->bus); pci_assign_unassigned_bus_resources(vmd->bus); From patchwork Fri Nov 20 22:51:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 1404190 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CdBdC3N3Hz9sVJ for ; Sat, 21 Nov 2020 09:52:15 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729074AbgKTWvz (ORCPT ); Fri, 20 Nov 2020 17:51:55 -0500 Received: from mga04.intel.com ([192.55.52.120]:15001 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728905AbgKTWvz (ORCPT ); Fri, 20 Nov 2020 17:51:55 -0500 IronPort-SDR: jPx03cqxmX60jmhcjoX0l35s0DW5rzgas7+3sBkKSUxsThKooXTmuAsZimEAoNVJAM89rbNqAh GsEPlr+qrpNw== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="168985727" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="168985727" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:54 -0800 IronPort-SDR: SoAnqL7h9NPD//Gask9LNY0SxdkDVwHPl4yYMZAlTDs6jta9v57xjICgpGxMqMySShRI9Tc5b2 a/k4f0nV6DeA== X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="357852074" Received: from sabakhle-mobl1.amr.corp.intel.com (HELO jderrick-mobl.amr.corp.intel.com) ([10.213.165.80]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:53 -0800 From: Jon Derrick To: Lorenzo Pieralisi Cc: , Bjorn Helgaas , Nirmal Patel , Sushma Kalakota , Jon Derrick Subject: [PATCH 2/5] PCI: Add a reset quirk for VMD Date: Fri, 20 Nov 2020 15:51:41 -0700 Message-Id: <20201120225144.15138-3-jonathan.derrick@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201120225144.15138-1-jonathan.derrick@intel.com> References: <20201120225144.15138-1-jonathan.derrick@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org VMD domains should be reset in-between special attachment such as VFIO users. VMD does not offer a reset, however the subdevice domain itself can be reset starting at the Root Bus. Add a Secondary Bus Reset on each of the individual root port devices immediately downstream of the VMD root bus. Signed-off-by: Jon Derrick --- drivers/pci/quirks.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index f70692a..ee58b51 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3744,6 +3744,49 @@ static int reset_ivb_igd(struct pci_dev *dev, int probe) return 0; } +/* Issues SBR to VMD domain to clear PCI configuration */ +static int reset_vmd_sbr(struct pci_dev *dev, int probe) +{ + char __iomem *cfgbar, *base; + int rp; + u16 ctl; + + if (probe) + return 0; + + if (dev->dev.driver) + return 0; + + cfgbar = pci_iomap(dev, 0, 0); + if (!cfgbar) + return -ENOMEM; + + /* + * Subdevice config space is mapped linearly using 4k config space + * increments. Use increments of 0x8000 to locate root port devices. + */ + for (rp = 0; rp < 4; rp++) { + base = cfgbar + rp * 0x8000; + if (readl(base + PCI_COMMAND) == 0xFFFFFFFF) + continue; + + /* pci_reset_secondary_bus() */ + ctl = readw(base + PCI_BRIDGE_CONTROL); + ctl |= PCI_BRIDGE_CTL_BUS_RESET; + writew(ctl, base + PCI_BRIDGE_CONTROL); + readw(base + PCI_BRIDGE_CONTROL); + msleep(2); + + ctl &= ~PCI_BRIDGE_CTL_BUS_RESET; + writew(ctl, base + PCI_BRIDGE_CONTROL); + readw(base + PCI_BRIDGE_CONTROL); + } + + ssleep(1); + pci_iounmap(dev, cfgbar); + return 0; +} + /* Device-specific reset method for Chelsio T4-based adapters */ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe) { @@ -3919,6 +3962,11 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe) reset_ivb_igd }, { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_IVB_M2_VGA, reset_ivb_igd }, + { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D, reset_vmd_sbr }, + { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0, reset_vmd_sbr }, + { PCI_VENDOR_ID_INTEL, 0x467f, reset_vmd_sbr }, + { PCI_VENDOR_ID_INTEL, 0x4c3d, reset_vmd_sbr }, + { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_9A0B, reset_vmd_sbr }, { PCI_VENDOR_ID_SAMSUNG, 0xa804, nvme_disable_and_flr }, { PCI_VENDOR_ID_INTEL, 0x0953, delay_250ms_after_flr }, { PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID, From patchwork Fri Nov 20 22:51:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 1404191 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CdBdC6skzz9sSs for ; Sat, 21 Nov 2020 09:52:15 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729080AbgKTWvz (ORCPT ); Fri, 20 Nov 2020 17:51:55 -0500 Received: from mga04.intel.com ([192.55.52.120]:15001 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728905AbgKTWvz (ORCPT ); Fri, 20 Nov 2020 17:51:55 -0500 IronPort-SDR: svz5TYfK2lqqQUyJWlhE1aJ6tij6wbFuvwrw51QIAHVJ62uIwaprMGkNKMVwwo9UW/+mJwPF0Z I+bbTf+N/52Q== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="168985730" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="168985730" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:55 -0800 IronPort-SDR: W1X+h2sLTf2kjhDDC+E0ecB0wnF/NMZr6QP88Rn//fziVyZnMp8PQLICnYLqGqYnehR8uSBj8A jGFHhfEzTzlw== X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="357852085" Received: from sabakhle-mobl1.amr.corp.intel.com (HELO jderrick-mobl.amr.corp.intel.com) ([10.213.165.80]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:54 -0800 From: Jon Derrick To: Lorenzo Pieralisi Cc: , Bjorn Helgaas , Nirmal Patel , Sushma Kalakota , Jon Derrick Subject: [PATCH 3/5] PCI: vmd: Add offset translation helper Date: Fri, 20 Nov 2020 15:51:42 -0700 Message-Id: <20201120225144.15138-4-jonathan.derrick@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201120225144.15138-1-jonathan.derrick@intel.com> References: <20201120225144.15138-1-jonathan.derrick@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Adds a helper for translating physical addresses to bridge window offsets. No functional changes. Signed-off-by: Jon Derrick --- drivers/pci/controller/vmd.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index c7b5614..b0504ee 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -483,6 +483,16 @@ static int vmd_find_free_domain(void) return domain + 1; } +static void vmd_phys_to_offset(struct vmd_dev *vmd, u64 phys1, u64 phys2, + resource_size_t *offset1, + resource_size_t *offset2) +{ + *offset1 = vmd->dev->resource[VMD_MEMBAR1].start - + (phys1 & PCI_BASE_ADDRESS_MEM_MASK); + *offset2 = vmd->dev->resource[VMD_MEMBAR2].start - + (phys2 & PCI_BASE_ADDRESS_MEM_MASK); +} + static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, resource_size_t *offset1, resource_size_t *offset2) @@ -507,8 +517,8 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, phys1 = readq(membar2 + MB2_SHADOW_OFFSET); phys2 = readq(membar2 + MB2_SHADOW_OFFSET + 8); pci_iounmap(dev, membar2); - } else - return 0; + vmd_phys_to_offset(vmd, phys1, phys2, offset1, offset2); + } } else { /* Hypervisor-Emulated Vendor-Specific Capability */ int pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); @@ -525,15 +535,10 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, pci_read_config_dword(dev, pos + 16, ®); pci_read_config_dword(dev, pos + 20, ®u); phys2 = (u64) regu << 32 | reg; - } else - return 0; + vmd_phys_to_offset(vmd, phys1, phys2, offset1, offset2); + } } - *offset1 = dev->resource[VMD_MEMBAR1].start - - (phys1 & PCI_BASE_ADDRESS_MEM_MASK); - *offset2 = dev->resource[VMD_MEMBAR2].start - - (phys2 & PCI_BASE_ADDRESS_MEM_MASK); - return 0; } From patchwork Fri Nov 20 22:51:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 1404192 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CdBdD3CX5z9sVH for ; Sat, 21 Nov 2020 09:52:16 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728905AbgKTWv5 (ORCPT ); Fri, 20 Nov 2020 17:51:57 -0500 Received: from mga04.intel.com ([192.55.52.120]:15001 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728060AbgKTWv4 (ORCPT ); Fri, 20 Nov 2020 17:51:56 -0500 IronPort-SDR: h53o+rEr4ut3p+rWgG1FTXpL2jjiya+0dT6HM6naHrQWhD5b8LKm3j9G5RP59tDnxOHVY28WL8 rj0lgsBde/XA== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="168985733" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="168985733" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:56 -0800 IronPort-SDR: 9XXB3RMwjG/P05XUB/Ry6ekr+jyrbY04OyLBsP3ifRzdKEaKjb6ZzYwFc2flCd1uDlxrPX5VQx 0OEeAug8CGbQ== X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="357852100" Received: from sabakhle-mobl1.amr.corp.intel.com (HELO jderrick-mobl.amr.corp.intel.com) ([10.213.165.80]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:55 -0800 From: Jon Derrick To: Lorenzo Pieralisi Cc: , Bjorn Helgaas , Nirmal Patel , Sushma Kalakota , Jon Derrick Subject: [PATCH 4/5] PCI: vmd: Pass features to vmd_get_phys_offsets() Date: Fri, 20 Nov 2020 15:51:43 -0700 Message-Id: <20201120225144.15138-5-jonathan.derrick@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201120225144.15138-1-jonathan.derrick@intel.com> References: <20201120225144.15138-1-jonathan.derrick@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Modifies the physical address offset parser to use the device-specific features member. No functional changes. Signed-off-by: Jon Derrick --- drivers/pci/controller/vmd.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index b0504ee..71aa002 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -493,14 +493,14 @@ static void vmd_phys_to_offset(struct vmd_dev *vmd, u64 phys1, u64 phys2, (phys2 & PCI_BASE_ADDRESS_MEM_MASK); } -static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, +static int vmd_get_phys_offsets(struct vmd_dev *vmd, unsigned long features, resource_size_t *offset1, resource_size_t *offset2) { struct pci_dev *dev = vmd->dev; u64 phys1, phys2; - if (native_hint) { + if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) { u32 vmlock; int ret; @@ -519,7 +519,7 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, pci_iounmap(dev, membar2); vmd_phys_to_offset(vmd, phys1, phys2, offset1, offset2); } - } else { + } else if (features & VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP) { /* Hypervisor-Emulated Vendor-Specific Capability */ int pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); u32 reg, regu; @@ -638,16 +638,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * and child devices. These registers will either return the host value * or 0, depending on an enable bit in the VMD device. */ - if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) { + if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; - ret = vmd_get_phys_offsets(vmd, true, &offset[0], &offset[1]); - if (ret) - return ret; - } else if (features & VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP) { - ret = vmd_get_phys_offsets(vmd, false, &offset[0], &offset[1]); - if (ret) - return ret; - } + + ret = vmd_get_phys_offsets(vmd, features, &offset[0], &offset[1]); + if (ret) + return ret; /* * Certain VMD devices may have a root port configuration option which From patchwork Fri Nov 20 22:51:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 1404193 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CdBdD6fJXz9sSs for ; Sat, 21 Nov 2020 09:52:16 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729082AbgKTWv6 (ORCPT ); Fri, 20 Nov 2020 17:51:58 -0500 Received: from mga04.intel.com ([192.55.52.120]:15001 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728720AbgKTWv5 (ORCPT ); Fri, 20 Nov 2020 17:51:57 -0500 IronPort-SDR: zdgG7ANR0ViG3dQXMr4vu+XapDxgR1o/+deN02h5zTRAA9wCKe80bLV82gbP9oLFnnTGfZceaA VyftQ/8lu1/Q== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="168985737" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="168985737" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:57 -0800 IronPort-SDR: mV0Cu6xvUcP88u7sUFxqy/KRA/NGv8ikhRyiLo/RtntHQRLtxLubgJe1yX/AiQeDczj11yJlo3 urCvfbqKGAkA== X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="357852122" Received: from sabakhle-mobl1.amr.corp.intel.com (HELO jderrick-mobl.amr.corp.intel.com) ([10.213.165.80]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 14:51:56 -0800 From: Jon Derrick To: Lorenzo Pieralisi Cc: , Bjorn Helgaas , Nirmal Patel , Sushma Kalakota , Jon Derrick Subject: [PATCH 5/5] PCI: vmd: Add legacy guest passthrough mode Date: Fri, 20 Nov 2020 15:51:44 -0700 Message-Id: <20201120225144.15138-6-jonathan.derrick@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201120225144.15138-1-jonathan.derrick@intel.com> References: <20201120225144.15138-1-jonathan.derrick@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Some hypervisors allow passthrough of VMD to guests, but don't supply the emulated vendor-specific capability. VMD users currently passing-through VMD rely on a preconfiguration of the VMD Root Ports to inform the guest of the physical addresses for offset mapping the bridge windows. This patch adds a non-visible module parameter to activate host or guest passthrough mode. In host mode, this patch will write out the VMD MEMBAR information into the root ports on module unload. Guest mode will use the direct-assign hints, saving the host-supplied root port information on VMD module load and restore on exit. It uses this information in the offset calculation for bridge windows. This is enabled by non-visible module parameter because it is non-standard use case for certain users for a legacy behavior. Link: https://lore.kernel.org/linux-pci/20200706091625.GA26377@e121166-lin.cambridge.arm.com/ Signed-off-by: Sushma Kalakota Signed-off-by: Jon Derrick --- drivers/pci/controller/vmd.c | 127 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 126 insertions(+), 1 deletion(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 71aa002..711bbee 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -35,6 +35,19 @@ #define MB2_SHADOW_OFFSET 0x2000 #define MB2_SHADOW_SIZE 16 +enum legacy_da_mode { + VMD_DA_NONE, + VMD_DA_HOST, + VMD_DA_GUEST, +}; + +static int legacy_da_mode; +static char legacy_da_mode_str[sizeof("guest")]; +module_param_string(legacy_da_mode, legacy_da_mode_str, + sizeof(legacy_da_mode_str), 0); +MODULE_PARM_DESC(legacy_da_mode, + "use legacy host-provided addressing hints in Root Ports to assist guest passthrough (off, host, guest)"); + enum vmd_features { /* * Device may contain registers which hint the physical location of the @@ -97,6 +110,12 @@ struct vmd_irq_list { unsigned int count; }; +struct root_port_addr { + int port; + u64 membase; + u64 pref_membase; +}; + struct vmd_dev { struct pci_dev *dev; @@ -112,6 +131,7 @@ struct vmd_dev { struct pci_bus *bus; u8 busn_start; u8 first_vec; + struct root_port_addr rp_addr; }; static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus) @@ -483,6 +503,97 @@ static int vmd_find_free_domain(void) return domain + 1; } +#define VMD_RP_BASE(vmd, port) ((vmd)->cfgbar + (port) * 8 * 4096) +static void vmd_save_root_port_info(struct vmd_dev *vmd) +{ + resource_size_t physical = 0; + char __iomem *addr; + int port; + + if (upper_32_bits(pci_resource_start(vmd->dev, VMD_MEMBAR1))) + return; + + for (port = 0; port < 4; port++) { + u32 membase; + + addr = VMD_RP_BASE(vmd, port) + PCI_MEMORY_BASE; + membase = readl(addr); + + /* Break on first found root port */ + if ((membase != 0xffffffff) && (membase != 0) && + (membase != 0x0000fff0)) + break; + } + + if (port >= 4) + return; + + vmd->rp_addr.port = port; + + /* Only save the first root port index in host mode */ + if (legacy_da_mode == VMD_DA_HOST) + return; + + addr = VMD_RP_BASE(vmd, port) + PCI_MEMORY_BASE; + physical = ((u64)readw(addr) & 0xfff0) << 16; + vmd->rp_addr.membase = physical; + + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_BASE_UPPER32; + physical = ((u64)readl(addr)) << 32; + vmd->rp_addr.pref_membase = physical; + + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_MEMORY_BASE; + physical |= ((u64)readw(addr) & 0xfff0) << 16; + vmd->rp_addr.pref_membase |= physical; + + writel(0, VMD_RP_BASE(vmd, port) + PCI_MEMORY_BASE); + writel(0, VMD_RP_BASE(vmd, port) + PCI_PREF_BASE_UPPER32); + writel(0, VMD_RP_BASE(vmd, port) + PCI_PREF_MEMORY_BASE); + writel(0, VMD_RP_BASE(vmd, port) + PCI_PREF_MEMORY_LIMIT); +} + +static void vmd_restore_root_port_info(struct vmd_dev *vmd) +{ + resource_size_t phyaddr; + char __iomem *addr; + u32 val; + int port; + + port = vmd->rp_addr.port; + if (legacy_da_mode == VMD_DA_HOST) { + /* Write the MEMBAR information to prepare the guest */ + phyaddr = pci_resource_start(vmd->dev, VMD_MEMBAR1); + if (upper_32_bits(phyaddr)) + return; + + addr = VMD_RP_BASE(vmd, port) + PCI_MEMORY_BASE; + val = (phyaddr >> 16) & 0xfff0; + writew(val, addr); + + phyaddr = pci_resource_start(vmd->dev, VMD_MEMBAR2); + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_BASE_UPPER32; + val = phyaddr >> 32; + writel(val, addr); + + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_MEMORY_BASE; + val = (phyaddr >> 16) & 0xfff0; + writew(val, addr); + } else if (legacy_da_mode == VMD_DA_GUEST) { + /* Restore information provided by Host */ + addr = VMD_RP_BASE(vmd, port) + PCI_MEMORY_BASE; + val = (vmd->rp_addr.membase >> 16) & 0xfff0; + writew(val, addr); + + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_BASE_UPPER32; + val = vmd->rp_addr.pref_membase >> 32; + writel(val, addr); + + addr = VMD_RP_BASE(vmd, port) + PCI_PREF_MEMORY_BASE; + val = (vmd->rp_addr.pref_membase >> 16) & 0xfff0; + writew(val, addr); + } +} + static void vmd_phys_to_offset(struct vmd_dev *vmd, u64 phys1, u64 phys2, resource_size_t *offset1, resource_size_t *offset2) @@ -500,7 +611,19 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, unsigned long features, struct pci_dev *dev = vmd->dev; u64 phys1, phys2; - if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) { + if (!strncmp(legacy_da_mode_str, "host", 4)) + legacy_da_mode = VMD_DA_HOST; + else if (!strncmp(legacy_da_mode_str, "guest", 5)) + legacy_da_mode = VMD_DA_GUEST; + + if (legacy_da_mode != VMD_DA_NONE) { + vmd_save_root_port_info(vmd); + if (legacy_da_mode == VMD_DA_GUEST) { + vmd_phys_to_offset(vmd, vmd->rp_addr.membase, + vmd->rp_addr.pref_membase, + offset1, offset2); + } + } else if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) { u32 vmlock; int ret; @@ -732,6 +855,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) if (!vmd->bus) { pci_free_resource_list(&resources); vmd_remove_irq_domain(vmd); + vmd_restore_root_port_info(vmd); return -ENODEV; } @@ -821,6 +945,7 @@ static void vmd_remove(struct pci_dev *dev) vmd_cleanup_srcu(vmd); vmd_detach_resources(vmd); vmd_remove_irq_domain(vmd); + vmd_restore_root_port_info(vmd); } #ifdef CONFIG_PM_SLEEP