From patchwork Mon Dec 18 05:02:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 849788 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0TSb5DFjz9s83 for ; Mon, 18 Dec 2017 16:03:11 +1100 (AEDT) Received: from localhost ([::1]:56799 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQna0-0000m0-Vg for incoming@patchwork.ozlabs.org; Mon, 18 Dec 2017 00:03:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37795) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnZJ-0000kT-Sl for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQnZJ-0001TR-6d for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:25 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45700) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eQnZJ-0001Sm-0s for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:25 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 311F1356C5 for ; Mon, 18 Dec 2017 05:02:24 +0000 (UTC) Received: from gimli.home (ovpn-116-49.phx2.redhat.com [10.3.116.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id E47DC78401; Mon, 18 Dec 2017 05:02:23 +0000 (UTC) From: Alex Williamson To: qemu-devel@nongnu.org Date: Sun, 17 Dec 2017 22:02:23 -0700 Message-ID: <20171218050223.13478.39223.stgit@gimli.home> In-Reply-To: <20171218040852.13478.19208.stgit@gimli.home> References: <20171218040852.13478.19208.stgit@gimli.home> User-Agent: StGit/0.18 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 18 Dec 2017 05:02:24 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC/RFT PATCH 1/5] vfio/pci: Fixup VFIOMSIXInfo comment X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: eric.auger@redhat.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The fields were removed in the referenced commit, but the comment still mentions them. Fixes: 2fb9636ebf24 ("vfio-pci: Remove unused fields from VFIOMSIXInfo") Signed-off-by: Alex Williamson --- hw/vfio/pci.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index a8fb3b34222c..3d753222ca4c 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -86,7 +86,7 @@ enum { VFIO_INT_MSIX = 3, }; -/* Cache of MSI-X setup plus extra mmap and memory region for split BAR map */ +/* Cache of MSI-X setup */ typedef struct VFIOMSIXInfo { uint8_t table_bar; uint8_t pba_bar; From patchwork Mon Dec 18 05:02:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 849790 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0TW670Nfz9sBW for ; Mon, 18 Dec 2017 16:05:22 +1100 (AEDT) Received: from localhost ([::1]:56807 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnc9-0002Ws-3m for incoming@patchwork.ozlabs.org; Mon, 18 Dec 2017 00:05:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37815) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnZS-0000oB-4n for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQnZP-0001bF-1w for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:34 -0500 Received: from mx1.redhat.com ([209.132.183.28]:59292) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eQnZO-0001a3-PG for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:30 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DD30237E72 for ; Mon, 18 Dec 2017 05:02:29 +0000 (UTC) Received: from gimli.home (ovpn-116-49.phx2.redhat.com [10.3.116.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id 91B5978401; Mon, 18 Dec 2017 05:02:29 +0000 (UTC) From: Alex Williamson To: qemu-devel@nongnu.org Date: Sun, 17 Dec 2017 22:02:29 -0700 Message-ID: <20171218050229.13478.65667.stgit@gimli.home> In-Reply-To: <20171218040852.13478.19208.stgit@gimli.home> References: <20171218040852.13478.19208.stgit@gimli.home> User-Agent: StGit/0.18 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 18 Dec 2017 05:02:29 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC/RFT PATCH 2/5] vfio/pci: Add base BAR MemoryRegion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: eric.auger@redhat.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add one more layer to our stack of MemoryRegions, this base region allows us to register BARs independently of the vfio region or to extend the size of BARs which do map to a region. This will be useful when we want hypervisor defined BARs or sections of BARs, for purposes such as relocating MSI-X emulation. We therefore call msix_init() based on this new base MemoryRegion, while the quirks, which only modify regions still operate on those sub-MemoryRegions. Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++------------- hw/vfio/pci.h | 3 ++ 2 files changed, 60 insertions(+), 17 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c977ee327f94..8f46fdd1d391 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1440,9 +1440,9 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) vdev->msix->pending = g_malloc0(BITS_TO_LONGS(vdev->msix->entries) * sizeof(unsigned long)); ret = msix_init(&vdev->pdev, vdev->msix->entries, - vdev->bars[vdev->msix->table_bar].region.mem, + vdev->bars[vdev->msix->table_bar].mr, vdev->msix->table_bar, vdev->msix->table_offset, - vdev->bars[vdev->msix->pba_bar].region.mem, + vdev->bars[vdev->msix->pba_bar].mr, vdev->msix->pba_bar, vdev->msix->pba_offset, pos, &err); if (ret < 0) { @@ -1482,8 +1482,8 @@ static void vfio_teardown_msi(VFIOPCIDevice *vdev) if (vdev->msix) { msix_uninit(&vdev->pdev, - vdev->bars[vdev->msix->table_bar].region.mem, - vdev->bars[vdev->msix->pba_bar].region.mem); + vdev->bars[vdev->msix->table_bar].mr, + vdev->bars[vdev->msix->pba_bar].mr); g_free(vdev->msix->pending); } } @@ -1500,12 +1500,11 @@ static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled) } } -static void vfio_bar_setup(VFIOPCIDevice *vdev, int nr) +static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) { VFIOBAR *bar = &vdev->bars[nr]; uint32_t pci_bar; - uint8_t type; int ret; /* Skip both unimplemented BARs and the upper half of 64bit BARS. */ @@ -1524,23 +1523,52 @@ static void vfio_bar_setup(VFIOPCIDevice *vdev, int nr) pci_bar = le32_to_cpu(pci_bar); bar->ioport = (pci_bar & PCI_BASE_ADDRESS_SPACE_IO); bar->mem64 = bar->ioport ? 0 : (pci_bar & PCI_BASE_ADDRESS_MEM_TYPE_64); - type = pci_bar & (bar->ioport ? ~PCI_BASE_ADDRESS_IO_MASK : - ~PCI_BASE_ADDRESS_MEM_MASK); + bar->type = pci_bar & (bar->ioport ? ~PCI_BASE_ADDRESS_IO_MASK : + ~PCI_BASE_ADDRESS_MEM_MASK); + bar->size = bar->region.size; +} - if (vfio_region_mmap(&bar->region)) { - error_report("Failed to mmap %s BAR %d. Performance may be slow", - vdev->vbasedev.name, nr); +static void vfio_bars_prepare(VFIOPCIDevice *vdev) +{ + int i; + + for (i = 0; i < PCI_ROM_SLOT; i++) { + vfio_bar_prepare(vdev, i); } +} - pci_register_bar(&vdev->pdev, nr, type, bar->region.mem); +static void vfio_bar_register(VFIOPCIDevice *vdev, int nr) +{ + VFIOBAR *bar = &vdev->bars[nr]; + char *name; + + if (!bar->size) { + return; + } + + bar->mr = g_new0(MemoryRegion, 1); + name = g_strdup_printf("%s base BAR %d", vdev->vbasedev.name, nr); + memory_region_init_io(bar->mr, OBJECT(vdev), NULL, NULL, name, bar->size); + g_free(name); + + if (bar->region.size) { + memory_region_add_subregion(bar->mr, 0, bar->region.mem); + + if (vfio_region_mmap(&bar->region)) { + error_report("Failed to mmap %s BAR %d. Performance may be slow", + vdev->vbasedev.name, nr); + } + } + + pci_register_bar(&vdev->pdev, nr, bar->type, bar->mr); } -static void vfio_bars_setup(VFIOPCIDevice *vdev) +static void vfio_bars_register(VFIOPCIDevice *vdev) { int i; for (i = 0; i < PCI_ROM_SLOT; i++) { - vfio_bar_setup(vdev, i); + vfio_bar_register(vdev, i); } } @@ -1549,8 +1577,13 @@ static void vfio_bars_exit(VFIOPCIDevice *vdev) int i; for (i = 0; i < PCI_ROM_SLOT; i++) { + VFIOBAR *bar = &vdev->bars[i]; + vfio_bar_quirk_exit(vdev, i); - vfio_region_exit(&vdev->bars[i].region); + vfio_region_exit(&bar->region); + if (bar->region.size) { + memory_region_del_subregion(bar->mr, bar->region.mem); + } } if (vdev->vga) { @@ -1564,8 +1597,14 @@ static void vfio_bars_finalize(VFIOPCIDevice *vdev) int i; for (i = 0; i < PCI_ROM_SLOT; i++) { + VFIOBAR *bar = &vdev->bars[i]; + vfio_bar_quirk_finalize(vdev, i); - vfio_region_finalize(&vdev->bars[i].region); + vfio_region_finalize(&bar->region); + if (bar->size) { + object_unparent(OBJECT(bar->mr)); + g_free(bar->mr); + } } if (vdev->vga) { @@ -2810,7 +2849,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) goto error; } - vfio_bars_setup(vdev); + vfio_bars_prepare(vdev); + vfio_bars_register(vdev); ret = vfio_add_capabilities(vdev, errp); if (ret) { diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 3d753222ca4c..dcdb1a806769 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -33,6 +33,9 @@ typedef struct VFIOQuirk { typedef struct VFIOBAR { VFIORegion region; + MemoryRegion *mr; + size_t size; + uint8_t type; bool ioport; bool mem64; QLIST_HEAD(, VFIOQuirk) quirks; From patchwork Mon Dec 18 05:02:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 849789 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0TTC3GZBz9sCZ for ; Mon, 18 Dec 2017 16:03:43 +1100 (AEDT) Received: from localhost ([::1]:56801 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnaW-0001CQ-71 for incoming@patchwork.ozlabs.org; Mon, 18 Dec 2017 00:03:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnZb-0000wC-CL for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQnZY-0001jq-5f for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:43 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49054) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eQnZX-0001j3-Vs for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:40 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 10B1D85540 for ; Mon, 18 Dec 2017 05:02:39 +0000 (UTC) Received: from gimli.home (ovpn-116-49.phx2.redhat.com [10.3.116.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id 10B0C18ED2; Mon, 18 Dec 2017 05:02:35 +0000 (UTC) From: Alex Williamson To: qemu-devel@nongnu.org Date: Sun, 17 Dec 2017 22:02:35 -0700 Message-ID: <20171218050234.13478.35203.stgit@gimli.home> In-Reply-To: <20171218040852.13478.19208.stgit@gimli.home> References: <20171218040852.13478.19208.stgit@gimli.home> User-Agent: StGit/0.18 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 18 Dec 2017 05:02:39 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC/RFT PATCH 3/5] vfio/pci: Emulate BARs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: eric.auger@redhat.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The kernel provides similar emulation of PCI BAR register access to QEMU, so up until now we've used that for things like BAR sizing and storing the BAR address. However, if we intend to resize BARs or add BARs that don't exist on the physical device, we need to switch to the pure QEMU emulation of the BAR. Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 8f46fdd1d391..c383b842da20 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2773,6 +2773,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) /* QEMU can choose to expose the ROM or not */ memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); + /* QEMU can also add or extend BARs */ + memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); /* * The PCI spec reserves vendor ID 0xffff as an invalid value. The From patchwork Mon Dec 18 05:02:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 849792 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0TXx3DP1z9s4s for ; Mon, 18 Dec 2017 16:06:56 +1100 (AEDT) Received: from localhost ([::1]:56820 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQndd-0003s9-Np for incoming@patchwork.ozlabs.org; Mon, 18 Dec 2017 00:06:53 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37972) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnZk-00014B-VN for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQnZh-0001wU-St for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:53 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49070) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eQnZh-0001vN-K5 for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:02:49 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BC59C85540 for ; Mon, 18 Dec 2017 05:02:48 +0000 (UTC) Received: from gimli.home (ovpn-116-49.phx2.redhat.com [10.3.116.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id 99DBC6758A; Mon, 18 Dec 2017 05:02:44 +0000 (UTC) From: Alex Williamson To: qemu-devel@nongnu.org Date: Sun, 17 Dec 2017 22:02:44 -0700 Message-ID: <20171218050244.13478.35553.stgit@gimli.home> In-Reply-To: <20171218040852.13478.19208.stgit@gimli.home> References: <20171218040852.13478.19208.stgit@gimli.home> User-Agent: StGit/0.18 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 18 Dec 2017 05:02:48 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC/RFT PATCH 4/5] qapi: Create DEFINE_PROP_OFF_AUTO_PCIBAR X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: eric.auger@redhat.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add an option which allows the user to specify a PCI BAR number, including an 'off' and 'auto' selection. Signed-off-by: Alex Williamson --- hw/core/qdev-properties.c | 11 +++++++++++ include/hw/qdev-properties.h | 4 ++++ qapi/common.json | 26 ++++++++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/hw/core/qdev-properties.c b/hw/core/qdev-properties.c index 1dc80fcea2af..e33184e5a342 100644 --- a/hw/core/qdev-properties.c +++ b/hw/core/qdev-properties.c @@ -1256,3 +1256,14 @@ const PropertyInfo qdev_prop_link = { .name = "link", .create = create_link_property, }; + +/* --- OffAutoPCIBAR off/auto/bar0/bar1/bar2/bar3/bar4/bar5 --- */ + +const PropertyInfo qdev_prop_off_auto_pcibar = { + .name = "OffAutoPCIBAR", + .description = "off/auto/bar0/bar1/bar2/bar3/bar4/bar5", + .enum_table = &OffAutoPCIBAR_lookup, + .get = get_enum, + .set = set_enum, + .set_default_value = set_default_value_enum, +}; diff --git a/include/hw/qdev-properties.h b/include/hw/qdev-properties.h index e2321f1cc1ec..9ca2af5e3625 100644 --- a/include/hw/qdev-properties.h +++ b/include/hw/qdev-properties.h @@ -32,6 +32,7 @@ extern const PropertyInfo qdev_prop_blocksize; extern const PropertyInfo qdev_prop_pci_host_devaddr; extern const PropertyInfo qdev_prop_arraylen; extern const PropertyInfo qdev_prop_link; +extern const PropertyInfo qdev_prop_off_auto_pcibar; #define DEFINE_PROP(_name, _state, _field, _prop, _type) { \ .name = (_name), \ @@ -212,6 +213,9 @@ extern const PropertyInfo qdev_prop_link; DEFINE_PROP(_n, _s, _f, qdev_prop_pci_host_devaddr, PCIHostDeviceAddress) #define DEFINE_PROP_MEMORY_REGION(_n, _s, _f) \ DEFINE_PROP(_n, _s, _f, qdev_prop_ptr, MemoryRegion *) +#define DEFINE_PROP_OFF_AUTO_PCIBAR(_n, _s, _f, _d) \ + DEFINE_PROP_SIGNED(_n, _s, _f, _d, qdev_prop_off_auto_pcibar, \ + OffAutoPCIBAR) #define DEFINE_PROP_END_OF_LIST() \ {} diff --git a/qapi/common.json b/qapi/common.json index 6eb01821ef59..d9b14dd429f3 100644 --- a/qapi/common.json +++ b/qapi/common.json @@ -100,3 +100,29 @@ { 'alternate': 'StrOrNull', 'data': { 's': 'str', 'n': 'null' } } + +## +# @OffAutoPCIBAR: +# +# An enumeration of options for specifying a PCI BAR +# +# @off: The specified feature is disabled +# +# @auto: The PCI BAR for the feature is automatically selected +# +# @bar0: PCI BAR0 is used for the feature +# +# @bar1: PCI BAR1 is used for the feature +# +# @bar2: PCI BAR2 is used for the feature +# +# @bar3: PCI BAR3 is used for the feature +# +# @bar4: PCI BAR4 is used for the feature +# +# @bar5: PCI BAR5 is used for the feature +# +# Since: 2.12 +## +{ 'enum': 'OffAutoPCIBAR', + 'data': [ 'off', 'auto', 'bar0', 'bar1', 'bar2', 'bar3', 'bar4', 'bar5' ] } From patchwork Mon Dec 18 05:02:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 849791 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0TW919H7z9s4s for ; Mon, 18 Dec 2017 16:05:25 +1100 (AEDT) Received: from localhost ([::1]:56808 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQncA-0002Xm-8d for incoming@patchwork.ozlabs.org; Mon, 18 Dec 2017 00:05:22 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38041) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eQnZw-0001EO-I8 for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:03:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eQnZt-0002Jv-Cd for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:03:04 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34488) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eQnZt-0002H5-3C for qemu-devel@nongnu.org; Mon, 18 Dec 2017 00:03:01 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3A6D35F7B7 for ; Mon, 18 Dec 2017 05:03:00 +0000 (UTC) Received: from gimli.home (ovpn-116-49.phx2.redhat.com [10.3.116.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A8196E72E; Mon, 18 Dec 2017 05:02:54 +0000 (UTC) From: Alex Williamson To: qemu-devel@nongnu.org Date: Sun, 17 Dec 2017 22:02:53 -0700 Message-ID: <20171218050253.13478.49457.stgit@gimli.home> In-Reply-To: <20171218040852.13478.19208.stgit@gimli.home> References: <20171218040852.13478.19208.stgit@gimli.home> User-Agent: StGit/0.18 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 18 Dec 2017 05:03:00 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC/RFT PATCH 5/5] vfio/pci: Allow relocating MSI-X MMIO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: eric.auger@redhat.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" With recently proposed kernel side vfio-pci changes, the MSI-X vector table area can be mmap'd from userspace, allowing direct access to non-MSI-X registers within the host page size of this area. However, we only get that direct access if QEMU isn't also emulating MSI-X within that same page. For x86/64 host, the system page size is 4K and the PCI spec recommends a minimum of 4K to 8K alignment to separate MSI-X from non-MSI-X registers, therefore only devices which don't honor this recommendation would see any improvement from this option. The real targets for this feature are hosts where the page size exceeds the PCI spec recommended alignment, such as ARM64 systems with 64K pages. This new x-msix-relocation option accepts the following options: off: Disable MSI-X relocation, use native device config (default) auto: Automaically relocate MSI-X MMIO to another BAR or offset based on minimum additional MMIO requirement bar0..bar5: Specify the target BAR, which will either be extended if the BAR exists or added if the BAR slot is available. Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 1 hw/vfio/trace-events | 2 + 3 files changed, 104 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c383b842da20..b4426abf297a 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1352,6 +1352,101 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev) } } +static void vfio_pci_relocate_msix(VFIOPCIDevice *vdev) +{ + int target_bar = -1; + size_t msix_sz; + + if (!vdev->msix || vdev->msix_relo == OFF_AUTOPCIBAR_OFF) { + return; + } + + /* The actual minimum size of MSI-X structures */ + msix_sz = (vdev->msix->entries * PCI_MSIX_ENTRY_SIZE) + + (QEMU_ALIGN_UP(vdev->msix->entries, 64) / 8); + /* Round up to host pages, we don't want to share a page */ + msix_sz = REAL_HOST_PAGE_ALIGN(msix_sz); + /* PCI BARs must be a power of 2 */ + msix_sz = pow2ceil(msix_sz); + + /* Auto: pick the BAR that incurs the least additional MMIO space */ + if (vdev->msix_relo == OFF_AUTOPCIBAR_AUTO) { + int i; + size_t best = UINT64_MAX; + + for (i = 0; i < PCI_ROM_SLOT; i++) { + size_t size; + + if (vdev->bars[i].ioport) { + continue; + } + + /* MSI-X MMIO must reside within first 32bit offset of BAR */ + if (vdev->bars[i].size > (UINT32_MAX / 2)) + continue; + + /* + * Must be pow2, so larger of double existing or double msix_sz, + * or if BAR unimplemented, msix_sz + */ + size = MAX(vdev->bars[i].size * 2, + vdev->bars[i].size ? msix_sz * 2 : msix_sz); + + trace_vfio_msix_relo_cost(vdev->vbasedev.name, i, size); + + if (size < best) { + best = size; + target_bar = i; + } + + if (vdev->bars[i].mem64) { + i++; + } + } + } else { + target_bar = (int)(vdev->msix_relo - OFF_AUTOPCIBAR_BAR0); + } + + if (target_bar < 0 || vdev->bars[target_bar].ioport || + (!vdev->bars[target_bar].size && + target_bar > 0 && vdev->bars[target_bar - 1].mem64)) { + return; /* Go BOOM? Plumb Error */ + } + + /* + * If adding a new BAR, test if we can make it 64bit. We make it + * prefetchable since QEMU MSI-X emulation has no read side effects + * and doing so makes mapping more flexible. + */ + if (!vdev->bars[target_bar].size) { + if (target_bar < (PCI_ROM_SLOT - 1) && + !vdev->bars[target_bar + 1].size) { + vdev->bars[target_bar].mem64 = true; + vdev->bars[target_bar].type = PCI_BASE_ADDRESS_MEM_TYPE_64; + } + vdev->bars[target_bar].type |= PCI_BASE_ADDRESS_MEM_PREFETCH; + vdev->bars[target_bar].size = msix_sz; + vdev->msix->table_offset = 0; + } else { + vdev->bars[target_bar].size = MAX(vdev->bars[target_bar].size * 2, + msix_sz * 2); + /* + * Due to above size calc, MSI-X always starts halfway into the BAR, + * which will always be a separate host page. + */ + vdev->msix->table_offset = vdev->bars[target_bar].size / 2; + } + + vdev->msix->table_bar = target_bar; + vdev->msix->pba_bar = target_bar; + /* Requires 8-byte alignment, but PCI_MSIX_ENTRY_SIZE guarantees that */ + vdev->msix->pba_offset = vdev->msix->table_offset + + (vdev->msix->entries * PCI_MSIX_ENTRY_SIZE); + + trace_vfio_msix_relo(vdev->vbasedev.name, + vdev->msix->table_bar, vdev->msix->table_offset); +} + /* * We don't have any control over how pci_add_capability() inserts * capabilities into the chain. In order to setup MSI-X we need a @@ -1430,6 +1525,8 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) vdev->msix = msix; vfio_pci_fixup_msix_region(vdev); + + vfio_pci_relocate_msix(vdev); } static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) @@ -2845,13 +2942,14 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vfio_pci_size_rom(vdev); + vfio_bars_prepare(vdev); + vfio_msix_early_setup(vdev, &err); if (err) { error_propagate(errp, err); goto error; } - vfio_bars_prepare(vdev); vfio_bars_register(vdev); ret = vfio_add_capabilities(vdev, errp); @@ -3041,6 +3139,8 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, nv_gpudirect_clique, qdev_prop_nv_gpudirect_clique, uint8_t), + DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, + OFF_AUTOPCIBAR_OFF), /* * TODO - support passed fds... is this necessary? * DEFINE_PROP_STRING("vfiofd", VFIOPCIDevice, vfiofd_name), diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index dcdb1a806769..588381f201b4 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -135,6 +135,7 @@ typedef struct VFIOPCIDevice { (1 << VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT) int32_t bootindex; uint32_t igd_gms; + OffAutoPCIBAR msix_relo; uint8_t pm_cap; uint8_t nv_gpudirect_clique; bool pci_aer; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index fae096c0724f..437ccdd29053 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -16,6 +16,8 @@ vfio_msix_pba_disable(const char *name) " (%s)" vfio_msix_pba_enable(const char *name) " (%s)" vfio_msix_disable(const char *name) " (%s)" vfio_msix_fixup(const char *name, int bar, uint64_t start, uint64_t end) " (%s) MSI-X region %d mmap fixup [0x%"PRIx64" - 0x%"PRIx64"]" +vfio_msix_relo_cost(const char *name, int bar, uint64_t cost) " (%s) BAR %d cost 0x%"PRIx64"" +vfio_msix_relo(const char *name, int bar, uint64_t offset) " (%s) BAR %d offset 0x%"PRIx64"" vfio_msi_enable(const char *name, int nr_vectors) " (%s) Enabled %d MSI vectors" vfio_msi_disable(const char *name) " (%s)" vfio_pci_load_rom(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s ROM:\n size: 0x%lx, offset: 0x%lx, flags: 0x%lx"