From patchwork Wed Sep 2 03:25:24 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 513279 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 5C824140285 for ; Wed, 2 Sep 2015 13:30:14 +1000 (AEST) Received: from localhost ([::1]:32986 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZWykW-00061K-99 for incoming@patchwork.ozlabs.org; Tue, 01 Sep 2015 23:30:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43303) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZWygE-0007Qo-US for qemu-devel@nongnu.org; Tue, 01 Sep 2015 23:25:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZWygD-0006nw-F4 for qemu-devel@nongnu.org; Tue, 01 Sep 2015 23:25:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57227) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZWygD-0006nc-4k for qemu-devel@nongnu.org; Tue, 01 Sep 2015 23:25:45 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id D615DC0EE04F; Wed, 2 Sep 2015 03:25:44 +0000 (UTC) Received: from jason-ThinkPad-T430s.nay.redhat.com (dhcp-14-122.nay.redhat.com [10.66.14.122]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t823PQfn024102; Tue, 1 Sep 2015 23:25:42 -0400 From: Jason Wang To: qemu-devel@nongnu.org Date: Wed, 2 Sep 2015 11:25:24 +0800 Message-Id: <1441164325-14417-8-git-send-email-jasowang@redhat.com> In-Reply-To: <1441164325-14417-1-git-send-email-jasowang@redhat.com> References: <1441164325-14417-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 209.132.183.28 Cc: cornelia.huck@de.ibm.com, Jason Wang , mst@redhat.com Subject: [Qemu-devel] [PATCH V2 7/8] virtio-pci: introduce pio notification capability for modern device X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We used to use mmio for notification. This could be slow on some arch (e.g on x86 without EPT). So this patch introduces pio bar and a pio notification cap for modern device. This ability is enabled through property "modern-pio-notify" for virtio pci devices and was disabled by default. Management can enable when it thinks it was needed. Benchmarks shows almost no obvious difference compared to legacy device on machines without ept. Thanks Wenli Quan for the benchmarking. Cc: Michael S. Tsirkin Signed-off-by: Jason Wang --- hw/virtio/virtio-pci.c | 122 ++++++++++++++++++++++++++++++++++++++++++------- hw/virtio/virtio-pci.h | 10 ++++ 2 files changed, 115 insertions(+), 17 deletions(-) diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 2d00d06..bf61acf 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -270,7 +270,9 @@ static int virtio_pci_set_host_notifier_internal(VirtIOPCIProxy *proxy, EventNotifier *notifier = virtio_queue_get_host_notifier(vq); bool legacy = !(proxy->flags & VIRTIO_PCI_FLAG_DISABLE_LEGACY); bool modern = !(proxy->flags & VIRTIO_PCI_FLAG_DISABLE_MODERN); + bool modern_pio = proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY; MemoryRegion *modern_mr = &proxy->notify.mr; + MemoryRegion *modern_notify_mr = &proxy->notify_pio.mr; MemoryRegion *legacy_mr = &proxy->bar; hwaddr modern_addr = QEMU_VIRTIO_PCI_QUEUE_MEM_MULT * virtio_get_queue_index(vq); @@ -288,6 +290,10 @@ static int virtio_pci_set_host_notifier_internal(VirtIOPCIProxy *proxy, if (modern) { memory_region_add_eventfd(modern_mr, modern_addr, 0, false, n, notifier); + if (modern_pio) { + memory_region_add_eventfd(modern_notify_mr, 0, 2, + true, n, notifier); + } } if (legacy) { memory_region_add_eventfd(legacy_mr, legacy_addr, 2, @@ -297,6 +303,10 @@ static int virtio_pci_set_host_notifier_internal(VirtIOPCIProxy *proxy, if (modern) { memory_region_del_eventfd(modern_mr, modern_addr, 0, false, n, notifier); + if (modern_pio) { + memory_region_del_eventfd(modern_notify_mr, 0, 2, + true, n, notifier); + } } if (legacy) { memory_region_del_eventfd(legacy_mr, legacy_addr, 2, @@ -1404,6 +1414,17 @@ static void virtio_pci_notify_write(void *opaque, hwaddr addr, } } +static void virtio_pci_notify_write_pio(void *opaque, hwaddr addr, + uint64_t val, unsigned size) +{ + VirtIODevice *vdev = opaque; + unsigned queue = val; + + if (queue < VIRTIO_QUEUE_MAX) { + virtio_queue_notify(vdev, queue); + } +} + static uint64_t virtio_pci_isr_read(void *opaque, hwaddr addr, unsigned size) { @@ -1497,6 +1518,16 @@ static void virtio_pci_modern_regions_init(VirtIOPCIProxy *proxy) }, .endianness = DEVICE_LITTLE_ENDIAN, }; + static const MemoryRegionOps notify_pio_ops = { + .read = virtio_pci_notify_read, + .write = virtio_pci_notify_write_pio, + .impl = { + .min_access_size = 1, + .max_access_size = 4, + }, + .endianness = DEVICE_LITTLE_ENDIAN, + }; + memory_region_init_io(&proxy->common.mr, OBJECT(proxy), &common_ops, @@ -1521,30 +1552,60 @@ static void virtio_pci_modern_regions_init(VirtIOPCIProxy *proxy) virtio_bus_get_device(&proxy->bus), "virtio-pci-notify", proxy->notify.size); + + memory_region_init_io(&proxy->notify_pio.mr, OBJECT(proxy), + ¬ify_pio_ops, + virtio_bus_get_device(&proxy->bus), + "virtio-pci-notify-pio", + proxy->notify.size); } static void virtio_pci_modern_region_map(VirtIOPCIProxy *proxy, VirtIOPCIRegion *region, - struct virtio_pci_cap *cap) + struct virtio_pci_cap *cap, + MemoryRegion *mr, + uint8_t bar) { - memory_region_add_subregion(&proxy->modern_bar, - region->offset, - ®ion->mr); + memory_region_add_subregion(mr, region->offset, ®ion->mr); cap->cfg_type = region->type; - cap->bar = proxy->modern_mem_bar; + cap->bar = bar; cap->offset = cpu_to_le32(region->offset); cap->length = cpu_to_le32(region->size); virtio_pci_add_mem_cap(proxy, cap); + +} + +static void virtio_pci_modern_mem_region_map(VirtIOPCIProxy *proxy, + VirtIOPCIRegion *region, + struct virtio_pci_cap *cap) +{ + virtio_pci_modern_region_map(proxy, region, cap, + &proxy->modern_bar, proxy->modern_mem_bar); } -static void virtio_pci_modern_region_unmap(VirtIOPCIProxy *proxy, - VirtIOPCIRegion *region) +static void virtio_pci_modern_io_region_map(VirtIOPCIProxy *proxy, + VirtIOPCIRegion *region, + struct virtio_pci_cap *cap) +{ + virtio_pci_modern_region_map(proxy, region, cap, + &proxy->io_bar, proxy->modern_io_bar); +} + +static void virtio_pci_modern_mem_region_unmap(VirtIOPCIProxy *proxy, + VirtIOPCIRegion *region) { memory_region_del_subregion(&proxy->modern_bar, ®ion->mr); } +static void virtio_pci_modern_io_region_unmap(VirtIOPCIProxy *proxy, + VirtIOPCIRegion *region) +{ + memory_region_del_subregion(&proxy->io_bar, + ®ion->mr); +} + /* This is called by virtio-bus just after the device is plugged. */ static void virtio_pci_device_plugged(DeviceState *d, Error **errp) { @@ -1552,6 +1613,7 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp) VirtioBusState *bus = &proxy->bus; bool legacy = !(proxy->flags & VIRTIO_PCI_FLAG_DISABLE_LEGACY); bool modern = !(proxy->flags & VIRTIO_PCI_FLAG_DISABLE_MODERN); + bool modern_pio = proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY; uint8_t *config; uint32_t size; VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); @@ -1590,16 +1652,31 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp) .cap.cap_len = sizeof cfg, .cap.cfg_type = VIRTIO_PCI_CAP_PCI_CFG, }; - struct virtio_pci_cfg_cap *cfg_mask; + struct virtio_pci_notify_cap notify_pio = { + .cap.cap_len = sizeof notify, + .notify_off_multiplier = cpu_to_le32(0x0), + }; - /* TODO: add io access for speed */ + struct virtio_pci_cfg_cap *cfg_mask; virtio_add_feature(&vdev->host_features, VIRTIO_F_VERSION_1); virtio_pci_modern_regions_init(proxy); - virtio_pci_modern_region_map(proxy, &proxy->common, &cap); - virtio_pci_modern_region_map(proxy, &proxy->isr, &cap); - virtio_pci_modern_region_map(proxy, &proxy->device, &cap); - virtio_pci_modern_region_map(proxy, &proxy->notify, ¬ify.cap); + + virtio_pci_modern_mem_region_map(proxy, &proxy->common, &cap); + virtio_pci_modern_mem_region_map(proxy, &proxy->isr, &cap); + virtio_pci_modern_mem_region_map(proxy, &proxy->device, &cap); + virtio_pci_modern_mem_region_map(proxy, &proxy->notify, ¬ify.cap); + + if (modern_pio) { + memory_region_init(&proxy->io_bar, OBJECT(proxy), + "virtio-pci-io", 0x4); + + pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar, + PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar); + + virtio_pci_modern_io_region_map(proxy, &proxy->notify_pio, + ¬ify_pio.cap); + } pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar, PCI_BASE_ADDRESS_SPACE_MEMORY | @@ -1652,14 +1729,18 @@ static void virtio_pci_device_unplugged(DeviceState *d) { VirtIOPCIProxy *proxy = VIRTIO_PCI(d); bool modern = !(proxy->flags & VIRTIO_PCI_FLAG_DISABLE_MODERN); + bool modern_pio = proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY; virtio_pci_stop_ioeventfd(proxy); if (modern) { - virtio_pci_modern_region_unmap(proxy, &proxy->common); - virtio_pci_modern_region_unmap(proxy, &proxy->isr); - virtio_pci_modern_region_unmap(proxy, &proxy->device); - virtio_pci_modern_region_unmap(proxy, &proxy->notify); + virtio_pci_modern_mem_region_unmap(proxy, &proxy->common); + virtio_pci_modern_mem_region_unmap(proxy, &proxy->isr); + virtio_pci_modern_mem_region_unmap(proxy, &proxy->device); + virtio_pci_modern_mem_region_unmap(proxy, &proxy->notify); + if (modern_pio) { + virtio_pci_modern_io_region_unmap(proxy, &proxy->notify_pio); + } } } @@ -1679,6 +1760,7 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) */ proxy->legacy_io_bar = 0; proxy->msix_bar = 1; + proxy->modern_io_bar = 2; proxy->modern_mem_bar = 4; proxy->common.offset = 0x0; @@ -1698,6 +1780,10 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) QEMU_VIRTIO_PCI_QUEUE_MEM_MULT * VIRTIO_QUEUE_MAX; proxy->notify.type = VIRTIO_PCI_CAP_NOTIFY_CFG; + proxy->notify_pio.offset = 0x0; + proxy->notify_pio.size = 0x4; + proxy->notify_pio.type = VIRTIO_PCI_CAP_NOTIFY_CFG; + /* subclasses can enforce modern, so do this unconditionally */ memory_region_init(&proxy->modern_bar, OBJECT(proxy), "virtio-pci", 2 * QEMU_VIRTIO_PCI_QUEUE_MEM_MULT * @@ -1744,6 +1830,8 @@ static Property virtio_pci_properties[] = { VIRTIO_PCI_FLAG_DISABLE_MODERN_BIT, true), DEFINE_PROP_BIT("migrate-extra", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_MIGRATE_EXTRA_BIT, true), + DEFINE_PROP_BIT("modern-pio-notify", VirtIOPCIProxy, flags, + VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY_BIT, false), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h index d890009..3d376f2 100644 --- a/hw/virtio/virtio-pci.h +++ b/hw/virtio/virtio-pci.h @@ -79,6 +79,13 @@ typedef struct VirtioBusClass VirtioPCIBusClass; #define VIRTIO_PCI_FLAG_MIGRATE_EXTRA_BIT 4 #define VIRTIO_PCI_FLAG_MIGRATE_EXTRA (1 << VIRTIO_PCI_FLAG_MIGRATE_EXTRA_BIT) +/* have pio notification for modern device ? */ +#define VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY_BIT 5 +#define VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY \ + (1 << VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY_BIT) + + + typedef struct { MSIMessage msg; int virq; @@ -123,11 +130,14 @@ struct VirtIOPCIProxy { VirtIOPCIRegion isr; VirtIOPCIRegion device; VirtIOPCIRegion notify; + VirtIOPCIRegion notify_pio; MemoryRegion modern_bar; + MemoryRegion io_bar; MemoryRegion modern_cfg; AddressSpace modern_as; uint32_t legacy_io_bar; uint32_t msix_bar; + uint32_t modern_io_bar; uint32_t modern_mem_bar; int config_cap; uint32_t flags;