From patchwork Sun Dec 12 15:02:06 2010 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 75251 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 898A2B7088 for ; Mon, 13 Dec 2010 02:07:00 +1100 (EST) Received: from localhost ([127.0.0.1]:33470 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PRnVa-00085P-QM for incoming@patchwork.ozlabs.org; Sun, 12 Dec 2010 10:06:42 -0500 Received: from [140.186.70.92] (port=36215 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PRnRT-0005z0-Bk for qemu-devel@nongnu.org; Sun, 12 Dec 2010 10:02:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PRnRQ-0004T1-D2 for qemu-devel@nongnu.org; Sun, 12 Dec 2010 10:02:26 -0500 Received: from mail-ww0-f53.google.com ([74.125.82.53]:39340) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PRnRP-0004Sf-U6 for qemu-devel@nongnu.org; Sun, 12 Dec 2010 10:02:24 -0500 Received: by wwi18 with SMTP id 18so5160094wwi.10 for ; Sun, 12 Dec 2010 07:02:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:date :message-id:x-mailer:in-reply-to:references; bh=u8/XumkOFwTIOw0+FvFel7zxdIsFV09zb/7OFSWwmLA=; b=MBI2hWfAArp27YZTuumVggNIe2mq6Kh77w1gXaZnSddcus7L56Xoiu3BHiKu+POTFJ F2xcsGmllsM7o+RgYWJ/V1h5BRlwqx8kbZwtxXPlyoOM1QASYEAgYfLnMLE8AttiDhnL zqORglX+yhG2sJF3HQOb+xq7COQ6DSqjmEIt0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references; b=cufW+3npGiF9nbroBZdhWah9My/Bi55M8BGGU+y2j3bOLGvhXeOz3nAoDdoWJ+MkQ0 8yr/AcdouxpJKs2yT9CojosPivQvb5hu7K/c6MSJMZztF4H8gVxHoHvxGkCU5HcWW0or s3evPcQ5vIMyw25joRWA3yEF9jEMCpEJecCXc= Received: by 10.216.175.147 with SMTP id z19mr1916566wel.27.1292166142811; Sun, 12 Dec 2010 07:02:22 -0800 (PST) Received: from localhost.localdomain ([109.224.133.37]) by mx.google.com with ESMTPS id m7sm2556987wer.42.2010.12.12.07.02.22 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 12 Dec 2010 07:02:22 -0800 (PST) From: Stefan Hajnoczi To: Date: Sun, 12 Dec 2010 15:02:06 +0000 Message-Id: <1292166128-10874-3-git-send-email-stefanha@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.2.3 In-Reply-To: <1292166128-10874-1-git-send-email-stefanha@linux.vnet.ibm.com> References: <1292166128-10874-1-git-send-email-stefanha@linux.vnet.ibm.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) Cc: Stefan Hajnoczi , "Michael S. Tsirkin" Subject: [Qemu-devel] [PATCH v5 2/4] virtio-pci: Use ioeventfd for virtqueue notify X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Virtqueue notify is currently handled synchronously in userspace virtio. This prevents the vcpu from executing guest code while hardware emulation code handles the notify. On systems that support KVM, the ioeventfd mechanism can be used to make virtqueue notify a lightweight exit by deferring hardware emulation to the iothread and allowing the VM to continue execution. This model is similar to how vhost receives virtqueue notifies. The result of this change is improved performance for userspace virtio devices. Virtio-blk throughput increases especially for multithreaded scenarios and virtio-net transmit throughput increases substantially. Some virtio devices are known to have guest drivers which expect a notify to be processed synchronously and spin waiting for completion. Only enable ioeventfd for virtio-blk and virtio-net for now. Care must be taken not to interfere with vhost-net, which uses host notifiers. If the set_host_notifier() API is used by a device virtio-pci will disable virtio-ioeventfd and let the device deal with host notifiers as it wishes. After migration and on VM change state (running/paused) virtio-ioeventfd will enable/disable itself. * VIRTIO_CONFIG_S_DRIVER_OK -> enable virtio-ioeventfd * !VIRTIO_CONFIG_S_DRIVER_OK -> disable virtio-ioeventfd * virtio_pci_set_host_notifier() -> disable virtio-ioeventfd * vm_change_state(running=0) -> disable virtio-ioeventfd * vm_change_state(running=1) -> enable virtio-ioeventfd Signed-off-by: Stefan Hajnoczi --- hw/virtio-pci.c | 190 ++++++++++++++++++++++++++++++++++++++++++++++++------- hw/virtio.c | 14 +++- hw/virtio.h | 1 + 3 files changed, 179 insertions(+), 26 deletions(-) diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c index 13dd391..f57c45a 100644 --- a/hw/virtio-pci.c +++ b/hw/virtio-pci.c @@ -83,6 +83,11 @@ /* Flags track per-device state like workarounds for quirks in older guests. */ #define VIRTIO_PCI_FLAG_BUS_MASTER_BUG (1 << 0) +/* Performance improves when virtqueue kick processing is decoupled from the + * vcpu thread using ioeventfd for some devices. */ +#define VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT 1 +#define VIRTIO_PCI_FLAG_USE_IOEVENTFD (1 << VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT) + /* QEMU doesn't strictly need write barriers since everything runs in * lock-step. We'll leave the calls to wmb() in though to make it obvious for * KVM or if kqemu gets SMP support. @@ -107,6 +112,8 @@ typedef struct { /* Max. number of ports we can have for a the virtio-serial device */ uint32_t max_virtserial_ports; virtio_net_conf net; + bool ioeventfd_started; + VMChangeStateEntry *vm_change_state_entry; } VirtIOPCIProxy; /* virtio device */ @@ -179,12 +186,131 @@ static int virtio_pci_load_queue(void * opaque, int n, QEMUFile *f) return 0; } +static int virtio_pci_set_host_notifier_internal(VirtIOPCIProxy *proxy, + int n, bool assign) +{ + VirtQueue *vq = virtio_get_queue(proxy->vdev, n); + EventNotifier *notifier = virtio_queue_get_host_notifier(vq); + int r; + if (assign) { + r = event_notifier_init(notifier, 1); + if (r < 0) { + return r; + } + r = kvm_set_ioeventfd_pio_word(event_notifier_get_fd(notifier), + proxy->addr + VIRTIO_PCI_QUEUE_NOTIFY, + n, assign); + if (r < 0) { + event_notifier_cleanup(notifier); + } + } else { + r = kvm_set_ioeventfd_pio_word(event_notifier_get_fd(notifier), + proxy->addr + VIRTIO_PCI_QUEUE_NOTIFY, + n, assign); + if (r < 0) { + return r; + } + + /* Handle the race condition where the guest kicked and we deassigned + * before we got around to handling the kick. + */ + if (event_notifier_test_and_clear(notifier)) { + virtio_queue_notify_vq(vq); + } + + event_notifier_cleanup(notifier); + } + return r; +} + +static void virtio_pci_host_notifier_read(void *opaque) +{ + VirtQueue *vq = opaque; + EventNotifier *n = virtio_queue_get_host_notifier(vq); + if (event_notifier_test_and_clear(n)) { + virtio_queue_notify_vq(vq); + } +} + +static void virtio_pci_set_host_notifier_fd_handler(VirtIOPCIProxy *proxy, + int n, bool assign) +{ + VirtQueue *vq = virtio_get_queue(proxy->vdev, n); + EventNotifier *notifier = virtio_queue_get_host_notifier(vq); + if (assign) { + qemu_set_fd_handler(event_notifier_get_fd(notifier), + virtio_pci_host_notifier_read, NULL, vq); + } else { + qemu_set_fd_handler(event_notifier_get_fd(notifier), + NULL, NULL, NULL); + } +} + +static int virtio_pci_start_ioeventfd(VirtIOPCIProxy *proxy) +{ + int n, r; + + if (!(proxy->flags & VIRTIO_PCI_FLAG_USE_IOEVENTFD) || + proxy->ioeventfd_started) { + return 0; + } + + for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) { + if (!virtio_queue_get_num(proxy->vdev, n)) { + continue; + } + + r = virtio_pci_set_host_notifier_internal(proxy, n, true); + if (r < 0) { + goto assign_error; + } + + virtio_pci_set_host_notifier_fd_handler(proxy, n, true); + } + proxy->ioeventfd_started = true; + return 0; + +assign_error: + while (--n >= 0) { + if (!virtio_queue_get_num(proxy->vdev, n)) { + continue; + } + + virtio_pci_set_host_notifier_fd_handler(proxy, n, false); + virtio_pci_set_host_notifier_internal(proxy, n, false); + } + proxy->ioeventfd_started = false; + proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD; + return r; +} + +static int virtio_pci_stop_ioeventfd(VirtIOPCIProxy *proxy) +{ + int n; + + if (!proxy->ioeventfd_started) { + return 0; + } + + for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) { + if (!virtio_queue_get_num(proxy->vdev, n)) { + continue; + } + + virtio_pci_set_host_notifier_fd_handler(proxy, n, false); + virtio_pci_set_host_notifier_internal(proxy, n, false); + } + proxy->ioeventfd_started = false; + return 0; +} + static void virtio_pci_reset(DeviceState *d) { VirtIOPCIProxy *proxy = container_of(d, VirtIOPCIProxy, pci_dev.qdev); + virtio_pci_stop_ioeventfd(proxy); virtio_reset(proxy->vdev); msix_reset(&proxy->pci_dev); - proxy->flags = 0; + proxy->flags &= ~VIRTIO_PCI_FLAG_BUS_MASTER_BUG; } static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) @@ -209,6 +335,7 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) case VIRTIO_PCI_QUEUE_PFN: pa = (target_phys_addr_t)val << VIRTIO_PCI_QUEUE_ADDR_SHIFT; if (pa == 0) { + virtio_pci_stop_ioeventfd(proxy); virtio_reset(proxy->vdev); msix_unuse_all_vectors(&proxy->pci_dev); } @@ -223,6 +350,12 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val) virtio_queue_notify(vdev, val); break; case VIRTIO_PCI_STATUS: + if (val & VIRTIO_CONFIG_S_DRIVER_OK) { + virtio_pci_start_ioeventfd(proxy); + } else { + virtio_pci_stop_ioeventfd(proxy); + } + virtio_set_status(vdev, val & 0xFF); if (vdev->status == 0) { virtio_reset(proxy->vdev); @@ -403,6 +536,7 @@ static void virtio_write_config(PCIDevice *pci_dev, uint32_t address, if (PCI_COMMAND == address) { if (!(val & PCI_COMMAND_MASTER)) { if (!(proxy->flags & VIRTIO_PCI_FLAG_BUS_MASTER_BUG)) { + virtio_pci_stop_ioeventfd(proxy); virtio_set_status(proxy->vdev, proxy->vdev->status & ~VIRTIO_CONFIG_S_DRIVER_OK); } @@ -480,30 +614,27 @@ assign_error: static int virtio_pci_set_host_notifier(void *opaque, int n, bool assign) { VirtIOPCIProxy *proxy = opaque; - VirtQueue *vq = virtio_get_queue(proxy->vdev, n); - EventNotifier *notifier = virtio_queue_get_host_notifier(vq); - int r; - if (assign) { - r = event_notifier_init(notifier, 1); - if (r < 0) { - return r; - } - r = kvm_set_ioeventfd_pio_word(event_notifier_get_fd(notifier), - proxy->addr + VIRTIO_PCI_QUEUE_NOTIFY, - n, assign); - if (r < 0) { - event_notifier_cleanup(notifier); - } + + /* Stop using ioeventfd for virtqueue kick if the device starts using host + * notifiers. This makes it easy to avoid stepping on each others' toes. + */ + if (proxy->ioeventfd_started) { + virtio_pci_stop_ioeventfd(proxy); + proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD; + } + + return virtio_pci_set_host_notifier_internal(proxy, n, assign); +} + +static void virtio_pci_vm_change_state_handler(void *opaque, int running, int reason) +{ + VirtIOPCIProxy *proxy = opaque; + + if (running && (proxy->vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) { + virtio_pci_start_ioeventfd(proxy); } else { - r = kvm_set_ioeventfd_pio_word(event_notifier_get_fd(notifier), - proxy->addr + VIRTIO_PCI_QUEUE_NOTIFY, - n, assign); - if (r < 0) { - return r; - } - event_notifier_cleanup(notifier); + virtio_pci_stop_ioeventfd(proxy); } - return r; } static const VirtIOBindings virtio_pci_bindings = { @@ -563,6 +694,10 @@ static void virtio_init_pci(VirtIOPCIProxy *proxy, VirtIODevice *vdev, proxy->host_features |= 0x1 << VIRTIO_F_NOTIFY_ON_EMPTY; proxy->host_features |= 0x1 << VIRTIO_F_BAD_FEATURE; proxy->host_features = vdev->get_features(vdev, proxy->host_features); + + proxy->vm_change_state_entry = qemu_add_vm_change_state_handler( + virtio_pci_vm_change_state_handler, + proxy); } static int virtio_blk_init_pci(PCIDevice *pci_dev) @@ -590,6 +725,9 @@ static int virtio_blk_init_pci(PCIDevice *pci_dev) static int virtio_exit_pci(PCIDevice *pci_dev) { + VirtIOPCIProxy *proxy = DO_UPCAST(VirtIOPCIProxy, pci_dev, pci_dev); + + qemu_del_vm_change_state_handler(proxy->vm_change_state_entry); return msix_uninit(pci_dev); } @@ -597,6 +735,7 @@ static int virtio_blk_exit_pci(PCIDevice *pci_dev) { VirtIOPCIProxy *proxy = DO_UPCAST(VirtIOPCIProxy, pci_dev, pci_dev); + virtio_pci_stop_ioeventfd(proxy); virtio_blk_exit(proxy->vdev); blockdev_mark_auto_del(proxy->block.bs); return virtio_exit_pci(pci_dev); @@ -658,6 +797,7 @@ static int virtio_net_exit_pci(PCIDevice *pci_dev) { VirtIOPCIProxy *proxy = DO_UPCAST(VirtIOPCIProxy, pci_dev, pci_dev); + virtio_pci_stop_ioeventfd(proxy); virtio_net_exit(proxy->vdev); return virtio_exit_pci(pci_dev); } @@ -705,6 +845,8 @@ static PCIDeviceInfo virtio_info[] = { .qdev.props = (Property[]) { DEFINE_PROP_HEX32("class", VirtIOPCIProxy, class_code, 0), DEFINE_BLOCK_PROPERTIES(VirtIOPCIProxy, block), + DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags, + VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true), DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2), DEFINE_VIRTIO_BLK_FEATURES(VirtIOPCIProxy, host_features), DEFINE_PROP_END_OF_LIST(), @@ -717,6 +859,8 @@ static PCIDeviceInfo virtio_info[] = { .exit = virtio_net_exit_pci, .romfile = "pxe-virtio.bin", .qdev.props = (Property[]) { + DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags, + VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true), DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3), DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features), DEFINE_NIC_PROPERTIES(VirtIOPCIProxy, nic), diff --git a/hw/virtio.c b/hw/virtio.c index 07dbf86..e40296a 100644 --- a/hw/virtio.c +++ b/hw/virtio.c @@ -575,11 +575,19 @@ int virtio_queue_get_num(VirtIODevice *vdev, int n) return vdev->vq[n].vring.num; } +void virtio_queue_notify_vq(VirtQueue *vq) +{ + if (vq->vring.desc) { + VirtIODevice *vdev = vq->vdev; + trace_virtio_queue_notify(vdev, vq - vdev->vq, vq); + vq->handle_output(vdev, vq); + } +} + void virtio_queue_notify(VirtIODevice *vdev, int n) { - if (n < VIRTIO_PCI_QUEUE_MAX && vdev->vq[n].vring.desc) { - trace_virtio_queue_notify(vdev, n, &vdev->vq[n]); - vdev->vq[n].handle_output(vdev, &vdev->vq[n]); + if (n < VIRTIO_PCI_QUEUE_MAX) { + virtio_queue_notify_vq(&vdev->vq[n]); } } diff --git a/hw/virtio.h b/hw/virtio.h index 02fa312..5ae521c 100644 --- a/hw/virtio.h +++ b/hw/virtio.h @@ -219,5 +219,6 @@ void virtio_queue_set_last_avail_idx(VirtIODevice *vdev, int n, uint16_t idx); VirtQueue *virtio_get_queue(VirtIODevice *vdev, int n); EventNotifier *virtio_queue_get_guest_notifier(VirtQueue *vq); EventNotifier *virtio_queue_get_host_notifier(VirtQueue *vq); +void virtio_queue_notify_vq(VirtQueue *vq); void virtio_irq(VirtQueue *vq); #endif