From patchwork Fri Oct 11 11:20:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175120 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQY93GmYz9sP3 for ; Fri, 11 Oct 2019 22:22:53 +1100 (AEDT) Received: from localhost ([::1]:48610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt0V-0008LB-4g for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:22:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43403) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsyL-00061G-Nm for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsyK-0001cg-9j for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:37 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37012) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsyK-0001cL-1h for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:36 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5134F3086272; Fri, 11 Oct 2019 11:20:35 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F365A5D6C8; Fri, 11 Oct 2019 11:20:27 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 01/10] qdev/qbus: add hidden device support Date: Fri, 11 Oct 2019 13:20:06 +0200 Message-Id: <20191011112015.11785-2-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Fri, 11 Oct 2019 11:20:35 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This adds support for hiding a device to the qbus and qdev APIs. The first user of this will be the virtio-net failover feature but the API introduced with this patch could be used to implement other features as well, for example hiding pci devices when a pci bus is powered off. qdev_device_add() is modified to check for a net_failover_pair_id argument in the option string. A DeviceListener callback should_be_hidden() is added. It can be used by a standby device to inform qdev that this device should not be added now. The standby device handler can store the device options to plug the device in at a later point in time. One reason for hiding the device is that we don't want to expose both devices to the guest kernel until the respective virtio feature bit VIRTIO_NET_F_STANDBY was negotiated and we know that the devices will be handled correctly by the guest. More information on the kernel feature this is using: https://www.kernel.org/doc/html/latest/networking/net_failover.html An example where the primary device is a vfio-pci device and the standby device is a virtio-net device: A device is hidden when it has an "net_failover_pair_id" option, e.g. -device virtio-net-pci,...,failover=on,... -device vfio-pci,...,net_failover_pair_id=net1,... Signed-off-by: Jens Freimann --- hw/core/qdev.c | 19 +++++++++++++++++++ include/hw/qdev-core.h | 9 +++++++++ qdev-monitor.c | 43 ++++++++++++++++++++++++++++++++++++++---- vl.c | 6 ++++-- 4 files changed, 71 insertions(+), 6 deletions(-) diff --git a/hw/core/qdev.c b/hw/core/qdev.c index cbad6c1d55..84fac591ca 100644 --- a/hw/core/qdev.c +++ b/hw/core/qdev.c @@ -212,6 +212,25 @@ void device_listener_unregister(DeviceListener *listener) QTAILQ_REMOVE(&device_listeners, listener, link); } +bool qdev_should_hide_device(QemuOpts *opts, Error **errp) +{ + bool res = false; + bool match_found = false; + DeviceListener *listener; + + QTAILQ_FOREACH(listener, &device_listeners, link) { + if (listener->should_be_hidden) { + listener->should_be_hidden(listener, opts, &match_found, &res); + } + + if (match_found) { + break; + } + } + + return res; +} + void qdev_set_legacy_instance_id(DeviceState *dev, int alias_id, int required_for_version) { diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index aa123f88cb..b61cf82ded 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -154,6 +154,13 @@ struct DeviceState { struct DeviceListener { void (*realize)(DeviceListener *listener, DeviceState *dev); void (*unrealize)(DeviceListener *listener, DeviceState *dev); + /* + * This callback is called just upon init of the DeviceState + * and can be used by a standby device for informing qdev if this + * device should be hidden by checking the device opts + */ + void (*should_be_hidden)(DeviceListener *listener, QemuOpts *device_opts, + bool *match_found, bool *res); QTAILQ_ENTRY(DeviceListener) link; }; @@ -451,4 +458,6 @@ static inline bool qbus_is_hotpluggable(BusState *bus) void device_listener_register(DeviceListener *listener); void device_listener_unregister(DeviceListener *listener); +bool qdev_should_hide_device(QemuOpts *opts, Error **errp); + #endif diff --git a/qdev-monitor.c b/qdev-monitor.c index 148df9cacf..9fc8331157 100644 --- a/qdev-monitor.c +++ b/qdev-monitor.c @@ -32,9 +32,11 @@ #include "qemu/help_option.h" #include "qemu/option.h" #include "qemu/qemu-print.h" +#include "qemu/option_int.h" #include "sysemu/block-backend.h" #include "sysemu/sysemu.h" #include "migration/misc.h" +#include "migration/migration.h" /* * Aliases were a bad idea from the start. Let's keep them @@ -562,14 +564,45 @@ void qdev_set_id(DeviceState *dev, const char *id) } } +static int is_failover_device(void *opaque, const char *name, const char *value, + Error **errp) +{ + if (strcmp(name, "net_failover_pair_id") == 0) { + QemuOpts *opts = (QemuOpts *)opaque; + + if (qdev_should_hide_device(opts, errp) && errp && !*errp) { + return 1; + } else if (errp && *errp) { + return -1; + } + } + + return 0; +} + +static bool should_hide_device(QemuOpts *opts, Error **err) +{ + if (qemu_opt_foreach(opts, is_failover_device, opts, err) == 0) { + return false; + } + return true; +} + DeviceState *qdev_device_add(QemuOpts *opts, Error **errp) { DeviceClass *dc; const char *driver, *path; - DeviceState *dev; + DeviceState *dev = NULL; BusState *bus = NULL; Error *err = NULL; + if (opts && should_hide_device(opts, &err)) { + if (err) { + goto err_del_dev; + } + return NULL; + } + driver = qemu_opt_get(opts, "driver"); if (!driver) { error_setg(errp, QERR_MISSING_PARAMETER, "driver"); @@ -648,8 +681,10 @@ DeviceState *qdev_device_add(QemuOpts *opts, Error **errp) err_del_dev: error_propagate(errp, err); - object_unparent(OBJECT(dev)); - object_unref(OBJECT(dev)); + if (dev) { + object_unparent(OBJECT(dev)); + object_unref(OBJECT(dev)); + } return NULL; } @@ -818,7 +853,7 @@ void qdev_unplug(DeviceState *dev, Error **errp) return; } - if (!migration_is_idle()) { + if (!migration_is_idle() && !migration_in_setup(migrate_get_current())) { error_setg(errp, "device_del not allowed while migrating"); return; } diff --git a/vl.c b/vl.c index 002bf4919e..898b9901c8 100644 --- a/vl.c +++ b/vl.c @@ -2204,10 +2204,12 @@ static int device_init_func(void *opaque, QemuOpts *opts, Error **errp) DeviceState *dev; dev = qdev_device_add(opts, errp); - if (!dev) { + if (!dev && *errp) { + error_report_err(*errp); return -1; + } else if (dev) { + object_unref(OBJECT(dev)); } - object_unref(OBJECT(dev)); return 0; } From patchwork Fri Oct 11 11:20:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175118 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQWM4mfWz9sP6 for ; Fri, 11 Oct 2019 22:21:19 +1100 (AEDT) Received: from localhost ([::1]:48580 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsyz-0006Ag-86 for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:21:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43427) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsyS-00068G-0X for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsyR-0001f3-1i for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:43 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45420) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsyQ-0001ev-SZ for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:42 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 26BFA316E535; Fri, 11 Oct 2019 11:20:42 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C861A60A9F; Fri, 11 Oct 2019 11:20:36 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 02/10] pci: mark devices partially unplugged Date: Fri, 11 Oct 2019 13:20:07 +0200 Message-Id: <20191011112015.11785-3-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Fri, 11 Oct 2019 11:20:42 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Only the guest unplug request was triggered. This is needed for the failover feature. In case of a failed migration we need to plug the device back to the guest. Signed-off-by: Jens Freimann --- hw/pci/pci.c | 2 ++ hw/pci/pcie.c | 3 +++ include/hw/pci/pci.h | 1 + 3 files changed, 6 insertions(+) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index aa05c2b9b2..c140b37765 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -2078,6 +2078,8 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp) Error *local_err = NULL; bool is_default_rom; + pci_dev->partially_hotplugged = false; + /* initialize cap_present for pci_is_express() and pci_config_size(), * Note that hybrid PCIs are not set automatically and need to manage * QEMU_PCI_CAP_EXPRESS manually */ diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c index a6beb567bd..19363ff8ce 100644 --- a/hw/pci/pcie.c +++ b/hw/pci/pcie.c @@ -456,6 +456,9 @@ static void pcie_unplug_device(PCIBus *bus, PCIDevice *dev, void *opaque) { HotplugHandler *hotplug_ctrl = qdev_get_hotplug_handler(DEVICE(dev)); + if (dev->partially_hotplugged) { + return; + } hotplug_handler_unplug(hotplug_ctrl, DEVICE(dev), &error_abort); object_unparent(OBJECT(dev)); } diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index f3f0ffd5fb..f3a39c9bbd 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -265,6 +265,7 @@ typedef struct PCIReqIDCache PCIReqIDCache; struct PCIDevice { DeviceState qdev; + bool partially_hotplugged; /* PCI config space */ uint8_t *config; From patchwork Fri Oct 11 11:20:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175119 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQWb2r07z9sP3 for ; Fri, 11 Oct 2019 22:21:31 +1100 (AEDT) Received: from localhost ([::1]:48584 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIszA-0006PV-Vs for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:21:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43443) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsyc-0006MN-20 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsyb-0001iw-2e for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:47024) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsya-0001ih-TA for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:20:53 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2666F3090FD1; Fri, 11 Oct 2019 11:20:52 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9FD036031D; Fri, 11 Oct 2019 11:20:43 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 03/10] pci: mark device having guest unplug request pending Date: Fri, 11 Oct 2019 13:20:08 +0200 Message-Id: <20191011112015.11785-4-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Fri, 11 Oct 2019 11:20:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Set pending_deleted_event in DeviceState for failover primary devices that were successfully unplugged by the Guest OS. Signed-off-by: Jens Freimann --- hw/pci/pcie.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c index 19363ff8ce..08718188bb 100644 --- a/hw/pci/pcie.c +++ b/hw/pci/pcie.c @@ -457,6 +457,7 @@ static void pcie_unplug_device(PCIBus *bus, PCIDevice *dev, void *opaque) HotplugHandler *hotplug_ctrl = qdev_get_hotplug_handler(DEVICE(dev)); if (dev->partially_hotplugged) { + dev->qdev.pending_deleted_event = false; return; } hotplug_handler_unplug(hotplug_ctrl, DEVICE(dev), &error_abort); @@ -476,6 +477,8 @@ void pcie_cap_slot_unplug_request_cb(HotplugHandler *hotplug_dev, return; } + dev->pending_deleted_event = true; + /* In case user cancel the operation of multi-function hot-add, * remove the function that is unexposed to guest individually, * without interaction with guest. From patchwork Fri Oct 11 11:20:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175123 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQbN1dbcz9sP3 for ; Fri, 11 Oct 2019 22:24:48 +1100 (AEDT) Received: from localhost ([::1]:48630 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt2L-0002TE-Mt for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:24:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43469) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsym-0006eX-08 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsyk-0001o1-W1 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43422) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsyk-0001nI-Qk for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:02 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 138B518C4265; Fri, 11 Oct 2019 11:21:01 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A2CA61001B09; Fri, 11 Oct 2019 11:20:53 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 04/10] qapi: add unplug primary event Date: Fri, 11 Oct 2019 13:20:09 +0200 Message-Id: <20191011112015.11785-5-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.62]); Fri, 11 Oct 2019 11:21:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This event is emitted when we sent a request to unplug a failover primary device from the Guest OS and it includes the device id of the primary device. Signed-off-by: Jens Freimann --- qapi/migration.json | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index 82feb5bd39..52e69e2868 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1448,3 +1448,22 @@ # Since: 3.0 ## { 'command': 'migrate-pause', 'allow-oob': true } + +## +# @UNPLUG_PRIMARY: +# +# Emitted from source side of a migration when migration state is +# WAIT_UNPLUG. Device was unplugged by guest operating system. +# Device resources in QEMU are kept on standby to be able to re-plug it in case +# of migration failure. +# +# @device_id: QEMU device id of the unplugged device +# +# Since: 4.2 +# +# Example: +# {"event": "UNPLUG_PRIMARY", "data": {"device_id": "hostdev0"} } +# +## +{ 'event': 'UNPLUG_PRIMARY', + 'data': { 'device_id': 'str' } } From patchwork Fri Oct 11 11:20:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175122 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQYj55WZz9sP3 for ; Fri, 11 Oct 2019 22:23:21 +1100 (AEDT) Received: from localhost ([::1]:48618 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt0x-0000bX-8n for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:23:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43513) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsyw-0006pI-Ay for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsyu-0001ul-Kx for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:47094) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsyu-0001uX-Cj for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:12 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A43533090FDE; Fri, 11 Oct 2019 11:21:11 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7F21B5D9C3; Fri, 11 Oct 2019 11:21:02 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 05/10] qapi: add failover negotiated event Date: Fri, 11 Oct 2019 13:20:10 +0200 Message-Id: <20191011112015.11785-6-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Fri, 11 Oct 2019 11:21:11 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This event is sent to let libvirt know that VIRTIO_NET_F_STANDBY feature was not negotiated during virtio feature negotiation. If this event is received it means any primary devices hotplugged before this were were never really added to QEMU devices. Signed-off-by: Jens Freimann --- qapi/net.json | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/qapi/net.json b/qapi/net.json index 728990f4fb..8c5f3f1fb2 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -737,3 +737,19 @@ ## { 'command': 'announce-self', 'boxed': true, 'data' : 'AnnounceParameters'} + +## +# @FAILOVER_NEGOTIATED: +# +# Emitted when VIRTIO_NET_F_STANDBY was negotiated during feature negotiation +# +# Since: 4.2 +# +# Example: +# +# <- { "event": "FAILOVER_NEGOTIATED", +# "data": {} } +# +## +{ 'event': 'FAILOVER_NEGOTIATED', + 'data': {} } From patchwork Fri Oct 11 11:20:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175121 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQYW6lYtz9sP3 for ; Fri, 11 Oct 2019 22:23:11 +1100 (AEDT) Received: from localhost ([::1]:48614 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt0n-0000Jy-0h for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:23:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43558) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIsz4-00071c-87 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIsz3-00021c-6k for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:22 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52196) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIsz3-0001ze-18 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:21 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4919C18C8909; Fri, 11 Oct 2019 11:21:20 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2AD165D713; Fri, 11 Oct 2019 11:21:13 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 06/10] migration: allow unplug during migration for failover devices Date: Fri, 11 Oct 2019 13:20:11 +0200 Message-Id: <20191011112015.11785-7-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.70]); Fri, 11 Oct 2019 11:21:20 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" In "b06424de62 migration: Disable hotplug/unplug during migration" we added a check to disable unplug for all devices until we have figured out what works. For failover primary devices qdev_unplug() is called from the migration handler, i.e. during migration. This patch adds a flag to DeviceState which is set to false for all devices and makes an exception for vfio-pci devices that are also primary devices in a failover pair. Signed-off-by: Jens Freimann --- hw/core/qdev.c | 1 + include/hw/qdev-core.h | 1 + qdev-monitor.c | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/hw/core/qdev.c b/hw/core/qdev.c index 84fac591ca..fb76c6eeaf 100644 --- a/hw/core/qdev.c +++ b/hw/core/qdev.c @@ -991,6 +991,7 @@ static void device_initfn(Object *obj) dev->instance_id_alias = -1; dev->realized = false; + dev->allow_unplug_during_migration = false; object_property_add_bool(obj, "realized", device_get_realized, device_set_realized, NULL); diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index b61cf82ded..f023c6e9c7 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -143,6 +143,7 @@ struct DeviceState { bool pending_deleted_event; QemuOpts *opts; int hotplugged; + bool allow_unplug_during_migration; BusState *parent_bus; QLIST_HEAD(, NamedGPIOList) gpios; QLIST_HEAD(, BusState) child_bus; diff --git a/qdev-monitor.c b/qdev-monitor.c index 9fc8331157..cf9d1fa901 100644 --- a/qdev-monitor.c +++ b/qdev-monitor.c @@ -853,7 +853,7 @@ void qdev_unplug(DeviceState *dev, Error **errp) return; } - if (!migration_is_idle() && !migration_in_setup(migrate_get_current())) { + if (!migration_is_idle() && !dev->allow_unplug_during_migration) { error_setg(errp, "device_del not allowed while migrating"); return; } From patchwork Fri Oct 11 11:20:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175125 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQc73TyFz9sP3 for ; Fri, 11 Oct 2019 22:25:27 +1100 (AEDT) Received: from localhost ([::1]:48638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt2z-00039l-32 for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:25:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43613) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIszF-0007IL-D5 for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIszC-00028b-SS for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:33 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44938) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIszC-00027r-KB for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:30 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D7C1630860C3; Fri, 11 Oct 2019 11:21:29 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C23265C1B2; Fri, 11 Oct 2019 11:21:21 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 07/10] migration: add new migration state wait-unplug Date: Fri, 11 Oct 2019 13:20:12 +0200 Message-Id: <20191011112015.11785-8-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Fri, 11 Oct 2019 11:21:29 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds a new migration state called wait-unplug. It is entered after the SETUP state and will transition into ACTIVE once all devices were succesfully unplugged from the guest. So if a guest doesn't respond or takes long to honor the unplug request the user will see the migration state 'wait-unplug'. In the migration thread we query failover devices if they're are still pending the guest unplug. When all are unplugged the migration continues. We give it a defined number of iterations including small waiting periods before we proceed. Signed-off-by: Jens Freimann --- include/migration/vmstate.h | 2 ++ migration/migration.c | 34 ++++++++++++++++++++++++++++++++++ migration/migration.h | 3 +++ migration/savevm.c | 36 ++++++++++++++++++++++++++++++++++++ migration/savevm.h | 1 + qapi/migration.json | 5 ++++- 6 files changed, 80 insertions(+), 1 deletion(-) diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index 1fbfd099dd..39ef125225 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -186,6 +186,8 @@ struct VMStateDescription { int (*pre_save)(void *opaque); int (*post_save)(void *opaque); bool (*needed)(void *opaque); + bool (*dev_unplug_pending)(void *opaque); + const VMStateField *fields; const VMStateDescription **subsections; }; diff --git a/migration/migration.c b/migration/migration.c index 5f7e4d15e9..a17d9fb990 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -52,9 +52,14 @@ #include "hw/qdev-properties.h" #include "monitor/monitor.h" #include "net/announce.h" +#include "qemu/queue.h" #define MAX_THROTTLE (32 << 20) /* Migration transfer speed throttling */ +/* Time in milliseconds to wait for guest OS to unplug PCI device */ +#define FAILOVER_GUEST_UNPLUG_WAIT 10000 +#define FAILOVER_UNPLUG_RETRIES 5 + /* Amount of time to allocate to each "chunk" of bandwidth-throttled * data. */ #define BUFFER_DELAY 100 @@ -954,6 +959,9 @@ static void fill_source_migration_info(MigrationInfo *info) case MIGRATION_STATUS_CANCELLED: info->has_status = true; break; + case MIGRATION_STATUS_WAIT_UNPLUG: + info->has_status = true; + break; } info->status = s->state; } @@ -1695,6 +1703,7 @@ bool migration_is_idle(void) case MIGRATION_STATUS_COLO: case MIGRATION_STATUS_PRE_SWITCHOVER: case MIGRATION_STATUS_DEVICE: + case MIGRATION_STATUS_WAIT_UNPLUG: return false; case MIGRATION_STATUS__MAX: g_assert_not_reached(); @@ -3224,6 +3233,8 @@ static void *migration_thread(void *opaque) int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST); MigThrError thr_error; bool urgent = false; + bool all_unplugged = true; + int i = 0; rcu_register_thread(); @@ -3260,6 +3271,27 @@ static void *migration_thread(void *opaque) qemu_savevm_state_setup(s->to_dst_file); + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, + MIGRATION_STATUS_WAIT_UNPLUG); + + while (i < FAILOVER_UNPLUG_RETRIES && + s->state == MIGRATION_STATUS_WAIT_UNPLUG) { + i++; + qemu_sem_timedwait(&s->wait_unplug_sem, FAILOVER_GUEST_UNPLUG_WAIT); + all_unplugged = qemu_savevm_state_guest_unplug_pending(); + if (all_unplugged) { + break; + } + } + + if (all_unplugged) { + migrate_set_state(&s->state, MIGRATION_STATUS_WAIT_UNPLUG, + MIGRATION_STATUS_ACTIVE); + } else { + migrate_set_state(&s->state, MIGRATION_STATUS_WAIT_UNPLUG, + MIGRATION_STATUS_CANCELLING); + } + s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE); @@ -3508,6 +3540,7 @@ static void migration_instance_finalize(Object *obj) qemu_mutex_destroy(&ms->qemu_file_lock); g_free(params->tls_hostname); g_free(params->tls_creds); + qemu_sem_destroy(&ms->wait_unplug_sem); qemu_sem_destroy(&ms->rate_limit_sem); qemu_sem_destroy(&ms->pause_sem); qemu_sem_destroy(&ms->postcopy_pause_sem); @@ -3553,6 +3586,7 @@ static void migration_instance_init(Object *obj) qemu_sem_init(&ms->postcopy_pause_rp_sem, 0); qemu_sem_init(&ms->rp_state.rp_sem, 0); qemu_sem_init(&ms->rate_limit_sem, 0); + qemu_sem_init(&ms->wait_unplug_sem, 0); qemu_mutex_init(&ms->qemu_file_lock); } diff --git a/migration/migration.h b/migration/migration.h index 4f2fe193dc..79b3dda146 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -206,6 +206,9 @@ struct MigrationState /* Flag set once the migration thread called bdrv_inactivate_all */ bool block_inactive; + /* Migration is waiting for guest to unplug device */ + QemuSemaphore wait_unplug_sem; + /* Migration is paused due to pause-before-switchover */ QemuSemaphore pause_sem; diff --git a/migration/savevm.c b/migration/savevm.c index bb9462a54d..26e5bde687 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -942,6 +942,20 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +static int qemu_savevm_nr_failover_devices(void) +{ + SaveStateEntry *se; + int n = 0; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + if (se->vmsd && se->vmsd->dev_unplug_pending) { + n++; + } + } + + return n; +} + void qemu_savevm_send_colo_enable(QEMUFile *f) { trace_savevm_send_colo_enable(); @@ -1113,6 +1127,28 @@ void qemu_savevm_state_header(QEMUFile *f) } } +bool qemu_savevm_state_guest_unplug_pending(void) +{ + int nr_failover_devs; + SaveStateEntry *se; + bool ret = false; + int n = 0; + + nr_failover_devs = qemu_savevm_nr_failover_devices(); + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + if (!se->vmsd || !se->vmsd->dev_unplug_pending) { + continue; + } + ret = se->vmsd->dev_unplug_pending(se->opaque); + if (!ret) { + n++; + } + } + + return n == nr_failover_devs; +} + void qemu_savevm_state_setup(QEMUFile *f) { SaveStateEntry *se; diff --git a/migration/savevm.h b/migration/savevm.h index 51a4b9caa8..ba64a7e271 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -31,6 +31,7 @@ bool qemu_savevm_state_blocked(Error **errp); void qemu_savevm_state_setup(QEMUFile *f); +bool qemu_savevm_state_guest_unplug_pending(void); int qemu_savevm_state_resume_prepare(MigrationState *s); void qemu_savevm_state_header(QEMUFile *f); int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy); diff --git a/qapi/migration.json b/qapi/migration.json index 52e69e2868..5a06cd489f 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -133,6 +133,9 @@ # @device: During device serialisation when pause-before-switchover is enabled # (since 2.11) # +# @wait-unplug: wait for device unplug request by guest OS to be completed. +# (since 4.2) +# # Since: 2.3 # ## @@ -140,7 +143,7 @@ 'data': [ 'none', 'setup', 'cancelling', 'cancelled', 'active', 'postcopy-active', 'postcopy-paused', 'postcopy-recover', 'completed', 'failed', 'colo', - 'pre-switchover', 'device' ] } + 'pre-switchover', 'device', 'wait-unplug' ] } ## # @MigrationInfo: From patchwork Fri Oct 11 11:20:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175127 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQg72D1lz9sP3 for ; Fri, 11 Oct 2019 22:28:03 +1100 (AEDT) Received: from localhost ([::1]:48678 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt5U-0006Qi-NR for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:28:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43639) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIszK-0007Pp-BC for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIszJ-0002G3-ET for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59216) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIszJ-0002Ew-8y for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:37 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7CD763065603; Fri, 11 Oct 2019 11:21:36 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9A818196B2; Fri, 11 Oct 2019 11:21:31 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 08/10] libqos: tolerate wait-unplug migration state Date: Fri, 11 Oct 2019 13:20:13 +0200 Message-Id: <20191011112015.11785-9-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Fri, 11 Oct 2019 11:21:36 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Jens Freimann --- tests/libqos/libqos.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/libqos/libqos.c b/tests/libqos/libqos.c index d71557c5cb..f229eb2cb8 100644 --- a/tests/libqos/libqos.c +++ b/tests/libqos/libqos.c @@ -125,7 +125,8 @@ void migrate(QOSState *from, QOSState *to, const char *uri) break; } - if ((strcmp(st, "setup") == 0) || (strcmp(st, "active") == 0)) { + if ((strcmp(st, "setup") == 0) || (strcmp(st, "active") == 0) + || (strcmp(st, "wait-unplug") == 0)) { qobject_unref(rsp); g_usleep(5000); continue; From patchwork Fri Oct 11 11:20:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175124 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQc02QvKz9sP3 for ; Fri, 11 Oct 2019 22:25:20 +1100 (AEDT) Received: from localhost ([::1]:48634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt2s-00030i-49 for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:25:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43689) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIszU-0007aN-Vo for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIszS-0002Lg-SQ for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35162) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIszS-0002LH-KB for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:46 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D5C48796EF; Fri, 11 Oct 2019 11:21:45 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 034F75D71C; Fri, 11 Oct 2019 11:21:37 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 09/10] net/virtio: add failover support Date: Fri, 11 Oct 2019 13:20:14 +0200 Message-Id: <20191011112015.11785-10-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 11 Oct 2019 11:21:45 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds support to handle failover device pairs of a virtio-net device and a vfio-pci device, where the virtio-net acts as the standby device and the vfio-pci device as the primary. The general idea is that we have a pair of devices, a vfio-pci and a emulated (virtio-net) device. Before migration the vfio device is unplugged and data flows to the emulated device, on the target side another vfio-pci device is plugged in to take over the data-path. In the guest the net_failover module will pair net devices with the same MAC address. To achieve this we need: 1. Provide a callback function for the should_be_hidden DeviceListener. It is called when the primary device is plugged in. Evaluate the QOpt passed in to check if it is the matching primary device. It returns two values: - one to signal if the device to be added is the matching primary device - another one to signal to qdev if it should actually continue with adding the device or skip it. In the latter case it stores the device options in the VirtioNet struct and the device is added once the VIRTIO_NET_F_STANDBY feature is negotiated during virtio feature negotiation. If the virtio-net devices are not realized at the time the vfio-pci devices are realized, we need to connect the devices later. This way we make sure primary and standby devices can be specified in any order. 2. Register a callback for migration status notifier. When called it will unplug its primary device before the migration happens. 3. Register a callback for the migration code that checks if a device needs to be unplugged from the guest. Signed-off-by: Jens Freimann --- hw/net/virtio-net.c | 267 +++++++++++++++++++++++++++++++++ include/hw/virtio/virtio-net.h | 12 ++ include/hw/virtio/virtio.h | 1 + 3 files changed, 280 insertions(+) diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 9f11422337..fb62e18b65 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -12,6 +12,7 @@ */ #include "qemu/osdep.h" +#include "qemu/atomic.h" #include "qemu/iov.h" #include "qemu/main-loop.h" #include "qemu/module.h" @@ -21,6 +22,10 @@ #include "net/tap.h" #include "qemu/error-report.h" #include "qemu/timer.h" +#include "qemu/option.h" +#include "qemu/option_int.h" +#include "qemu/config-file.h" +#include "qapi/qmp/qdict.h" #include "hw/virtio/virtio-net.h" #include "net/vhost_net.h" #include "net/announce.h" @@ -28,11 +33,15 @@ #include "qapi/error.h" #include "qapi/qapi-events-net.h" #include "hw/qdev-properties.h" +#include "qapi/qapi-types-migration.h" +#include "qapi/qapi-events-migration.h" #include "hw/virtio/virtio-access.h" #include "migration/misc.h" #include "standard-headers/linux/ethtool.h" #include "sysemu/sysemu.h" #include "trace.h" +#include "monitor/qdev.h" +#include "hw/pci/pci.h" #define VIRTIO_NET_VM_VERSION 11 @@ -746,9 +755,85 @@ static inline uint64_t virtio_net_supported_guest_offloads(VirtIONet *n) return virtio_net_guest_offloads_by_features(vdev->guest_features); } +static void failover_add_primary(VirtIONet *n) +{ + Error *err = NULL; + + n->primary_device_opts = qemu_opts_find(qemu_find_opts("device"), + n->primary_device_id); + if (n->primary_device_opts) { + n->primary_dev = qdev_device_add(n->primary_device_opts, &err); + if (err) { + qemu_opts_del(n->primary_device_opts); + } + if (n->primary_dev) { + n->primary_bus = n->primary_dev->parent_bus; + if (err) { + qdev_unplug(n->primary_dev, &err); + qdev_set_id(n->primary_dev, ""); + + } + } + } + if (err) { + error_report_err(err); + } +} + +static int is_my_primary(void *opaque, QemuOpts *opts, Error **errp) +{ + VirtIONet *n = opaque; + int ret = 0; + + const char *standby_id = qemu_opt_get(opts, "net_failover_pair_id"); + + if (standby_id != NULL && (g_strcmp0(standby_id, n->netclient_name) == 0)) { + n->primary_device_id = g_strdup(opts->id); + ret = 1; + } + + return ret; +} + +static DeviceState *virtio_net_find_primary(VirtIONet *n, Error *err) +{ + DeviceState *dev = NULL; + + if (qemu_opts_foreach(qemu_find_opts("device"), + is_my_primary, n, &err)) { + if (n->primary_device_id) { + dev = qdev_find_recursive(sysbus_get_default(), + n->primary_device_id); + } else { + return NULL; + } + } + return dev; +} + + + +static DeviceState *virtio_connect_failover_devices(VirtIONet *n, + DeviceState *dev, + Error **errp) +{ + DeviceState *prim_dev = NULL; + + prim_dev = virtio_net_find_primary(n, *errp); + if (prim_dev) { + n->primary_device_id = g_strdup(prim_dev->id); + n->primary_device_opts = prim_dev->opts; + } else { + error_setg(errp, "connect: virtio_net: Could not find primary device"); + } + + return prim_dev; +} + static void virtio_net_set_features(VirtIODevice *vdev, uint64_t features) { VirtIONet *n = VIRTIO_NET(vdev); + Error *err = NULL; int i; if (n->mtu_bypass_backend && @@ -790,6 +875,22 @@ static void virtio_net_set_features(VirtIODevice *vdev, uint64_t features) } else { memset(n->vlans, 0xff, MAX_VLAN >> 3); } + + if (virtio_has_feature(features, VIRTIO_NET_F_STANDBY)) { + atomic_set(&n->primary_should_be_hidden, false); + failover_add_primary(n); + if (!n->primary_dev) { + n->primary_dev = virtio_connect_failover_devices(n, n->qdev, &err); + if (!n->primary_dev) { + failover_add_primary(n); + n->primary_dev = virtio_connect_failover_devices(n, n->qdev, + &err); + } + } + } + if (err) { + error_report_err(err); + } } static int virtio_net_handle_rx_mode(VirtIONet *n, uint8_t cmd, @@ -2630,6 +2731,135 @@ void virtio_net_set_netclient_name(VirtIONet *n, const char *name, n->netclient_type = g_strdup(type); } +static bool failover_unplug_primary(VirtIONet *n) +{ + HotplugHandler *hotplug_ctrl; + PCIDevice *pci_dev; + Error *err = NULL; + + hotplug_ctrl = qdev_get_hotplug_handler(n->primary_dev); + if (hotplug_ctrl) { + pci_dev = PCI_DEVICE(n->primary_dev); + pci_dev->partially_hotplugged = true; + hotplug_handler_unplug_request(hotplug_ctrl, n->primary_dev, &err); + if (err) { + error_report_err(err); + return false; + } + } else { + return false; + } + return true; +} + +static bool failover_replug_primary(VirtIONet *n, Error **errp) +{ + HotplugHandler *hotplug_ctrl; + PCIDevice *pdev = PCI_DEVICE(n->primary_dev); + + if (!pdev->partially_hotplugged) { + return true; + } + if (!n->primary_device_opts) { + n->primary_device_opts = qemu_opts_from_qdict( + qemu_find_opts("device"), + n->primary_device_dict, errp); + } + if (n->primary_device_opts) { + qdev_set_parent_bus(n->primary_dev, n->primary_bus); + n->primary_should_be_hidden = false; + qemu_opt_set_bool(n->primary_device_opts, + "partially_hotplugged", true, errp); + hotplug_ctrl = qdev_get_hotplug_handler(n->primary_dev); + if (hotplug_ctrl) { + hotplug_handler_pre_plug(hotplug_ctrl, n->primary_dev, errp); + hotplug_handler_plug(hotplug_ctrl, n->primary_dev, errp); + } + if (!n->primary_dev) { + error_setg(errp, "virtio_net: couldn't find primary device"); + } + } + return *errp != NULL; +} + +static void virtio_net_handle_migration_primary(VirtIONet *n, + MigrationState *s) +{ + bool should_be_hidden; + Error *err = NULL; + + should_be_hidden = atomic_read(&n->primary_should_be_hidden); + + if (!n->primary_dev) { + n->primary_dev = virtio_connect_failover_devices(n, n->qdev, &err); + if (!n->primary_dev) { + return; + } + } + + if (migration_in_setup(s) && !should_be_hidden && + n->primary_dev) { + if (failover_unplug_primary(n)) { + vmstate_unregister(n->primary_dev, qdev_get_vmsd(n->primary_dev), + n->primary_dev); + qapi_event_send_unplug_primary(n->primary_device_id); + atomic_set(&n->primary_should_be_hidden, true); + } else { + warn_report("virtio_net: Couldn't unplug primary device"); + } + } else if (migration_has_failed(s)) { + /* We already unplugged the device let's plugged it back */ + if (!failover_replug_primary(n, &err)) { + if (err) { + error_report_err(err); + } + } + } +} + +static void virtio_net_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *s = data; + VirtIONet *n = container_of(notifier, VirtIONet, migration_state); + virtio_net_handle_migration_primary(n, s); +} + +static void virtio_net_primary_should_be_hidden(DeviceListener *listener, + QemuOpts *device_opts, bool *match_found, bool *hide) +{ + VirtIONet *n = container_of(listener, VirtIONet, primary_listener); + + n->primary_device_dict = qemu_opts_to_qdict(device_opts, + n->primary_device_dict); + if (n->primary_device_dict) { + g_free(n->standby_id); + n->standby_id = g_strdup(qdict_get_try_str(n->primary_device_dict, + "net_failover_pair_id")); + } + if (device_opts && g_strcmp0(n->standby_id, n->netclient_name) == 0) { + *match_found = true; + } else { + *match_found = false; + *hide = false; + g_free(n->standby_id); + n->primary_device_dict = NULL; + return; + } + + n->primary_device_opts = device_opts; + + /* primary_should_be_hidden is set during feature negotiation */ + *hide = atomic_read(&n->primary_should_be_hidden); + + if (n->primary_device_dict) { + g_free(n->primary_device_id); + n->primary_device_id = g_strdup(qdict_get_try_str( + n->primary_device_dict, "id")); + if (!n->primary_device_id) { + warn_report("primary_device_id NULL"); + } + } +} static void virtio_net_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); @@ -2660,6 +2890,16 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp) n->host_features |= (1ULL << VIRTIO_NET_F_SPEED_DUPLEX); } + if (n->failover) { + n->primary_listener.should_be_hidden = + virtio_net_primary_should_be_hidden; + atomic_set(&n->primary_should_be_hidden, true); + device_listener_register(&n->primary_listener); + n->migration_state.notify = virtio_net_migration_state_notifier; + add_migration_state_change_notifier(&n->migration_state); + n->host_features |= (1ULL << VIRTIO_NET_F_STANDBY); + } + virtio_net_set_config_size(n, n->host_features); virtio_init(vdev, "virtio-net", VIRTIO_ID_NET, n->config_size); @@ -2782,6 +3022,13 @@ static void virtio_net_device_unrealize(DeviceState *dev, Error **errp) g_free(n->mac_table.macs); g_free(n->vlans); + if (n->failover) { + g_free(n->primary_device_id); + g_free(n->standby_id); + qobject_unref(n->primary_device_dict); + n->primary_device_dict = NULL; + } + max_queues = n->multiqueue ? n->max_queues : 1; for (i = 0; i < max_queues; i++) { virtio_net_del_queue(n, i); @@ -2819,6 +3066,23 @@ static int virtio_net_pre_save(void *opaque) return 0; } +static bool primary_unplug_pending(void *opaque) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VirtIONet *n = VIRTIO_NET(vdev); + + return n->primary_dev ? n->primary_dev->pending_deleted_event : false; +} + +static bool dev_unplug_pending(void *opaque) +{ + DeviceState *dev = opaque; + VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(dev); + + return vdc->primary_unplug_pending(dev); +} + static const VMStateDescription vmstate_virtio_net = { .name = "virtio-net", .minimum_version_id = VIRTIO_NET_VM_VERSION, @@ -2828,6 +3092,7 @@ static const VMStateDescription vmstate_virtio_net = { VMSTATE_END_OF_LIST() }, .pre_save = virtio_net_pre_save, + .dev_unplug_pending = dev_unplug_pending, }; static Property virtio_net_properties[] = { @@ -2889,6 +3154,7 @@ static Property virtio_net_properties[] = { true), DEFINE_PROP_INT32("speed", VirtIONet, net_conf.speed, SPEED_UNKNOWN), DEFINE_PROP_STRING("duplex", VirtIONet, net_conf.duplex_str), + DEFINE_PROP_BOOL("failover", VirtIONet, failover, false), DEFINE_PROP_END_OF_LIST(), }; @@ -2913,6 +3179,7 @@ static void virtio_net_class_init(ObjectClass *klass, void *data) vdc->guest_notifier_pending = virtio_net_guest_notifier_pending; vdc->legacy_features |= (0x1 << VIRTIO_NET_F_GSO); vdc->vmsd = &vmstate_virtio_net_device; + vdc->primary_unplug_pending = primary_unplug_pending; } static const TypeInfo virtio_net_info = { diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h index b96f0c643f..3da4ca382a 100644 --- a/include/hw/virtio/virtio-net.h +++ b/include/hw/virtio/virtio-net.h @@ -18,6 +18,7 @@ #include "standard-headers/linux/virtio_net.h" #include "hw/virtio/virtio.h" #include "net/announce.h" +#include "qemu/option_int.h" #define TYPE_VIRTIO_NET "virtio-net-device" #define VIRTIO_NET(obj) \ @@ -43,6 +44,7 @@ typedef struct virtio_net_conf int32_t speed; char *duplex_str; uint8_t duplex; + char *primary_id_str; } virtio_net_conf; /* Coalesced packets type & status */ @@ -185,6 +187,16 @@ struct VirtIONet { AnnounceTimer announce_timer; bool needs_vnet_hdr_swap; bool mtu_bypass_backend; + QemuOpts *primary_device_opts; + QDict *primary_device_dict; + DeviceState *primary_dev; + BusState *primary_bus; + char *primary_device_id; + char *standby_id; + bool primary_should_be_hidden; + bool failover; + DeviceListener primary_listener; + Notifier migration_state; }; void virtio_net_set_netclient_name(VirtIONet *n, const char *name, diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index 48e8d04ff6..0c857ecf1a 100644 --- a/include/hw/virtio/virtio.h +++ b/include/hw/virtio/virtio.h @@ -159,6 +159,7 @@ typedef struct VirtioDeviceClass { void (*save)(VirtIODevice *vdev, QEMUFile *f); int (*load)(VirtIODevice *vdev, QEMUFile *f, int version_id); const VMStateDescription *vmsd; + bool (*primary_unplug_pending)(void *opaque); } VirtioDeviceClass; void virtio_instance_init_common(Object *proxy_obj, void *data, From patchwork Fri Oct 11 11:20:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 1175126 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46qQfF0VxSz9sP3 for ; Fri, 11 Oct 2019 22:27:17 +1100 (AEDT) Received: from localhost ([::1]:48664 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIt4k-0005So-De for incoming@patchwork.ozlabs.org; Fri, 11 Oct 2019 07:27:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43725) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iIszd-0007kW-Ja for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iIszc-0002R5-DW for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:57 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57608) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iIszc-0002Qd-5U for qemu-devel@nongnu.org; Fri, 11 Oct 2019 07:21:56 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 67E1A1DCD; Fri, 11 Oct 2019 11:21:55 +0000 (UTC) Received: from localhost (dhcp-192-238.str.redhat.com [10.33.192.238]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4DEFB5C1B2; Fri, 11 Oct 2019 11:21:47 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v3 10/10] vfio: unplug failover primary device before migration Date: Fri, 11 Oct 2019 13:20:15 +0200 Message-Id: <20191011112015.11785-11-jfreimann@redhat.com> In-Reply-To: <20191011112015.11785-1-jfreimann@redhat.com> References: <20191011112015.11785-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.71]); Fri, 11 Oct 2019 11:21:55 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: parav@mellanox.com, mst@redhat.com, aadam@redhat.com, dgilbert@redhat.com, alex.williamson@redhat.com, laine@redhat.com, ailan@redhat.com, ehabkost@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" As usual block all vfio-pci devices from being migrated, but make an exception for failover primary devices. This is achieved by setting unmigratable to 0 but also add a migration blocker for all vfio-pci devices except failover primary devices. These will be unplugged before migration happens by the migration handler of the corresponding virtio-net standby device. Signed-off-by: Jens Freimann --- hw/vfio/pci.c | 35 ++++++++++++++++++++++++++++++++++- hw/vfio/pci.h | 2 ++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c5e6fe61cb..64cf8e07d9 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -40,6 +40,9 @@ #include "pci.h" #include "trace.h" #include "qapi/error.h" +#include "migration/blocker.h" +#include "qemu/option.h" +#include "qemu/option_int.h" #define TYPE_VFIO_PCI "vfio-pci" #define PCI_VFIO(obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI) @@ -2698,6 +2701,12 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static int has_net_failover_arg(void *opaque, const char *name, + const char *value, Error **errp) +{ + return (strcmp(name, "net_failover_pair_id") == 0); +} + static void vfio_realize(PCIDevice *pdev, Error **errp) { VFIOPCIDevice *vdev = PCI_VFIO(pdev); @@ -2710,6 +2719,20 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) int groupid; int i, ret; bool is_mdev; + uint16_t class_id; + + if (qemu_opt_foreach(pdev->qdev.opts, has_net_failover_arg, + (void *) pdev->qdev.opts, &err) == 0) { + error_setg(&vdev->migration_blocker, + "VFIO device doesn't support migration"); + ret = migrate_add_blocker(vdev->migration_blocker, &err); + if (err) { + error_propagate(errp, err); + error_free(vdev->migration_blocker); + } + } else { + pdev->qdev.allow_unplug_during_migration = true; + } if (!vdev->vbasedev.sysfsdev) { if (!(~vdev->host.domain || ~vdev->host.bus || @@ -2812,6 +2835,14 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) goto error; } + if (vdev->net_failover_pair_id != NULL) { + class_id = pci_get_word(pdev->config + PCI_CLASS_DEVICE); + if (class_id != PCI_CLASS_NETWORK_ETHERNET) { + error_setg(errp, "failover device is not an Ethernet device"); + goto error; + } + } + /* vfio emulates a lot for us, but some bits need extra love */ vdev->emulated_config_bits = g_malloc0(vdev->config_size); @@ -3110,6 +3141,8 @@ static Property vfio_pci_dev_properties[] = { display, ON_OFF_AUTO_OFF), DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), + DEFINE_PROP_STRING("net_failover_pair_id", VFIOPCIDevice, + net_failover_pair_id), DEFINE_PROP_UINT32("x-intx-mmap-timeout-ms", VFIOPCIDevice, intx.mmap_timeout, 1100), DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features, @@ -3152,7 +3185,7 @@ static Property vfio_pci_dev_properties[] = { static const VMStateDescription vfio_pci_vmstate = { .name = "vfio-pci", - .unmigratable = 1, + .unmigratable = 0, }; static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 834a90d646..da4417071a 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -134,6 +134,7 @@ typedef struct VFIOPCIDevice { PCIHostDeviceAddress host; EventNotifier err_notifier; EventNotifier req_notifier; + char *net_failover_pair_id; int (*resetfn)(struct VFIOPCIDevice *); uint32_t vendor_id; uint32_t device_id; @@ -168,6 +169,7 @@ typedef struct VFIOPCIDevice { bool no_vfio_ioeventfd; bool enable_ramfb; VFIODisplay *dpy; + Error *migration_blocker; } VFIOPCIDevice; uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len);