From patchwork Mon May 14 16:54:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913103 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="UNV8SBKf"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6KJ4Skjz9s0W for ; Tue, 15 May 2018 02:56:04 +1000 (AEST) Received: from localhost ([::1]:47260 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlW-0007zC-77 for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 12:56:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33269) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkH-0007Tc-T4 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkG-0005Cy-OW for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:46 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:37692) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkG-0005CJ-GL for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:44 -0400 Received: by mail-pg0-x244.google.com with SMTP id a13-v6so5736565pgu.4 for ; Mon, 14 May 2018 09:54:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5au+ODjbKfxZVvObusqN7oQlE8vKtVmpv6Xpp6lUnLc=; b=UNV8SBKfx6XbXm3kILoSRSutwHBkMHWZYuumSg9l0XWJ9IpRFp+QGHJNsuj41CS+YF lzAO77koHoYgOqkcWII20XqdQiHYb+qq+MPqeOuNzFXZuDg++AnTnfoyhmhtu8DvbhpG 8pg2kVSwHvzTNX220WjVnW4BgJsCs9NK+eBrrQ7+0c/E/WH6Fz+qD7Sx7PG0WjXEN5+8 0Leog0tFVB04ksdYh7bHMMMd2ARegYh1TWzSfXBnhZwPO1ZLOAXRDTnSeTMkIzK7H2I6 8YPv4z2AUlgPT2TbluTPs0qE8t835MfUkmHRCjIs6Hf94XsXkX1lwSCRFPBxISfodVvm uKPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5au+ODjbKfxZVvObusqN7oQlE8vKtVmpv6Xpp6lUnLc=; b=EL5Eh1Iaoo5gcX8Tr4BmFzL0+QvKcukWH0lt0ccmDYRwJIRiyP6Tnd7IZdbYrw53Vj O7zIV4prWotj6JFk7JjI4t3oHYOBXOZ9iD5rDcdqWXjacY1WpscuOhTIIFT/3nHiiN2M 1lM8yUeATEHKmCm/snNVlgWhYepcjUIL3mYGDXkvKQ9M4nc0CY6RZIPzhF7zJEEf5jjl IJShEw62j0Pmj33w+KcY9w7wOWGrLg5dICMep4SG5z3Pe4WQ3MXCU4d4ulf8E/VMtdMQ RCL3KrSeZC6yrctS6mqxAEwelhNempNL2Ke3Do6/mYHGbXXDhgkNxLnXP84a46f/vE0T /3Rg== X-Gm-Message-State: ALKqPweaowXSuBX3c8g0wUwqYAG1Vkztl/eMS1j0nVwVSs0tX1SpgW8D FTv+GoBpgNLTZYCK24DC2n/gUg== X-Google-Smtp-Source: AB8JxZr6pluQPARI7zK4G9kgGcHgoOWZqGlJDeL5Q5NIAy4W5s4mCmyyUHvd8Y5nOFrQzF0eWGOtJQ== X-Received: by 2002:a63:824a:: with SMTP id w71-v6mr9272778pgd.37.1526316883079; Mon, 14 May 2018 09:54:43 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.54.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:54:42 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:08 +0800 Message-Id: <20180514165424.12884-2-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V7 RESEND 01/17] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index da6c36dcf7..cd118510c5 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384 @@ -86,6 +87,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=1,ack=1' */ } Connection; uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 62dad2d773..0909a9a8af 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -99,15 +99,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=1' packets from client side, it acks 'fin=1, ack=1' + * packet from server side. From this point, we can ensure that there + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this rare + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_table. + * + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=1, ack=1' packet from client side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = htonl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; } return 0; } /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -139,8 +168,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=1' packets from server side, it acks 'fin=1, ack=1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=1,ack=1' packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=1, ack=1' packet from server side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = ntohl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; } - return 0; } @@ -190,7 +245,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, if (sender == nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +258,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); From patchwork Mon May 14 16:54:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913108 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NqqV40K0"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Mr0Gm7z9s0W for ; Tue, 15 May 2018 02:58:16 +1000 (AEST) Received: from localhost ([::1]:47269 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGnd-0001Qd-LI for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 12:58:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33308) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkM-0007W7-KK for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkL-0005GX-DK for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:50 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:34626) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkL-0005Fq-47 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:49 -0400 Received: by mail-pf0-x241.google.com with SMTP id a14-v6so6308194pfi.1 for ; Mon, 14 May 2018 09:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bPWakVKwKVXcOSvOX/0oAFOpDmI9fYM3+lTTL5xYEss=; b=NqqV40K0pDEIK3N7LXjuUIqUykYmKbx0fUhL9SDHtQXlK4aZZnz6kLFBBOmc2NOtm+ kqAGZs2B7R8h3GxhK51q4p9AzXaApAPUaj1FI3/DXIdxCqMghPUL6ml+UUqfMHqd/O6R WZCWcIODaEBIWSI6pLXWK7YdbfNLOI5X6dqUUNbPaO4TERxGutbALhx48lxuEj2dBbPU R29VhEUNe18KdQ2toODjPksBV4eIqqAMg12K1p7Kl8inOP52QGW8U0VxfScKDfX4HW1o ndWS1Kpf08L61geBz75hhnqjfxx32JpVeIDOy3Jez3r3dPFNNIoLZRL2mpck/uiCyNnF F4NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bPWakVKwKVXcOSvOX/0oAFOpDmI9fYM3+lTTL5xYEss=; b=A7iBGmjIblfbWW78gg3d3n5gP8ZELDIoFsSXck1u2EELCXWx5UEOG/ISn68P4YZ0uJ bmEXB7lqYgr/KmWUJvjE4omVK1ekWx4p73x72s3nq4Nj8tm1s1P1l13HgCdFz/ElkwHR DHpHK3Bb1uoiAXHaqRGeyOu2CvrJEQfitrhHT3gKQnttyNaTSc+GmpOH6U5c+64SWN7p /UgwaHMdWZu6DlUOeZih7Y1E31RoE7lnYMGQa/mse9/ewzD+08DZGTNNIJCuaf0+ewEm w4NrLX+e4MWOoPtrCc3NIsjvz2zu44I+cPfbe3WN/cpoY+DDR8hcPhXX6IxgcqFl+eOl Hi4w== X-Gm-Message-State: ALKqPwfvEtilJx7fn9IghkQwPq8E6KX6kFkZCP1V7TUgqJtwwXdw093a k+QRF2MDl3IDURZCSvffpW+V2Q== X-Google-Smtp-Source: AB8JxZqgycV8hskdWkKpc5UE+wPQAEnFfTv938Wz7BnRfnZFE2oqX1nmngb/0FsB0MQ94c83QAZ9Zw== X-Received: by 2002:a63:7807:: with SMTP id t7-v6mr1985874pgc.125.1526316887728; Mon, 14 May 2018 09:54:47 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.54.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:54:46 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:09 +0800 Message-Id: <20180514165424.12884-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 RESEND 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c4cc..7ff3ae8904 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon May 14 16:54:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913109 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DzJ7vdup"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6NV5tFjz9s0W for ; Tue, 15 May 2018 02:58:50 +1000 (AEST) Received: from localhost ([::1]:47271 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGoC-0001wx-EE for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 12:58:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33362) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkU-0007dL-8c for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkP-0005Jk-9V for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:58 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:44320) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkP-0005J0-3u for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:53 -0400 Received: by mail-pf0-x241.google.com with SMTP id q22-v6so6302003pff.11 for ; Mon, 14 May 2018 09:54:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PYeU+Tc1rcAKSmqU2XR30cyGjmPFM8SQWmmnAuMGfD8=; b=DzJ7vdupTyQhJlgdHAG6YrxIVlIK/+sYRIqFEiOq9T8VEb/dfMUhHxoyI6PEp7GFf3 EJISAMAMEOZbZTDkzScjqY8W5bBjCfKkZlA+KEo0tR7tKPx3S7SpYxQa4FVNvt9hR+ee 44nJXMFUZ8/SJpE3eK89z3658yIRK0X/IAOChNUJ9JiYR6wNO8FX9pY9pOeT8SF5shCQ r7P0DzcZnGegiHtUZI1kKw9MJIThfExssuZljHsgAXnNNGAP60Ce2HLyovQnXubNK967 KoRSUdyj18IJcRjmNuDwAHgFgdM+kFppYT8DOWhH/TgYpCL/ODBZCnEVBMXCYba9d+9+ y+Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PYeU+Tc1rcAKSmqU2XR30cyGjmPFM8SQWmmnAuMGfD8=; b=rvDqRLB0U0/TAdpLhYga9f15mtAC5iDxWf+HDJreW3bjAaxnJ7I7Xuml2wqdy76Phe mfUo86FUP0KYcr/qnSlth0CBdaiwiI7Yt5i0h7oSODILR+waEaba+LV3635ZoGPJAlJP uIrJoaEwhq3rrXamhG7DKmdw4XT88tHIIQr48x1Z8+gqOcyJZXCzYidwHa9HWRU5Ec6x cd6O0LYF9uilHHliGoToOeE7eOk9Zz8mno0CReuP0ZANN+K0q4DsFkAGAbdssU7WHdhP mY/XYKA6G/bk7SMhbhjOBUhCq2m6azpwVf+TkUnACSOxtXl+LoluLSw2tLHAddAkg2Xz LSuQ== X-Gm-Message-State: ALKqPweY7vMHe41CYvc50XaYH3LEMy77pk3Wm3Hx+A/5DxD9TDCXM9ja KH/yuWSwMc1/RCzSKWtZwLvAUQ== X-Google-Smtp-Source: AB8JxZr954ZoScELuK6++Tdg4uEz8GquYSxb3zFH/RSKV5E5dIUfBbZ9T/pZPmdErupDD98flWQuMg== X-Received: by 2002:a65:4dca:: with SMTP id q10-v6mr9192153pgt.22.1526316891838; Mon, 14 May 2018 09:54:51 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.54.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:54:50 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:10 +0800 Message-Id: <20180514165424.12884-4-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 RESEND 03/17] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 7ff3ae8904..05061cd1c4 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static NotifierList colo_compare_notifiers = + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -561,8 +565,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result = NULL; int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -573,10 +593,7 @@ static int colo_old_packet_check_one_conn(Connection *conn, if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } @@ -620,11 +637,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76aea..22ddd512e2 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); #endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon May 14 16:54:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913117 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="pmJe1XZU"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Rz4YfWz9s15 for ; Tue, 15 May 2018 03:01:51 +1000 (AEST) Received: from localhost ([::1]:47294 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGr7-0004Wt-7j for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:01:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33377) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkW-0007fB-Dc for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkV-0005OJ-Cb for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:00 -0400 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:33414) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkV-0005Nm-4y for qemu-devel@nongnu.org; Mon, 14 May 2018 12:54:59 -0400 Received: by mail-pg0-x242.google.com with SMTP id v7-v6so2947399pgs.0 for ; Mon, 14 May 2018 09:54:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DN7Vg/Rz4yeeVoLD+Gpj6knxQgRzYlVwI1aTpWs9rTI=; b=pmJe1XZUVzQbbOZB7316hScjgU0UQKtuw/TDcGIPfeSS+l7zKlsboD8dj7ITK6oSe1 yrdSe+/nFiuDdhPSExiqSr3C8HTnwrdUbXMwvlCglt4zii2waSl5IpfdS8ZDE5LeZqjj yOYSOHeZ7BYie2ynx8CaKIHl4mjxc49GY931/NuwBGttk2V7md8+g2ItQwqd88uUP2B1 UZYvMPEQc724cEyXLCmLYEi5dnSG7WXvrbKRq1ZNzJtYM9BcsrLoPo2WW0tvSiVcCDZz NSg+ex1G04xC/qehl6V+bIh8gSiJaxox0cgoWjbi+nWl5S7uVLfkL6ogrt4JT94shAJJ D/ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DN7Vg/Rz4yeeVoLD+Gpj6knxQgRzYlVwI1aTpWs9rTI=; b=hJEaxnxImX53XrgWvPmE2voJqipCiq8YJTZav+THBVSrjpvlu5mgXJWISRn/Kqp9KH DHIRTATw1TtGjIfhxT8YhfCmqgyyN2W6gmWPQueCKZjeTO+OUhPDWsG/xMkhBcLXslwk AhNus4oJY2GG+N4o1zmd+IatCbJoUNWeRksoMV//ntebKiarDN8eEg8+T6mqRxcfa8Ab Ydij6CF7hxw5NgnlbvVdboejdiyhsZHqJQVx6jyHdnwfigWyNqIXIjiGOPrecYlqkq23 qtbMjeVJQ3jh/9bSaAAJL+9NrAT66SqadtPcGbpbz+rzlU0K2JnS5xV9miE5M8bgJ67a 5rpA== X-Gm-Message-State: ALKqPweKEJ3AKe306orACJEJPcSHVCLtKdKKeNgRAWbYY9i6U4X/4NlX fzuH7Ip9us8jRK6hrNaBAqiPqw== X-Google-Smtp-Source: AB8JxZpeu1lLfWp9x7nSi4MXTI7N1s9p8nJ1Z14hYCeCd7qgs/SkyXFAsIcyuRqd4OPMCCw5IDoI8Q== X-Received: by 2002:a62:a391:: with SMTP id q17-v6mr11411879pfl.87.1526316897807; Mon, 14 May 2018 09:54:57 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.54.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:54:56 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:11 +0800 Message-Id: <20180514165424.12884-5-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V7 RESEND 04/17] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067ed4..081df1835f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier; #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb = qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 35f2781b03..bca187275a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -76,9 +76,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid mode. + * Note: Please change this default value to 20000 when we support hybrid mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 From patchwork Mon May 14 16:54:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913120 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="q7kfXU0z"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Sl35M8z9s0W for ; Tue, 15 May 2018 03:02:31 +1000 (AEST) Received: from localhost ([::1]:47299 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGrl-0006eU-5C for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:02:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33432) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkg-0007py-EF for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGka-0005SA-9Q for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:10 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:45544) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGka-0005RU-1c for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:04 -0400 Received: by mail-pf0-x241.google.com with SMTP id c10-v6so6297463pfi.12 for ; Mon, 14 May 2018 09:55:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CAdi+nmrpwATpkP19hOEO/WiYaqfDSv7fKtIae/GoS0=; b=q7kfXU0zipkt3fmICr1MjRm7YAsmaA7KLoD5wgeMPCNT3Nl6EBOre2OmQCYDWbeVpK YFfl0SLg4dWKFNDV6Uu1Ib5KeE96OinCSJukryGfTPjRQVGh/z4PnrtUa38+BkteFFul XQlthPmzDbplPeGno2qi6ZlnJcxGZgyxZrLobLJ80rTG8gL6psjGkS6Jyf16kMDDW2EZ rfgEaUpev8F9lhVuaUoWYyn1rCH3/Xo0pIJBWwcrnqojvOYG4ntnAF8VmbIREcaz+EL8 TZnP8BjoW8Adn6djPIm/TdzTlH9Zl+/CkRlVeG7Edtr7gzveGTaKWb8mzCCQ+TdxPbwk IodA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CAdi+nmrpwATpkP19hOEO/WiYaqfDSv7fKtIae/GoS0=; b=AlI7LmVUa5FmSt9R2CxfI0LI2o3pEpislF2f1ErpcY8RyXgetulZd38uFylL40AmA2 bIanXrFVs73jJagfmprTy29qioEKpY6bjPtAV5W826RsZN9xvvvRNYoPHXbYQm1W+9SC kwHSDhdImziztQKL//pcV5guOdcZQ6I+8Gerd6QNIOV4ZuY935YS+YvBbQmQDCCGd7qT RbkG9jynfvIjcKmtfCeGQFXjig8RQ1aNpwhk1BpeeZCasMFtgK6PNciJf7tJjT87iF0o SQ+nCTXOiNl5JESueZ86D+6N3An1i97ucKq2qaekH4V574KNZd9WWiCgldQVyFd2Pd56 kIwg== X-Gm-Message-State: ALKqPwe1pDiNMpbvEXrSwqJXZFQs1ioskJmS2RSEPO3zbe++XKBXHe8k 79a3KqzcSNCupgGvPkA26J/rvg== X-Google-Smtp-Source: AB8JxZofjOJGVbHscwcyEmoMLtCPnM1vLTVglHdszN055Jf3RrEk0M7CBdV3r41ApUstTNLyg79ntQ== X-Received: by 2002:a65:550d:: with SMTP id f13-v6mr9248392pgr.324.1526316902671; Mon, 14 May 2018 09:55:02 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.54.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:01 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:12 +0800 Message-Id: <20180514165424.12884-6-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 RESEND 05/17] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df1835f..e06640c3d6 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis = migration_incoming_get_current(); + Error *local_err = NULL; /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s = migrate_get_current(); int old_state; + Error *local_err = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err = NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index bca187275a..ddd0c4b988 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -357,6 +357,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis = migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err = NULL; assert(mis->from_src_file); mis->largest_page_size = qemu_ram_pagesize_largest(); @@ -388,6 +389,14 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); From patchwork Mon May 14 16:54:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913123 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="g2RSprR5"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Xd5tHBz9s0W for ; Tue, 15 May 2018 03:05:53 +1000 (AEST) Received: from localhost ([::1]:47311 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGv1-0000l7-Dc for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:05:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33434) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkg-0007q3-H3 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkf-0005VJ-2w for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:10 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:44866) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGke-0005Uc-Qj for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:09 -0400 Received: by mail-pl0-x242.google.com with SMTP id e6-v6so7702529plt.11 for ; Mon, 14 May 2018 09:55:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YCBkyWH/2taRzE54aLAbNlgqZB58DJwEyni5oiMZt8o=; b=g2RSprR5O32PanyHWlwMNzvq+8t4Uv5SSoGYZw41DnOZ35yP5IdaYWKxMG8sDHHpnf 57o21p32UyHUP33ddzcnpbeUH25m7EnRbrPxJTgO+Zyo5ycpsdNl+i1cAMIa20JYOMYY DMvGxRFkbqtkzMH2wGJ27JGCoKsQoVKO4CBSY+e9qv2jZwJPxoN/N2USflWtIMLZl4tv LBn6l76HW/TVXu1bWSDLDNbQCGhc3r5dVVq5mvtMUrC/pwzEMUXgIv5pLpb9CcGjS9M5 56nG82uThspKODcCV4pvwyp6ys41HxK/vVaPoFV2Go5+KrRqO9Il5j2BXdMcutz+rnHh r7Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YCBkyWH/2taRzE54aLAbNlgqZB58DJwEyni5oiMZt8o=; b=ndSHVbaGWAO1jPhHKXpOVTLdG6R96xqyHoAkQoYj5REyHXguc2YsmEXSxoRTRb+jtA e78/0kzHB0O6k+8Eh6FUgYQJq9AOG3vgxQdepe/CMzYk+2stea0yx1OAsVf4TXJshPAq mGro96wfApamWWT79/oQUmk0UNT4SuTqZwE4GuFXZziPuvE96s5UPY9vTVEkn1ZtN0B4 hdCN0aEesV/851BCglLqx0M+33JMxmfIktfUbY4zXGGVKsIlFpF2MbpMVrLCn+9KkvsS GgpkEtCNrIN1OT4C/TmBVkJcuvFB3yDXT6ZQxlCl7uDdzgNKCsSbAHushuiRWmw86OcW UXoQ== X-Gm-Message-State: ALKqPwfKz+nxiNhnT1w02Q6czgRIGseUcpG7fsQtLIYrOZG8vmYbK7vG KgK9c1C5GyODSUnRcKv0YucOlw== X-Google-Smtp-Source: AB8JxZrfZ5uiBUbjJAMUXewrSJeymljiz63+4+//lc2aJfM1yYEWpkJwQX9CIDT1g16GoJtwWx03Ig== X-Received: by 2002:a17:902:b58e:: with SMTP id a14-v6mr10769841pls.261.1526316907410; Mon, 14 May 2018 09:55:07 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:06 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:13 +0800 Message-Id: <20180514165424.12884-7-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V7 RESEND 06/17] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 20 +++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 60 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fcf4c..99ce17aca7 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47ba8..a4f3bafd86 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4dfe7..0000000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s = opaque; - - s->colo_requested = migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state = { - .name = "COLOState", - .version_id = 1, - .minimum_version_id = 1, - .pre_save = colo_info_pre_save, - .needed = colo_info_need, - .fields = (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested = false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c3d6..c083d3696f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); return NULL; } diff --git a/migration/migration.c b/migration/migration.c index ddd0c4b988..8dee7dd309 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -277,6 +277,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled = true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -388,7 +404,7 @@ static void process_incoming_migration_co(void *opaque) } /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2431,6 +2447,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index e2be02afe4..c43d220220 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -55,6 +55,8 @@ #include "io/channel-buffer.h" #include "io/channel-file.h" #include "sysemu/replay.h" +#include "migration/colo.h" + #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -81,6 +83,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; @@ -836,6 +841,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1793,6 +1804,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis) return ret; } +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1866,6 +1883,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } return 0; diff --git a/migration/savevm.h b/migration/savevm.h index cf4f0d37ca..c6d46b37a2 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,6 +52,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index d6be74b7a7..9295b4cf40 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index 12e31d1aa9..a1576d2045 100644 --- a/vl.c +++ b/vl.c @@ -4437,8 +4437,6 @@ int main(int argc, char **argv, char **envp) #endif } - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); From patchwork Mon May 14 16:54:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913104 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="M9iZv947"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6KQ6D7yz9s0W for ; Tue, 15 May 2018 02:56:10 +1000 (AEST) Received: from localhost ([::1]:47262 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlc-00081N-BI for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 12:56:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33449) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkl-0007xG-Gq for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkk-0005bv-1l for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:15 -0400 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:41026) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkj-0005b7-Kl for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:13 -0400 Received: by mail-pg0-x243.google.com with SMTP id w4-v6so4101507pgq.8 for ; Mon, 14 May 2018 09:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LLU1ZYKEOtTds6nN3aIiShkRr9VFW9YDW2HjIIdtFvI=; b=M9iZv947tyf+2+mYfdmrMjppsiFBEg9qHCQYyJluT7ONkmiPHWeEH7itlaRi0rRJzX ui+dcpdiiqqonRcBL6UffRnN1Daecj/l0kcrloiyzw2OqeY1zouQV+tZyys0LhBRH+yE X86Wh0/nLL3TvOZAPuZxxJ7Amhca2S8+sK6gTkOvtHIvHFZpotz0B5JCus5v4XiGtdN4 ozVahucKg7wZLqPz8RyNoIRdAau6SIatXaoURzlWNGNFrgWaMA7Xj5a//waVaJFTT1ux vQiqiPGI8CCP4219iXooEHfG666bkp/q+sFzpPEx+oykujpb+zST9KLcGxLABvbBDQ+5 059Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LLU1ZYKEOtTds6nN3aIiShkRr9VFW9YDW2HjIIdtFvI=; b=EcKpI6r1mwiLedSRiXO3JXAmveFcX4uTb0x+3nT3IyoQ2nGob1cqe0bkHDDWMqUb9N yznukOH79UrAqm5IX02waerBIDVbn/3XWNYsQJ4y3P0NLOqXhBeC5s9YaU9z8YOd0eC9 ECb4a7JgOS0k3mYJEU5f+V//updQsWYIlbS6BC7xxg6Joz7JTPUFIyaaSlr3WCSR9Xxc MlGiHv2IiAkzJXRE783YRJOTXG9vJuoB3loJPSCrkT1xLXtZZOc4QH0D0CEm0LsAmXkz kw8byHRuuQToiOS297xhFsbNYn4lZQsKVkR0UjhEmMdoMYpk4KCjEEZcAQ8o0ZXm/uSa 0FHg== X-Gm-Message-State: ALKqPweUUn6/T1FMXVTU8V5q85l1o7ZfonOQeu1bsMjFFkUgHBQGqj7w 4tyj/HzHva/csqpne0U/HEWaQw== X-Google-Smtp-Source: AB8JxZrdSflHKmt2dwVt1JtogwnQyNj0L+MK0fIWOvKY1ul42jZMwawiVQaYN2Za2qh5kpWYC4a5zw== X-Received: by 2002:a63:b746:: with SMTP id w6-v6mr8876040pgt.193.1526316912217; Mon, 14 May 2018 09:55:12 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:11 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:14 +0800 Message-Id: <20180514165424.12884-8-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V7 RESEND 07/17] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++++++++++++++++++++++++++++++++++++++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 104 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf2446a176..51ec153a57 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 8dee7dd309..cfc1b958b9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -421,6 +421,8 @@ static void process_incoming_migration_co(void *opaque) /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 912810c18e..7ca845f8a9 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2520,6 +2520,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block, return block->host + offset; } +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2724,6 +2738,57 @@ static void decompress_data_with_multi_threads(QEMUFile *f, qemu_mutex_unlock(&decomp_done_lock); } +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache = qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2740,6 +2805,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } @@ -2753,6 +2819,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap = NULL; } + return 0; } @@ -2966,7 +3033,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host = NULL; + void *host = NULL, *host_bak = NULL; uint8_t ch; addr = qemu_get_be64(f); @@ -2986,13 +3053,36 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(f, flags); - host = host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_cache + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects the + * speed of the migration, but it obviously reduce the downtime of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host = colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak = host; + } + } + if (!migration_incoming_in_colo_state()) { + host = host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret = -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } @@ -3087,6 +3177,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) if (!ret) { ret = qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } ret |= wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 5030be110a..66e9b86ff0 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -64,4 +64,8 @@ bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_offset); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t nr); +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index c43d220220..ec0bff09ce 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1807,7 +1807,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } /* From patchwork Mon May 14 16:54:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913128 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Y6+YMKae"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6cF1HDcz9s0W for ; Tue, 15 May 2018 03:09:01 +1000 (AEST) Received: from localhost ([::1]:47327 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGy2-0003Ey-Pt for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:08:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33461) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGkq-00081D-Rr for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGkp-0005fF-9x for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:20 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:33924) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGko-0005eb-Pq for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:19 -0400 Received: by mail-pl0-x243.google.com with SMTP id ay10-v6so7712632plb.1 for ; Mon, 14 May 2018 09:55:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bJnSGsF2KcrHZo4L+iZB0TIE8jxs9cp4Oz7/H3wjRgc=; b=Y6+YMKaeRzVojzG6mQu5JyHs9I0we/T5FbEgyvml9m6K1uFkCQj5+Vr9CwDh0elUsi SJYiKNjdzcuTdu6Lr7z2BBTUwTkUEnvO4R1uhbrMF6rX/yE5IKIe1SYEwRLUyr5en1sF bKnJzcwmgRIa3fUGl8/RECPEDl0XJjmdfatEWpwwlFy4dcqrdF+2QTTrfHghs9B1R5A7 Sw+ZSHhIvw9YUxuenMZpcobLSPdXrno1L2u/sBy4jji7WmhuOdivdU66fHFlksoOgY8d xtJCRuHmYrs2qGZQbWc+4Y5zvD0B+AU5oDUGfCAMw0Km5gM4rY+Dtqswdz7tcrBT+FMl IP1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bJnSGsF2KcrHZo4L+iZB0TIE8jxs9cp4Oz7/H3wjRgc=; b=dstLpp0QHXxPqlccF5QWMC6wArnPrdu+4Skjc7TFOJAJqvKYMhFUNeGdyJ1ZsZ2IQ1 YMXIygujdKdUzxcdvLEcyOb6XNuvhS1Z9za3GW6emGUBHtBRT+gQJzOfkJ+1kKC8F7CA WHT9jMKq8WXiE0MX1L6URh80toyeBs21Hb7oxMON1+7ltXnmU6waH+Nu+NbqNsfmUDCc CaVt0aPmPQCxKmxUxrkwCC16ltmierUNYLZPNAQQhSJ1v36bAITXnzjy3vI40SvPUvsL dAE3C6xTPwR7dgU0c5VCeoqGtdkT/60SdnmANbpvbz4bQj0/fyfJZ0NXwg/XnKpFb9yw 4Mhg== X-Gm-Message-State: ALKqPwc6cWibTCtsfbJuWUq/uFAXjfQdW2XPMd5rQM/rMlI/+ME03iUH EuBnUrdShiORmKe8LSS5YwoSqA== X-Google-Smtp-Source: AB8JxZpzINtX9s+zukcPHfavDO3y6vs1J1hyB+F9J6i5iIMcajAz/+pTg0Aq9SVg9HM9JEMzmeOf4A== X-Received: by 2002:a17:902:9349:: with SMTP id g9-v6mr10451285plp.375.1526316917417; Mon, 14 May 2018 09:55:17 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:16 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:15 +0800 Message-Id: <20180514165424.12884-9-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V7 RESEND 08/17] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 7ca845f8a9..e35dfee06e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2531,6 +2531,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } @@ -2760,6 +2769,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap together + * with to decide which page in cache should be flushed into SVM's RAM. Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages = block->max_length >> TARGET_PAGE_BITS; + + block->bmap = bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state = g_new0(RAMState, 1); + ram_state->migration_dirty_pages = 0; + return 0; out_locked: @@ -2779,6 +2806,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap = NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2787,6 +2818,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state = NULL; } /** From patchwork Mon May 14 16:54:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913113 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="co9Csyja"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6RJ4Fnsz9s0W for ; Tue, 15 May 2018 03:01:16 +1000 (AEST) Received: from localhost ([::1]:47289 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGqY-000455-5H for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:01:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33482) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGku-00083s-Us for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGku-0005iW-2v for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:25 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:34628) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGkt-0005hl-GU for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:23 -0400 Received: by mail-pf0-x241.google.com with SMTP id a14-v6so6308984pfi.1 for ; Mon, 14 May 2018 09:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6fPviG/TdAWgY7B4YSvh1j6C30LwuMZ+AXnMP/6ZFpM=; b=co9CsyjaoYmr+pZUjdfs0btodDsNSgURVcgZfdxJjRrKElYrKcpm8jD7h6sqOSvUaP s3WfBPoDyk/YYNx0MF/H/SwLaOWgYMovJicrafofHG0CohpaoSFSth4d6YOeyfVJsjZi c0bRDuN7m3958WUQjoYmCXuPK+uZMAGAPGJ4nPUIR+elJifvQFFO/jdOVOstpfI09LIU HTnVktneL79LjqDANnnNGfvHvu5oUR1Zm5o8Gtl1+hQsEe7QfexyufFDSi01A7RUT+si ZORT7oSHDqtSFlId9pFBSxTIuf/iwyH3m+mJ05+4P9l2gMlW84fFVo4JmUSifB5N70nW MHyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6fPviG/TdAWgY7B4YSvh1j6C30LwuMZ+AXnMP/6ZFpM=; b=TeZ+beHnQ6lnfS1konyCPwAZwzO/q8BRqsvGl7l5xucsdjVcIfXYK6hh4UwKeqKjqx h4uRCatF4oJuhEWwZIa5cX0Mit4zNuO2TZUeTbdbe+EKTpKIJyWxro3janSaWqen8Doh IBhXEB7Y2/cu+eZaf9PmLeLIRcuZf6KtimkpqIp4sKrwEhCDj5ye/Vpl99J/huLBdQxA 0KncM+fHVVYrb64n5G0cqFPZgempfAvRVL0RAqfetjjetKoXbBFKaDRlj0c3n7SkBDhK E9DtQVJ9mKKfb8b/EwBax8Pn39BsDo6oiDg5qDY7d9bMkLBrKDAL0sraLKhLyGH+sYU2 uOgQ== X-Gm-Message-State: ALKqPwcDTrZ1JXIeu0XmwvwWEODWAs/9lu0GH7AEEfOA3a95d7hCcV0C AWjzE9zn3zkjvZ4ZDws/BxpctA== X-Google-Smtp-Source: AB8JxZqLkReBtzkJ/ovAN7sjZQP5XqW+sEAYvIto6bhQj1+ob8c3Q1ZCpNSae4g7c9WqbV/tXVO9NA== X-Received: by 2002:a63:9812:: with SMTP id q18-v6mr9001068pgd.170.1526316921897; Mon, 14 May 2018 09:55:21 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:20 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:16 +0800 Message-Id: <20180514165424.12884-10-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 RESEND 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index e35dfee06e..4235a8f24d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3031,6 +3031,40 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages == 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -3043,6 +3077,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) bool postcopy_running = postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); + bool need_flush = false; seq_iter++; @@ -3218,6 +3253,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index 9295b4cf40..8e2f9749e0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Mon May 14 16:54:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913110 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ql0bgdWE"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Nb6Qylz9s0W for ; Tue, 15 May 2018 02:58:55 +1000 (AEST) Received: from localhost ([::1]:47274 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGoH-00021S-HV for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 12:58:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33536) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGl3-0008Bj-4o for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGky-0005lz-KA for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:33 -0400 Received: from mail-pf0-x22d.google.com ([2607:f8b0:400e:c00::22d]:39941) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGky-0005lJ-Dk for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:28 -0400 Received: by mail-pf0-x22d.google.com with SMTP id f189-v6so6306562pfa.7 for ; Mon, 14 May 2018 09:55:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1k4ztbdsYCmsp1MgR4MJL4AoI9gtOnP24l8VMwnkbUw=; b=Ql0bgdWEr84tfIWpOn0lQENrcKtCjwpyaTWlhb7Z5hJlpegDm1LUGda1bNc1PZZvdp Dc+NBi2DScGCEDj2uyUYRg5r3JFKnhDekrWj2qAlyp18eHQ3dJh9mAI11yxI5wVyChsA GWzHuYyhvUsaMw1LrdnmBsr3rnc7zGEUwUY964+hCRvTZ5omdjuqYwAHQY6Dgv9PiPc/ /ShOA4+gbdmnNjXskZXL2efcWzCdX8eAVhKyvdeuPTzbKlG8UvzIoQ/gNiFHMJ0WTDrb 4patKrSfME8iVI9IbzLUfq4R1nHTVMqASlBHUNuIfx6fSt6xpcIGTGA3A9v+gmULIbcm spRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1k4ztbdsYCmsp1MgR4MJL4AoI9gtOnP24l8VMwnkbUw=; b=BXcto+QIIXPRAIJJn9JpkgK9ymowuc04KbR/Ooh4lbqBbSi+NPuvPjM54m+1RwIwRX Y7YygnEm/naa6Lq8xq+l//F9JJz/Ng1f4Aw833MFfXoaCkZzd7VieCyp7kFmU9QslY1v PvWAZk9kPnw5EZ6b+t6fNzd2BIpgFUbJeaBOJRqe+wYfWlehOJu4PQpnO3VmW/rPqnhT WqKQjiWAY7kVlrUMuW5XBsJx7UMyWGrLXZjao7/3vRVlEa5xuTGksvCu69q3IjG57gbe Un/OzR8k21mxRBC2w6zuFFe4oA/5/okVYkXhb+026K9xW2fcaCzkVojhR6hZxDFIkLxe vZtg== X-Gm-Message-State: ALKqPwc+uP1F0+zPM5N+3Dkf8u0Z/loi5vP/ybO3LQbR+TlaIFq5VGKP 9PvTwd4BS7tVk8jA9Cm2nQEtCQ== X-Google-Smtp-Source: AB8JxZrlLC+84x51R+zu5NyZxe6fVQT1wD/xybb5vHWGC5nhysJW0LlTMVbEnyHTD1fbsxmsRc2TtQ== X-Received: by 2002:a62:883:: with SMTP id 3-v6mr11471404pfi.154.1526316927076; Mon, 14 May 2018 09:55:27 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:25 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:17 +0800 Message-Id: <20180514165424.12884-11-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::22d Subject: [Qemu-devel] [PATCH V7 RESEND 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 20 ++++++++++++++++++++ qapi/migration.json | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d3696f..8ca63813c2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,18 @@ out: qemu_fclose(fb); } + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() == FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -744,6 +757,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() == FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } if (fb) { qemu_fclose(fb); diff --git a/qapi/migration.json b/qapi/migration.json index f3974c6807..55dae48089 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -874,6 +874,43 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.13 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: # From patchwork Mon May 14 16:54:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913132 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JuJqtQ8+"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6gd3ybpz9s0q for ; Tue, 15 May 2018 03:11:57 +1000 (AEST) Received: from localhost ([::1]:47351 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIH0t-0005jV-3G for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:11:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33548) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGl4-0008D4-G4 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGl3-0005qU-O8 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:34 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:42013) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGl3-0005oh-IC for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:33 -0400 Received: by mail-pg0-x241.google.com with SMTP id p9-v6so5731580pgc.9 for ; Mon, 14 May 2018 09:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=c3c1I96ArOBhkzzXNc0WYEI8Zrc5NZ/DICc6d2S57jo=; b=JuJqtQ8+IVbZH1vRvkeC+nHT1bJjdeSOt4yM3PC5uxJQZf+KIhZoZYtAsrjjvBqqaF t1TMKp8NhVQ3KD5/xcPjUVLdvAjDpSy94r/23MNWly2oUzFX/jjCzq8MWX5mQFyWhTy2 fDHTS04p6Zer9va1euYrLNp3xbXMoLjPCvbeNm88F4imJi0riemKp0YBpfVmTRpj82oA Lgg6cgS7pLb7vzvN47/xJdZvQKdiYxIE2YXzqvJmrRL3pJNXNxe45f/eVngAd/ekp5ip g7D5S7IIevz3SuTrEkxA+4MB0v1ohIAnWbdS3JzByEsgkIBd2eHaGHhlOuvVIHlPhfQR U2qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=c3c1I96ArOBhkzzXNc0WYEI8Zrc5NZ/DICc6d2S57jo=; b=g7S6WtLhV5PQE5VtuHn6c1S0YUzrIjGfDnjXCVM7IAX1/H/EakNfUpDtAUkpIrXDiL P1fNCsPcB7lOA7k3WhxONvy+VX7nH7t35WoiS2hZYq1ZcCflTKPCHJ5BdHAfba+VfDu9 RUcQiIgr4cevBpC4dTBefIkXBC9GOagyZ6S7FFp2IHwDMAK4Pfl7lHoUyGkmY565ewTn W+DTxmUad/s/sfZrnre7UFCG1WV13QjPbL9e15M8xRD+9hmICgQmAbMvAP0WB0XK1Mo+ YcEThJdvqAPxaqxNsMKRrzy7ti13ueEnNNZ9iq0FZnZgzUwtpGXhoN0x6JCU4gVGU8/H +YkA== X-Gm-Message-State: ALKqPwfWi2tOBeK0U9Qx1uXAJYv7rLX62ErmvkyrW5TwmBG/C8HQAzQ7 LjIcgNY5USTKqp9jWoPsXVrLcA== X-Google-Smtp-Source: AB8JxZqYTUk8VbETAzmbzcbDYkv9JqGe5bkDhK3rjVRK0vjWt0JpkngxbezYKj5TAQbXPPPidYOxeg== X-Received: by 2002:a65:6604:: with SMTP id w4-v6mr9175411pgv.102.1526316931776; Mon, 14 May 2018 09:55:31 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:30 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:18 +0800 Message-Id: <20180514165424.12884-12-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V7 RESEND 11/17] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 34 ++++++++++++++++++++++++++++++++++ qapi/migration.json | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8ca63813c2..cdff0a2490 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,39 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } +COLOStatus *qmp_query_colo_status(Error **errp) +{ + int state; + COLOStatus *s = g_new0(COLOStatus, 1); + + if (get_colo_mode() == COLO_MODE_UNKNOWN) { + error_setg(errp, QERR_FEATURE_DISABLED, "colo"); + s->colo_running = false; + goto out; + } else if (get_colo_mode() == COLO_MODE_PRIMARY) { + state = migrate_get_current()->state; + } else { + state = migration_incoming_get_current()->state; + } + s->colo_running = state == MIGRATION_STATUS_COLO; + +out: + s->mode = get_colo_mode(); + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason = COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason = COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason = COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 55dae48089..13589ba948 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1220,3 +1220,36 @@ # Since: 2.9 ## { 'command': 'xen-colo-do-checkpoint' } + +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @colo-running: true if COLO is running. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'colo-running': 'bool', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "colo-running": true, "reason": "request" } } +# +# Since: 2.13 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } From patchwork Mon May 14 16:54:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913118 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Dc2e5XW0"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6S25ZYwz9s0W for ; Tue, 15 May 2018 03:01:54 +1000 (AEST) Received: from localhost ([::1]:47295 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGrA-0004cW-Ey for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:01:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33577) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGl9-0008Hj-Cy for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGl8-0005u3-6W for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:39 -0400 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:46014) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGl7-0005tY-Tm for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:38 -0400 Received: by mail-pg0-x243.google.com with SMTP id w3-v6so5729883pgv.12 for ; Mon, 14 May 2018 09:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ekv/jKAygJzBdwEjjRtgSgnac/JnStAGMbf8vYEf0r8=; b=Dc2e5XW0tr5Ay3eqSko7BdvCkAaAPtFjdRpwpkBe/zccar8XiNfpzMKZoAAPL5KaAF su1wWhcMfKWw6Dm33et2QWrwSUZ8HSPShJSmbhCwXE86tN3hG1Km/SYmwGht1c7nBnsh GyMl+MQcSgr/csgvDRAisMwnV522EoOwKEqoa5SVI0oG3yJu+OcjI8F/V/tjYhaAnZIq dA4QltUcgHVCfKWOYNe6si/2EY8CPexkLge+rKbR4zRRMc3TuT76shZ+y0Wq1zeRJ0WY s9zFUpLWtirIMJr62VPOans2C2l+fM/KimksOxl76W+o7+5/Q3xHmI2DYtJxiYF/25Ww ukew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ekv/jKAygJzBdwEjjRtgSgnac/JnStAGMbf8vYEf0r8=; b=K1MioqD5oyBFAxMLOgCozqd0Jg13uoCRru9XbmMyPR2fHQe8/yCAC9If6FOL2iqjsH W2307omJPyVU1D8LnsPe4AQQMLHf6AwLs7lRMN2/m/Q8Xskd/iLPED2Gx+wdLQP0fdcJ t+cq15WGh7c23gXXijYiPTYFGbcfxeblk9qDoSmZKlFKLmmRtSnEgNDOeqNQBu3NyTWa 2UYYNbw9y/U6bpbhamOI1EqWZJQkzcfhOImPNXqXuvewFi4L0QVD6yMmUQDd4DAMPe0O WbR4ZTQzWTKAE1o8eNPxZL+px3zIpSHWXGdXHZiY5v2ZwH60yIrjRSwfuh+SPARIlc5c 6WzA== X-Gm-Message-State: ALKqPwfpDgq79M7Ek2nEqwSx0ckrWRb+J64ehShov8eoSFmXjJaBKzR4 5rdKIBh7X1dzjz+VtNwhc8lXvw== X-Google-Smtp-Source: AB8JxZqmgJFdg4XhOKCgwA4+HqbIbmyhNK0uQvG1f7fc+jDpSSspVMeuv9NxMv0giD4XSyjLErKFew== X-Received: by 2002:a65:628a:: with SMTP id f10-v6mr9215021pgv.6.1526316936530; Mon, 14 May 2018 09:55:36 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:35 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:19 +0800 Message-Id: <20180514165424.12884-13-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V7 RESEND 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index cdff0a2490..5b055f79f1 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -414,23 +415,30 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -643,6 +651,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -715,6 +724,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -748,8 +767,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index ec0bff09ce..0f61239429 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1332,13 +1332,20 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1394,8 +1401,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2075,7 +2080,7 @@ void qemu_loadvm_state_cleanup(void) } } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2229,6 +2234,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index c6d46b37a2..cf7935dd68 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -53,8 +53,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif From patchwork Mon May 14 16:54:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913122 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="TKtcI3PK"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Wt3ccZz9s0W for ; Tue, 15 May 2018 03:05:14 +1000 (AEST) Received: from localhost ([::1]:47309 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGuO-0000Jn-3T for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:05:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33591) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlE-0008MB-ED for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGlC-0005x3-Qi for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:44 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:40660) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGlC-0005wP-K1 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:42 -0400 Received: by mail-pf0-x244.google.com with SMTP id f189-v6so6306966pfa.7 for ; Mon, 14 May 2018 09:55:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aX3sDwjfakqhtRP0Z49uFUz5sIyRodddMtGk3by/K9Y=; b=TKtcI3PKuvoSJkDXKDbpDjUUC47gLvuhVO16WZuHre5n4dm3XTHWPPD1m9zt9R+uzW fchrb4wcmtxOYgADjKVuif6Mpzjeet7MkhCngT2Jt/7lf8H/SRoF+bzM8SUwWStcA6mk DbjtbDKn5p2TfIIbasNN7oXP3JrJC2T8kbEgcZiTCf0mBSFzon6pPiq23fnZn1Lpnoo7 YvN6sPnIXb/VoKOP67ygRlNI797E4i2puma+0DPOUBIpkV7Y2c6cQ7N3dj5gt5hG/bV1 M5VKmG8DvQXuwF3B5TsXRbMAZAVAPUZIDoNNec0KT2Zsh2GVuq+kfw42IBE0R2d44TJr p1Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aX3sDwjfakqhtRP0Z49uFUz5sIyRodddMtGk3by/K9Y=; b=tdGN+19jJlxTsznxd8EKyVI27sc+5BnP0V8T9v3WjG7F99XJMhltSegR5YLfBz2rM7 v6UyKwtheR3pa9x5qtmWF+mzmM8pcp+S0bERHlzx75Awq/O1Zl9k7NTm2BPbx0ueasHa InqyKIasAHehBDgLL2wxehF4fGIAlEKa600EB/bxMnrzWCqwaNgOERRlhcfMaH4Ht1zw A/096c0T3uUBmmPrP1laXSBoJxzPk5BSw5OjpdTIgs0WYbRQxo3Wgl8N5s/Q3WLjuYWu FkK926cWSrs78a9gE+pU/wuRqf9sxU6WBXEzMmRdgBEyXVUQZVV/sKme/bObXwJ2n5DG VQ0A== X-Gm-Message-State: ALKqPwcLRRoUfj5Ulit7sAFKomZWyc7IGA6SnxtWIdsAuBVUz+CZvulh XL5Hnrch2TfJF0lBdpPKuW3LKA== X-Google-Smtp-Source: AB8JxZpQyR0UNL7v7YFFoImwiJJ//uqqePh3taKMHH1zwETKiSUWwNd6vF4qgfJzeZqUBRRgL7PUPg== X-Received: by 2002:a63:6fc8:: with SMTP id k191-v6mr1107216pgc.153.1526316941277; Mon, 14 May 2018 09:55:41 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:40 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:20 +0800 Message-Id: <20180514165424.12884-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V7 RESEND 13/17] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4235a8f24d..21027c5b4d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2786,6 +2786,7 @@ int colo_init_ram_cache(void) } ram_state = g_new0(RAMState, 1); ram_state->migration_dirty_pages = 0; + memory_global_dirty_log_start(); return 0; @@ -2806,10 +2807,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap = NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3042,6 +3045,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset = 0; + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_length); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block = QLIST_FIRST_RCU(&ram_list.blocks); From patchwork Mon May 14 16:54:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913121 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OcPQGn5l"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6W61HgWz9s0W for ; Tue, 15 May 2018 03:04:34 +1000 (AEST) Received: from localhost ([::1]:47305 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGtj-00084Y-Q7 for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:04:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33608) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlI-0008PC-7Q for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGlH-0005zU-D1 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:48 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:36600) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGlH-0005z2-69 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:47 -0400 Received: by mail-pg0-x241.google.com with SMTP id z70-v6so5738390pgz.3 for ; Mon, 14 May 2018 09:55:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uSLxxx8sgpwe+c/fwWitSvkqKpBqgINTlXyIVHjl6NU=; b=OcPQGn5l/N/Kb47XMUQOEo7FeHi2eh0Ow3h29mlbDIRj/aOHurZsseXy1K1RLeFapc +KyFuQBt8bIml/k9PtdmORToE1j23rEh/EHCK5UIrlOhIjlDFQlOuiiagbv6btPfM9Bc q+z4rnA29ppI2fwaYIY2eLj7ltTvVjf/crN909VUPt8rZNQqvBIz+Y6ZRIxtiz046teQ 4mHvBwaFBdytH+r3i9fewCq/apteqF0sWnqTF4OpL3h/0FP89nEkbEdWzOoMCvcVAPGd cXVHx26UYprp7pyKvFzWu68chj9XTChEtDs5CDuC2pzf1wsVKnMucWR/XBQY9UwYYkZl TRSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uSLxxx8sgpwe+c/fwWitSvkqKpBqgINTlXyIVHjl6NU=; b=YILIYPBNgfMWqhtnW8XrGLMB3/G9bsANrDDm8o4HQZvA2C8S2I/Bvq6EPDxXU6/s6d LG7kVkp026lK86grzyD6CrXqw2hpvAdZT3rOSW8wF1OpLjBz3O6MVTXxtV/bF0jCcZ5y Y91IEImTsCvzcnM8jkF0kp8bpw4XqdMpK21E556m7iViI9p12vElz58lOObArd1DzfYy gSZMuJf8db9WghHoReRK3pC/YKh/7jIZ0ahWTAQewIwsJP11MZxV9a1rcdJeYSwoRCkA EX+92K16bCpDFlVVWSxPyz6I1jIFVYsMxqIydaGvg9SLD0vJPTXe9tJxV2YUP5rYpFW0 kaxw== X-Gm-Message-State: ALKqPwfTuzCYT1d/hLQo57DqIPA6v70DRcOBqdlhDeDF81mcNQ3HEQCW +T26MM5DtDjXlxLFngJgpdbOvA== X-Google-Smtp-Source: AB8JxZq2fKL7BtMb198gjCAtQ/lZJ/pgZaE5jPZgMTRWnOq3JrJTGMAolN0MKTNSJL211yqOdP5zUg== X-Received: by 2002:a65:61c8:: with SMTP id j8-v6mr9129085pgv.370.1526316945859; Mon, 14 May 2018 09:55:45 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:45 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:21 +0800 Message-Id: <20180514165424.12884-15-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V7 RESEND 14/17] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6f82..49da666ac0 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class; @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender, int iovcnt, void *opaque); +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d663..0f17eba143 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc = NETFILTER_CLASS(oc); ucc->complete = netfilter_complete; + nfc->handle_event = dummy_handle_event; } static const TypeInfo netfilter_info = { diff --git a/net/net.c b/net/net.c index 29f83983e5..d58691bb8e 100644 --- a/net/net.c +++ b/net/net.c @@ -1335,6 +1335,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc = NULL; + Error *local_err = NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer = nc->peer; + type = nc->info->type; + if (!peer || type != NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc = NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; From patchwork Mon May 14 16:54:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913126 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sXKsiAvH"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6Zm2CTcz9s0W for ; Tue, 15 May 2018 03:07:44 +1000 (AEST) Received: from localhost ([::1]:47322 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGwo-0002D8-0Y for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:07:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33678) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlR-00005F-42 for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGlM-00062s-9p for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:57 -0400 Received: from mail-pl0-x22a.google.com ([2607:f8b0:400e:c01::22a]:46484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGlM-00062b-3K for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:52 -0400 Received: by mail-pl0-x22a.google.com with SMTP id 30-v6so2871319pld.13 for ; Mon, 14 May 2018 09:55:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RDzK/IZdtuxv+giKDkRCJEMEUmWV17NczqputA7O5qY=; b=sXKsiAvHDlzXlLeUeoIaqy4D9XjKRqKUNEci7ahMytJTWRBo2gBwUUz5ZoeJX4bBkZ kopekERu4Ti7LFrORo6421ndAyYqkdwNGZJOCwwhiuNnAQbOwbqSf+w51O1nq6t7hKHN wQwcuyBkrfg18ZilnoB8DR9FAvZLQZndQnyxFLViIh97r/dsk0Xgo6F2O6Qxu3rOArSZ hAqr8Vm39ECFzsv0lLYPitdke31rK0siehWgpvwwNlAPaOhXdDuz6rTKcC4wu5rdAToD pwnxB/lovGYauJzwaQq9zqdZT+EiuIHvh2ZDiQ4uj6gB9/7nu1hpcA6UJlXQu1ppdKdf ZLjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RDzK/IZdtuxv+giKDkRCJEMEUmWV17NczqputA7O5qY=; b=iwTxv14U0pRSE4dJ4FG17pE73dURGVyW/VJ9EyhmJvICnziJF3kSPReNh/N5oKrN+u nLgQsmnh89OLbgzpkslnp7a1T4mfuNxZxEy6pRAN8YAT9/4mtpJg7h/ecYhOvsuF+shB IL5LzKnNM+q95zTPHgm+7/Fn3yal+oyAhurrACN/TvXlonjMMzGiJo40HcR9Q872TP8H jCfCzu+uBYa5iJwMa2oTMYWacXy3bgwPhQdFtHoEpVoNqepikSNhcQBrfrqr4ycitRYd +liSaR0TTYF3bX5sVlfUKy8fP8vzO+avpqiaEMalEjdxx5Lb4AigHD5tgeIhT9fkl00S oKFQ== X-Gm-Message-State: ALKqPweXIoQhLnTbipLTQkpv1bJeXjliRzpUMLvC/10GcIwmC1a2OYRW 0/H+8Yla6qlK+zuz4/Wm0szYJQ== X-Google-Smtp-Source: AB8JxZoAkCr6mr0GdI/hePw8GPuy2gBtCkC1F2Wnubmb6rYsU5oRFCIXWX96Wr3DO+u0fQGxzGQ/vw== X-Received: by 2002:a17:902:683:: with SMTP id 3-v6mr7270174plh.291.1526316950750; Mon, 14 May 2018 09:55:50 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:49 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:22 +0800 Message-Id: <20180514165424.12884-16-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22a Subject: [Qemu-devel] [PATCH V7 RESEND 15/17] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 5b055f79f1..3dfd84d897 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -82,6 +83,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); @@ -786,6 +792,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 0909a9a8af..f3c306cc89 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,6 +20,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -277,6 +279,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, return 0; } +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_data) +{ + Connection *conn = (Connection *)value; + + conn->offset = 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn = (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs = FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -332,6 +371,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, void *data) nfc->setup = colo_rewriter_setup; nfc->cleanup = colo_rewriter_cleanup; nfc->receive_iov = colo_rewriter_receive_iov; + nfc->handle_event = colo_rewriter_handle_event; } static const TypeInfo colo_rewriter_info = { From patchwork Mon May 14 16:54:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913127 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tOr44K6h"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6bb0H2Gz9s0W for ; Tue, 15 May 2018 03:08:27 +1000 (AEST) Received: from localhost ([::1]:47325 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGxU-0002lb-LX for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:08:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33679) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlR-00005G-4D for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGlQ-00066D-CN for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:57 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:41045) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGlQ-00065U-5K for qemu-devel@nongnu.org; Mon, 14 May 2018 12:55:56 -0400 Received: by mail-pl0-x244.google.com with SMTP id az12-v6so7706019plb.8 for ; Mon, 14 May 2018 09:55:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9neXvScEEfL9qiWRCyo9yJFqn66XsqhGjUPalXsacLs=; b=tOr44K6h719ni8ZCdz+n+5st5TfaqgjnZhXT5Ad3N3EWr8QgzCw+mlLIHZNeBbLQ/3 tFClRF2YppQ7kDSoXbmh9Cq+P88A+/JDM6EnN+/y0p5rTTDZRueHZwpWYs9KK0VHyQot rQ4at9g/pJDeLBfQXwMeuN8T6nVV6z1ayWpgGJ0iIjI5Ux48NLZvBNDhytgji5lZIqK5 0J5rqVYssIoFuK77ZufSws+E8QwNorYhTVyA3y1Gd1FD5SaSdVCTbiAtonHgZxhgbd4Y QAzJTTOOWfcJSGYIjTjW6Fc1h+slxjPhavOCN4pvdHW5GQUE4XtH61j1iIl1lBETJksa PlAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9neXvScEEfL9qiWRCyo9yJFqn66XsqhGjUPalXsacLs=; b=cH0k6rz3rRc97r4GxkdgrUfRO6f0E6A2tIhsd8GddR35ThyK6pUt85bRkYauaJGyJd xAaDtmuHfH2Qdffl9MNPHWaTplTKHrEHTi+VDC1cSKZOlbOWnEpjnFDIPpNj25NGE6xR SFNnZsj4iHqsJRV85LJYHftBNdTBsHoN9z8pQjdoj7XWVtX9953APw3XtrqPxTd2Ndj4 cxE7hjWMvANHgTvoe8UDJxdSBR7ILUkK5ErMykF7ErPWuBsz5oZdoMJBhSWGsSB1zpif 8QULYfSNTmY3/Waq4YsEOmbo6pJy5IoibJVLyjF6c5t0I2bi3lbffxBcoaz6C4UCFsfT VUYA== X-Gm-Message-State: ALKqPwevZPOrdv016ccT8gYC2Q1cHSyzU+olAkE0Sw00jc44urtah1E4 WBMRkvResoqc+wdsyqQDx/Avww== X-Google-Smtp-Source: AB8JxZqg618qqer0tu3TtP0Pmz5Se/wayKrbwYjsJD6mryhZd02Iaha+nSFlhYRR1kMCoocek5RFgg== X-Received: by 2002:a17:902:59ce:: with SMTP id d14-v6mr10875167plj.253.1526316954842; Mon, 14 May 2018 09:55:54 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:53 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:23 +0800 Message-Id: <20180514165424.12884-17-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V7 RESEND 16/17] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 3dfd84d897..15463e2823 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -88,6 +88,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); @@ -799,6 +804,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); From patchwork Mon May 14 16:54:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 913124 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JX6g+rJ4"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40l6YD41NTz9s0W for ; Tue, 15 May 2018 03:06:24 +1000 (AEST) Received: from localhost ([::1]:47318 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGvW-0001DF-5y for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 13:06:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33707) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGlV-00008q-4k for qemu-devel@nongnu.org; Mon, 14 May 2018 12:56:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGlU-00068y-IR for qemu-devel@nongnu.org; Mon, 14 May 2018 12:56:01 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:41043) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGlU-00068d-Cx for qemu-devel@nongnu.org; Mon, 14 May 2018 12:56:00 -0400 Received: by mail-pl0-x241.google.com with SMTP id az12-v6so7706119plb.8 for ; Mon, 14 May 2018 09:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=j4C59cjBT4ZORdV4kWXXNsgacKu2nLRUSS7fPUbvGfE=; b=JX6g+rJ4lXzoQ69Q/xbLKerCp/lIARl5jMrUfx0q3FfmVy7MbqrfOBtuM4gTm9Xoa9 i+NiBSp6avnH3DdDLDK1VCL8JNMGvNAnovWzSKG8sHWpKhThEoiHgM8AtJfTAnBh7dTW 0J7gw2d3nXxU3TDDIxL7xyg4/hMVuYANalZB6ydOUASMR/8gUUEam1htVJzL0D03TIhl itgHqx5sJk37z60tlo/6KEgzMN8LXzWXF3QHcdHq6p6or3Odtun/hoP5eNPe042n9aiK yRigfCdl7Sg9WBfiHEWSL0+UIycje1mqd5WyvUmqliyg2PBOaHxfsXbq1hhhYv+glwLx 0ssg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=j4C59cjBT4ZORdV4kWXXNsgacKu2nLRUSS7fPUbvGfE=; b=XrrqZ3MN04aR2mVcSOhbeTA41vjaPzD9+sWLOvotgZwfe2ghbsXTQ3AQgTrEV/NEJ9 C9yzYWUdyIa1W7IdBEBv+xFOokaF9S83nyh9Vu7ChPodPIsrBB8RI0ibxa29/VgTqPJk 4zNZ9IrmQzjZSVb9NHR4VapqadRjVJ+BOk0ZOeU1EUk8oI4MP/M7th4YVZ3TE0m35Vnt tl7c826tM7/4TNFybHBc5TuBp73dSUlAIFCOaMH4s1mFWBjumPkOI58twVpek8BRcLIw dvZ3u3mtZGLzuyXy5ERlLrVI04x8jjFQOLIiYh2hb1r5xffsbXW8IDGqsZezywEtH+D2 IRlQ== X-Gm-Message-State: ALKqPwdrudDywDEjqBRtfgdzcndmHhPWu7czvpypaOiXi3zUHWmFcjce Geuka2IiaLQ19v9qrb9vfg2D5Q== X-Google-Smtp-Source: AB8JxZoqyaEh/RpirlFTKxirV+nj5Ft//dpL9e/N3izP6zkzBvJEPKWcyxuWsJ8Yzmp+GdZCB1MYHA== X-Received: by 2002:a17:902:a986:: with SMTP id bh6-v6mr10889045plb.245.1526316959124; Mon, 14 May 2018 09:55:59 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id u69-v6sm631448pgb.42.2018.05.14.09.55.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 09:55:58 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Tue, 15 May 2018 00:54:24 +0800 Message-Id: <20180514165424.12884-18-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180514165424.12884-1-zhangckid@gmail.com> References: <20180514165424.12884-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V7 RESEND 17/17] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 15463e2823..16def4865c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -135,6 +135,11 @@ static void primary_vm_do_failover(void) migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); /* * Wake up COLO thread which may blocked in recv() or send(), @@ -552,6 +557,9 @@ static void colo_process_checkpoint(MigrationState *s) qemu_sem_wait(&s->colo_checkpoint_sem); + if (s->state != MIGRATION_STATUS_COLO) { + goto out; + } ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out;