From patchwork Mon May 14 03:23:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912693 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CxqegToA"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqcX6L3qz9s0q for ; Mon, 14 May 2018 15:53:15 +1000 (AEST) Received: from localhost ([::1]:35552 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Q4-0007Da-8x for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:53:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42583) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6MR-0003Bo-TT for qemu-devel@nongnu.org; Mon, 14 May 2018 01:49:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI45s-0007by-6z for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:13 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:46790) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI45r-0007aZ-Us for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:12 -0400 Received: by mail-pl0-x243.google.com with SMTP id 30-v6so1669802pld.13 for ; Sun, 13 May 2018 20:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=CxqegToANe3LnF8mURdXa5jtrJmw0qYwzG973jU7paGIhlukut598I1l0bYd+wX8fq oM3asM/pdnLyIvCjrKEwIAlx1Rovy3BtR+5lZBpYqytHXxBBnncKi6tH9qlCJTJrYZvm DpmKBz5Vxry5G8YX+AcsoxL4YW1J1ERi5PaU0rZALKS47VLd5OwlEDjJzXoDtzBVFLm7 zpJgHQ6QQRtBPmW0AWZDAzvYvmSxSRkMhHbTNL2mztgVWc495ke1nQkAxya9EFzKZv/Y FgCEf6tjEotoj5RrsWiX+yHplNzA6xYUrPYR8ig8m9B6cLdnaFyK1616lh1GOprX2cdQ tWtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=VmrDA24eNNFBy9wZxLjR2BbJRm6CLg8PlEXFDp624o2HflPKFonKo5wzZBZEkxQt2Y 3X/9nOAJuSNOPBIK5P8P2E1jZ0k+uIIU1SJ/+ucdhJBfLJjTPSJTKOaEmIUNTWAYORLC MJYgDj+HeGtqjBmdwTWNhXniXpDQ63AyrBidZCIl0mVkX9MyeNyPHZ+3mqrejOtK5U+e 3yd0FwWaCwqpmGjIQDe7PMnL9uSOJG57BcL5V3FJpNFifLI+31fnJfFdhqmayIko6wss LA4tw7h/obhkp0+1mVDVNdeo6Y46ww7d4GRX16Rmx9vwq9UcFw6UWepGGzdET87C+n/W LzTA== X-Gm-Message-State: ALKqPwe0HtMEc4cCYEmvrPoeTNlO9t88RzOfgbzSh9bpOm7DvEN1UPvi aLk+15lf6NY3VMA/0t7Z4V5HWA== X-Google-Smtp-Source: AB8JxZr/6PcrqlIRZZMElyjMxc5lZ3FaHJOIMp/OkV+nAkvQ3oX5pGsKqpWAusfys31/ErBsF3ZplQ== X-Received: by 2002:a17:902:aa95:: with SMTP id d21-v6mr7896217plr.73.1526268250578; Sun, 13 May 2018 20:24:10 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:09 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:32 +0800 Message-Id: <1526268228-27951-2-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V7 01/17] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 +++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index da6c36d..cd11851 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384 @@ -86,6 +87,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=1,ack=1' */ } Connection; uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 62dad2d..0909a9a 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -99,15 +99,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=1' packets from client side, it acks 'fin=1, ack=1' + * packet from server side. From this point, we can ensure that there + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this rare + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_table. + * + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=1, ack=1' packet from client side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = htonl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; } return 0; } /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -139,8 +168,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=1' packets from server side, it acks 'fin=1, ack=1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=1,ack=1' packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=1, ack=1' packet from server side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = ntohl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; } - return 0; } @@ -190,7 +245,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, if (sender == nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +258,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); From patchwork Mon May 14 03:23:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912698 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lbKAzcsa"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqmd2X1Zz9s0q for ; Mon, 14 May 2018 16:00:17 +1000 (AEST) Received: from localhost ([::1]:36037 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Ws-0003RG-TE for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:00:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47153) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6W3-000254-1O for qemu-devel@nongnu.org; Mon, 14 May 2018 01:59:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI45x-0007m9-Fv for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:21 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:46790) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI45x-0007kq-7T for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:17 -0400 Received: by mail-pl0-x242.google.com with SMTP id 30-v6so1669895pld.13 for ; Sun, 13 May 2018 20:24:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=lbKAzcsasAAsaD/drYIfqGqvKG8HMVSc8NbfHgHLaabJOdukT/w8Rp7W17eJpW7BBY qUNlwqdYgKb6h7+SO7y6iksPOztRTG6XqL92NcCdELw4CL5THyF4C6BUajJKX9wxs786 Zy+5bAUz3JP3c9zoQ/ALzvXVpqADkz3tJ/R0pWTiHQVmILkNK16Injg+SNTAR6/m6uJL jpuizW/p7pKfKDJbAPxFQmO6HMsOcYhT9txauQcGm2QTwVQ2DBprEFzn440UK1739GFQ i3Wm0yNbLF3maLt8zB/vO4LKMA6ui5D58ZrJd6Kfe3734itNjPc8Sfp/gVkQvWld2LgZ XNzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=FcVUz2tivfkCJasT/hREgDmzndQ7YbVPVdKu8ws904ud7W6wj1tp1Q8rsVtw0ce7e3 iMURHg+TIcXoHYRz/eiyWG4ViBn9W/EVODzFSYPsjMurcO7qyxezBf/2HjClP2OgRjwl SiASfreWeyWNqYd449t3sc8Znd+t3vaV6uNYWxZS3Pf6UTEnOJv53lB20wWWDbLJjfzm LuDBiFrFL/vBBVX7lr6suPijnd+BVEjsdGJ9u7RLw7NhB/dT8pgocC9jg2+lnc77X15q 0jpgXpHTlMhSl9LeZ3XRLb5qAZx08ThcKHRLoY5FWskip0ijA1x0RwMQTXn67Ysye81u C+fg== X-Gm-Message-State: ALKqPwesPRR9xjnAhJpqzUi/nZAwmwHkj01NQz7fKqxTxaS/fTyDMxZD cqLrtTC1LevHt3BIGuCFaPZcMA== X-Google-Smtp-Source: AB8JxZqSr5Lh5wtsFCkvQAODn6b7HuNkBEp2NArsZGorj7oKSjMJsnrsxU6v1mTHw/78nzyzIu38iQ== X-Received: by 2002:a17:902:2f43:: with SMTP id s61-v6mr7778256plb.99.1526268255881; Sun, 13 May 2018 20:24:15 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:15 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:33 +0800 Message-Id: <1526268228-27951-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V7 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad..fefb2fc 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c..7ff3ae8 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon May 14 03:23:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912696 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="N/3zHME9"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqhM43kzz9s02 for ; Mon, 14 May 2018 15:56:35 +1000 (AEST) Received: from localhost ([::1]:35735 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6TJ-0000Ic-6p for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:56:33 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43705) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Or-0007Ny-Az for qemu-devel@nongnu.org; Mon, 14 May 2018 01:51:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI461-0007tt-Uh for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:22 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:42181) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI461-0007s8-NQ for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:21 -0400 Received: by mail-pf0-x242.google.com with SMTP id p14-v6so5308071pfh.9 for ; Sun, 13 May 2018 20:24:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=N/3zHME9iV4kBYLEG6u4Y9RPVTrve9SSa9W4LH+RQexMTnv5Kuf/IuwdyY1/kPlQUD n7D/GigZWMnxKJPoFDvPft5IIpfPsN9KISV3mbk1FANwDBtNSPfCwAnfXBl87mFrC/fG FXENlJs/XWR5xtR4lw1auHUPBm57MFi58AgSVdjLBh4BmhlvsQx6gBz1bx7c7BujvlOQ nQWpGVUw8P14+Da65J+uOpSkb29eRaF4cTZwK9BCowBqS73w7keSovag0AEQP3IftMjg inzS79/gHa0GAvYhCvZGKDufa5NqPN+3pbbq7O96xTKbSOvT0lrfOHs69jA4y24zH089 v1VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=G3Yd/Y5jIjMVPh2AWgiyraerN1zAm5SXtEviTcP1B5YJN5+xgHEJo1xRx2I9fY8M8C 1msLRybj7Sz+8eHHAOC4MBwKXfb2uLaz71XvuVJ3JRum/k7mjwy/QU9Hzdg9puJJ4Qpv VPN8yC+u3lRO4SIvnyueFAvnurc+DQs5wpynKk3OsPezg2oEGhg6+yL2l8Wgm1ELD7AL atZO6L2jv1cKOK1E6TPgxy5lSWFhNTMn7u7QL4LHSi/ULhsmL5wMhHtcyFAv1QwfjjHE lP3OkCh8i5qWJO/s3eIeuu3ZQLI+Fm1U59w6PExq+n+N1/u0i+S8MdAZi/i1rUqhwazX u07g== X-Gm-Message-State: ALKqPwfg+amFA5NoycXVaYQbJy6e8wAMfSCcO1Qu9rmCFNgdPkTBwpNc kSHR+nUzz1ViX48IaA3Ohill4w== X-Google-Smtp-Source: AB8JxZqsKPQwVC4T7HMwe9QESZ6gSO35sK3zwSmhjdFMQVpGtsVy7PTOCDvc4hvDp/MTy98vIXL/Pg== X-Received: by 2002:a62:48d1:: with SMTP id q78-v6mr8633885pfi.70.1526268260513; Sun, 13 May 2018 20:24:20 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:19 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:34 +0800 Message-Id: <1526268228-27951-4-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V7 03/17] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 7ff3ae8..05061cd 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static NotifierList colo_compare_notifiers = + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -561,8 +565,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result = NULL; int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -573,10 +593,7 @@ static int colo_old_packet_check_one_conn(Connection *conn, if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } @@ -620,11 +637,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76..22ddd51 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); #endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon May 14 03:23:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912701 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nw2qDcoA"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kr9D5G21z9s02 for ; Mon, 14 May 2018 16:18:08 +1000 (AEST) Received: from localhost ([::1]:41543 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6oA-00087C-Cf for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:18:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52695) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6kx-00068E-4x for qemu-devel@nongnu.org; Mon, 14 May 2018 02:14:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI467-00080B-0c for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:31 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:43650) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI466-0007z6-P8 for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:26 -0400 Received: by mail-pl0-x242.google.com with SMTP id c41-v6so1290598plj.10 for ; Sun, 13 May 2018 20:24:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=C/zVYMfTHy2EkjKxubXlCaZxOky+2bZU5mgWGn/KNIA=; b=nw2qDcoAhLjDqErjCUHSb65PZ1x3dmUuDjiEV+qwFvIYDzn/JPj1jfLgB1ZbLz5+af ltR0At8G5HE1RZIpfkVyeDyIb5hKT/xe0td6gyu8xq8nrxEXn5bYeglGvRqnRx4oIA2j agmXew5BRyvuAyTK8CJjtjDllWX3lvgor+n9+k2k6FhHu/wG2dPr0Pb2VolllEbn+AAP 3LR42t/mi2krQ+7j7XQp8IcwkMwSG901Axk/Ei4x4RN9/bKcpk6GRk7pLRQT4n08eGho 61gfz5gQSFOXojgp2rfm1x0IQgNQOWMGN/fU88gn6kzNahmLZn50lFuLAl3OwKslFJ6d ItVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=C/zVYMfTHy2EkjKxubXlCaZxOky+2bZU5mgWGn/KNIA=; b=NGq2AdZpAwucwQPnhtCr72n9MN8EQFaSDIDC3sFhAXRlK1eHVoY9Rcsjn6HQzJlggI CZb2fXeemQZz8dP1Q1S/o6IN0ONr5lAi6i6FR6mNOy8e/SZfUYOwYmZQIwc1QSY0yJUi 9kVApfcD5GjVHA+dOuCw1VG36fXHygit01xch9aGGzGfiUH4w+5abMW9qErIvq3r+2Uc Typlh/ABf0pTY9Xgl5X5rVUVn83crK0N29vT/0zWy/C48KtgUPTepSOrGXITkZVXpmh7 Q2ln4GVMrO6flWd0vbo+kwRCF53qFVlZnceoHjXR7Xx7BY5Z8S5FikiF41WpJjAWor7w 4b5w== X-Gm-Message-State: ALKqPwdKP14r4+uG+EIUtYo2ODnCVAX1OTFFKQ0pUlTaO3BbJww2l5ex ChNNFhnc4LXUWA5yCZ/+o/RWzg== X-Google-Smtp-Source: AB8JxZpZIWtoDSBtlse6iObvGYZwQ6GFPnkrEsaZzmqpdHcckmsSiPD81QvSorV1wMTjy+05a/y4JQ== X-Received: by 2002:a17:902:d20a:: with SMTP id t10-v6mr7926527ply.364.1526268265602; Sun, 13 May 2018 20:24:25 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:24 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:35 +0800 Message-Id: <1526268228-27951-5-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V7 04/17] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067..081df18 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier; #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb = qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 35f2781..bca1872 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -76,9 +76,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid mode. + * Note: Please change this default value to 20000 when we support hybrid mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 From patchwork Mon May 14 03:23:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912691 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kqHy1HAC"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqXm64mGz9s0q for ; Mon, 14 May 2018 15:50:00 +1000 (AEST) Received: from localhost ([::1]:35366 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Mw-0006jJ-FI for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:49:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41614) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Kq-0000xX-17 for qemu-devel@nongnu.org; Mon, 14 May 2018 01:47:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46C-00086Z-7y for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:33 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:34225) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46C-00085k-07 for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:32 -0400 Received: by mail-pl0-x241.google.com with SMTP id ay10-v6so6508957plb.1 for ; Sun, 13 May 2018 20:24:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1J1mLAwnuTRC3m88rK8lPvstVX30OXBz9RePvtM5MSU=; b=kqHy1HACrbdDMgcYu5J43pweoPmieXIN7u3/7ReS4TQE715FEf1Oy7jWDz63+EltoF g0SSv9bOAc6Z7KaoqtTWxgz/2ywen0oeteBTfd/zgWBGqbDcDZEClcMduYHxchrOy1wa FaeCBm9uaUrwCMapHHwpiRVmLX3K7mBkJ+DSmd/dVXGWpFthlo817nl6jIjG3irCfIYD IrA0TKHbDbbAUjvyhboEG9Rx4NxgP5X6JC4CA34j67VRIeLYACsU55rjDbUbJiBgUVJD eYjcQOR5NKKDlBhS6BNRVjoP53Tv+wUqvhKn/R640y7IGnatjEKfT7zvErm8YhM29fNZ JLWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1J1mLAwnuTRC3m88rK8lPvstVX30OXBz9RePvtM5MSU=; b=JF46v033lGvyWgPotSV2Xt2yAkzVCShb793T0zpU6AEKGtjwUKx6sd591ZDEvwsTKF bjmZlQux9QQgVpoLJXRkdlGius/K2BRdEWLhUrezNARWD7v4Z2lFXlcBZySACHSlYT7E rAsMBebX5pkSHFTXpB72uY4rnV7AYmdzkdGNliN5PPLeDVntwDUX+GMx84hiURSnWJ3O IhZlsNmC7QQKlTXpIKjEi1QkNASLDmPbFcvFHUqXfKcgdmhzlKWSh3gGaEEE6yck145c YrK9yfcyU3A6nKhTN+DWPQdhOWg9E/g7gmQxbCF1xUbF0OgM4XXgHeWCyoca7L/99nW1 Cf3Q== X-Gm-Message-State: ALKqPwd+mTXNGtg/q8WGFageBYeO0E952niXQMZf0qxMuSy6sLQA1lYX 5OnSgciogQ4nDhFkXtwWio081Q== X-Google-Smtp-Source: AB8JxZouPHrtJGPP6sD5HV7AwWKxMSNIuummLUia4KgiM4Y0xBE0Fy1YVRNKW54XMk5EgigorehRfQ== X-Received: by 2002:a17:902:8d81:: with SMTP id v1-v6mr7667877plo.383.1526268270803; Sun, 13 May 2018 20:24:30 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:29 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:36 +0800 Message-Id: <1526268228-27951-6-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V7 05/17] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df18..e06640c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis = migration_incoming_get_current(); + Error *local_err = NULL; /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s = migrate_get_current(); int old_state; + Error *local_err = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err = NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index bca1872..ddd0c4b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -357,6 +357,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis = migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err = NULL; assert(mis->from_src_file); mis->largest_page_size = qemu_ram_pagesize_largest(); @@ -388,6 +389,14 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); From patchwork Mon May 14 03:23:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912686 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aLPCrJht"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqVK1kc2z9s0q for ; Mon, 14 May 2018 15:47:51 +1000 (AEST) Received: from localhost ([::1]:35269 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Kp-0007fM-RH for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:47:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40760) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6K8-0007Oe-Jc for qemu-devel@nongnu.org; Mon, 14 May 2018 01:47:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46J-0008L0-Gg for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:43 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:40440) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46J-0008Jt-7i for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:39 -0400 Received: by mail-pl0-x244.google.com with SMTP id t12-v6so6513285plo.7 for ; Sun, 13 May 2018 20:24:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/j3laNioBf9HnLTZepjNcqt4aLTxBX4q6cRmJI0mLPw=; b=aLPCrJhtTNbc3OEe7vLm7/JeawUBCvJ0wPDe218X0S7cLXnGAI5hRde0smbxwE8HIR 64CNxUtG4qKg+0ClHfQQGiockRF0aT4aBT/WbvWWZOFm7sbayrptt6UA4p/9vJGPMX/C q4EocDKZ90HmICCnkvn/gBvAJlKrZtyFsHQb4OWNbND7sZ0+AXxC4L7EWY8JlLpGiwyc O6RZTPkBQrPU30mVboufVOYGjxF62EeAspavT8VOTNeps30lT9Wkbt3IjrPSU3aMqQiW /RSB4gdL/7MnhMiAVMt5fZ5fKSuUnJctS5fEP+gp/W9fRokjiJBiBHGuguywbtbVODKc B5kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/j3laNioBf9HnLTZepjNcqt4aLTxBX4q6cRmJI0mLPw=; b=VLomH989awpRXpy3vUxpV7GmdyhWy4wQDk4FXQfnAvVgJ++qsppoCeoZ9h5jEFZpQc RnlJywE8oAVyvieMOk3y9IqhID+N4mdPyjK/PuWghF3SnfigfPuDEf3ATIOjNiMJYh6c hUxBnw9rjCZQg8l22Byk+zwNmu3woQpgtBUZiWFP3HXjQZ6JtVix60fkrcuYzVsFN9OM P2K0xWInSiGAUAIZhxK355AWROZ/MYDvT552zKPYOC5SEcZ2SNFPb8jaNLPxxaxr0BYP tz6rpLe5hUGO2ehDU+hAhRpM/vKzR+mZTzjHToLAEtw3iqqdTFHj53kesyFfWrbAxY3T saWA== X-Gm-Message-State: ALKqPwf4j+cTydsCmbKtEW+/49tR6SpKxZdteMtLtBF+Kf5b2D27oWQw pdXbBcV6jUVjVdv2muQY8/AQ6w== X-Google-Smtp-Source: AB8JxZqexR6EnqCrWcXgWtu2USUVeVEWKV1j6KPh6ZElFYH5XSFCVPgnOrmPiHajAYJp884rytYEXQ== X-Received: by 2002:a17:902:7b97:: with SMTP id w23-v6mr8090767pll.116.1526268277896; Sun, 13 May 2018 20:24:37 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:36 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:37 +0800 Message-Id: <1526268228-27951-7-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V7 06/17] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ------------------------------------------------ migration/colo.c | 13 ++++++++- migration/migration.c | 23 ++++++++++++++- migration/savevm.c | 20 +++++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 60 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fc..99ce17a 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47..a4f3baf 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4d..0000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s = opaque; - - s->colo_requested = migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state = { - .name = "COLOState", - .version_id = 1, - .minimum_version_id = 1, - .pre_save = colo_info_pre_save, - .needed = colo_info_need, - .fields = (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested = false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c..c083d36 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); return NULL; } diff --git a/migration/migration.c b/migration/migration.c index ddd0c4b..8dee7dd 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -277,6 +277,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled = true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -388,7 +404,7 @@ static void process_incoming_migration_co(void *opaque) } /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2431,6 +2447,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index e2be02a..c43d220 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -55,6 +55,8 @@ #include "io/channel-buffer.h" #include "io/channel-file.h" #include "sysemu/replay.h" +#include "migration/colo.h" + #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -81,6 +83,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; @@ -836,6 +841,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1793,6 +1804,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis) return ret; } +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1866,6 +1883,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } return 0; diff --git a/migration/savevm.h b/migration/savevm.h index cf4f0d3..c6d46b3 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,6 +52,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index d6be74b..9295b4c 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index 12e31d1..a1576d2 100644 --- a/vl.c +++ b/vl.c @@ -4437,8 +4437,6 @@ int main(int argc, char **argv, char **envp) #endif } - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); From patchwork Mon May 14 03:23:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912697 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VB9hRaU/"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqkP30zlz9s0W for ; Mon, 14 May 2018 15:58:20 +1000 (AEST) Received: from localhost ([::1]:35867 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Uy-0007C8-QI for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:58:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Ta-0000tF-Rz for qemu-devel@nongnu.org; Mon, 14 May 2018 01:56:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46P-0008Ti-4p for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:46 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:40438) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46O-0008SR-TD for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:45 -0400 Received: by mail-pl0-x241.google.com with SMTP id t12-v6so6513408plo.7 for ; Sun, 13 May 2018 20:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NvJKNR9Wq1PeJQQvkq1rZxYoXKrt82SUmxhiUuORfOQ=; b=VB9hRaU/IZeWgRtmoNV0mX4w1cl9UDs63887Rjn3U1UpkthTPqtSjxkLxfUpTTbrEt yZfrM/XFwJPPPYRXtHLoMKxo0uf42MCa8ddiTYh0+Li504dkk5h5kmF6C5D2iPS8GNEp W7vUgVxKr+yAFn02niZIlbgA3mu5Hfw+l3+Q8cdCmjr6N5sGUrghdvNSl4rCjGgzaJnT LIEWtHtLBbcJwspiHU5MdLdtvDCC2ADtAqinf/hXQr+TfEankeCfLM8HXvcI7AENkA+8 YKer1QIZoBZn0eEpTM0HsstaGjJsCC1K7WO/WF0D/veP07L9jAbFZo1FmZOd3OnAz5aO 313w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NvJKNR9Wq1PeJQQvkq1rZxYoXKrt82SUmxhiUuORfOQ=; b=PFS5/5gAA8F7KzrCoVXUnTJ/VrznOzsLKxeopRWe437pVvFlDl/DLQkOKlXTOywpmQ zeWGFGXyFdqONVdHCnH9cC/kJq0O3SSTghSdXFCo5cKdAkd2l8DZDu4llqd9Cd8IruaN asJvZTErfikCyrOGot0NSuXdVz60PQuwrFMCJu053yt1FiYmBG7X9mOLfUGF76M8cFgt SjUT5kfJQ57m4Ou4NIIzG/pVbxnbynXgA/bbYDN7M9zM4P0F3IL1Nord8+pc4+pGWE0A ifDhkeW6v1VfvGW+MJf8xVXnnArKZdfUDt7XbNGw9AT+sfayhjsRSobvbm8zEdqyhfNo 6c4w== X-Gm-Message-State: ALKqPwcXVsM6PF1p2z3P0nfDUMY1OkCL6cehmJULGubJL/QV/GurlneR gqs9uQLw4X3sd00EJGZ/TDGfzw== X-Google-Smtp-Source: AB8JxZoIxI1cqXGdnjKJ0svU0SmjlRySt5MT42tNntspHwBAMQL1zmZtt1M+5jfhAuvIbxF4JxfxUA== X-Received: by 2002:a17:902:b681:: with SMTP id c1-v6mr7915952pls.286.1526268283700; Sun, 13 May 2018 20:24:43 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:42 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:38 +0800 Message-Id: <1526268228-27951-8-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V7 07/17] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 104 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf2446a..51ec153 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 8dee7dd..cfc1b95 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -421,6 +421,8 @@ static void process_incoming_migration_co(void *opaque) /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 912810c..7ca845f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2520,6 +2520,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block, return block->host + offset; } +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2724,6 +2738,57 @@ static void decompress_data_with_multi_threads(QEMUFile *f, qemu_mutex_unlock(&decomp_done_lock); } +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache = qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2740,6 +2805,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } @@ -2753,6 +2819,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap = NULL; } + return 0; } @@ -2966,7 +3033,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host = NULL; + void *host = NULL, *host_bak = NULL; uint8_t ch; addr = qemu_get_be64(f); @@ -2986,13 +3053,36 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(f, flags); - host = host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_cache + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects the + * speed of the migration, but it obviously reduce the downtime of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host = colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak = host; + } + } + if (!migration_incoming_in_colo_state()) { + host = host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret = -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } @@ -3087,6 +3177,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) if (!ret) { ret = qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } ret |= wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 5030be1..66e9b86 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -64,4 +64,8 @@ bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_offset); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t nr); +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index c43d220..ec0bff0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1807,7 +1807,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } /* From patchwork Mon May 14 03:23:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912716 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nOu9gd51"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krpD0BbHz9s0W for ; Mon, 14 May 2018 16:46:43 +1000 (AEST) Received: from localhost ([::1]:44043 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI7Fp-0001iM-Jz for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:46:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6e7-0006dK-9B for qemu-devel@nongnu.org; Mon, 14 May 2018 02:07:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46T-00008p-S4 for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:53 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:36132) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46T-00007R-MY for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:49 -0400 Received: by mail-pl0-x243.google.com with SMTP id v24-v6so6521943plo.3 for ; Sun, 13 May 2018 20:24:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IH1bphaxcG2LjizEZBLNyCH28G/Lq2miAkn26IAeS2k=; b=nOu9gd51rRQ1aYwlLGYDUREmL4E/SXvEwtN94yn9wdazJHZNjtaevT6Pe3INaRfTcm PUtumi3cz7VXDYb6IWhywmv8lFJCzhZuYuLZrf6OvnKI9okv/2KzqQLxf0PLlw+Quwoz D55fQKLqhdmpBy8DDAx9oDt1z7M10D/yam4jsBGrgpldtSdNlZJNHrOrq/eC/DCPtiUa zmi1hw0R1/gRHvW3VxAjy1omSno9aaSvlkpSzpcCRiUZXaTVa3dUuaM5uX9MdHja88ii 5m1oUynHhRdGY2ZQwoXcBYcZHaMyDGCk9URA8oc3NORMoiJIYRVZQyWeMhePrWiNE2Cy wN2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IH1bphaxcG2LjizEZBLNyCH28G/Lq2miAkn26IAeS2k=; b=M8x2V0h/DeLPEpw4D1jQ9L1pjsuw9foVKrRy8s1bjrJz83i8X9v/e8DwG3D8Z2TRqy agTb4LO3xNpR/TVfCDB1884Q48umR5lk1gUUh6yqxNwNcdSxap8SCPD4yWFkydWj0LJ2 FdZah3gKvf3LDzH3NwR9oIyeyQuoF8wEPsNRMFhiqhz+cc6QuWBvwsLI+zL9CfJr5p3V S+uT2DBJM5M4rbA4nXYG0OhlAAhgdfsarv2laAw4xO9jIdFJofMZK+KdJc0JkVakAetJ 4JNVfermyNAcucOROOvnYBA6Vg2zTfno20GxWAWTc+fNO2SzsN+vC5uKLYV5HDNMP25B epWQ== X-Gm-Message-State: ALKqPwdyPQi6koVa0e7hqXj5XcLtsFddmu/3R94D6A+/WHOBivfsc+dh izGzRwY6HNmpy7D1gEn5PGaURg== X-Google-Smtp-Source: AB8JxZpFlLf2htTQ6YxGNRtT+PvBalrwu2T5M6KfP0+tNMZHOefr66a3NZjNZZNuV6JLWinOFo5APw== X-Received: by 2002:a17:902:bc49:: with SMTP id t9-v6mr8078951plz.109.1526268288498; Sun, 13 May 2018 20:24:48 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:47 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:39 +0800 Message-Id: <1526268228-27951-9-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V7 08/17] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 7ca845f..e35dfee 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2531,6 +2531,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } @@ -2760,6 +2769,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap together + * with to decide which page in cache should be flushed into SVM's RAM. Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages = block->max_length >> TARGET_PAGE_BITS; + + block->bmap = bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state = g_new0(RAMState, 1); + ram_state->migration_dirty_pages = 0; + return 0; out_locked: @@ -2779,6 +2806,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap = NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2787,6 +2818,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state = NULL; } /** From patchwork Mon May 14 03:23:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912694 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="IWd/XJob"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqdn2L3Gz9s02 for ; Mon, 14 May 2018 15:54:21 +1000 (AEST) Received: from localhost ([::1]:35655 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6R8-0001Yd-T4 for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:54:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43336) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Nr-00005o-7V for qemu-devel@nongnu.org; Mon, 14 May 2018 01:50:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46Y-0000GE-UI for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:56 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:45466) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46Y-0000EU-NH for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:54 -0400 Received: by mail-pf0-x241.google.com with SMTP id c10-v6so5305257pfi.12 for ; Sun, 13 May 2018 20:24:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g819tnk+8Qd28uIgGno/qnvKAxrfmnG7hWA5eEBBsqQ=; b=IWd/XJobKRbRQS8cc3Ocl93cUeHRLdQligV0vdDX8agDaPL90GwCochTGCXaK5NC/d hMnWhHHKye9Y0h46pGDViwDkh8Bu2Bf8uzua4MJttLumpt8bJeEnCKi/Ks6qZl2MH7S1 D/0qT05SanrChgpzmnW7UkaLMMyBv25ZgPYuwp3ldsrmDSyu+ZPRxc6duzmmHNR8GeiO GFXoXw/0wpJjUZcgzEYePrJn8nUl7YZHCFYvfrA+87ZgFmryyrH+jHtB/D9ZXHD7yXBh 8GqhD61msGLEoJyuyJaSUfOOpmq4rqKmHA9dC7VPtm8GA0EtGMNss494cLZ1EJZkH89o wblw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=g819tnk+8Qd28uIgGno/qnvKAxrfmnG7hWA5eEBBsqQ=; b=nxm4/v1Z1xABOmCKkfs4yxoThQrdh1K7Mc68PU+U1uB8QLdFml25gvnLsQMQyrM6W8 sU91ZMIPsiQPf3xDDaWzgnUdha3cq0HaJ/fLtlcaSJ4oZLdEtdOrvAFuwmtMp55wve6d fQrz3W5RvWN3rNJlER6XSuMtGZp6YeuTU0amor1Y2ApiRVuNywjl8nWOE2s6HeLHAL8R gwJVksnyhoTgLK7oGebJa/sJDrvUlys7NrEBiE8/htJILzcQHkyV4iKTFCoB/uD+O30K rtAWBe4gh6atRCyO72RC8e0TDHiIz6HUybJujE2lChK4AmFxDxx7TnPbZOJOvczIAvXq pe7A== X-Gm-Message-State: ALKqPwcittSxjbkl99ToupbktQgKLqh/Rxe/StcUw5aKZ0E+LRX0mAWS ebuiHXxjADz9m7J5PJzVE/PGIw== X-Google-Smtp-Source: AB8JxZp2Tl9B4xljrm8rZkS07kWpKVEFjioBIYbS9Jkht/nQYH1ClKsdMFg3CZxLqAIiIMUVRckNQQ== X-Received: by 2002:a65:424d:: with SMTP id d13-v6mr5780818pgq.234.1526268293489; Sun, 13 May 2018 20:24:53 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:52 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:40 +0800 Message-Id: <1526268228-27951-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index e35dfee..4235a8f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3031,6 +3031,40 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages == 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -3043,6 +3077,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) bool postcopy_running = postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); + bool need_flush = false; seq_iter++; @@ -3218,6 +3253,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index 9295b4c..8e2f974 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Mon May 14 03:23:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912718 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QbLA5Fyw"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krvV266gz9s0W for ; Mon, 14 May 2018 16:51:18 +1000 (AEST) Received: from localhost ([::1]:44490 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI7KF-0004Vx-Vz for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:51:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6fX-0006dK-0R for qemu-devel@nongnu.org; Mon, 14 May 2018 02:09:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46d-0000O8-Sk for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:03 -0400 Received: from mail-pg0-x234.google.com ([2607:f8b0:400e:c05::234]:38339) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46d-0000N6-Mq for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:59 -0400 Received: by mail-pg0-x234.google.com with SMTP id n9-v6so4846470pgq.5 for ; Sun, 13 May 2018 20:24:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=P8+FLOfvMorXstTMNMIRaiUcPBB8BGJsymX5MoajP1I=; b=QbLA5FywbCgWKlVUUr00lR8emGmytcDJaF6OJcQ62riLYZEn0CVPJC5Y1l3BDl9SQs yBe1a8CCN61ej29FK9doekxX5pouz41Km0FU6skTVVZK8PZCd3lXlu1b+OcTc6fOGjyj 1ldO0p9QCltoa/cYAiE2B2OyhpyGAl+dzWjibRLszhpHBI0EaQzTr4R79iV2/ErhfECk Ea0vpvgNprP6x5mMkZtNBtP9cQxHx+LYD9fuFt/2UQHQyNTLZEo8xJ6xD2QD1nXP7rO4 4hSjv5oiab8hVqcWPPqOWSWXjTQ6R+7SFyW8z6A0u3TlrA01LDDaZ4e1QsQA5Juo1f6W EMxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=P8+FLOfvMorXstTMNMIRaiUcPBB8BGJsymX5MoajP1I=; b=YjxruLF8woH5TUzUb+L4RI7wBmOxYhPYQaXWRldAiDF9HjxfyRo8UL/HoryPmRHZgJ 5AkhNVKmkgRBb5XWiYJeakGZZkyvY4YHGhkSnAv6pNWEcpnoLVOnFhAP1xv9TibfoVaY Zu8kAY+zY3C6N52MJT/F4C3N/cl4EtOgl6n/bP5B9WX7EPm/sWuZhygsjWn8UTr2I9OJ Ny6dpinukgV8QHbwtqcYnJtkcPveQpw/rleWDRf6TqaSHiRb22+hHNdQQxzZCkUr5lrJ fwvfrWaNPLSyqVDQizufjkirwQAPGO7r0KrKXFpoHHRNQx0r1j6syeu2KrU4OQmf4ekZ zocw== X-Gm-Message-State: ALKqPwft5m3yZgIvI+bd/4Yial0YUVIZ8NIt+i58xWj2gcgn42uoV+2E 7yPLfAcrRvMnJNoEGT9dRwuTCw== X-Google-Smtp-Source: AB8JxZpsUi/bQIz+eaBsPVm+7fZ4NTuaMbZw65s36sx4NOOyWfG4GUT5d/oFjkEfIXDmx9P/iLLkYA== X-Received: by 2002:a62:1fc8:: with SMTP id l69-v6mr8586787pfj.49.1526268298441; Sun, 13 May 2018 20:24:58 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:57 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:41 +0800 Message-Id: <1526268228-27951-11-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::234 Subject: [Qemu-devel] [PATCH V7 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 20 ++++++++++++++++++++ qapi/migration.json | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d36..8ca6381 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,18 @@ out: qemu_fclose(fb); } + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() == FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -744,6 +757,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() == FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } if (fb) { qemu_fclose(fb); diff --git a/qapi/migration.json b/qapi/migration.json index f3974c6..55dae48 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -875,6 +875,43 @@ 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.13 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + +## # @x-colo-lost-heartbeat: # # Tell qemu that heartbeat is lost, request it to do takeover procedures. From patchwork Mon May 14 03:23:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912714 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="K01uZL1D"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krd03Nwnz9s08 for ; Mon, 14 May 2018 16:38:44 +1000 (AEST) Received: from localhost ([::1]:43085 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI786-0003FO-3M for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:38:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6WX-0006dK-OR for qemu-devel@nongnu.org; Mon, 14 May 2018 01:59:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46i-0000Uc-NS for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:05 -0400 Received: from mail-pg0-x22a.google.com ([2607:f8b0:400e:c05::22a]:45501) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46i-0000Te-Hm for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:04 -0400 Received: by mail-pg0-x22a.google.com with SMTP id w3-v6so4844595pgv.12 for ; Sun, 13 May 2018 20:25:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kOzj4YnKHIPIR9M+mgiQwcTIHh9IZ4z+ZsihpF83qNw=; b=K01uZL1DJtlRTzc874f4cu+oRfXfNi6wWAnt2pfBM7t8ve45nD18q8G1VE+0rwKxgP MxV35+RKCQj+i8yqdRfZMHZaeITAALScNlfCM/FBJsmlBUPM1cnsFscKSfhGBTOywDa8 49uCXFnC166YTaynfFo4fk0HoqLEpxDw2C3uU1HQbjxqPqghTg/9rdS098nA+WaYPYLW 0LWV+eFqx7ytmb1htaNKNa5a1IpBRiSK6A3mzNEkbw0dWNWFSBmyT8PXHTs4/k6pHg4c Qu+CZ0mH+WMtYNOSNuQPi8trX5eK9/L1cP1z4oxj8KhMix7jxMrlu3dius9VwaT2pSvP I/sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kOzj4YnKHIPIR9M+mgiQwcTIHh9IZ4z+ZsihpF83qNw=; b=fiUtcn8d2R14leIyF2fat/VCN493F/KozbK6Mz6IMoRGmZ97w+E/XNYS/u2heHhR/B xFmk0e2LUQzI+UFV77QwRzc3yewlFWdDmmte2t/2K4S1T8agOOmasn/bFHXMMdMu9D6q nNaUe9CQxIKN9Vt2vuNQLFAVeo94CS9s5b+WE9CcGG/guwiqwbLZFGpuZ2KG791gVSta YJoLB13Nlyn+W5kKCvUpoS+XHaEWn3bVRIVxhSbMHeWUjQp+wXiZOggoEkEniJOSm1gX HnxDVxeP+zz5xyzyMgQaQy3dIKRP5BNDB5OIx+tOfu9F1yROIdl4KmRKCPuklx9NVI02 33TA== X-Gm-Message-State: ALKqPweS799Fh3A47QJDk4TY98VZh3q36Uqip50xG+FKkFS2IOUcmuOP KTdwK9boRqx6sFOSz7n4A0xpvw== X-Google-Smtp-Source: AB8JxZprnPgqJ846ugguJ901NCBdasg2xOVv473s9yQadN1uuxp5wormgggai99FPPiBfBjYQirwZA== X-Received: by 2002:a63:6110:: with SMTP id v16-v6mr6893624pgb.292.1526268303357; Sun, 13 May 2018 20:25:03 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:02 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:42 +0800 Message-Id: <1526268228-27951-12-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22a Subject: [Qemu-devel] [PATCH V7 11/17] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 34 ++++++++++++++++++++++++++++++++++ qapi/migration.json | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8ca6381..314344c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,39 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } +COLOStatus *qmp_query_colo_status(Error **errp) +{ + int state; + COLOStatus *s = g_new0(COLOStatus, 1); + + if (get_colo_mode() == COLO_MODE_UNKNOWN) { + error_setg(errp, QERR_FEATURE_DISABLED, "colo"); + s->colo_running = false; + goto out; + } else if (get_colo_mode() == COLO_MODE_PRIMARY) { + state = migrate_get_current()->state; + } else { + state = migration_incoming_get_current()->state; + } + s->colo_running = state == MIGRATION_STATUS_COLO; + +out: + s->mode = get_colo_mode(); + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason = COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason = COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason = COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 55dae48..13589ba 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1220,3 +1220,36 @@ # Since: 2.9 ## { 'command': 'xen-colo-do-checkpoint' } + +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @colo-running: true if COLO is running. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'colo-running': 'bool', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "colo-running": true, "reason": "request" } } +# +# Since: 2.13 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } From patchwork Mon May 14 03:23:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912708 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="dlexpN5Y"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krRK2RTHz9s02 for ; Mon, 14 May 2018 16:30:21 +1000 (AEST) Received: from localhost ([::1]:42540 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6zy-0004hW-SM for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:30:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6g5-0006dK-7A for qemu-devel@nongnu.org; Mon, 14 May 2018 02:09:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46o-0000aS-7l for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:14 -0400 Received: from mail-pl0-x236.google.com ([2607:f8b0:400e:c01::236]:34424) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46o-0000Zr-04 for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:10 -0400 Received: by mail-pl0-x236.google.com with SMTP id ay10-v6so6509678plb.1 for ; Sun, 13 May 2018 20:25:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yWFrcXEFvFX0MP9vrdrR6VQWCgO9lRgtlgnt3l4KLCs=; b=dlexpN5YeXldTTFHvkTSE2YOTQ/0NoPwsS7q/02qzgG0OLxQl34xQltKCO1OSN96xr QwgnebvupASEJyBhmaEXd4J8NLb11BRP/AaE/w3ZXtv5Icidkmlua9V/fdh1L4kVgIf5 HhJlPMN73SPdzLTtZfSwaWetXqkH1nRuILXQJseATUBr5xmfcf+qoMK8e7Uh5FcmyUKE HEoCu1h/FUgmkmTaTUO97kksSsXQFQ01w4UKH68gOoLM35nCrUrQJk8K3cyJV5kSNuTK aOBuxH7q3DjcQNE7OTlotm1A5aSGteK3hpczL4pqw95YhKPLeoP2TT8a8jgISkShOha4 Om4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yWFrcXEFvFX0MP9vrdrR6VQWCgO9lRgtlgnt3l4KLCs=; b=THLwzANcdNCZGYZLilIBO4Xr7ek80v82eY6bbwDxOVR85D3ifpqO2DzwkREjJjHSNN CTNjxPPIHw+qfsY5Dvw8hY71E4X8X2kPrl971EpmllpFnZGbwMIBEAKTxKyT5mAIr52I MlAe6zRywMF61I97TyJIJi0KvcZZrgSydY4MUf/NXFvlppkElwPi1jvV2KkcOYzW7Umf W3nKG+YeYOTgAyBtxk/Ue0Bff+I2hAzr93ZO72odbz81oCYOEcOXgXfUl409foNnKFOj 6t2fHmWw7nhSmTPHZ+fmqo+CSAJ4VbuXIEaQ+WuatjGTWQdxaSteDixo/b+a4NoC90fV lkaQ== X-Gm-Message-State: ALKqPwcz+6btZuPSKGiptu8kCuv/ufurfJb3LosI3PQ+WVNjjya0nibX RgLKnYDVW6wPA9XyWIu9W0F5vQ== X-Google-Smtp-Source: AB8JxZoAdEyt16cM+RkVes9eWYci0eJXFPgRK9fAwUhrfSozpI7KHhuTM6RCY0it8+iPbGrcBiNpeA== X-Received: by 2002:a17:902:6041:: with SMTP id a1-v6mr7944117plt.59.1526268308701; Sun, 13 May 2018 20:25:08 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:07 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:43 +0800 Message-Id: <1526268228-27951-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::236 Subject: [Qemu-devel] [PATCH V7 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 314344c..7869d55 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -414,24 +415,31 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -643,6 +651,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -715,6 +724,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -748,8 +767,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index ec0bff0..0f61239 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1332,13 +1332,20 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1394,8 +1401,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2075,7 +2080,7 @@ void qemu_loadvm_state_cleanup(void) } } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2229,6 +2234,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index c6d46b3..cf7935d 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -53,8 +53,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif From patchwork Mon May 14 03:23:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912692 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NvpR+7X8"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kqYc1rN8z9s0q for ; Mon, 14 May 2018 15:50:44 +1000 (AEST) Received: from localhost ([::1]:35420 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Nd-0006Be-Te for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 01:50:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42399) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6M8-0004LU-Ak for qemu-devel@nongnu.org; Mon, 14 May 2018 01:49:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46s-0000eD-Ir for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:15 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:37404) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46s-0000da-CV for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:14 -0400 Received: by mail-pl0-x242.google.com with SMTP id w19-v6so3895300plq.4 for ; Sun, 13 May 2018 20:25:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=l/UMpu2HwkV3Omzpo3jpft43+BYwdzJb/7h+AAtuQ0A=; b=NvpR+7X8oXnF5ccberM8zFXIWCKh5ya2wePXmIF8pcJ+Hhnob3VP9/JeZuXLTRqUx4 XkyCOgTSoUv3+AxSZExzh5RRwGfJnr0fRsZEhyDvE7kl7ADwpr2I4+zGE1WVA6xElrf4 EuGBIY1+jTgm5LfPHsKC+k0AeGUe8HPFsSONrNNpxUj/CcC5u/AvgP9HsBjj7wc/t9Mt 9EPRDYffPKtxismk7e6MTO/rd9PpWa7/4Z0fhxoNjF29ZZ9Cp68R8A6t6jVoYxWKfnbr MEKNV1QjQL8FLnuKx+Re75DPgAWBaRW+YoxtX+4xNSekfyIyju2K09cvL0BakocJ6sXu Gzsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=l/UMpu2HwkV3Omzpo3jpft43+BYwdzJb/7h+AAtuQ0A=; b=WAXzjUbFRfvP5yOjYyn1xwFz93ikI3ceibizsIl0T5Ub3ngClmvhpAozKx7UYTi8gi IsI6UHOg7kjHUSfR41y8lfmTQifjIaXGphxEFIO3h5wAJu6PrJUahNE1cqyXay5AWW2S fZqJmQci9yftt23NmQwVuc0AW2ZXptyr4o1M2VqCcZsDhT0uOB0H5ZhZ3o/qf6n+ZlNZ YAVYw7mVS2aK0nE9b/wEFTow4WSBm+CZSH5WinNP6TvAccv1fq6hh7GBSJ8fsjAjswGG 9gHilXLQ7FKRP3rUSJTr43wt3y6Dh6uc6Rm+ZqnmAqZH+PQHnjAjt6R3c3yAp2NzAAQh +1DQ== X-Gm-Message-State: ALKqPwdLyBkLVJCRc4VH5Zfy8BGK8Df/0+vnJLGJm6OWiSPVH2poSAdi YzKHNSAT1ru7ezXH0ch1D1eMfA== X-Google-Smtp-Source: AB8JxZoWV1O0Ffyi2sV0NuG/0HfFoAwBAN40PKYmYVRcGEP6bg5zRu/LBDQ4PNfz6y8NQWftN9ToMw== X-Received: by 2002:a17:902:28ab:: with SMTP id f40-v6mr8058039plb.208.1526268313229; Sun, 13 May 2018 20:25:13 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:12 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:44 +0800 Message-Id: <1526268228-27951-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V7 13/17] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4235a8f..21027c5 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2786,6 +2786,7 @@ int colo_init_ram_cache(void) } ram_state = g_new0(RAMState, 1); ram_state->migration_dirty_pages = 0; + memory_global_dirty_log_start(); return 0; @@ -2806,10 +2807,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap = NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3042,6 +3045,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset = 0; + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_length); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block = QLIST_FIRST_RCU(&ram_list.blocks); From patchwork Mon May 14 03:23:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912713 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tcf6ejgy"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krYh54Bpz9s02 for ; Mon, 14 May 2018 16:35:52 +1000 (AEST) Received: from localhost ([::1]:42794 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI75K-0000oI-BG for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:35:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47224) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6WF-000694-D2 for qemu-devel@nongnu.org; Mon, 14 May 2018 01:59:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46x-0000mh-9a for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:20 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:33326) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46x-0000lI-2q for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:19 -0400 Received: by mail-pf0-x241.google.com with SMTP id a20-v6so1369984pfo.0 for ; Sun, 13 May 2018 20:25:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EtLn1VlplhjRL+Oqyu8HuYlcRW/kPez7euXvHhOKC1c=; b=tcf6ejgyzpcp1PysxYqP8jQGledRR3XvQQBBqGpLmnYCi2fjBG9SU9+Ar4nWlEKG7w J6Ihxnh9Zn+6lv0yn26anAsi+OrRcQCXdcb5QiL34VEJXPiwWzRemFkrOhdAvYa6QS0H R0YC0Fnb1/VXdVjd+LE2RIZeq9mkYg8LOXQvglY1pNo7c+MFgsApcUC1MBFG1DouoegJ wplp7zcRXqx4jRu3MQ+jAx4TBKSTVzR5/ADTNiK+46T4/qF6/s/gEr3CUgqwsJb0MK2l m2IOilrnDgfaWebNXO7kwaMcWqW0cCTv4ODk9TX0AGR0cgWUEZkME1GK/7VfKl0zN6aE qV9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EtLn1VlplhjRL+Oqyu8HuYlcRW/kPez7euXvHhOKC1c=; b=gUi069TqPcFsfDXICxBT3IZKpxxUKQ9/42fPjevtQR042YzLTC1IsshXc9kY1Nj11M vSGUjQKZyBzbQ+2YZixUXzFDjFyL9Dl7kgH91eEf25gGCc/z24A5dKUU5momvbCEhdZy Jo93sSRTwIjtvg4/rt7Kn1f4OtkpBtMxvKYuXigUw0IY2N0WbHCA2Fas1OarcjQCvU+1 Lx3JpiWd+0JW3D9xVD05OReA5tcSFXYOOUtZuuCiTuoRvVCBoKoh7HmNDWr7L4ZH/qaO TFTQnJqOodj7AnxOAlREUZJJreOGlCAtk90Y50pQpeseW+pL4U/z9t09TntU3BtnoZgD teOg== X-Gm-Message-State: ALKqPwcBhuG/hpRT64C5gjpAihpMkXLfkltLerClchhJZqEtKpU5e3XY CHR0jjQmRzZLw2g24+x8iHNTfg== X-Google-Smtp-Source: AB8JxZrhq41pmOMFkSQjhszUGol5U2b3AtPji62jwT+CqkQ3rQUhIdqkdE+/OzAjAxUVVNgrUmzkbA== X-Received: by 2002:a63:618b:: with SMTP id v133-v6mr6834948pgb.285.1526268317915; Sun, 13 May 2018 20:25:17 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:17 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:45 +0800 Message-Id: <1526268228-27951-15-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 14/17] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6..49da666 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class; @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender, int iovcnt, void *opaque); +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d..0f17eba 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc = NETFILTER_CLASS(oc); ucc->complete = netfilter_complete; + nfc->handle_event = dummy_handle_event; } static const TypeInfo netfilter_info = { diff --git a/net/net.c b/net/net.c index 29f8398..d58691b 100644 --- a/net/net.c +++ b/net/net.c @@ -1335,6 +1335,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc = NULL; + Error *local_err = NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer = nc->peer; + type = nc->info->type; + if (!peer || type != NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc = NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; From patchwork Mon May 14 03:23:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912710 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRmAD62y"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krWY6dFdz9s02 for ; Mon, 14 May 2018 16:34:01 +1000 (AEST) Received: from localhost ([::1]:42691 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI73X-0007dE-I0 for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:33:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47368) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Y3-0006aO-6W for qemu-devel@nongnu.org; Mon, 14 May 2018 02:01:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI472-0000sj-4I for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:25 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:40443) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI471-0000rz-UX for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:24 -0400 Received: by mail-pl0-x243.google.com with SMTP id t12-v6so6514205plo.7 for ; Sun, 13 May 2018 20:25:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/weR4mu/aSbI0SOgvZGIaau+uyj4HGEo84Dt7VF0Cx8=; b=tRmAD62yUA2rgtx/oPQlWbPPJS5CBXfhF9bd5RqEcnNRIe2yLvy1qpj9RZL2SSV2Nj 6ul6q1e+KKwcKheIkO1GD+yH0VOpdWD3W6eO2StHLmTDs73KUhS7Mn3BL0W5T7HoLc67 1koyu1DnIz60CePobxMQbRGIA5qxXYyUNSSC3Jvst66nLs/pgLjt4+p6rYpKDlF0hxHF UqCB8tRrDip96c3xqnYmxK+GKRchqCJTUX4qB4f4dSeT15qAOAc2exauaVuBFCiZ/exY fJoHfu76CfgGwlQOu98FY9456J35mLgBDGD4nTloVkwbmn3IxcWnP6kVMKot1OgBk1WL AidA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/weR4mu/aSbI0SOgvZGIaau+uyj4HGEo84Dt7VF0Cx8=; b=bt5XPBDsEgJuO+YFfGBc9M83d6CH1FJtII7qRsjKYX5JFMJQVm/1TBKczniqstywyB hBQivUgcnfE+TWOTVegGsSXwml1vtJRLsoDtrMLwMPK8WBKmxgAgUmXc0Pki64pZbdgu kNEhGjTo2FVPWWsZH12tWHHvyTYQOHIypBFvDYADykl+egaEA22Yd8LpdyFHbukvn9r3 eXqUF0AVC4eg8FTj5s4TpMWlcJq6U9ni+1TvXy6fu3q9XtRrKRIMj/Al3gxLNnkKdmOm +DfcXvXkZyG0FbJm9niqe0FzPkyh5sky69BRUUDlF3vzMyQIF8+ledFCsZuIPevBswte UODw== X-Gm-Message-State: ALKqPwdAelCBWOM1QPUlPjjNayZ89Q+D6fruFij4nhRjzBx6jOGmgVtY sk22R0ojU2XViqXpuVRmIrah+Q== X-Google-Smtp-Source: AB8JxZqm9qaLKVO2pJLP3bbruwb8xya3JY8rySoHRPRnVBms4zWwFiP2a+1PWqXLW+DBwPVFOdFAjg== X-Received: by 2002:a17:902:ab93:: with SMTP id f19-v6mr8140376plr.392.1526268322785; Sun, 13 May 2018 20:25:22 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:21 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:46 +0800 Message-Id: <1526268228-27951-16-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V7 15/17] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 7869d55..5e517dc 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -82,6 +83,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); @@ -786,6 +792,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 0909a9a..f3c306c 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,6 +20,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -277,6 +279,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, return 0; } +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_data) +{ + Connection *conn = (Connection *)value; + + conn->offset = 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn = (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs = FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -332,6 +371,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, void *data) nfc->setup = colo_rewriter_setup; nfc->cleanup = colo_rewriter_cleanup; nfc->receive_iov = colo_rewriter_receive_iov; + nfc->handle_event = colo_rewriter_handle_event; } static const TypeInfo colo_rewriter_info = { From patchwork Mon May 14 03:23:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912699 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Bgbip5Ei"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40kr6p1knkz9s0W for ; Mon, 14 May 2018 16:16:02 +1000 (AEST) Received: from localhost ([::1]:41465 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6m5-0006cG-4f for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:15:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52586) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6kx-00060s-7W for qemu-devel@nongnu.org; Mon, 14 May 2018 02:14:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI477-0000zJ-DU for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:30 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:37404) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI476-0000xq-Ja for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:28 -0400 Received: by mail-pl0-x241.google.com with SMTP id w19-v6so3895626plq.4 for ; Sun, 13 May 2018 20:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=W3Pb33nZV84BYKajEA+RBOCsx/y0k0R3XgNyV0JqEZ0=; b=Bgbip5EiqVatpmhRwi3vf/Chri0Yo5RPtzuG2aPapOoJS6jPU1mZ0nhrV4f23zaO6Z /v+GvKp0nVfZeFmWa/Ni4MgAzL1PSANGJSa1T+/HdtqI0l/OzFnPNaOiFQ3dVYK4fD5p ZyZ6J0A7dluhWmvBvtDFF/aAz9TbOGU58RghxMNkALSwLCHkByGemYe6YAkDcg6KAQNJ JyRlbYoQaT4p/LiMJJIwFCBGqGY+CYrHNVAZQn8M7/Krr5eDvmre2aDg57edp4bkFxiI 1MdqTXtcFaKCi3dPG4hEjDrwCFOnMGx/kzhfgbVfrfjraSuE/E3MZzZBCpLwHbueZTcr X7Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=W3Pb33nZV84BYKajEA+RBOCsx/y0k0R3XgNyV0JqEZ0=; b=Nt5C0wM1gFrXNFbvc4ao/JaSox6GiQxs4iiXRaLZoz7mtp4Y9dM0u2K9mYW6HagefL q5hudQSKCGmoKwXaRFpqZgcMKmn4f4Box3ebfxe3J2+umLh6GkTY5fybZnzdgBNAGwE5 vTh1LV3D6Ml4ZIz167zbbH7B4z0QGFGPwfRt+MMZ29uEScwD62AcvwQ/L0vJ90k+Kd99 qwtVK+YIu7T17go6z83ZEliyLGJoR1fm6tKjeceZV//cYjxUa6ycZX4lbnkZkSX3EnEN cTsFir36Xrr1g5qQELR7RgGGLyNUKzzRxa2P3sxocBqf6XumwiT85UF+MEJcXWHdqyp0 rnyA== X-Gm-Message-State: ALKqPwddgaqf3jKXirMmcM06YJjuY5AueNK493LAk+j4xT7w2O3MWqii yvnYBkjNz329xfkIOZGeQpCfLw== X-Google-Smtp-Source: AB8JxZqmRavX7FEpx+AVaXZ6fuMT02RiYRgfMABks+swfhR3rdQYvUbq6UJyvNCYrxlrQ36PTBh3lw== X-Received: by 2002:a17:902:2f43:: with SMTP id s61-v6mr7782053plb.99.1526268327522; Sun, 13 May 2018 20:25:27 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:26 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:47 +0800 Message-Id: <1526268228-27951-17-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V7 16/17] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 5e517dc..d67bdc2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -88,6 +88,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); @@ -799,6 +804,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); From patchwork Mon May 14 03:23:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 912719 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hem/EieZ"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40krxm4xb3z9s0q for ; Mon, 14 May 2018 16:53:16 +1000 (AEST) Received: from localhost ([::1]:44664 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI7MA-0005zp-E0 for incoming@patchwork.ozlabs.org; Mon, 14 May 2018 02:53:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6gN-0006dK-1O for qemu-devel@nongnu.org; Mon, 14 May 2018 02:10:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI47B-00015W-PU for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:34 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:45470) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI47B-00014F-Hm for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:33 -0400 Received: by mail-pf0-x244.google.com with SMTP id c10-v6so5305914pfi.12 for ; Sun, 13 May 2018 20:25:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gDOaczUBMb7s+whdZ/MTEfFb21DM/MqUI5E3f4sM7Ww=; b=hem/EieZW4ZIiu9BITbCdXmjXkbxzZb0ivTNXBF1bLEOFjjAIOgzboe+GgZH3bBKpB GWKeZO2R7Gyi3Xrh5Ub2UfKZDwDjAfBBoFyFwUDXUY5kZn+cKQS4OfFag7o0BwR7lODA NG/3aEjvOymk/MBAEiSgw8nFZ70urZ52y6Z8i+iOwcHXTJnrBgCzEHEqS9G1PKyz+5ee S6b/xGbW3cr0ox2Z95+TCFmnZNEmSbG5YLYwaZofeUrp5yb/twWdhHsXmkaF2zzGUsU5 HeEqCq8XEU8uJBAG8myD2mfCs8qX70zPIZcuQOq0MXvWGrvXz5FL9ho0IoehBLvEdf/j V2kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gDOaczUBMb7s+whdZ/MTEfFb21DM/MqUI5E3f4sM7Ww=; b=XJtFXAz1WPuok3IHnO8jZW5iz2qB9FWMZjUsZkrr5WNFYESyrXQQyQEptYwJHBwsaI TbJCREs9o2dWVCvAKJ3Z4LdygsvLOC3VgjnoT3vHl/N+Rf8/e1MCndZDEUNkwM+yMFTg R1Nj1Xy2S70dwIfoZGoDdmRaNxqmZZxQae92NbsZ9N9qknH8UPp2HzIU2QGa16X1xpcj KwAkAC7CrnUjZcBcsiN9g+0XC5vyOrvTLZAtO/Ka9MvMXgyGtdhtDKWrDpla/iNJabZ/ tnXsO4uE7UG09S+l6FY+dp7gv/tUAoPyhiTGXebvXZT5Rov9LKc47HX/Tm9AHv7oXSOv 5kDA== X-Gm-Message-State: ALKqPweH2NF6l2FlTmxVL1wnD8PD2SI4nuOTYcQs5QW8yvZk6VT6lopM quB9McGPS8LVXgEMZrzpHswvbA== X-Google-Smtp-Source: AB8JxZop2Ka5NFUCzW+dg9VKHb1R5eZ1ZIo3Qx7OQsRyRsxD2v/rJtSFGH3Ff0un1wsDzsdfEcQijQ== X-Received: by 2002:a62:d9d9:: with SMTP id b86-v6mr8604009pfl.41.1526268332374; Sun, 13 May 2018 20:25:32 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:31 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:48 +0800 Message-Id: <1526268228-27951-18-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V7 17/17] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index d67bdc2..a7e0eaa 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -135,6 +135,11 @@ static void primary_vm_do_failover(void) migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); /* * Wake up COLO thread which may blocked in recv() or send(), @@ -552,6 +557,9 @@ static void colo_process_checkpoint(MigrationState *s) qemu_sem_wait(&s->colo_checkpoint_sem); + if (s->state != MIGRATION_STATUS_COLO) { + goto out; + } ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out;