From patchwork Fri Jan 19 13:44:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 863510 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="meoRyNcA"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zNMZN5Npvz9s7g for ; Sat, 20 Jan 2018 00:47:08 +1100 (AEDT) Received: from localhost ([::1]:50327 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0c-0003Yg-PP for incoming@patchwork.ozlabs.org; Fri, 19 Jan 2018 08:47:06 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46549) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzm-0003X4-Ek for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzl-0002dc-04 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:14 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:41983) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzk-0002dH-O9 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:12 -0500 Received: by mail-pg0-x244.google.com with SMTP id 136so1458621pgd.8 for ; Fri, 19 Jan 2018 05:46:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Hu9Feno9vRYER8WMKcaLr1DBR5y+XFmncbgvti/KUhs=; b=meoRyNcAG2CEYg7+pSAMffurB+HME9fceh0eeiEVxpVk7y7zaiReXJWrLJN2kSiFeE jsugzJQgm11B99bdk6tbOlOg0cdXIZVlJWfsoT1FQp4ZLprHWuz2qrFtaCIKsPRCwnrs 7xnW1QcpRUOGA2oPg7roaq4NYdnEh2vJuAslTd6fm1LVsKZKpn4U0fNRntDpZoJbjRr8 CuFb2vD0KbxiRj/+suwJiSTPdTaqcE4xBcW5hrTxDjAqk+lTyN0iwHmyyW0ysyzGwsX0 uWQ72K1i1pKcUM0phgUxKhxU/J6tL34gEE1iSI0B02X267LdI7ozmxHWUcV8Ouda117p dpow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Hu9Feno9vRYER8WMKcaLr1DBR5y+XFmncbgvti/KUhs=; b=CyOJ2WmA2y2BnyNo4/RK1VS22A6TgRE9HpxTlML/DJCcpzkqaRTKeTN40/RjOPt1dv 1S/oYaenLysEhylcww3HChwY1SGmi1LmxU17jZ6MBJu1QUa/Ydb8Q06ZhFYJEc2QeqAC FB0dXvkXZu+TQk9mYA8AiviJcILe1ptYxnBAy7B0IKrX4hKgMrKoM/Yn2P36f/ObZPj3 5fq9h/ueHbQxXA6bLJlM8ONRQes75pw7OdBr9B8rK1VvagkakbdxPTLxteebmpSgMmum X7ZF268PMpkh8M2I02nu1hbILk3QcZe1Q3Vjz9QwwFSuoUj4c8HPuISuR00bgBA0KtsX kHjw== X-Gm-Message-State: AKwxytfUqcSWaUoNAkhorVbnyONUTbVn22KVWWJk2RmRZfDC+vDUh20M eOZNy3CsNGUUU9nTHaO18qb6P7DP X-Google-Smtp-Source: ACJfBosLH8ZyXcSiMLQJCmP2VUkY5hPIVbt/iXvDI0+yOKkbDva8z+sCeAm/1xzDPB5RC5C1c3OwTw== X-Received: by 10.99.125.74 with SMTP id m10mr26448995pgn.354.1516369571389; Fri, 19 Jan 2018 05:46:11 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:10 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:31 +0800 Message-Id: <1516369485-5374-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V4 02/16] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index ff9874e..6adf3a5 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -15,6 +15,12 @@ #include "qemu-common.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 0ebdec9..4bceca8 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,17 +29,26 @@ #include "qapi-visit.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -86,6 +95,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -631,6 +645,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -651,6 +684,28 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -664,6 +719,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -821,8 +877,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -885,6 +946,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -892,6 +954,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -904,6 +976,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */