From patchwork Thu May 21 08:12:58 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhanghailiang X-Patchwork-Id: 474864 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 276F114027F for ; Thu, 21 May 2015 18:20:06 +1000 (AEST) Received: from localhost ([::1]:55846 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLi0-0002uC-9Q for incoming@patchwork.ozlabs.org; Thu, 21 May 2015 04:20:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52369) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLcj-0002Cb-D6 for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YvLcg-0002iE-31 for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:37 -0400 Received: from szxga03-in.huawei.com ([119.145.14.66]:24110) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLcf-0002gq-7o for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:33 -0400 Received: from 172.24.2.119 (EHLO szxeml427-hub.china.huawei.com) ([172.24.2.119]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id BGI98528; Thu, 21 May 2015 16:14:21 +0800 (CST) Received: from localhost (10.177.22.69) by szxeml427-hub.china.huawei.com (10.82.67.182) with Microsoft SMTP Server id 14.3.158.1; Thu, 21 May 2015 16:14:09 +0800 From: zhanghailiang To: Date: Thu, 21 May 2015 16:12:58 +0800 Message-ID: <1432196001-10352-7-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 1.9.2.msysgit.0 In-Reply-To: <1432196001-10352-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1432196001-10352-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.22.69] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020206.555D93DE.005D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-05-26 15:14:31, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 1c966b027c7dc3135f14c00e42c2f59e X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 119.145.14.66 Cc: lizhijian@cn.fujitsu.com, quintela@redhat.com, yunhong.jiang@intel.com, eddie.dong@intel.com, peter.huangpeng@huawei.com, dgilbert@redhat.com, zhanghailiang , arei.gonglei@huawei.com, amit.shah@redhat.com, Lai Jiangshan , Yang Hongyang , david@gibson.dropbear.id.au Subject: [Qemu-devel] [PATCH COLO-Frame v5 06/29] COLO: Implement colo checkpoint protocol X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We need communications protocol of user-defined to control the checkpoint process. The new checkpoint request is started by Primary VM, and the interactive process like below: Checkpoint synchronizing points, Primary Secondary NEW @ Suspend SUSPENDED @ Suspend&Save state SEND @ Send state Receive state RECEIVED @ Flush network Load state LOADED @ Resume Resume Start Comparing NOTE: 1) '@' who sends the message 2) Every sync-point is synchronized by two sides with only one handshake(single direction) for low-latency. If more strict synchronization is required, a opposite direction sync-point should be added. 3) Since sync-points are single direction, the remote side may go forward a lot when this side just receives the sync-point. Signed-off-by: Yang Hongyang Signed-off-by: Lai Jiangshan Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Gonglei --- migration/colo.c | 237 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 33d3105..7663144 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -15,6 +15,41 @@ #include "trace.h" #include "qemu/error-report.h" +enum { + COLO_CHECPOINT_READY = 0x46, + + /* + * Checkpoint synchronizing points. + * + * Primary Secondary + * NEW @ + * Suspend + * SUSPENDED @ + * Suspend&Save state + * SEND @ + * Send state Receive state + * RECEIVED @ + * Flush network Load state + * LOADED @ + * Resume Resume + * + * Start Comparing + * NOTE: + * 1) '@' who sends the message + * 2) Every sync-point is synchronized by two sides with only + * one handshake(single direction) for low-latency. + * If more strict synchronization is required, a opposite direction + * sync-point should be added. + * 3) Since sync-points are single direction, the remote side may + * go forward a lot when this side just receives the sync-point. + */ + COLO_CHECKPOINT_NEW, + COLO_CHECKPOINT_SUSPENDED, + COLO_CHECKPOINT_SEND, + COLO_CHECKPOINT_RECEIVED, + COLO_CHECKPOINT_LOADED, +}; + static QEMUBH *colo_bh; static Coroutine *colo; @@ -29,19 +64,136 @@ bool migrate_in_colo_state(void) return (s->state == MIGRATION_STATUS_COLO); } +/* colo checkpoint control helper */ +static int colo_ctl_put(QEMUFile *f, uint64_t request) +{ + int ret = 0; + + qemu_put_be64(f, request); + qemu_fflush(f); + + ret = qemu_file_get_error(f); + + return ret; +} + +static int colo_ctl_get_value(QEMUFile *f, uint64_t *value) +{ + int ret = 0; + uint64_t temp; + + temp = qemu_get_be64(f); + + ret = qemu_file_get_error(f); + if (ret < 0) { + return -1; + } + + *value = temp; + return 0; +} + +static int colo_ctl_get(QEMUFile *f, uint64_t require) +{ + int ret; + uint64_t value; + + ret = colo_ctl_get_value(f, &value); + if (ret < 0) { + return ret; + } + + if (value != require) { + error_report("unexpected state! expected: %"PRIu64 + ", received: %"PRIu64, require, value); + exit(1); + } + + return ret; +} + +static int colo_do_checkpoint_transaction(MigrationState *s, QEMUFile *control) +{ + int ret; + + ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW); + if (ret < 0) { + goto out; + } + + ret = colo_ctl_get(control, COLO_CHECKPOINT_SUSPENDED); + if (ret < 0) { + goto out; + } + + /* TODO: suspend and save vm state to colo buffer */ + + ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND); + if (ret < 0) { + goto out; + } + + /* TODO: send vmstate to slave */ + + ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED); + if (ret < 0) { + goto out; + } + trace_colo_receive_message("COLO_CHECKPOINT_RECEIVED"); + + ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED); + if (ret < 0) { + goto out; + } + trace_colo_receive_message("COLO_CHECKPOINT_LOADED"); + + /* TODO: resume master */ + +out: + return ret; +} + static void *colo_thread(void *opaque) { MigrationState *s = opaque; + QEMUFile *colo_control = NULL; + int ret; + + colo_control = qemu_fopen_socket(qemu_get_fd(s->file), "rb"); + if (!colo_control) { + error_report("Open colo_control failed!"); + goto out; + } + + /* + * Wait for slave finish loading vm states and enter COLO + * restore. + */ + ret = colo_ctl_get(colo_control, COLO_CHECPOINT_READY); + if (ret < 0) { + goto out; + } + trace_colo_receive_message("COLO_CHECPOINT_READY"); qemu_mutex_lock_iothread(); vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); - /*TODO: COLO checkpoint savevm loop*/ + while (s->state == MIGRATION_STATUS_COLO) { + /* start a colo checkpoint */ + if (colo_do_checkpoint_transaction(s, colo_control)) { + goto out; + } + } +out: migrate_set_state(s, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + if (colo_control) { + qemu_fclose(colo_control); + } + qemu_mutex_lock_iothread(); qemu_bh_schedule(s->cleanup_bh); qemu_mutex_unlock_iothread(); @@ -74,14 +226,95 @@ void colo_init_checkpointer(MigrationState *s) qemu_bh_schedule(colo_bh); } +/* + * return: + * 0: start a checkpoint + * -1: some error happened, exit colo restore + */ +static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request) +{ + int ret; + uint64_t cmd; + + ret = colo_ctl_get_value(f, &cmd); + if (ret < 0) { + return -1; + } + + switch (cmd) { + case COLO_CHECKPOINT_NEW: + *checkpoint_request = 1; + return 0; + default: + return -1; + } +} + void *colo_process_incoming_checkpoints(void *opaque) { + struct colo_incoming *colo_in = opaque; + QEMUFile *f = colo_in->file; + int fd = qemu_get_fd(f); + QEMUFile *ctl = NULL; + int ret; colo = qemu_coroutine_self(); assert(colo != NULL); - /* TODO: COLO checkpoint restore loop */ + ctl = qemu_fopen_socket(fd, "wb"); + if (!ctl) { + error_report("Can't open incoming channel!"); + goto out; + } + ret = colo_ctl_put(ctl, COLO_CHECPOINT_READY); + if (ret < 0) { + goto out; + } + /* TODO: in COLO mode, slave is runing, so start the vm */ + while (true) { + int request = 0; + int ret = colo_wait_handle_cmd(f, &request); + + if (ret < 0) { + break; + } else { + if (!request) { + continue; + } + } + /* TODO: suspend guest */ + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_SUSPENDED); + if (ret < 0) { + goto out; + } + + ret = colo_ctl_get(f, COLO_CHECKPOINT_SEND); + if (ret < 0) { + goto out; + } + trace_colo_receive_message("COLO_CHECKPOINT_SEND"); + + /* TODO: read migration data into colo buffer */ + + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_RECEIVED); + if (ret < 0) { + goto out; + } + trace_colo_receive_message("COLO_CHECKPOINT_RECEIVED"); + + /* TODO: load vm state */ + + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_LOADED); + if (ret < 0) { + goto out; + } +} + +out: colo = NULL; + if (ctl) { + qemu_fclose(ctl); + } loadvm_exit_colo(); return NULL;