From patchwork Thu May 21 08:13:01 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhanghailiang X-Patchwork-Id: 474861 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 847E814077A for ; Thu, 21 May 2015 18:17:40 +1000 (AEST) Received: from localhost ([::1]:55820 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLfe-00077u-HR for incoming@patchwork.ozlabs.org; Thu, 21 May 2015 04:17:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52317) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLcg-0002BG-CN for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YvLce-0002hQ-Ez for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:34 -0400 Received: from szxga02-in.huawei.com ([119.145.14.65]:10911) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvLcd-0002go-Kx for qemu-devel@nongnu.org; Thu, 21 May 2015 04:14:32 -0400 Received: from 172.24.2.119 (EHLO szxeml434-hub.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id CLR90067; Thu, 21 May 2015 16:14:21 +0800 (CST) Received: from localhost (10.177.22.69) by szxeml434-hub.china.huawei.com (10.82.67.225) with Microsoft SMTP Server id 14.3.158.1; Thu, 21 May 2015 16:14:11 +0800 From: zhanghailiang To: Date: Thu, 21 May 2015 16:13:01 +0800 Message-ID: <1432196001-10352-10-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 1.9.2.msysgit.0 In-Reply-To: <1432196001-10352-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1432196001-10352-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.22.69] X-CFilter-Loop: Reflected X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 119.145.14.65 Cc: lizhijian@cn.fujitsu.com, quintela@redhat.com, yunhong.jiang@intel.com, eddie.dong@intel.com, peter.huangpeng@huawei.com, dgilbert@redhat.com, zhanghailiang , arei.gonglei@huawei.com, amit.shah@redhat.com, Lai Jiangshan , Yang Hongyang , david@gibson.dropbear.id.au Subject: [Qemu-devel] [PATCH COLO-Frame v5 09/29] COLO: Save VM state to slave when do checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We should save PVM's RAM/device to slave when needed. For VM state, we will cache them in slave, we use QEMUSizedBuffer to store the data, we need know the data size of VM state, so in master, we use qsb to store VM state temporarily, and then migrate the data to slave. Signed-off-by: zhanghailiang Signed-off-by: Yang Hongyang Signed-off-by: Gonglei Signed-off-by: Lai Jiangshan Signed-off-by: Li Zhijian --- arch_init.c | 48 +++++++++++++++++++++++++++++++++---------- migration/colo.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++---- savevm.c | 2 +- 3 files changed, 96 insertions(+), 16 deletions(-) diff --git a/arch_init.c b/arch_init.c index 23d3feb..6fbc82d 100644 --- a/arch_init.c +++ b/arch_init.c @@ -54,6 +54,7 @@ #include "hw/acpi/acpi.h" #include "qemu/host-utils.h" #include "qemu/rcu_queue.h" +#include "migration/migration-colo.h" #ifdef DEBUG_ARCH_INIT #define DPRINTF(fmt, ...) \ @@ -1185,16 +1186,8 @@ static void reset_ram_globals(void) #define MAX_WAIT 50 /* ms, half buffered_file limit */ - -/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has - * long-running RCU critical section. When rcu-reclaims in the code - * start to become numerous it will be necessary to reduce the - * granularity of these critical sections. - */ - -static int ram_save_setup(QEMUFile *f, void *opaque) +static int ram_save_init_globals(void) { - RAMBlock *block; int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */ mig_throttle_on = false; @@ -1253,6 +1246,31 @@ static int ram_save_setup(QEMUFile *f, void *opaque) migration_bitmap_sync(); qemu_mutex_unlock_ramlist(); qemu_mutex_unlock_iothread(); + rcu_read_unlock(); + + return 0; +} + +/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has + * long-running RCU critical section. When rcu-reclaims in the code + * start to become numerous it will be necessary to reduce the + * granularity of these critical sections. + */ + +static int ram_save_setup(QEMUFile *f, void *opaque) +{ + RAMBlock *block; + + /* + * migration has already setup the bitmap, reuse it. + */ + if (!migrate_in_colo_state()) { + if (ram_save_init_globals() < 0) { + return -1; + } + } + + rcu_read_lock(); qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE); @@ -1352,7 +1370,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) while (true) { int pages; - pages = ram_find_and_save_block(f, true, &bytes_transferred); + pages = ram_find_and_save_block(f, !migrate_in_colo_state(), + &bytes_transferred); /* no more blocks to sent */ if (pages == 0) { break; @@ -1361,7 +1380,14 @@ static int ram_save_complete(QEMUFile *f, void *opaque) flush_compressed_data(f); ram_control_after_iterate(f, RAM_CONTROL_FINISH); - migration_end(); + + /* + * Since we need to reuse dirty bitmap in colo, + * don't cleanup the bitmap. + */ + if (!migrate_enable_colo() || migration_has_failed(migrate_get_current())) { + migration_end(); + } rcu_read_unlock(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); diff --git a/migration/colo.c b/migration/colo.c index 7663144..8ff03e7 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -52,6 +52,9 @@ enum { static QEMUBH *colo_bh; static Coroutine *colo; +/* colo buffer */ +#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) +QEMUSizedBuffer *colo_buffer; bool colo_supported(void) { @@ -115,6 +118,8 @@ static int colo_ctl_get(QEMUFile *f, uint64_t require) static int colo_do_checkpoint_transaction(MigrationState *s, QEMUFile *control) { int ret; + size_t size; + QEMUFile *trans = NULL; ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW); if (ret < 0) { @@ -125,16 +130,47 @@ static int colo_do_checkpoint_transaction(MigrationState *s, QEMUFile *control) if (ret < 0) { goto out; } + /* Reset colo buffer and open it for write */ + qsb_set_length(colo_buffer, 0); + trans = qemu_bufopen("w", colo_buffer); + if (!trans) { + error_report("Open colo buffer for write failed"); + goto out; + } + + /* suspend and save vm state to colo buffer */ + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + qemu_mutex_unlock_iothread(); + DPRINTF("vm is stoped\n"); + + /* Disable block migration */ + s->params.blk = 0; + s->params.shared = 0; + qemu_savevm_state_begin(trans, &s->params); + qemu_mutex_lock_iothread(); + qemu_savevm_state_complete(trans); + qemu_mutex_unlock_iothread(); - /* TODO: suspend and save vm state to colo buffer */ + qemu_fflush(trans); ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND); if (ret < 0) { goto out; } + /* we send the total size of the vmstate first */ + size = qsb_get_length(colo_buffer); + ret = colo_ctl_put(s->file, size); + if (ret < 0) { + goto out; + } - /* TODO: send vmstate to slave */ - + qsb_put_buffer(s->file, colo_buffer, size); + qemu_fflush(s->file); + ret = qemu_file_get_error(s->file); + if (ret < 0) { + goto out; + } ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED); if (ret < 0) { goto out; @@ -147,9 +183,18 @@ static int colo_do_checkpoint_transaction(MigrationState *s, QEMUFile *control) } trace_colo_receive_message("COLO_CHECKPOINT_LOADED"); - /* TODO: resume master */ + ret = 0; + /* resume master */ + qemu_mutex_lock_iothread(); + vm_start(); + qemu_mutex_unlock_iothread(); + DPRINTF("vm resume to run again\n"); out: + if (trans) { + qemu_fclose(trans); + } + return ret; } @@ -175,6 +220,12 @@ static void *colo_thread(void *opaque) } trace_colo_receive_message("COLO_CHECPOINT_READY"); + colo_buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE); + if (colo_buffer == NULL) { + error_report("Failed to allocate colo buffer!"); + goto out; + } + qemu_mutex_lock_iothread(); vm_start(); qemu_mutex_unlock_iothread(); @@ -190,6 +241,9 @@ static void *colo_thread(void *opaque) out: migrate_set_state(s, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + qsb_free(colo_buffer); + colo_buffer = NULL; + if (colo_control) { qemu_fclose(colo_control); } diff --git a/savevm.c b/savevm.c index 3b0e222..cd7ec27 100644 --- a/savevm.c +++ b/savevm.c @@ -42,7 +42,7 @@ #include "qemu/iov.h" #include "block/snapshot.h" #include "block/qapi.h" - +#include "migration/migration-colo.h" #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035