From patchwork Mon Oct 21 01:14:17 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrhines@linux.vnet.ibm.com X-Patchwork-Id: 285060 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 80C6A2C00BA for ; Mon, 21 Oct 2013 12:19:08 +1100 (EST) Received: from localhost ([::1]:37916 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VY49C-0001Ki-Jb for incoming@patchwork.ozlabs.org; Sun, 20 Oct 2013 21:19:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50524) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VY45I-0003xm-1r for qemu-devel@nongnu.org; Sun, 20 Oct 2013 21:15:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VY458-00047G-W5 for qemu-devel@nongnu.org; Sun, 20 Oct 2013 21:15:03 -0400 Received: from e36.co.us.ibm.com ([32.97.110.154]:44735) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VY458-00047C-OF for qemu-devel@nongnu.org; Sun, 20 Oct 2013 21:14:54 -0400 Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 20 Oct 2013 19:14:54 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sun, 20 Oct 2013 19:14:51 -0600 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 366AD3E4003F for ; Sun, 20 Oct 2013 19:14:51 -0600 (MDT) Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r9L1Eofn389778 for ; Sun, 20 Oct 2013 19:14:50 -0600 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r9L1I3Ph000825 for ; Sun, 20 Oct 2013 19:18:03 -0600 Received: from mahler.ibm.com ([9.80.101.39]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r9L1HikU032531; Sun, 20 Oct 2013 19:18:01 -0600 From: mrhines@linux.vnet.ibm.com To: qemu-devel@nongnu.org Date: Mon, 21 Oct 2013 01:14:17 +0000 Message-Id: <1382318062-6288-8-git-send-email-mrhines@linux.vnet.ibm.com> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1382318062-6288-1-git-send-email-mrhines@linux.vnet.ibm.com> References: <1382318062-6288-1-git-send-email-mrhines@linux.vnet.ibm.com> X-TM-AS-MML: No X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13102101-3532-0000-0000-000002537E4F X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 32.97.110.154 Cc: aliguori@us.ibm.com, quintela@redhat.com, owasserm@redhat.com, onom@us.ibm.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com Subject: [Qemu-devel] [RFC PATCH v1: 07/12] mc: introduce state machine error handling and migration_bitmap prep X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Michael R. Hines" Since MC will repeatedly call the pre-existing live migration call path over and over again (forever), the migration_bitmap initialization only needs to happen once and the destruction of the bitmap needs to be avoided in successive checkpoints. Also, there some additional state machine error handling to prepare for before introducing the MC core logic. Signed-off-by: Michael R. Hines --- arch_init.c | 29 ++++++++++++++++++++++++----- include/migration/migration.h | 19 +++++++++++++++++++ include/migration/qemu-file.h | 1 + migration.c | 33 +++++++++++++++++++-------------- 4 files changed, 63 insertions(+), 19 deletions(-) diff --git a/arch_init.c b/arch_init.c index 9cf7d18..d47b38b 100644 --- a/arch_init.c +++ b/arch_init.c @@ -795,13 +795,13 @@ static void ram_migration_cancel(void *opaque) migration_end(); } -static void reset_ram_globals(void) +static void reset_ram_globals(bool reset_bulk_stage) { last_seen_block = NULL; last_sent_block = NULL; last_offset = 0; last_version = ram_list.version; - ram_bulk_stage = true; + ram_bulk_stage = reset_bulk_stage; } #define MAX_WAIT 50 /* ms, half buffered_file limit */ @@ -811,6 +811,15 @@ static int ram_save_setup(QEMUFile *f, void *opaque) RAMBlock *block; int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS; + /* + * RAM stays open during micro-checkpointing for the next transaction. + */ + if (migration_is_mc(migrate_get_current())) { + qemu_mutex_lock_ramlist(); + reset_ram_globals(false); + goto skip_setup; + } + migration_bitmap = bitmap_new(ram_pages); bitmap_set(migration_bitmap, 0, ram_pages); migration_dirty_pages = ram_pages; @@ -833,12 +842,14 @@ static int ram_save_setup(QEMUFile *f, void *opaque) qemu_mutex_lock_iothread(); qemu_mutex_lock_ramlist(); bytes_transferred = 0; - reset_ram_globals(); + reset_ram_globals(true); memory_global_dirty_log_start(); migration_bitmap_sync(); qemu_mutex_unlock_iothread(); +skip_setup: + qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE); QTAILQ_FOREACH(block, &ram_list.blocks, next) { @@ -867,7 +878,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) qemu_mutex_lock_ramlist(); if (ram_list.version != last_version) { - reset_ram_globals(); + reset_ram_globals(true); } ram_control_before_iterate(f, RAM_CONTROL_ROUND); @@ -948,7 +959,15 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } ram_control_after_iterate(f, RAM_CONTROL_FINISH); - migration_end(); + + /* + * Only cleanup at the end of normal migrations + * or if the MC destination failed and we got an error. + * Otherwise, we are (or will be soon) in MIG_STATE_MC. + */ + if(!migrate_use_mc() || migration_has_failed(migrate_get_current())) { + migration_end(); + } qemu_mutex_unlock_ramlist(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); diff --git a/include/migration/migration.h b/include/migration/migration.h index 0e7f121..fcf7684 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -95,6 +95,8 @@ int migrate_fd_close(MigrationState *s); void add_migration_state_change_notifier(Notifier *notify); void remove_migration_state_change_notifier(Notifier *notify); bool migration_in_setup(MigrationState *); +bool migration_is_active(MigrationState *); +bool migration_is_mc(MigrationState *s); bool migration_has_finished(MigrationState *); bool migration_has_failed(MigrationState *); MigrationState *migrate_get_current(void); @@ -126,6 +128,15 @@ void migration_bitmap_worker_start(MigrationState *s); void migration_bitmap_worker_stop(MigrationState *s); void migrate_set_state(MigrationState *s, int old_state, int new_state); +enum { + MIG_STATE_ERROR = -1, + MIG_STATE_NONE, + MIG_STATE_SETUP, + MIG_STATE_CANCELLED, + MIG_STATE_ACTIVE, + MIG_STATE_MC, + MIG_STATE_COMPLETED, +}; void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); /** @@ -194,4 +205,12 @@ int ram_control_copy_page(QEMUFile *f, ram_addr_t block_offset_source, ram_addr_t offset_source, long size); + +int migrate_use_mc(void); +int migrate_use_mc_rdma_copy(void); + +#define MC_VERSION 1 + +void qemu_rdma_info_save(QEMUFile *f, void *opaque); +int qemu_rdma_info_load(QEMUFile *f, void *opaque, int version_id); #endif diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h index d67e97a..b547de9 100644 --- a/include/migration/qemu-file.h +++ b/include/migration/qemu-file.h @@ -71,6 +71,7 @@ typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags); #define RAM_CONTROL_ROUND 1 #define RAM_CONTROL_HOOK 2 #define RAM_CONTROL_FINISH 3 +#define RAM_CONTROL_FLUSH 4 /* * This function allows override of where the RAM page diff --git a/migration.c b/migration.c index 2b1ab20..62dded3 100644 --- a/migration.c +++ b/migration.c @@ -36,15 +36,6 @@ do { } while (0) #endif -enum { - MIG_STATE_ERROR = -1, - MIG_STATE_NONE, - MIG_STATE_SETUP, - MIG_STATE_CANCELLED, - MIG_STATE_ACTIVE, - MIG_STATE_COMPLETED, -}; - #define MAX_THROTTLE (32 << 20) /* Migration speed throttling */ /* Amount of time to allocate to each "chunk" of bandwidth-throttled @@ -270,7 +261,7 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, MigrationState *s = migrate_get_current(); MigrationCapabilityStatusList *cap; - if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP) { + if (migration_is_active(s)) { error_set(errp, QERR_MIGRATION_ACTIVE); return; } @@ -282,6 +273,17 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, /* shared migration helpers */ +bool migration_is_mc(MigrationState *s) +{ + return s->state == MIG_STATE_MC; +} + +bool migration_is_active(MigrationState *s) +{ + return (s->state == MIG_STATE_ACTIVE) || migration_in_setup(s) + || migration_is_mc(s); +} + static void migrate_fd_cleanup(void *opaque) { MigrationState *s = opaque; @@ -299,7 +301,7 @@ static void migrate_fd_cleanup(void *opaque) s->file = NULL; } - assert(s->state != MIG_STATE_ACTIVE); + assert(!migration_is_active(s)); if (s->state != MIG_STATE_COMPLETED) { qemu_savevm_state_cancel(); @@ -308,7 +310,7 @@ static void migrate_fd_cleanup(void *opaque) notifier_list_notify(&migration_state_notifiers, s); } -static void migrate_set_state(MigrationState *s, int old_state, int new_state) +void migrate_set_state(MigrationState *s, int old_state, int new_state) { if (atomic_cmpxchg(&s->state, old_state, new_state) == new_state) { trace_migrate_set_state(new_state); @@ -405,7 +407,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, params.blk = has_blk && blk; params.shared = has_inc && inc; - if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP) { + if (migration_is_active(s)) { error_set(errp, QERR_MIGRATION_ACTIVE); return; } @@ -594,7 +596,10 @@ static void *migration_thread(void *opaque) } if (!qemu_file_get_error(s->file)) { - migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COMPLETED); + if (!migrate_use_mc()) { + migrate_set_state(s, + MIG_STATE_ACTIVE, MIG_STATE_COMPLETED); + } break; } }