From patchwork Tue Oct 30 08:33:11 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Isaku Yamahata X-Patchwork-Id: 195420 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 320412C00B0 for ; Tue, 30 Oct 2012 21:37:31 +1100 (EST) Received: from localhost ([::1]:48523 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TT7Hr-0006Wq-1V for incoming@patchwork.ozlabs.org; Tue, 30 Oct 2012 04:35:03 -0400 Received: from eggs.gnu.org ([208.118.235.92]:60911) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TT7GQ-0004tO-HP for qemu-devel@nongnu.org; Tue, 30 Oct 2012 04:33:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TT7GE-0000bK-1I for qemu-devel@nongnu.org; Tue, 30 Oct 2012 04:33:34 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:44752) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TT7GD-0000ZW-CN for qemu-devel@nongnu.org; Tue, 30 Oct 2012 04:33:21 -0400 Received: from ps.local.valinux.co.jp (vagw.valinux.co.jp [210.128.90.14]) by mail.valinux.co.jp (Postfix) with SMTP id 143EE181D5; Tue, 30 Oct 2012 17:33:15 +0900 (JST) Received: (nullmailer pid 29501 invoked by uid 1000); Tue, 30 Oct 2012 08:33:14 -0000 From: Isaku Yamahata To: qemu-devel@nongnu.org, kvm@vger.kernel.org Date: Tue, 30 Oct 2012 17:33:11 +0900 Message-Id: X-Mailer: git-send-email 1.7.10.4 In-Reply-To: References: In-Reply-To: References: X-Virus-Scanned: clamav-milter 0.95.2 at va-mail.local.valinux.co.jp X-Virus-Status: Clean X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 210.128.90.3 Cc: benoit.hudzia@gmail.com, aarcange@redhat.com, aliguori@us.ibm.com, quintela@redhat.com, stefanha@gmail.com, t.hirofuchi@aist.go.jp, dlaor@redhat.com, satoshi.itoh@aist.go.jp, mdroth@linux.vnet.ibm.com, yoshikawa.takuya@oss.ntt.co.jp, owasserm@redhat.com, avi@redhat.com, pbonzini@redhat.com, chegu_vinod@hp.com Subject: [Qemu-devel] [PATCH v3 35/35] postcopy: pre+post optimization outgoing side X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Signed-off-by: Isaku Yamahata --- arch_init.c | 6 ++-- migration-postcopy.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++--- migration.h | 1 + 3 files changed, 94 insertions(+), 7 deletions(-) diff --git a/arch_init.c b/arch_init.c index 49fbaff..f9bd483 100644 --- a/arch_init.c +++ b/arch_init.c @@ -502,8 +502,10 @@ bool ram_save_block(QEMUFile *f, bool last_stage) if (offset >= block->length) { offset = 0; block = QLIST_NEXT(block, next); - if (!block) + if (!block) { block = QLIST_FIRST(&ram_list.blocks); + migrate_get_current()->precopy_count++; + } } } while (block != last_block || offset != last_offset); @@ -619,7 +621,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) acct_clear(); } - if (!params->postcopy) { + if (!(params->postcopy && params->precopy_count == 0)) { memory_global_dirty_log_start(); migration_bitmap_sync(); } diff --git a/migration-postcopy.c b/migration-postcopy.c index 8a43c42..3f63385 100644 --- a/migration-postcopy.c +++ b/migration-postcopy.c @@ -322,6 +322,10 @@ int postcopy_outgoing_create_read_socket(MigrationState *s) void postcopy_outgoing_state_begin(QEMUFile *f, const MigrationParams *params) { uint64_t options = 0; + if (params->precopy_count > 0) { + options |= POSTCOPY_OPTION_PRECOPY; + } + qemu_put_ubyte(f, QEMU_VM_POSTCOPY_INIT); qemu_put_be32(f, sizeof(options)); qemu_put_be64(f, options); @@ -337,12 +341,36 @@ void postcopy_outgoing_state_complete( int postcopy_outgoing_ram_save_iterate(QEMUFile *f, void *opaque) { - qemu_put_be64(f, RAM_SAVE_FLAG_EOS); - return 1; + int ret; + MigrationState *s = migrate_get_current(); + if (s->params.precopy_count == 0) { + qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + return 1; + } + + ret = ram_save_iterate(f); + if (ret < 0) { + return ret; + } + if (ret == 1) { + DPRINTF("precopy worked\n"); + return ret; + } + if (ram_bytes_remaining() == 0) { + DPRINTF("no more precopy\n"); + return 1; + } + return s->precopy_count >= s->params.precopy_count? 1: 0; } int postcopy_outgoing_ram_save_complete(QEMUFile *f, void *opaque) { + MigrationState *s = migrate_get_current(); + if (s->params.precopy_count > 0) { + /* Make sure all dirty bits are set */ + migration_bitmap_sync(); + memory_global_dirty_log_stop(); + } qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; } @@ -544,6 +572,7 @@ static void postcopy_outgoing_recv_handler(void *opaque) PostcopyOutgoingState *postcopy_outgoing_begin(MigrationState *ms) { PostcopyOutgoingState *s = g_new(PostcopyOutgoingState, 1); + const RAMBlock *block; DPRINTF("outgoing begin\n"); qemu_buffered_file_drain(ms->file); @@ -553,9 +582,64 @@ PostcopyOutgoingState *postcopy_outgoing_begin(MigrationState *ms) s->mig_read = ms->file_read; s->mig_buffered_write = ms->file; - /* Make sure all dirty bits are set */ - memory_global_dirty_log_stop(); - migration_bitmap_init(); + if (ms->params.precopy_count > 0) { + QEMUFile *f = ms->file; + uint64_t last_long = + BITS_TO_LONGS(last_ram_offset() >> TARGET_PAGE_BITS); + + /* send dirty bitmap */ + qemu_mutex_lock_ramlist(); + QLIST_FOREACH(block, &ram_list.blocks, next) { + const unsigned long *bitmap = migration_bitmap_get(); + uint64_t length; + uint64_t start; + uint64_t end; + uint64_t i; + + qemu_put_byte(f, strlen(block->idstr)); + qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr)); + qemu_put_be64(f, block->offset); + qemu_put_be64(f, block->length); + + start = (block->offset >> TARGET_PAGE_BITS); + end = (block->offset + block->length) >> TARGET_PAGE_BITS; + + length = BITS_TO_LONGS(end - (start & ~63)) * sizeof(unsigned long); + length = DIV_ROUND_UP(length, sizeof(uint64_t)) * sizeof(uint64_t); + qemu_put_be64(f, length); + DPRINTF("dirty bitmap %s 0x%"PRIx64" 0x%"PRIx64" 0x%"PRIx64"\n", + block->idstr, block->offset, block->length, length); + + start /= BITS_PER_LONG; + end = DIV_ROUND_UP(end, BITS_PER_LONG); + assert(end <= last_long); + + for (i = start; i < end; + i += sizeof(uint64_t) / sizeof(unsigned long)) { + uint64_t val; +#if HOST_LONG_BITS == 64 + val = bitmap[i]; +#elif HOST_LONG_BITS == 32 + if (i + 1 < last_long) { + val = bitmap[i] | ((uint64_t)bitmap[i + 1] << 32); + } else { + val = bitmap[i]; + } +#else +# error "unsupported" +#endif + qemu_put_be64(f, val); + } + } + qemu_mutex_unlock_ramlist(); + + /* terminator */ + qemu_put_byte(f, 0); /* idstr len */ + qemu_put_be64(f, 0); /* block offset */ + qemu_put_be64(f, 0); /* block length */ + qemu_put_be64(f, 0); /* bitmap len */ + DPRINTF("sent dirty bitmap\n"); + } qemu_set_fd_handler(s->fd_read, &postcopy_outgoing_recv_handler, NULL, s); diff --git a/migration.h b/migration.h index c4d7b0a..9bd4062 100644 --- a/migration.h +++ b/migration.h @@ -58,6 +58,7 @@ struct MigrationState int fd_read; QEMUFile *file_read; /* connection from the detination */ PostcopyOutgoingState *postcopy; + int precopy_count; }; void process_incoming_migration(QEMUFile *f);