From patchwork Wed Apr 10 04:29:39 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrhines@linux.vnet.ibm.com X-Patchwork-Id: 235295 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id E1CF72C00BB for ; Wed, 10 Apr 2013 14:44:35 +1000 (EST) Received: from localhost ([::1]:53251 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmgi-00019f-SK for incoming@patchwork.ozlabs.org; Wed, 10 Apr 2013 00:31:12 -0400 Received: from eggs.gnu.org ([208.118.235.92]:51770) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmfv-0000z8-EE for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UPmfs-0001H9-Ik for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:23 -0400 Received: from e39.co.us.ibm.com ([32.97.110.160]:47466) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmfs-0001H2-9Z for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:20 -0400 Received: from /spool/local by e39.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Apr 2013 22:30:19 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e39.co.us.ibm.com (192.168.1.139) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 9 Apr 2013 22:30:18 -0600 Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 96F503E40045 for ; Tue, 9 Apr 2013 22:30:04 -0600 (MDT) Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r3A4UEsE119466 for ; Tue, 9 Apr 2013 22:30:15 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r3A4UEYU022715 for ; Tue, 9 Apr 2013 22:30:14 -0600 Received: from mrhinesdev.klabtestbed.com (klinux.watson.ibm.com [9.2.208.21]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r3A4U3RM021687; Tue, 9 Apr 2013 22:30:12 -0600 From: mrhines@linux.vnet.ibm.com To: qemu-devel@nongnu.org Date: Wed, 10 Apr 2013 00:29:39 -0400 Message-Id: <1365568180-19593-7-git-send-email-mrhines@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1365568180-19593-1-git-send-email-mrhines@linux.vnet.ibm.com> References: <1365568180-19593-1-git-send-email-mrhines@linux.vnet.ibm.com> X-TM-AS-MML: No X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13041004-3620-0000-0000-000001F808DA X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 32.97.110.160 Cc: aliguori@us.ibm.com, mst@redhat.com, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com Subject: [Qemu-devel] [RFC PATCH RDMA support v6: 6/7] send pc.ram over RDMA X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Michael R. Hines" All that is left for this part of the patch is: 1. use the new (optionally defined) save_ram_page function pointer to decide what to do with the page if RDMA is enable or not and return ENOTSUP as agreed. 2. invoke hooks from QEMURamControlOps function pointers to hook into the RDMA protocol at the right points in order to perform dynamic page registration. Signed-off-by: Michael R. Hines --- arch_init.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/arch_init.c b/arch_init.c index 769ce77..a7d5b16 100644 --- a/arch_init.c +++ b/arch_init.c @@ -115,6 +115,7 @@ const uint32_t arch_type = QEMU_ARCH; #define RAM_SAVE_FLAG_EOS 0x10 #define RAM_SAVE_FLAG_CONTINUE 0x20 #define RAM_SAVE_FLAG_XBZRLE 0x40 +#define RAM_SAVE_FLAG_REGISTER 0x80 /* perform hook during iteration */ static struct defconfig_file { @@ -170,6 +171,13 @@ static struct { .cache = NULL, }; +#ifdef CONFIG_RDMA +void qemu_ram_registration_start(QEMUFile *f, void *opaque, int section) +{ + DPRINTF("start section: %d\n", section); + qemu_put_be64(f, RAM_SAVE_FLAG_REGISTER); +} +#endif int64_t xbzrle_cache_resize(int64_t new_size) { @@ -447,15 +455,22 @@ static int ram_save_block(QEMUFile *f, bool last_stage) ram_bulk_stage = false; } } else { + bool zero; uint8_t *p; int cont = (block == last_sent_block) ? RAM_SAVE_FLAG_CONTINUE : 0; p = memory_region_get_ram_ptr(mr) + offset; + /* use capability now, defaults to true */ + zero = migrate_check_for_zero() ? is_zero_page(p) : false; + /* In doubt sent page as normal */ bytes_sent = -1; - if (is_zero_page(p)) { + if ((bytes_sent = ram_control_save_page(f, block->offset, + offset, cont, TARGET_PAGE_SIZE, zero)) >= 0) { + acct_info.norm_pages++; + } else if (zero) { acct_info.dup_pages++; if (!ram_bulk_stage) { bytes_sent = save_block_hdr(f, block, offset, cont, @@ -476,7 +491,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage) } /* XBZRLE overflow or normal page */ - if (bytes_sent == -1) { + if (bytes_sent == -1 || bytes_sent == -ENOTSUP) { bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE); qemu_put_buffer_async(f, p, TARGET_PAGE_SIZE); bytes_sent += TARGET_PAGE_SIZE; @@ -598,6 +613,18 @@ static int ram_save_setup(QEMUFile *f, void *opaque) } qemu_mutex_unlock_ramlist(); + + /* + * These following calls generate reserved messages for future expansion of the RDMA + * protocol. If the ops are not defined, nothing will happen. + * + * Please leave in place. They are intended to be used to pre-register + * memory in the future to mitigate the extremely high cost of dynamic page + * registration. + */ + ram_control_before_iterate(f, RAM_CONTROL_SETUP); + ram_control_after_iterate(f, RAM_CONTROL_SETUP); + qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; @@ -616,6 +643,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) reset_ram_globals(); } + ram_control_before_iterate(f, RAM_CONTROL_ROUND); + t0 = qemu_get_clock_ns(rt_clock); i = 0; while ((ret = qemu_file_rate_limit(f)) == 0) { @@ -646,6 +675,12 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) qemu_mutex_unlock_ramlist(); + /* + * must occur before EOS (or any QEMUFile operation) + * because of RDMA protocol + */ + ram_control_after_iterate(f, RAM_CONTROL_ROUND); + if (ret < 0) { bytes_transferred += total_sent; return ret; @@ -663,6 +698,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) qemu_mutex_lock_ramlist(); migration_bitmap_sync(); + ram_control_before_iterate(f, RAM_CONTROL_FINISH); + /* try transferring iterative blocks of memory */ /* flush all remaining blocks regardless of rate limiting */ @@ -676,6 +713,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } bytes_transferred += bytes_sent; } + + ram_control_after_iterate(f, RAM_CONTROL_FINISH); migration_end(); qemu_mutex_unlock_ramlist(); @@ -864,6 +903,8 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret = -EINVAL; goto done; } + } else if (flags & RAM_SAVE_FLAG_REGISTER) { + ram_control_register_iterate(f, RAM_CONTROL_REGISTER); } error = qemu_file_get_error(f); if (error) {