From patchwork Thu Mar 3 10:44:28 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Liang Z" X-Patchwork-Id: 591386 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3E855140C3C for ; Thu, 3 Mar 2016 21:53:34 +1100 (AEDT) Received: from localhost ([::1]:33817 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abQsu-0004df-D4 for incoming@patchwork.ozlabs.org; Thu, 03 Mar 2016 05:53:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34441) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abQqO-0008KW-0d for qemu-devel@nongnu.org; Thu, 03 Mar 2016 05:50:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1abQqK-0006Ts-MV for qemu-devel@nongnu.org; Thu, 03 Mar 2016 05:50:55 -0500 Received: from mga02.intel.com ([134.134.136.20]:36812) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abQqK-0006Tb-Bx for qemu-devel@nongnu.org; Thu, 03 Mar 2016 05:50:52 -0500 Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP; 03 Mar 2016 02:50:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,532,1449561600"; d="scan'208";a="663095279" Received: from ll.sh.intel.com (HELO localhost) ([10.239.13.27]) by FMSMGA003.fm.intel.com with ESMTP; 03 Mar 2016 02:50:47 -0800 From: Liang Li To: quintela@redhat.com, amit.shah@redhat.com, qemu-devel@nongnu.org, linux-kernel@vger.kernel.org Date: Thu, 3 Mar 2016 18:44:28 +0800 Message-Id: <1457001868-15949-5-git-send-email-liang.z.li@intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1457001868-15949-1-git-send-email-liang.z.li@intel.com> References: <1457001868-15949-1-git-send-email-liang.z.li@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 134.134.136.20 Cc: ehabkost@redhat.com, kvm@vger.kernel.org, mst@redhat.com, Liang Li , dgilbert@redhat.com, virtualization@lists.linux-foundation.org, linux-mm@kvack.org, pbonzini@redhat.com, akpm@linux-foundation.org, rth@twiddle.net Subject: [Qemu-devel] [RFC qemu 4/4] migration: filter out guest's free pages in ram bulk stage X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Get the free pages information through virtio and filter out the free pages in the ram bulk stage. This can significantly reduce the total live migration time as well as network traffic. Signed-off-by: Liang Li --- migration/ram.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 6 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index ee2547d..819553b 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -40,6 +40,7 @@ #include "trace.h" #include "exec/ram_addr.h" #include "qemu/rcu_queue.h" +#include "sysemu/balloon.h" #ifdef DEBUG_MIGRATION_RAM #define DPRINTF(fmt, ...) \ @@ -241,6 +242,7 @@ static struct BitmapRcu { struct rcu_head rcu; /* Main migration bitmap */ unsigned long *bmap; + unsigned long *free_pages_bmap; /* bitmap of pages that haven't been sent even once * only maintained and used in postcopy at the moment * where it's used to send the dirtymap at the start @@ -561,12 +563,7 @@ ram_addr_t migration_bitmap_find_dirty(RAMBlock *rb, unsigned long next; bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap; - if (ram_bulk_stage && nr > base) { - next = nr + 1; - } else { - next = find_next_bit(bitmap, size, nr); - } - + next = find_next_bit(bitmap, size, nr); *ram_addr_abs = next << TARGET_PAGE_BITS; return (next - base) << TARGET_PAGE_BITS; } @@ -1415,6 +1412,9 @@ void free_xbzrle_decoded_buf(void) static void migration_bitmap_free(struct BitmapRcu *bmap) { g_free(bmap->bmap); + if (balloon_free_pages_support()) { + g_free(bmap->free_pages_bmap); + } g_free(bmap->unsentmap); g_free(bmap); } @@ -1873,6 +1873,28 @@ err: return ret; } +static void filter_out_guest_free_pages(unsigned long *free_pages_bmap) +{ + RAMBlock *block; + DirtyMemoryBlocks *blocks; + unsigned long end, page; + + blocks = atomic_rcu_read(&ram_list.dirty_memory[DIRTY_MEMORY_MIGRATION]); + block = QLIST_FIRST_RCU(&ram_list.blocks); + end = TARGET_PAGE_ALIGN(block->offset + + block->used_length) >> TARGET_PAGE_BITS; + page = block->offset >> TARGET_PAGE_BITS; + + while (page < end) { + unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE; + unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE; + unsigned long num = MIN(end - page, DIRTY_MEMORY_BLOCK_SIZE - offset); + unsigned long *p = free_pages_bmap + BIT_WORD(page); + + slow_bitmap_complement(blocks->blocks[idx], p, num); + page += num; + } +} /* Each of ram_save_setup, ram_save_iterate and ram_save_complete has * long-running RCU critical section. When rcu-reclaims in the code @@ -1884,6 +1906,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) { RAMBlock *block; int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */ + uint64_t free_pages_count = 0; dirty_rate_high_cnt = 0; bitmap_sync_count = 0; @@ -1931,6 +1954,9 @@ static int ram_save_setup(QEMUFile *f, void *opaque) ram_bitmap_pages = last_ram_offset() >> TARGET_PAGE_BITS; migration_bitmap_rcu = g_new0(struct BitmapRcu, 1); migration_bitmap_rcu->bmap = bitmap_new(ram_bitmap_pages); + if (balloon_free_pages_support()) { + migration_bitmap_rcu->free_pages_bmap = bitmap_new(ram_bitmap_pages); + } if (migrate_postcopy_ram()) { migration_bitmap_rcu->unsentmap = bitmap_new(ram_bitmap_pages); @@ -1945,6 +1971,20 @@ static int ram_save_setup(QEMUFile *f, void *opaque) DIRTY_MEMORY_MIGRATION); } memory_global_dirty_log_start(); + + if (balloon_free_pages_support() && + balloon_get_free_pages(migration_bitmap_rcu->free_pages_bmap, + &free_pages_count) == 0) { + qemu_mutex_unlock_iothread(); + while (balloon_get_free_pages(migration_bitmap_rcu->free_pages_bmap, + &free_pages_count) == 0) { + usleep(1000); + } + qemu_mutex_lock_iothread(); + + filter_out_guest_free_pages(migration_bitmap_rcu->free_pages_bmap); + } + migration_bitmap_sync(); qemu_mutex_unlock_ramlist(); qemu_mutex_unlock_iothread();