From patchwork Tue May 23 11:31:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Perevalov X-Patchwork-Id: 765918 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wXDfF0jXvz9sPG for ; Tue, 23 May 2017 22:01:57 +1000 (AEST) Received: from localhost ([::1]:48289 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dD8Ve-0005PY-PT for incoming@patchwork.ozlabs.org; Tue, 23 May 2017 08:01:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48782) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dD82L-0006lr-C1 for qemu-devel@nongnu.org; Tue, 23 May 2017 07:31:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dD82H-0003fa-Ep for qemu-devel@nongnu.org; Tue, 23 May 2017 07:31:37 -0400 Received: from mailout1.w1.samsung.com ([210.118.77.11]:24896) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dD82H-0003ez-7D for qemu-devel@nongnu.org; Tue, 23 May 2017 07:31:33 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OQE0016WLCJ5K70@mailout1.w1.samsung.com> for qemu-devel@nongnu.org; Tue, 23 May 2017 12:31:31 +0100 (BST) Received: from eusmges3.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170523113131eucas1p1163909297f4e469c03fa696046a6cd76~BOOFAsSTg1972819728eucas1p1T; Tue, 23 May 2017 11:31:31 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges3.samsung.com (EUCPMTA) with SMTP id CF.32.17464.39D14295; Tue, 23 May 2017 12:31:31 +0100 (BST) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170523113130eucas1p1babac9d8659c10abe22ddc7d5b9526ab~BOOESWxGU1797217972eucas1p1k; Tue, 23 May 2017 11:31:30 +0000 (GMT) X-AuditID: cbfec7f2-f797e6d000004438-3f-59241d934f3a Received: from eusync3.samsung.com ( [203.254.199.213]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 64.D1.20206.29D14295; Tue, 23 May 2017 12:31:30 +0100 (BST) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync3.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OQE003QBLC1VWB0@eusync3.samsung.com>; Tue, 23 May 2017 12:31:30 +0100 (BST) From: Alexey Perevalov To: qemu-devel@nongnu.org Date: Tue, 23 May 2017 14:31:08 +0300 Message-id: <1495539071-12995-8-git-send-email-a.perevalov@samsung.com> X-Mailer: git-send-email 1.9.1 In-reply-to: <1495539071-12995-1-git-send-email-a.perevalov@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrNIsWRmVeSWpSXmKPExsWy7djPc7qTZVUiDZ5dZ7WYe/c8i0Xvtnvs FhPfrme1uNL+k91iy/5v7BbHe3ewOLB5PLm2mcnj/b6rbB59W1YxBjBHcdmkpOZklqUW6dsl cGV8/d/EXNBrUPH7ZAdzA+M35S5GTg4JAROJeUe/s0PYYhIX7q1n62Lk4hASWMoosfL/fHYI 5zOjRP/Dl4wwHR8a50JVLWOUmLXjMyuE080kcWhvM5DDwcEmYCCx754tSIOIgKTE767TzCA1 zALNjBKHHv9kBkkIC9hLNL78zQpiswioSsy+vY0FpJdXwF3ixQlXiGVyEiePTQYr4RTwkHh3 +B4LyBwJgftsEv2tO9hA6iUEZCU2HWCGqHeRON44iQ3CFpZ4dXwL1GsyEpcnd0P1tjNKdO/s ZIVwJjBKnJn+F6rKXuLUzatMIDazAJ/EpG3TmSEW8Ep0tAlBmB4Si9YlQFQ7Suzt3cAE8fts Rom+y9NZJzDKLGBkWMUoklpanJueWmysV5yYW1yal66XnJ+7iREYpaf/Hf+0g/HrCatDjAIc jEo8vBqPlSKFWBPLiitzDzFKcDArifD6CKlECvGmJFZWpRblxxeV5qQWH2KU5mBREuflOnUt QkggPbEkNTs1tSC1CCbLxMEp1cCoNathBavO8YKYiJbwuFOqO/jjt55hz/V+dktJ1fTvqZLF PBKmp6vsMt4/i2DgTHL8wB0n8mRqlZAG06dERdEDEzZsXSx35Y3gpg6P9bdefO/TjNd16fw7 VaJk4ivb1+35f64fjQ7b52S28J7Fx5dvV634vN7f7cTmrW7RSdxvT2c85jSIeH9eiaU4I9FQ i7moOBEAN9BqEc4CAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrFLMWRmVeSWpSXmKPExsVy+t/xq7qTZFUiDfbdEbGYe/c8i0Xvtnvs FhPfrme1uNL+k91iy/5v7BbHe3ewOLB5PLm2mcnj/b6rbB59W1YxBjBHudlkpCampBYppOYl 56dk5qXbKoWGuOlaKCnkJeam2ipF6PqGBCkplCXmlAJ5RgZowME5wD1YSd8uwS3j6/8m5oJe g4rfJzuYGxi/KXcxcnJICJhIfGicywZhi0lcuLceyObiEBJYwiix79wyFginl0ni/YPJTF2M HBxsAgYS++7ZgjSICEhK/O46zQxSwyzQzCjRuroNbJKwgL1E48vfrCA2i4CqxOzb21hAenkF 3CVenHCFWCYncfLYZLASTgEPiXeH77GA2EJAJacnXmedwMi7gJFhFaNIamlxbnpusZFecWJu cWleul5yfu4mRmC4bjv2c8sOxq53wYcYBTgYlXh4NR4rRQqxJpYVV+YeYpTgYFYS4fURUokU 4k1JrKxKLcqPLyrNSS0+xGgKdNNEZinR5HxgLOWVxBuaGJpbGhoZW1iYGxkpifNO/XAlXEgg PbEkNTs1tSC1CKaPiYNTqoHRwEf5fJimUejnaNU2sWbN3+scQ+fJh80Q1te+4SDyaoFOzZM/ LCW3Pl+tdphcePL1+oUxe89OOuBxdk/j0cdpeytDVouc380bf0OQw+hF8GT9QtnjD3n2+x6p O5V0cfLuhLmWHycbuf+axrL0hbF86+eEhkXWG5fv83+ZeivdPvUkk+Ap98PxSizFGYmGWsxF xYkAKmi+Jm0CAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170523113130eucas1p1babac9d8659c10abe22ddc7d5b9526ab X-Msg-Generator: CA X-Sender-IP: 182.198.249.180 X-Local-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG+yCvOyEseyghOyekBtTZW5pb3IgRW5naW5lZXI=?= X-Global-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG1NhbXN1bmcgRWxlY3Ryb25pY3MbU2VuaW9yIEVuZ2luZWVy?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170523113130eucas1p1babac9d8659c10abe22ddc7d5b9526ab X-RootMTR: 20170523113130eucas1p1babac9d8659c10abe22ddc7d5b9526ab References: <1495539071-12995-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 210.118.77.11 Subject: [Qemu-devel] [PATCH V6 07/10] migration: add bitmap for copied page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: i.maximets@samsung.com, dgilbert@redhat.com, peterx@redhat.com, Alexey Perevalov Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds ability to track down already copied pages, it's necessary for calculation vCPU block time in postcopy migration feature and maybe for restore after postcopy migration failure. Functions which work with RAMBlock are placed into ram.c, due to TARGET_WORDS_BIGENDIAN is poisoned int postcopy-ram.c - hardware independed code. Signed-off-by: Alexey Perevalov --- include/migration/migration.h | 16 +++++++++++ migration/postcopy-ram.c | 22 ++++++++++++--- migration/ram.c | 63 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+), 4 deletions(-) diff --git a/include/migration/migration.h b/include/migration/migration.h index 449cb07..4e05c83 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -101,6 +101,20 @@ struct MigrationIncomingState { LoadStateEntry_Head loadvm_handlers; /* + * bitmap indicates whether page copied, + * based on ramblock offset + * now it is using only for blocktime calculation in + * postcopy migration, so livetime of this entry: + * since user requested blocktime calculation, + * till the end of postcopy migration + * as an example it could represend following memory map + * ___________________________________ + * |4k pages | hugepages | 4k pages + * + * */ + unsigned long *copied_pages; + + /* * PostcopyBlocktimeContext to keep information for postcopy * live migration, to calculate vCPU block time * */ @@ -279,6 +293,8 @@ int migrate_compress_threads(void); int migrate_decompress_threads(void); bool migrate_use_events(void); bool migrate_postcopy_blocktime(void); +unsigned long int get_copied_bit_offset(ram_addr_t addr); +unsigned long int *copied_pages_bitmap_new(void); /* Sending on the return path - generic and then for each message type */ void migrate_send_rp_message(MigrationIncomingState *mis, diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 5435a40..d647769 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -94,23 +94,34 @@ static void destroy_blocktime_context(struct PostcopyBlocktimeContext *ctx) static void postcopy_migration_cb(Notifier *n, void *data) { - PostcopyBlocktimeContext *ctx = container_of(n, PostcopyBlocktimeContext, - postcopy_notifier); MigrationState *s = data; if (migration_has_finished(s) || migration_has_failed(s)) { + MigrationIncomingState *mis = migration_incoming_get_current(); + PostcopyBlocktimeContext *ctx = mis->blocktime_ctx; + + if (!ctx) { + return; + } + g_free(ctx->page_fault_vcpu_time); /* g_free is NULL robust */ ctx->page_fault_vcpu_time = NULL; g_free(ctx->vcpu_addr); ctx->vcpu_addr = NULL; + g_free(mis->copied_pages); + mis->copied_pages = NULL; } } static void migration_exit_cb(Notifier *n, void *data) { - PostcopyBlocktimeContext *ctx = container_of(n, PostcopyBlocktimeContext, - exit_notifier); + MigrationIncomingState *mis = migration_incoming_get_current(); + PostcopyBlocktimeContext *ctx = mis->blocktime_ctx; + if (!ctx) { + return; + } destroy_blocktime_context(ctx); + mis->blocktime_ctx = NULL; } static struct PostcopyBlocktimeContext *blocktime_context_new(void) @@ -227,6 +238,9 @@ static bool ufd_check_and_apply(int ufd, MigrationIncomingState *mis) mis->blocktime_ctx = blocktime_context_new(); } + if (!mis->copied_pages) { + mis->copied_pages = copied_pages_bitmap_new(); + } asked_features |= UFFD_FEATURE_THREAD_ID; } #endif diff --git a/migration/ram.c b/migration/ram.c index f59fdd4..1abb6bb 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2661,6 +2661,69 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) return ret; } +static unsigned long get_total_bits_per_page(ram_addr_t mem_length, + size_t page_size) +{ + unsigned long page_size_bit = find_last_bit((unsigned long *)&page_size, + BITS_PER_LONG); + unsigned long total_bits = mem_length >> page_size_bit; + if (mem_length % page_size) { + total_bits += 1; + } + return total_bits; +} + +/* + * this function allocates bitmap for copied pages, + * also it calculates + * how many entries do we need + * */ +unsigned long int *copied_pages_bitmap_new(void) +{ + RAMBlock *block; + unsigned long int total_bits = 0; + + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + /* in general case used_length may not be aligned + * by page_size */ + + total_bits += get_total_bits_per_page(block->used_length, + block->page_size); + } + rcu_read_unlock(); + + return bitmap_new(total_bits); +} + +unsigned long int get_copied_bit_offset(ram_addr_t addr) +{ + RAMBlock *block; + unsigned long int iter_bit = 0; + + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + /* in general case used_length may not be aligned + * by page_size */ + if (block->host == NULL) { + continue; + } + if (addr - (ram_addr_t)block->host < block->max_length) { + unsigned long page_size_bit = find_last_bit( + (unsigned long *)&block->page_size, + BITS_PER_LONG); + ram_addr_t offset = addr - (ram_addr_t)block->host; + iter_bit += offset >> page_size_bit; + break; + } + iter_bit += get_total_bits_per_page(block->used_length, + block->page_size); + } + rcu_read_unlock(); + + return iter_bit; +} + static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,