From patchwork Mon Sep 25 14:55:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818275 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16cN0bWhz9tXC for ; Tue, 26 Sep 2017 00:56:52 +1000 (AEST) Received: from localhost ([::1]:42830 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoU-0005ks-46 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 10:56:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53964) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUnd-0005hm-N8 for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:55:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUnc-00039h-Tw for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:55:57 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52382) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUnV-00035S-S1; Mon, 25 Sep 2017 10:55:49 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C73D781E07; Mon, 25 Sep 2017 14:55:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com C73D781E07 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8FCA778384; Mon, 25 Sep 2017 14:55:46 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:07 -0500 Message-Id: <20170925145526.32690-2-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 25 Sep 2017 14:55:48 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 01/20] block: Make bdrv_img_create() size selection easier to read X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" All callers of bdrv_img_create() pass in a size, or -1 to read the size from the backing file. We then set that size as the QemuOpt default, which means we will reuse that default rather than the final parameter to qemu_opt_get_size() several lines later. But it is rather confusing to read subsequent checks of 'size == -1' when it looks (without seeing the full context) like size defaults to 0; it also doesn't help that a size of 0 is valid (for some formats). Rework the logic to make things more legible. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v6: Combine into a series rather than being a standalone patch (more for ease of tracking than for being on topic) --- block.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block.c b/block.c index 5c65fac672..528cda7b2c 100644 --- a/block.c +++ b/block.c @@ -4488,7 +4488,7 @@ void bdrv_img_create(const char *filename, const char *fmt, /* The size for the image must always be specified, unless we have a backing * file and we have not been forbidden from opening it. */ - size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, 0); + size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, img_size); if (backing_file && !(flags & BDRV_O_NO_BACKING)) { BlockDriverState *bs; char *full_backing = g_new0(char, PATH_MAX); From patchwork Mon Sep 25 14:55:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818280 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16gJ4phMz9t67 for ; Tue, 26 Sep 2017 00:59:24 +1000 (AEST) Received: from localhost ([::1]:42840 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUqw-00080A-Mi for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 10:59:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54075) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUns-0005s8-Hm for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUnm-0003Ex-Jm for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:12 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34010) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUnd-00039w-Pb; Mon, 25 Sep 2017 10:55:57 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9ED8052748; Mon, 25 Sep 2017 14:55:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 9ED8052748 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 19A9D6F13E; Mon, 25 Sep 2017 14:55:48 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:08 -0500 Message-Id: <20170925145526.32690-3-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 25 Sep 2017 14:55:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 02/20] hbitmap: Rename serialization_granularity to serialization_align X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The only client of hbitmap_serialization_granularity() is dirty-bitmap's bdrv_dirty_bitmap_serialization_align(). Keeping the two names consistent is worthwhile, and the shorter name is more representative of what the function returns (the required alignment to be used for start/count of other serialization functions, where violating the alignment causes assertion failures). Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: new patch --- include/qemu/hbitmap.h | 8 ++++---- block/dirty-bitmap.c | 2 +- tests/test-hbitmap.c | 10 +++++----- util/hbitmap.c | 8 ++++---- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h index d3a74a21fc..81e78043d1 100644 --- a/include/qemu/hbitmap.h +++ b/include/qemu/hbitmap.h @@ -159,16 +159,16 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item); bool hbitmap_is_serializable(const HBitmap *hb); /** - * hbitmap_serialization_granularity: + * hbitmap_serialization_align: * @hb: HBitmap to operate on. * - * Granularity of serialization chunks, used by other serialization functions. - * For every chunk: + * Required alignment of serialization chunks, used by other serialization + * functions. For every chunk: * 1. Chunk start should be aligned to this granularity. * 2. Chunk size should be aligned too, except for last chunk (for which * start + count == hb->size) */ -uint64_t hbitmap_serialization_granularity(const HBitmap *hb); +uint64_t hbitmap_serialization_align(const HBitmap *hb); /** * hbitmap_serialization_size: diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 30462d4f9a..0490ca3aff 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -617,7 +617,7 @@ uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap, uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap) { - return hbitmap_serialization_granularity(bitmap->bitmap); + return hbitmap_serialization_align(bitmap->bitmap); } void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap, diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c index 1acb353889..af41642346 100644 --- a/tests/test-hbitmap.c +++ b/tests/test-hbitmap.c @@ -738,15 +738,15 @@ static void test_hbitmap_meta_one(TestHBitmapData *data, const void *unused) } } -static void test_hbitmap_serialize_granularity(TestHBitmapData *data, - const void *unused) +static void test_hbitmap_serialize_align(TestHBitmapData *data, + const void *unused) { int r; hbitmap_test_init(data, L3 * 2, 3); g_assert(hbitmap_is_serializable(data->hb)); - r = hbitmap_serialization_granularity(data->hb); + r = hbitmap_serialization_align(data->hb); g_assert_cmpint(r, ==, 64 << 3); } @@ -974,8 +974,8 @@ int main(int argc, char **argv) hbitmap_test_add("/hbitmap/meta/word", test_hbitmap_meta_word); hbitmap_test_add("/hbitmap/meta/sector", test_hbitmap_meta_sector); - hbitmap_test_add("/hbitmap/serialize/granularity", - test_hbitmap_serialize_granularity); + hbitmap_test_add("/hbitmap/serialize/align", + test_hbitmap_serialize_align); hbitmap_test_add("/hbitmap/serialize/basic", test_hbitmap_serialize_basic); hbitmap_test_add("/hbitmap/serialize/part", diff --git a/util/hbitmap.c b/util/hbitmap.c index 21535cc90b..2f9d0fdbd0 100644 --- a/util/hbitmap.c +++ b/util/hbitmap.c @@ -413,14 +413,14 @@ bool hbitmap_is_serializable(const HBitmap *hb) { /* Every serialized chunk must be aligned to 64 bits so that endianness * requirements can be fulfilled on both 64 bit and 32 bit hosts. - * We have hbitmap_serialization_granularity() which converts this + * We have hbitmap_serialization_align() which converts this * alignment requirement from bitmap bits to items covered (e.g. sectors). * That value is: * 64 << hb->granularity * Since this value must not exceed UINT64_MAX, hb->granularity must be * less than 58 (== 64 - 6, where 6 is ld(64), i.e. 1 << 6 == 64). * - * In order for hbitmap_serialization_granularity() to always return a + * In order for hbitmap_serialization_align() to always return a * meaningful value, bitmaps that are to be serialized must have a * granularity of less than 58. */ @@ -437,7 +437,7 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item) return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0; } -uint64_t hbitmap_serialization_granularity(const HBitmap *hb) +uint64_t hbitmap_serialization_align(const HBitmap *hb) { assert(hbitmap_is_serializable(hb)); @@ -454,7 +454,7 @@ static void serialization_chunk(const HBitmap *hb, unsigned long **first_el, uint64_t *el_count) { uint64_t last = start + count - 1; - uint64_t gran = hbitmap_serialization_granularity(hb); + uint64_t gran = hbitmap_serialization_align(hb); assert((start & (gran - 1)) == 0); assert((last >> hb->granularity) < hb->size); From patchwork Mon Sep 25 14:55:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818276 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16cc0BVsz9t67 for ; Tue, 26 Sep 2017 00:57:04 +1000 (AEST) Received: from localhost ([::1]:42832 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUog-0005wQ-2B for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 10:57:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54042) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUnm-0005mJ-D8 for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUnl-0003EN-IL for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:06 -0400 Received: from mx1.redhat.com ([209.132.183.28]:8748) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUng-0003B9-6e; Mon, 25 Sep 2017 10:56:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 26A607E429; Mon, 25 Sep 2017 14:55:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 26A607E429 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE95C6F13E; Mon, 25 Sep 2017 14:55:56 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:09 -0500 Message-Id: <20170925145526.32690-4-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Mon, 25 Sep 2017 14:55:59 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 03/20] qcow2: Ensure bitmap serialization is aligned X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" When subdividing a bitmap serialization, the code in hbitmap.c enforces that start/count parameters are aligned (except that count can end early at end-of-bitmap). We exposed this required alignment through bdrv_dirty_bitmap_serialization_align(), but forgot to actually check that we comply with it. Fortunately, qcow2 is never dividing bitmap serialization smaller than one cluster (which is a minimum of 512 bytes); so we are always compliant with the serialization alignment (which insists that we partition at least 64 bits per chunk) because we are doing at least 4k bits per chunk. Still, it's safer to add an assertion (for the unlikely case that we'd ever support a cluster smaller than 512 bytes, or if the hbitmap implementation changes what it considers to be aligned), rather than leaving bdrv_dirty_bitmap_serialization_align() without a caller. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: new patch --- block/qcow2-bitmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index e8d3bdbd6e..b3ee4c794a 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -274,10 +274,13 @@ static int free_bitmap_clusters(BlockDriverState *bs, Qcow2BitmapTable *tb) static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s, const BdrvDirtyBitmap *bitmap) { - uint32_t sector_granularity = + uint64_t sector_granularity = bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS; + uint64_t sbc = sector_granularity * (s->cluster_size << 3); - return (uint64_t)sector_granularity * (s->cluster_size << 3); + assert(QEMU_IS_ALIGNED(sbc, + bdrv_dirty_bitmap_serialization_align(bitmap))); + return sbc; } /* load_bitmap_data From patchwork Mon Sep 25 14:55:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818285 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16kZ4QWXz9t67 for ; Tue, 26 Sep 2017 01:02:13 +1000 (AEST) Received: from localhost ([::1]:42862 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUte-0003A8-SL for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:02:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54117) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUnv-0005u4-ME for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUnu-0003Ho-GT for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:15 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54190) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUnn-0003FH-Vg; Mon, 25 Sep 2017 10:56:08 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EAF1181E12; Mon, 25 Sep 2017 14:56:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com EAF1181E12 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AD486F13E; Mon, 25 Sep 2017 14:55:59 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:10 -0500 Message-Id: <20170925145526.32690-5-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 25 Sep 2017 14:56:07 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 04/20] dirty-bitmap: Drop unused functions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We had several functions that no one is currently using, and which use sector-based interfaces. I'm trying to convert towards byte-based interfaces, so it's easier to just drop the unused functions: bdrv_dirty_bitmap_get_meta bdrv_dirty_bitmap_get_meta_locked bdrv_dirty_bitmap_reset_meta bdrv_dirty_bitmap_meta_granularity Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: rebase to Vladimir's persistent bitmaps (bdrv_dirty_bitmap_size now in use), dropped R-b v3: rebase to upstream changes (bdrv_dirty_bitmap_get_meta_locked was added in b64bd51e with no clients), kept R-b v2: tweak commit message based on review, no code change --- include/block/dirty-bitmap.h | 10 ---------- block/dirty-bitmap.c | 44 -------------------------------------------- 2 files changed, 54 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index a79a58d2c3..8fd842eac9 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -34,7 +34,6 @@ void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap); BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs); uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs); uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap); -uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap); bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap); bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap); const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap); @@ -44,15 +43,6 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, int64_t cur_sector, int64_t nr_sectors); void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap, int64_t cur_sector, int64_t nr_sectors); -int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors); -int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors); -void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors); BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap); BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap, uint64_t first_sector); diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 0490ca3aff..42a55e4a4b 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -173,45 +173,6 @@ void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap) qemu_mutex_unlock(bitmap->mutex); } -int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors) -{ - uint64_t i; - int sectors_per_bit = 1 << hbitmap_granularity(bitmap->meta); - - /* To optimize: we can make hbitmap to internally check the range in a - * coarse level, or at least do it word by word. */ - for (i = sector; i < sector + nb_sectors; i += sectors_per_bit) { - if (hbitmap_get(bitmap->meta, i)) { - return true; - } - } - return false; -} - -int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors) -{ - bool dirty; - - qemu_mutex_lock(bitmap->mutex); - dirty = bdrv_dirty_bitmap_get_meta_locked(bs, bitmap, sector, nb_sectors); - qemu_mutex_unlock(bitmap->mutex); - - return dirty; -} - -void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs, - BdrvDirtyBitmap *bitmap, int64_t sector, - int nb_sectors) -{ - qemu_mutex_lock(bitmap->mutex); - hbitmap_reset(bitmap->meta, sector, nb_sectors); - qemu_mutex_unlock(bitmap->mutex); -} - int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap) { return bitmap->size; @@ -511,11 +472,6 @@ uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap) return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap); } -uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap) -{ - return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->meta); -} - BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap, uint64_t first_sector) { From patchwork Mon Sep 25 14:55:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818289 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16nr141dz9t67 for ; Tue, 26 Sep 2017 01:05:04 +1000 (AEST) Received: from localhost ([::1]:42877 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUwQ-00063C-61 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:05:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54116) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUnv-0005u3-Lw for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUnu-0003Ht-IJ for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:15 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35182) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUnq-0003G3-NI; Mon, 25 Sep 2017 10:56:10 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A2F892CE910; Mon, 25 Sep 2017 14:56:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A2F892CE910 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id BED047AA70; Mon, 25 Sep 2017 14:56:05 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:11 -0500 Message-Id: <20170925145526.32690-6-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 25 Sep 2017 14:56:09 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 05/20] dirty-bitmap: Avoid size query failure during truncate X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We've previously fixed several places where we failed to account for possible errors from bdrv_nb_sectors(). Fix another one by making bdrv_dirty_bitmap_truncate() take the new size from the caller instead of querying itself; then adjust the sole caller bdrv_truncate() to pass the size just determined by a successful resize, or to skip the bitmap resize on failure, thus avoiding sizing the bitmaps to -1. This also fixes a bug where not all failure paths in bdrv_truncate() would set errp. Note that bdrv_truncate() is still a bit awkward. We may want to revisit it later and clean up things to better guarantee that a resize attempt either fails cleanly up front, or cannot fail after guest-visible changes have been made (if temporary changes are made, then they need to be cleanly rolled back). But that is a task for another day; for now, the goal is the bare minimum fix to ensure that just bdrv_dirty_bitmap_truncate() cannot fail. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: John Snow Reviewed-by: Fam Zheng --- v10: always resize bitmap as before [John], enhance commit message to point out errp bugfix [Vladimir] v9: skip only bdrv_dirty_bitmap_truncate on error [Fam] v8: retitle and rework to avoid possibility of secondary failure [John] v7: new patch [Kevin] --- include/block/dirty-bitmap.h | 2 +- block.c | 16 +++++++++++----- block/dirty-bitmap.c | 6 +++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 8fd842eac9..7a27590047 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -83,7 +83,7 @@ int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter); void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t sector_num); int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap); int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap); -void bdrv_dirty_bitmap_truncate(BlockDriverState *bs); +void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes); bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap); bool bdrv_has_readonly_bitmaps(BlockDriverState *bs); bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap); diff --git a/block.c b/block.c index 528cda7b2c..ef5af81f66 100644 --- a/block.c +++ b/block.c @@ -3545,12 +3545,18 @@ int bdrv_truncate(BdrvChild *child, int64_t offset, PreallocMode prealloc, assert(!(bs->open_flags & BDRV_O_INACTIVE)); ret = drv->bdrv_truncate(bs, offset, prealloc, errp); - if (ret == 0) { - ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS); - bdrv_dirty_bitmap_truncate(bs); - bdrv_parent_cb_resize(bs); - atomic_inc(&bs->write_gen); + if (ret < 0) { + return ret; } + ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS); + if (ret < 0) { + error_setg_errno(errp, -ret, "Could not refresh total sector count"); + } else { + offset = bs->total_sectors * BDRV_SECTOR_SIZE; + } + bdrv_dirty_bitmap_truncate(bs, offset); + bdrv_parent_cb_resize(bs); + atomic_inc(&bs->write_gen); return ret; } diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 42a55e4a4b..ee164fb518 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -1,7 +1,7 @@ /* * Block Dirty Bitmap * - * Copyright (c) 2016 Red Hat. Inc + * Copyright (c) 2016-2017 Red Hat. Inc * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -302,10 +302,10 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs, * Truncates _all_ bitmaps attached to a BDS. * Called with BQL taken. */ -void bdrv_dirty_bitmap_truncate(BlockDriverState *bs) +void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes) { BdrvDirtyBitmap *bitmap; - uint64_t size = bdrv_nb_sectors(bs); + int64_t size = DIV_ROUND_UP(bytes, BDRV_SECTOR_SIZE); bdrv_dirty_bitmaps_lock(bs); QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) { From patchwork Mon Sep 25 14:55:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818282 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16gj5QL8z9t5c for ; Tue, 26 Sep 2017 00:59:45 +1000 (AEST) Received: from localhost ([::1]:42844 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUrH-0008QH-OT for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 10:59:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54180) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUo4-000635-JU for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUo3-0003Lq-Et for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35844) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUnx-0003J8-Qe; Mon, 25 Sep 2017 10:56:18 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BC3FE2C973B; Mon, 25 Sep 2017 14:56:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BC3FE2C973B Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 38BB66F13C; Mon, 25 Sep 2017 14:56:09 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:12 -0500 Message-Id: <20170925145526.32690-7-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 25 Sep 2017 14:56:16 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 06/20] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We're already reporting bytes for bdrv_dirty_bitmap_granularity(); mixing bytes and sectors in our return values is a recipe for confusion. A later cleanup will convert dirty bitmap internals to be entirely byte-based, but in the meantime, we should report the bitmap size in bytes. The only external caller in qcow2-bitmap.c is temporarily more verbose (because it is still using sector-based math), but will later be switched to track progress by bytes instead of sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v8: no change v7: split external from internal change [Kevin], drop R-b v6: no change v5: fix bdrv_dirty_bitmap_truncate [John], drop R-b v4: retitle from "Track size in bytes", rebase to persistent bitmaps, round up when converting bytes to sectors v3: no change v2: tweak commit message, no code change --- block/dirty-bitmap.c | 2 +- block/qcow2-bitmap.c | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index ee164fb518..755555af32 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -175,7 +175,7 @@ void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap) int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap) { - return bitmap->size; + return bitmap->size * BDRV_SECTOR_SIZE; } const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index b3ee4c794a..65122e9ae1 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -295,10 +295,11 @@ static int load_bitmap_data(BlockDriverState *bs, BDRVQcow2State *s = bs->opaque; uint64_t sector, sbc; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); + uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); uint8_t *buf = NULL; uint64_t i, tab_size = size_to_clusters(s, - bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size)); + bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors)); if (tab_size != bitmap_table_size || tab_size > BME_MAX_TABLE_SIZE) { return -EINVAL; @@ -307,7 +308,7 @@ static int load_bitmap_data(BlockDriverState *bs, buf = g_malloc(s->cluster_size); sbc = sectors_covered_by_bitmap_cluster(s, bitmap); for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) { - uint64_t count = MIN(bm_size - sector, sbc); + uint64_t count = MIN(bm_sectors - sector, sbc); uint64_t entry = bitmap_table[i]; uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK; @@ -1077,13 +1078,14 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, int64_t sector; uint64_t sbc; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); + uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); const char *bm_name = bdrv_dirty_bitmap_name(bitmap); uint8_t *buf = NULL; BdrvDirtyBitmapIter *dbi; uint64_t *tb; uint64_t tb_size = size_to_clusters(s, - bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size)); + bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors)); if (tb_size > BME_MAX_TABLE_SIZE || tb_size * s->cluster_size > BME_MAX_PHYS_SIZE) @@ -1101,7 +1103,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, dbi = bdrv_dirty_iter_new(bitmap, 0); buf = g_malloc(s->cluster_size); sbc = sectors_covered_by_bitmap_cluster(s, bitmap); - assert(DIV_ROUND_UP(bm_size, sbc) == tb_size); + assert(DIV_ROUND_UP(bm_sectors, sbc) == tb_size); while ((sector = bdrv_dirty_iter_next(dbi)) != -1) { uint64_t cluster = sector / sbc; @@ -1109,7 +1111,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, int64_t off; sector = cluster * sbc; - end = MIN(bm_size, sector + sbc); + end = MIN(bm_sectors, sector + sbc); write_size = bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector); assert(write_size <= s->cluster_size); @@ -1141,7 +1143,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, goto fail; } - if (end >= bm_size) { + if (end >= bm_sectors) { break; } From patchwork Mon Sep 25 14:55:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818283 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16h92p9Kz9t67 for ; Tue, 26 Sep 2017 01:00:09 +1000 (AEST) Received: from localhost ([::1]:42848 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUrf-0000QW-DQ for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:00:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54202) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUo7-00067o-UT for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUo6-0003Mq-Sa for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:28 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56042) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUo0-0003Km-OQ; Mon, 25 Sep 2017 10:56:20 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AEA648123F; Mon, 25 Sep 2017 14:56:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com AEA648123F Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0AC256F120; Mon, 25 Sep 2017 14:56:16 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:13 -0500 Message-Id: <20170925145526.32690-8-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 25 Sep 2017 14:56:19 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 07/20] dirty-bitmap: Track bitmap size by bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We are still using an internal hbitmap that tracks a size in sectors, with the granularity scaled down accordingly, because it lets us use a shortcut for our iterators which are currently sector-based. But there's no reason we can't track the dirty bitmap size in bytes, since it is (mostly) an internal-only variable (remember, the size is how many bytes are covered by the bitmap, not how many bytes the bitmap occupies). A later cleanup will convert dirty bitmap internals to be entirely byte-based, eliminating the intermediate sector rounding added here; and technically, since bdrv_getlength() already rounds up to sectors, our use of DIV_ROUND_UP is more for theoretical completeness than for any actual rounding. Use is_power_of_2() while at it, instead of open-coding that. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v8: rebase to earlier truncate changes, R-b kept v7: split external from internal [Kevin], drop R-b v6: no change v5: fix bdrv_dirty_bitmap_truncate [John], drop R-b v4: retitle from "Track size in bytes", rebase to persistent bitmaps, round up when converting bytes to sectors v3: no change v2: tweak commit message, no code change --- block/dirty-bitmap.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 755555af32..6d32554b4e 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -42,7 +42,7 @@ struct BdrvDirtyBitmap { HBitmap *meta; /* Meta dirty bitmap */ BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */ char *name; /* Optional non-empty unique ID */ - int64_t size; /* Size of the bitmap (Number of sectors) */ + int64_t size; /* Size of the bitmap, in bytes */ bool disabled; /* Bitmap is disabled. It ignores all writes to the device */ int active_iterators; /* How many iterators are active */ @@ -115,17 +115,14 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, { int64_t bitmap_size; BdrvDirtyBitmap *bitmap; - uint32_t sector_granularity; - assert((granularity & (granularity - 1)) == 0); + assert(is_power_of_2(granularity) && granularity >= BDRV_SECTOR_SIZE); if (name && bdrv_find_dirty_bitmap(bs, name)) { error_setg(errp, "Bitmap already exists: %s", name); return NULL; } - sector_granularity = granularity >> BDRV_SECTOR_BITS; - assert(sector_granularity); - bitmap_size = bdrv_nb_sectors(bs); + bitmap_size = bdrv_getlength(bs); if (bitmap_size < 0) { error_setg_errno(errp, -bitmap_size, "could not get length of device"); errno = -bitmap_size; @@ -133,7 +130,12 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, } bitmap = g_new0(BdrvDirtyBitmap, 1); bitmap->mutex = &bs->dirty_bitmap_mutex; - bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(sector_granularity)); + /* + * TODO - let hbitmap track full granularity. For now, it is tracking + * only sector granularity, as a shortcut for our iterators. + */ + bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap_size, BDRV_SECTOR_SIZE), + ctz32(granularity) - BDRV_SECTOR_BITS); bitmap->size = bitmap_size; bitmap->name = g_strdup(name); bitmap->disabled = false; @@ -175,7 +177,7 @@ void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap) int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap) { - return bitmap->size * BDRV_SECTOR_SIZE; + return bitmap->size; } const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap) @@ -305,14 +307,13 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs, void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes) { BdrvDirtyBitmap *bitmap; - int64_t size = DIV_ROUND_UP(bytes, BDRV_SECTOR_SIZE); bdrv_dirty_bitmaps_lock(bs); QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) { assert(!bdrv_dirty_bitmap_frozen(bitmap)); assert(!bitmap->active_iterators); - hbitmap_truncate(bitmap->bitmap, size); - bitmap->size = size; + hbitmap_truncate(bitmap->bitmap, DIV_ROUND_UP(bytes, BDRV_SECTOR_SIZE)); + bitmap->size = bytes; } bdrv_dirty_bitmaps_unlock(bs); } @@ -549,7 +550,8 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out) hbitmap_reset_all(bitmap->bitmap); } else { HBitmap *backup = bitmap->bitmap; - bitmap->bitmap = hbitmap_alloc(bitmap->size, + bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap->size, + BDRV_SECTOR_SIZE), hbitmap_granularity(backup)); *out = backup; } From patchwork Mon Sep 25 14:55:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818281 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16gV0w6Nz9t5c for ; Tue, 26 Sep 2017 00:59:34 +1000 (AEST) Received: from localhost ([::1]:42842 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUr6-0008AG-3y for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 10:59:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54258) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoD-0006DP-KT for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoC-0003P3-8J for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:33 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49410) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUo7-0003Mi-7a; Mon, 25 Sep 2017 10:56:27 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 307AC1F57F; Mon, 25 Sep 2017 14:56:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 307AC1F57F Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id F401A6F124; Mon, 25 Sep 2017 14:56:19 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:14 -0500 Message-Id: <20170925145526.32690-9-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 25 Sep 2017 14:56:26 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 08/20] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Right now, the dirty-bitmap code exposes the fact that we use a scale of sector granularity in the underlying hbitmap to anything that wants to serialize a dirty bitmap. It's nicer to uniformly expose bytes as our dirty-bitmap interface, matching the previous change to bitmap size. The only caller to serialization is currently qcow2-cluster.c, which becomes a bit more verbose because it is still tracking sectors for other reasons, but a later patch will fix that to more uniformly use byte offsets everywhere. Likewise, within dirty-bitmap, we have to add more assertions that we are not truncating incorrectly, which can go away once the internal hbitmap is byte-based rather than sector-based. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: new patch --- include/block/dirty-bitmap.h | 14 +++++++------- block/dirty-bitmap.c | 37 ++++++++++++++++++++++++------------- block/qcow2-bitmap.c | 22 ++++++++++++++-------- 3 files changed, 45 insertions(+), 28 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 7a27590047..5f34a1a3c7 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -49,19 +49,19 @@ BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap, void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter); uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count); + uint64_t offset, uint64_t bytes); uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap); void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap, - uint8_t *buf, uint64_t start, - uint64_t count); + uint8_t *buf, uint64_t offset, + uint64_t bytes); void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap, - uint8_t *buf, uint64_t start, - uint64_t count, bool finish); + uint8_t *buf, uint64_t offset, + uint64_t bytes, bool finish); void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count, + uint64_t offset, uint64_t bytes, bool finish); void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count, + uint64_t offset, uint64_t bytes, bool finish); void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap); diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 6d32554b4e..555c736024 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -568,42 +568,53 @@ void bdrv_undo_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *in) } uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count) + uint64_t offset, uint64_t bytes) { - return hbitmap_serialization_size(bitmap->bitmap, start, count); + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + return hbitmap_serialization_size(bitmap->bitmap, + offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS); } uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap) { - return hbitmap_serialization_align(bitmap->bitmap); + return hbitmap_serialization_align(bitmap->bitmap) * BDRV_SECTOR_SIZE; } void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap, - uint8_t *buf, uint64_t start, - uint64_t count) + uint8_t *buf, uint64_t offset, + uint64_t bytes) { - hbitmap_serialize_part(bitmap->bitmap, buf, start, count); + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + hbitmap_serialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS); } void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap, - uint8_t *buf, uint64_t start, - uint64_t count, bool finish) + uint8_t *buf, uint64_t offset, + uint64_t bytes, bool finish) { - hbitmap_deserialize_part(bitmap->bitmap, buf, start, count, finish); + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + hbitmap_deserialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, finish); } void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count, + uint64_t offset, uint64_t bytes, bool finish) { - hbitmap_deserialize_zeroes(bitmap->bitmap, start, count, finish); + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + hbitmap_deserialize_zeroes(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, finish); } void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap, - uint64_t start, uint64_t count, + uint64_t offset, uint64_t bytes, bool finish) { - hbitmap_deserialize_ones(bitmap->bitmap, start, count, finish); + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + hbitmap_deserialize_ones(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, finish); } void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 65122e9ae1..92098bfa49 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -278,7 +278,7 @@ static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s, bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS; uint64_t sbc = sector_granularity * (s->cluster_size << 3); - assert(QEMU_IS_ALIGNED(sbc, + assert(QEMU_IS_ALIGNED(sbc << BDRV_SECTOR_BITS, bdrv_dirty_bitmap_serialization_align(bitmap))); return sbc; } @@ -299,7 +299,7 @@ static int load_bitmap_data(BlockDriverState *bs, uint8_t *buf = NULL; uint64_t i, tab_size = size_to_clusters(s, - bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors)); + bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size)); if (tab_size != bitmap_table_size || tab_size > BME_MAX_TABLE_SIZE) { return -EINVAL; @@ -316,7 +316,9 @@ static int load_bitmap_data(BlockDriverState *bs, if (offset == 0) { if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) { - bdrv_dirty_bitmap_deserialize_ones(bitmap, sector, count, + bdrv_dirty_bitmap_deserialize_ones(bitmap, + sector * BDRV_SECTOR_SIZE, + count * BDRV_SECTOR_SIZE, false); } else { /* No need to deserialize zeros because the dirty bitmap is @@ -327,7 +329,9 @@ static int load_bitmap_data(BlockDriverState *bs, if (ret < 0) { goto finish; } - bdrv_dirty_bitmap_deserialize_part(bitmap, buf, sector, count, + bdrv_dirty_bitmap_deserialize_part(bitmap, buf, + sector * BDRV_SECTOR_SIZE, + count * BDRV_SECTOR_SIZE, false); } } @@ -1085,7 +1089,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, uint64_t *tb; uint64_t tb_size = size_to_clusters(s, - bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors)); + bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size)); if (tb_size > BME_MAX_TABLE_SIZE || tb_size * s->cluster_size > BME_MAX_PHYS_SIZE) @@ -1112,8 +1116,8 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, sector = cluster * sbc; end = MIN(bm_sectors, sector + sbc); - write_size = - bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector); + write_size = bdrv_dirty_bitmap_serialization_size(bitmap, + sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE); assert(write_size <= s->cluster_size); off = qcow2_alloc_clusters(bs, s->cluster_size); @@ -1125,7 +1129,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, } tb[cluster] = off; - bdrv_dirty_bitmap_serialize_part(bitmap, buf, sector, end - sector); + bdrv_dirty_bitmap_serialize_part(bitmap, buf, + sector * BDRV_SECTOR_SIZE, + (end - sector) * BDRV_SECTOR_SIZE); if (write_size < s->cluster_size) { memset(buf + write_size, 0, s->cluster_size - write_size); } From patchwork Mon Sep 25 14:55:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818292 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16s60gzwz9t67 for ; Tue, 26 Sep 2017 01:07:53 +1000 (AEST) Received: from localhost ([::1]:42890 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUz8-0000Bo-RA for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:07:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54273) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoE-0006EM-FT for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoD-0003Pr-Fr for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:34 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45188) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUo9-0003Nr-T9; Mon, 25 Sep 2017 10:56:30 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D2CA4806A3; Mon, 25 Sep 2017 14:56:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D2CA4806A3 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 796C57836D; Mon, 25 Sep 2017 14:56:26 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:15 -0500 Message-Id: <20170925145526.32690-10-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 25 Sep 2017 14:56:28 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 09/20] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Change the qcow2 bitmap helper function sectors_covered_by_bitmap_cluster(), renaming it to bytes_covered_by_bitmap_cluster() in the process. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: new patch --- block/qcow2-bitmap.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 92098bfa49..4475273d8c 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -269,18 +269,16 @@ static int free_bitmap_clusters(BlockDriverState *bs, Qcow2BitmapTable *tb) return 0; } -/* This function returns the number of disk sectors covered by a single qcow2 - * cluster of bitmap data. */ -static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s, - const BdrvDirtyBitmap *bitmap) +/* Return the disk size covered by a single qcow2 cluster of bitmap data. */ +static uint64_t bytes_covered_by_bitmap_cluster(const BDRVQcow2State *s, + const BdrvDirtyBitmap *bitmap) { - uint64_t sector_granularity = - bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS; - uint64_t sbc = sector_granularity * (s->cluster_size << 3); + uint64_t granularity = bdrv_dirty_bitmap_granularity(bitmap); + uint64_t limit = granularity * (s->cluster_size << 3); - assert(QEMU_IS_ALIGNED(sbc << BDRV_SECTOR_BITS, + assert(QEMU_IS_ALIGNED(limit, bdrv_dirty_bitmap_serialization_align(bitmap))); - return sbc; + return limit; } /* load_bitmap_data @@ -293,7 +291,7 @@ static int load_bitmap_data(BlockDriverState *bs, { int ret = 0; BDRVQcow2State *s = bs->opaque; - uint64_t sector, sbc; + uint64_t sector, limit, sbc; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); uint8_t *buf = NULL; @@ -306,7 +304,8 @@ static int load_bitmap_data(BlockDriverState *bs, } buf = g_malloc(s->cluster_size); - sbc = sectors_covered_by_bitmap_cluster(s, bitmap); + limit = bytes_covered_by_bitmap_cluster(s, bitmap); + sbc = limit >> BDRV_SECTOR_BITS; for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) { uint64_t count = MIN(bm_sectors - sector, sbc); uint64_t entry = bitmap_table[i]; @@ -1080,7 +1079,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, int ret; BDRVQcow2State *s = bs->opaque; int64_t sector; - uint64_t sbc; + uint64_t limit, sbc; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); const char *bm_name = bdrv_dirty_bitmap_name(bitmap); @@ -1106,8 +1105,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, dbi = bdrv_dirty_iter_new(bitmap, 0); buf = g_malloc(s->cluster_size); - sbc = sectors_covered_by_bitmap_cluster(s, bitmap); - assert(DIV_ROUND_UP(bm_sectors, sbc) == tb_size); + limit = bytes_covered_by_bitmap_cluster(s, bitmap); + sbc = limit >> BDRV_SECTOR_BITS; + assert(DIV_ROUND_UP(bm_size, limit) == tb_size); while ((sector = bdrv_dirty_iter_next(dbi)) != -1) { uint64_t cluster = sector / sbc; From patchwork Mon Sep 25 14:55:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818287 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16lz3TlSz9tXC for ; Tue, 26 Sep 2017 01:03:27 +1000 (AEST) Received: from localhost ([::1]:42872 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUur-0004ai-K3 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:03:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54298) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoG-0006Gz-Sw for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoF-0003RJ-OY for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57215) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoC-0003Os-Mm; Mon, 25 Sep 2017 10:56:32 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AA4BB90906; Mon, 25 Sep 2017 14:56:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com AA4BB90906 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 26F9F6F113; Mon, 25 Sep 2017 14:56:29 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:16 -0500 Message-Id: <20170925145526.32690-11-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 25 Sep 2017 14:56:31 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 10/20] dirty-bitmap: Set iterator start by offset, not sector X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Jeff Cody , Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" All callers to bdrv_dirty_iter_new() passed 0 for their initial starting point, drop that parameter. Most callers to bdrv_set_dirty_iter() were scaling a byte offset to a sector number; the exception qcow2-bitmap will be converted later to use byte rather than sector iteration. Move the scaling to occur internally to dirty bitmap code instead, so that callers now pass in bytes. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: rebase to persistent bitmaps v3: no change v2: no change --- include/block/dirty-bitmap.h | 5 ++--- block/backup.c | 5 ++--- block/dirty-bitmap.c | 9 ++++----- block/mirror.c | 4 ++-- block/qcow2-bitmap.c | 4 ++-- 5 files changed, 12 insertions(+), 15 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 5f34a1a3c7..757fc4c5b8 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -44,8 +44,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap, int64_t cur_sector, int64_t nr_sectors); BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap); -BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap, - uint64_t first_sector); +BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap); void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter); uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap, @@ -80,7 +79,7 @@ void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, int64_t cur_sector, int64_t nr_sectors); int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter); -void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t sector_num); +void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t offset); int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap); int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap); void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes); diff --git a/block/backup.c b/block/backup.c index 517c300a49..ac9c018717 100644 --- a/block/backup.c +++ b/block/backup.c @@ -372,7 +372,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job) granularity = bdrv_dirty_bitmap_granularity(job->sync_bitmap); clusters_per_iter = MAX((granularity / job->cluster_size), 1); - dbi = bdrv_dirty_iter_new(job->sync_bitmap, 0); + dbi = bdrv_dirty_iter_new(job->sync_bitmap); /* Find the next dirty sector(s) */ while ((offset = bdrv_dirty_iter_next(dbi) * BDRV_SECTOR_SIZE) >= 0) { @@ -403,8 +403,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job) /* If the bitmap granularity is smaller than the backup granularity, * we need to advance the iterator pointer to the next cluster. */ if (granularity < job->cluster_size) { - bdrv_set_dirty_iter(dbi, - cluster * job->cluster_size / BDRV_SECTOR_SIZE); + bdrv_set_dirty_iter(dbi, cluster * job->cluster_size); } last_cluster = cluster - 1; diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 555c736024..84509476ba 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -473,11 +473,10 @@ uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap) return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap); } -BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap, - uint64_t first_sector) +BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap) { BdrvDirtyBitmapIter *iter = g_new(BdrvDirtyBitmapIter, 1); - hbitmap_iter_init(&iter->hbi, bitmap->bitmap, first_sector); + hbitmap_iter_init(&iter->hbi, bitmap->bitmap, 0); iter->bitmap = bitmap; bitmap->active_iterators++; return iter; @@ -645,9 +644,9 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, /** * Advance a BdrvDirtyBitmapIter to an arbitrary offset. */ -void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t sector_num) +void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset) { - hbitmap_iter_init(&iter->hbi, iter->hbi.hb, sector_num); + hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset >> BDRV_SECTOR_BITS); } int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap) diff --git a/block/mirror.c b/block/mirror.c index 6f5cb9f26c..0c705e0b4f 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -373,7 +373,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; if (next_dirty > next_offset || next_dirty < 0) { /* The bitmap iterator's cache is stale, refresh it */ - bdrv_set_dirty_iter(s->dbi, next_offset >> BDRV_SECTOR_BITS); + bdrv_set_dirty_iter(s->dbi, next_offset); next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; } assert(next_dirty == next_offset); @@ -796,7 +796,7 @@ static void coroutine_fn mirror_run(void *opaque) } assert(!s->dbi); - s->dbi = bdrv_dirty_iter_new(s->dirty_bitmap, 0); + s->dbi = bdrv_dirty_iter_new(s->dirty_bitmap); for (;;) { uint64_t delay_ns = 0; int64_t cnt, delta; diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 4475273d8c..44329fc74f 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -1103,7 +1103,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, return NULL; } - dbi = bdrv_dirty_iter_new(bitmap, 0); + dbi = bdrv_dirty_iter_new(bitmap); buf = g_malloc(s->cluster_size); limit = bytes_covered_by_bitmap_cluster(s, bitmap); sbc = limit >> BDRV_SECTOR_BITS; @@ -1153,7 +1153,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, break; } - bdrv_set_dirty_iter(dbi, end); + bdrv_set_dirty_iter(dbi, end * BDRV_SECTOR_SIZE); } *bitmap_table_size = tb_size; From patchwork Mon Sep 25 14:55:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818294 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16vv6hYWz9t67 for ; Tue, 26 Sep 2017 01:10:19 +1000 (AEST) Received: from localhost ([::1]:42898 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV1V-0001yj-U9 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:10:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54408) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoQ-0006Ov-Qg for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoP-0003XJ-P9 for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38382) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoI-0003Sg-JX; Mon, 25 Sep 2017 10:56:38 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8B63C5AFED; Mon, 25 Sep 2017 14:56:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8B63C5AFED Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 02D036F113; Mon, 25 Sep 2017 14:56:31 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:17 -0500 Message-Id: <20170925145526.32690-12-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 25 Sep 2017 14:56:37 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 11/20] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Jeff Cody , Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Thanks to recent cleanups, most callers were scaling a return value of sectors into bytes (the exception, in qcow2-bitmap, will be converted to byte-based iteration later). Update the interface to do the scaling internally instead. In qcow2-bitmap, the code was specifically checking for an error return of -1. To avoid a regression, we either have to make sure we continue to return -1 (rather than a scaled -512) on error, or we have to fix the caller to treat all negative values as error rather than just one magic value. It's easy enough to make both changes at the same time, even though either one in isolation would work. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v8: tweak commit message, add R-b v7: return -1, not -512; and fix qcow2-bitmap to check all negatives [Kevin] v5-v6: no change v4: rebase to persistent bitmap v3: no change v2: no change --- block/backup.c | 2 +- block/dirty-bitmap.c | 3 ++- block/mirror.c | 8 ++++---- block/qcow2-bitmap.c | 2 +- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/block/backup.c b/block/backup.c index ac9c018717..06ddbfd03d 100644 --- a/block/backup.c +++ b/block/backup.c @@ -375,7 +375,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job) dbi = bdrv_dirty_iter_new(job->sync_bitmap); /* Find the next dirty sector(s) */ - while ((offset = bdrv_dirty_iter_next(dbi) * BDRV_SECTOR_SIZE) >= 0) { + while ((offset = bdrv_dirty_iter_next(dbi)) >= 0) { cluster = offset / job->cluster_size; /* Fake progress updates for any clusters we skipped */ diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 84509476ba..e451916187 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -503,7 +503,8 @@ void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter) int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter) { - return hbitmap_iter_next(&iter->hbi); + int64_t ret = hbitmap_iter_next(&iter->hbi); + return ret < 0 ? -1 : ret * BDRV_SECTOR_SIZE; } /* Called within bdrv_dirty_bitmap_lock..unlock */ diff --git a/block/mirror.c b/block/mirror.c index 0c705e0b4f..de0a02778c 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -336,10 +336,10 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) int max_io_bytes = MAX(s->buf_size / MAX_IN_FLIGHT, MAX_IO_BYTES); bdrv_dirty_bitmap_lock(s->dirty_bitmap); - offset = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; + offset = bdrv_dirty_iter_next(s->dbi); if (offset < 0) { bdrv_set_dirty_iter(s->dbi, 0); - offset = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; + offset = bdrv_dirty_iter_next(s->dbi); trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap) * BDRV_SECTOR_SIZE); assert(offset >= 0); @@ -370,11 +370,11 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) break; } - next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; + next_dirty = bdrv_dirty_iter_next(s->dbi); if (next_dirty > next_offset || next_dirty < 0) { /* The bitmap iterator's cache is stale, refresh it */ bdrv_set_dirty_iter(s->dbi, next_offset); - next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE; + next_dirty = bdrv_dirty_iter_next(s->dbi); } assert(next_dirty == next_offset); nb_chunks++; diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 44329fc74f..b09010b1d3 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -1109,7 +1109,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, sbc = limit >> BDRV_SECTOR_BITS; assert(DIV_ROUND_UP(bm_size, limit) == tb_size); - while ((sector = bdrv_dirty_iter_next(dbi)) != -1) { + while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) >= 0) { uint64_t cluster = sector / sbc; uint64_t end, write_size; int64_t off; From patchwork Mon Sep 25 14:55:18 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818291 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16q81xtyz9t67 for ; Tue, 26 Sep 2017 01:06:12 +1000 (AEST) Received: from localhost ([::1]:42883 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUxW-0007Rx-Bh for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:06:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54436) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoS-0006Qg-Bg for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoR-0003Xv-2e for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:9922) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoO-0003Vn-7Z; Mon, 25 Sep 2017 10:56:44 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2FFE7806A3; Mon, 25 Sep 2017 14:56:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 2FFE7806A3 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id D54306F124; Mon, 25 Sep 2017 14:56:37 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:18 -0500 Message-Id: <20170925145526.32690-13-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 25 Sep 2017 14:56:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 12/20] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Juan Quintela , Jeff Cody , "Dr. David Alan Gilbert" , Max Reitz , Stefan Hajnoczi , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Thanks to recent cleanups, all callers were scaling a return value of sectors into bytes; do the scaling internally instead. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v8: no change, add R-b v7: fix one more trace caller [Kevin] v4-v6: no change v3: no change, add R-b v2: no change --- block/dirty-bitmap.c | 4 ++-- block/mirror.c | 16 ++++++---------- migration/block.c | 2 +- 3 files changed, 9 insertions(+), 13 deletions(-) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index e451916187..8322e23f0d 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -423,7 +423,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs) QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { BlockDirtyInfo *info = g_new0(BlockDirtyInfo, 1); BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1); - info->count = bdrv_get_dirty_count(bm) << BDRV_SECTOR_BITS; + info->count = bdrv_get_dirty_count(bm); info->granularity = bdrv_dirty_bitmap_granularity(bm); info->has_name = !!bm->name; info->name = g_strdup(bm->name); @@ -652,7 +652,7 @@ void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset) int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap) { - return hbitmap_count(bitmap->bitmap); + return hbitmap_count(bitmap->bitmap) << BDRV_SECTOR_BITS; } int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap) diff --git a/block/mirror.c b/block/mirror.c index de0a02778c..c5212d2c8d 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -340,8 +340,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) if (offset < 0) { bdrv_set_dirty_iter(s->dbi, 0); offset = bdrv_dirty_iter_next(s->dbi); - trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap) * - BDRV_SECTOR_SIZE); + trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap)); assert(offset >= 0); } bdrv_dirty_bitmap_unlock(s->dirty_bitmap); @@ -811,11 +810,10 @@ static void coroutine_fn mirror_run(void *opaque) cnt = bdrv_get_dirty_count(s->dirty_bitmap); /* s->common.offset contains the number of bytes already processed so - * far, cnt is the number of dirty sectors remaining and + * far, cnt is the number of dirty bytes remaining and * s->bytes_in_flight is the number of bytes currently being * processed; together those are the current total operation length */ - s->common.len = s->common.offset + s->bytes_in_flight + - cnt * BDRV_SECTOR_SIZE; + s->common.len = s->common.offset + s->bytes_in_flight + cnt; /* Note that even when no rate limit is applied we need to yield * periodically with no pending I/O so that bdrv_drain_all() returns. @@ -827,8 +825,7 @@ static void coroutine_fn mirror_run(void *opaque) s->common.iostatus == BLOCK_DEVICE_IO_STATUS_OK) { if (s->in_flight >= MAX_IN_FLIGHT || s->buf_free_count == 0 || (cnt == 0 && s->in_flight > 0)) { - trace_mirror_yield(s, cnt * BDRV_SECTOR_SIZE, - s->buf_free_count, s->in_flight); + trace_mirror_yield(s, cnt, s->buf_free_count, s->in_flight); mirror_wait_for_io(s); continue; } else if (cnt != 0) { @@ -869,7 +866,7 @@ static void coroutine_fn mirror_run(void *opaque) * whether to switch to target check one last time if I/O has * come in the meanwhile, and if not flush the data to disk. */ - trace_mirror_before_drain(s, cnt * BDRV_SECTOR_SIZE); + trace_mirror_before_drain(s, cnt); bdrv_drained_begin(bs); cnt = bdrv_get_dirty_count(s->dirty_bitmap); @@ -888,8 +885,7 @@ static void coroutine_fn mirror_run(void *opaque) } ret = 0; - trace_mirror_before_sleep(s, cnt * BDRV_SECTOR_SIZE, - s->synced, delay_ns); + trace_mirror_before_sleep(s, cnt, s->synced, delay_ns); if (!s->synced) { block_job_sleep_ns(&s->common, QEMU_CLOCK_REALTIME, delay_ns); if (block_job_is_cancelled(&s->common)) { diff --git a/migration/block.c b/migration/block.c index 9171f60028..a3512945da 100644 --- a/migration/block.c +++ b/migration/block.c @@ -667,7 +667,7 @@ static int64_t get_remaining_dirty(void) aio_context_release(blk_get_aio_context(bmds->blk)); } - return dirty << BDRV_SECTOR_BITS; + return dirty; } From patchwork Mon Sep 25 14:55:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818296 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16yh6ZvGz9t67 for ; Tue, 26 Sep 2017 01:12:42 +1000 (AEST) Received: from localhost ([::1]:42911 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV3o-0003mt-3t for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:12:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54544) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoY-0006Wl-Lx for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoX-0003cV-OX for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:56:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46182) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoT-0003ZY-OX; Mon, 25 Sep 2017 10:56:49 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BD8C2806B1; Mon, 25 Sep 2017 14:56:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BD8C2806B1 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76EFC6F113; Mon, 25 Sep 2017 14:56:43 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:19 -0500 Message-Id: <20170925145526.32690-14-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 25 Sep 2017 14:56:48 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 13/20] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Juan Quintela , Jeff Cody , "Dr. David Alan Gilbert" , Max Reitz , Stefan Hajnoczi , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Half the callers were already scaling bytes to sectors; the other half can eventually be simplified to use byte iteration. Both callers were already using the result as a bool, so make that explicit. Making the change also makes it easier for a future dirty-bitmap patch to offload scaling over to the internal hbitmap. Remember, asking whether a byte is dirty is effectively asking whether the entire granularity containing the byte is dirty, since we only track dirtiness by granularity. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Juan Quintela Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v4: only context change v3: rebase to _locked rename was straightforward enough that R-b kept v2: tweak commit message, no code change --- include/block/dirty-bitmap.h | 4 ++-- block/dirty-bitmap.c | 8 ++++---- block/mirror.c | 3 +-- migration/block.c | 3 ++- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 757fc4c5b8..9e39537e4b 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -72,8 +72,8 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap, /* Functions that require manual locking. */ void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap); void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap); -int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, - int64_t sector); +bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, + int64_t offset); void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, int64_t cur_sector, int64_t nr_sectors); void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 8322e23f0d..ad559c62b1 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -438,13 +438,13 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs) } /* Called within bdrv_dirty_bitmap_lock..unlock */ -int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, - int64_t sector) +bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, + int64_t offset) { if (bitmap) { - return hbitmap_get(bitmap->bitmap, sector); + return hbitmap_get(bitmap->bitmap, offset >> BDRV_SECTOR_BITS); } else { - return 0; + return false; } } diff --git a/block/mirror.c b/block/mirror.c index c5212d2c8d..545dfc50aa 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -361,8 +361,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) int64_t next_offset = offset + nb_chunks * s->granularity; int64_t next_chunk = next_offset / s->granularity; if (next_offset >= s->bdev_length || - !bdrv_get_dirty_locked(source, s->dirty_bitmap, - next_offset >> BDRV_SECTOR_BITS)) { + !bdrv_get_dirty_locked(source, s->dirty_bitmap, next_offset)) { break; } if (test_bit(next_chunk, s->in_flight_bitmap)) { diff --git a/migration/block.c b/migration/block.c index a3512945da..b618869661 100644 --- a/migration/block.c +++ b/migration/block.c @@ -530,7 +530,8 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds, blk_mig_unlock(); } bdrv_dirty_bitmap_lock(bmds->dirty_bitmap); - if (bdrv_get_dirty_locked(bs, bmds->dirty_bitmap, sector)) { + if (bdrv_get_dirty_locked(bs, bmds->dirty_bitmap, + sector * BDRV_SECTOR_SIZE)) { if (total_sectors - sector < BDRV_SECTORS_PER_DIRTY_CHUNK) { nr_sectors = total_sectors - sector; } else { From patchwork Mon Sep 25 14:55:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818295 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16wR6bS9z9t67 for ; Tue, 26 Sep 2017 01:10:47 +1000 (AEST) Received: from localhost ([::1]:42902 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV1x-0002JT-Vg for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:10:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54609) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoi-0006gp-Tk for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoh-0003i2-JQ for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51290) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoY-0003cj-Kk; Mon, 25 Sep 2017 10:56:54 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 93D06267C6; Mon, 25 Sep 2017 14:56:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 93D06267C6 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C0636F113; Mon, 25 Sep 2017 14:56:48 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:20 -0500 Message-Id: <20170925145526.32690-15-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 25 Sep 2017 14:56:53 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 14/20] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Juan Quintela , Jeff Cody , "Dr. David Alan Gilbert" , Max Reitz , Stefan Hajnoczi , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Some of the callers were already scaling bytes to sectors; others can be easily converted to pass byte offsets, all in our shift towards a consistent byte interface everywhere. Making the change will also make it easier to write the hold-out callers to use byte rather than sectors for their iterations; it also makes it easier for a future dirty-bitmap patch to offload scaling over to the internal hbitmap. Although all callers happen to pass sector-aligned values, make the internal scaling robust to any sub-sector requests. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: only context change v4: only context change, due to rebasing to persistent bitmaps v3: rebase to addition of _locked interfaces; complex enough that I dropped R-b v2: no change --- include/block/dirty-bitmap.h | 8 ++++---- block/dirty-bitmap.c | 22 ++++++++++++++-------- block/mirror.c | 16 ++++++++-------- migration/block.c | 7 +++++-- 4 files changed, 31 insertions(+), 22 deletions(-) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 9e39537e4b..3579a7597c 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -40,9 +40,9 @@ const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap); int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap); DirtyBitmapStatus bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap); void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors); + int64_t offset, int64_t bytes); void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors); + int64_t offset, int64_t bytes); BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap); BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap); void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter); @@ -75,9 +75,9 @@ void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap); bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t offset); void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors); + int64_t offset, int64_t bytes); void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors); + int64_t offset, int64_t bytes); int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter); void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t offset); int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap); diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index ad559c62b1..117837b3cc 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -509,35 +509,41 @@ int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter) /* Called within bdrv_dirty_bitmap_lock..unlock */ void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors) + int64_t offset, int64_t bytes) { + int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); + assert(bdrv_dirty_bitmap_enabled(bitmap)); assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors); + hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, + end_sector - (offset >> BDRV_SECTOR_BITS)); } void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors) + int64_t offset, int64_t bytes) { bdrv_dirty_bitmap_lock(bitmap); - bdrv_set_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors); + bdrv_set_dirty_bitmap_locked(bitmap, offset, bytes); bdrv_dirty_bitmap_unlock(bitmap); } /* Called within bdrv_dirty_bitmap_lock..unlock */ void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors) + int64_t offset, int64_t bytes) { + int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); + assert(bdrv_dirty_bitmap_enabled(bitmap)); assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors); + hbitmap_reset(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, + end_sector - (offset >> BDRV_SECTOR_BITS)); } void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap, - int64_t cur_sector, int64_t nr_sectors) + int64_t offset, int64_t bytes) { bdrv_dirty_bitmap_lock(bitmap); - bdrv_reset_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors); + bdrv_reset_dirty_bitmap_locked(bitmap, offset, bytes); bdrv_dirty_bitmap_unlock(bitmap); } diff --git a/block/mirror.c b/block/mirror.c index 545dfc50aa..5c0f79f56a 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -141,8 +141,7 @@ static void mirror_write_complete(void *opaque, int ret) if (ret < 0) { BlockErrorAction action; - bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset >> BDRV_SECTOR_BITS, - op->bytes >> BDRV_SECTOR_BITS); + bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset, op->bytes); action = mirror_error_action(s, false, -ret); if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) { s->ret = ret; @@ -161,8 +160,7 @@ static void mirror_read_complete(void *opaque, int ret) if (ret < 0) { BlockErrorAction action; - bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset >> BDRV_SECTOR_BITS, - op->bytes >> BDRV_SECTOR_BITS); + bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset, op->bytes); action = mirror_error_action(s, true, -ret); if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) { s->ret = ret; @@ -382,8 +380,8 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) * calling bdrv_get_block_status_above could yield - if some blocks are * marked dirty in this window, we need to know. */ - bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, offset >> BDRV_SECTOR_BITS, - nb_chunks * sectors_per_chunk); + bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, offset, + nb_chunks * s->granularity); bdrv_dirty_bitmap_unlock(s->dirty_bitmap); bitmap_set(s->in_flight_bitmap, offset / s->granularity, nb_chunks); @@ -625,7 +623,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) if (base == NULL && !bdrv_has_zero_init(target_bs)) { if (!bdrv_can_write_zeroes_with_unmap(target_bs)) { - bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, end); + bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, s->bdev_length); return 0; } @@ -681,7 +679,9 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) n = count >> BDRV_SECTOR_BITS; assert(n > 0); if (ret == 1) { - bdrv_set_dirty_bitmap(s->dirty_bitmap, sector_num, n); + bdrv_set_dirty_bitmap(s->dirty_bitmap, + sector_num * BDRV_SECTOR_SIZE, + n * BDRV_SECTOR_SIZE); } sector_num += n; } diff --git a/migration/block.c b/migration/block.c index b618869661..725f1974c3 100644 --- a/migration/block.c +++ b/migration/block.c @@ -329,7 +329,8 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds) blk->aiocb = blk_aio_preadv(bb, cur_sector * BDRV_SECTOR_SIZE, &blk->qiov, 0, blk_mig_read_cb, blk); - bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector, nr_sectors); + bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector * BDRV_SECTOR_SIZE, + nr_sectors * BDRV_SECTOR_SIZE); aio_context_release(blk_get_aio_context(bmds->blk)); qemu_mutex_unlock_iothread(); @@ -537,7 +538,9 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds, } else { nr_sectors = BDRV_SECTORS_PER_DIRTY_CHUNK; } - bdrv_reset_dirty_bitmap_locked(bmds->dirty_bitmap, sector, nr_sectors); + bdrv_reset_dirty_bitmap_locked(bmds->dirty_bitmap, + sector * BDRV_SECTOR_SIZE, + nr_sectors * BDRV_SECTOR_SIZE); bdrv_dirty_bitmap_unlock(bmds->dirty_bitmap); blk = g_new(BlkMigBlock, 1); From patchwork Mon Sep 25 14:55:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818293 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16sp3lJtz9t67 for ; Tue, 26 Sep 2017 01:08:30 +1000 (AEST) Received: from localhost ([::1]:42893 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUzk-0000eR-Jl for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:08:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54616) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUoj-0006hU-Ed for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUoi-0003iT-Kc for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51920) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoe-0003gU-Nq; Mon, 25 Sep 2017 10:57:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AA6C325C28; Mon, 25 Sep 2017 14:56:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com AA6C325C28 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id D4E3B779CF; Mon, 25 Sep 2017 14:56:53 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:21 -0500 Message-Id: <20170925145526.32690-16-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 25 Sep 2017 14:56:59 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 15/20] mirror: Switch mirror_dirty_init() to byte-based iteration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Jeff Cody , Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v6: no change v5: rebase to earlier changes v2-v4: no change --- block/mirror.c | 38 ++++++++++++++------------------------ 1 file changed, 14 insertions(+), 24 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index 5c0f79f56a..459b80f8f3 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -612,15 +612,13 @@ static void mirror_throttle(MirrorBlockJob *s) static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) { - int64_t sector_num, end; + int64_t offset; BlockDriverState *base = s->base; BlockDriverState *bs = s->source; BlockDriverState *target_bs = blk_bs(s->target); - int ret, n; + int ret; int64_t count; - end = s->bdev_length / BDRV_SECTOR_SIZE; - if (base == NULL && !bdrv_has_zero_init(target_bs)) { if (!bdrv_can_write_zeroes_with_unmap(target_bs)) { bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, s->bdev_length); @@ -628,9 +626,9 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) } s->initial_zeroing_ongoing = true; - for (sector_num = 0; sector_num < end; ) { - int nb_sectors = MIN(end - sector_num, - QEMU_ALIGN_DOWN(INT_MAX, s->granularity) >> BDRV_SECTOR_BITS); + for (offset = 0; offset < s->bdev_length; ) { + int bytes = MIN(s->bdev_length - offset, + QEMU_ALIGN_DOWN(INT_MAX, s->granularity)); mirror_throttle(s); @@ -646,9 +644,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) continue; } - mirror_do_zero_or_discard(s, sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, false); - sector_num += nb_sectors; + mirror_do_zero_or_discard(s, offset, bytes, false); + offset += bytes; } mirror_wait_for_all_io(s); @@ -656,10 +653,10 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) } /* First part, loop on the sectors and initialize the dirty bitmap. */ - for (sector_num = 0; sector_num < end; ) { + for (offset = 0; offset < s->bdev_length; ) { /* Just to make sure we are not exceeding int limit. */ - int nb_sectors = MIN(INT_MAX >> BDRV_SECTOR_BITS, - end - sector_num); + int bytes = MIN(s->bdev_length - offset, + QEMU_ALIGN_DOWN(INT_MAX, s->granularity)); mirror_throttle(s); @@ -667,23 +664,16 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s) return 0; } - ret = bdrv_is_allocated_above(bs, base, sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, &count); + ret = bdrv_is_allocated_above(bs, base, offset, bytes, &count); if (ret < 0) { return ret; } - /* TODO: Relax this once bdrv_is_allocated_above and dirty - * bitmaps no longer require sector alignment. */ - assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); - n = count >> BDRV_SECTOR_BITS; - assert(n > 0); + assert(count); if (ret == 1) { - bdrv_set_dirty_bitmap(s->dirty_bitmap, - sector_num * BDRV_SECTOR_SIZE, - n * BDRV_SECTOR_SIZE); + bdrv_set_dirty_bitmap(s->dirty_bitmap, offset, count); } - sector_num += n; + offset += count; } return 0; } From patchwork Mon Sep 25 14:55:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818286 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16lL5lG6z9t67 for ; Tue, 26 Sep 2017 01:02:54 +1000 (AEST) Received: from localhost ([::1]:42870 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUuK-0004FB-PS for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:02:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54671) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUos-0006qS-Ck for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUor-0003mS-CF for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40488) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUoh-0003hQ-8c; Mon, 25 Sep 2017 10:57:03 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 36FEB4E4C3; Mon, 25 Sep 2017 14:57:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 36FEB4E4C3 Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC9077053E; Mon, 25 Sep 2017 14:56:59 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:22 -0500 Message-Id: <20170925145526.32690-17-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Mon, 25 Sep 2017 14:57:02 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 16/20] qcow2: Switch qcow2_measure() to byte-based iteration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is new code, but it is easier to read if it makes passes over the image using bytes rather than sectors (and will get easier in the future when bdrv_get_block_status is converted to byte-based). Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v7: tweak constant given to MIN (no semantic change, R-b kept) [Kevin] v6: separate bug fix to earlier patch v5: new patch --- block/qcow2.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index d33fb3ecdd..4cb3189ae5 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -3648,20 +3648,19 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, */ required = virtual_size; } else { - int cluster_sectors = cluster_size / BDRV_SECTOR_SIZE; - int64_t sector_num; + int64_t offset; int pnum = 0; - for (sector_num = 0; - sector_num < ssize / BDRV_SECTOR_SIZE; - sector_num += pnum) { - int nb_sectors = MIN(ssize / BDRV_SECTOR_SIZE - sector_num, - BDRV_REQUEST_MAX_SECTORS); + for (offset = 0; offset < ssize; + offset += pnum * BDRV_SECTOR_SIZE) { + int nb_sectors = MIN(ssize - offset, + BDRV_REQUEST_MAX_BYTES) / BDRV_SECTOR_SIZE; BlockDriverState *file; int64_t ret; ret = bdrv_get_block_status_above(in_bs, NULL, - sector_num, nb_sectors, + offset >> BDRV_SECTOR_BITS, + nb_sectors, &pnum, &file); if (ret < 0) { error_setg_errno(&local_err, -ret, @@ -3674,12 +3673,11 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, } else if ((ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) == (BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) { /* Extend pnum to end of cluster for next iteration */ - pnum = ROUND_UP(sector_num + pnum, cluster_sectors) - - sector_num; + pnum = (ROUND_UP(offset + pnum * BDRV_SECTOR_SIZE, + cluster_size) - offset) >> BDRV_SECTOR_BITS; /* Count clusters we've seen */ - required += (sector_num % cluster_sectors + pnum) * - BDRV_SECTOR_SIZE; + required += offset % cluster_size + pnum * BDRV_SECTOR_SIZE; } } } From patchwork Mon Sep 25 14:55:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818297 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16zL64fFz9tXC for ; Tue, 26 Sep 2017 01:13:18 +1000 (AEST) Received: from localhost ([::1]:42917 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV4O-0004P5-SM for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:13:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54775) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUp6-00071U-56 for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUp5-0003st-6h for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:28 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40552) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUow-0003oR-OB; Mon, 25 Sep 2017 10:57:18 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B6D2D5F7AF; Mon, 25 Sep 2017 14:57:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B6D2D5F7AF Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7A2507053E; Mon, 25 Sep 2017 14:57:02 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:23 -0500 Message-Id: <20170925145526.32690-18-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 25 Sep 2017 14:57:17 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 17/20] qcow2: Switch load_bitmap_data() to byte-based iteration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v5: no change v4: new patch --- block/qcow2-bitmap.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index b09010b1d3..692ce0de88 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -291,9 +291,8 @@ static int load_bitmap_data(BlockDriverState *bs, { int ret = 0; BDRVQcow2State *s = bs->opaque; - uint64_t sector, limit, sbc; + uint64_t offset, limit; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); - uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); uint8_t *buf = NULL; uint64_t i, tab_size = size_to_clusters(s, @@ -305,32 +304,27 @@ static int load_bitmap_data(BlockDriverState *bs, buf = g_malloc(s->cluster_size); limit = bytes_covered_by_bitmap_cluster(s, bitmap); - sbc = limit >> BDRV_SECTOR_BITS; - for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) { - uint64_t count = MIN(bm_sectors - sector, sbc); + for (i = 0, offset = 0; i < tab_size; ++i, offset += limit) { + uint64_t count = MIN(bm_size - offset, limit); uint64_t entry = bitmap_table[i]; - uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK; + uint64_t data_offset = entry & BME_TABLE_ENTRY_OFFSET_MASK; assert(check_table_entry(entry, s->cluster_size) == 0); - if (offset == 0) { + if (data_offset == 0) { if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) { - bdrv_dirty_bitmap_deserialize_ones(bitmap, - sector * BDRV_SECTOR_SIZE, - count * BDRV_SECTOR_SIZE, + bdrv_dirty_bitmap_deserialize_ones(bitmap, offset, count, false); } else { /* No need to deserialize zeros because the dirty bitmap is * already cleared */ } } else { - ret = bdrv_pread(bs->file, offset, buf, s->cluster_size); + ret = bdrv_pread(bs->file, data_offset, buf, s->cluster_size); if (ret < 0) { goto finish; } - bdrv_dirty_bitmap_deserialize_part(bitmap, buf, - sector * BDRV_SECTOR_SIZE, - count * BDRV_SECTOR_SIZE, + bdrv_dirty_bitmap_deserialize_part(bitmap, buf, offset, count, false); } } From patchwork Mon Sep 25 14:55:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818290 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y16pd480Kz9t67 for ; Tue, 26 Sep 2017 01:05:45 +1000 (AEST) Received: from localhost ([::1]:42878 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUx5-0006zA-K0 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:05:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54853) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUpB-00076R-Be for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUpA-0003vm-Dp for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:33 -0400 Received: from mx1.redhat.com ([209.132.183.28]:13559) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUp2-0003qy-Q1; Mon, 25 Sep 2017 10:57:24 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BB4007EA91; Mon, 25 Sep 2017 14:57:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BB4007EA91 Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3585779D7; Mon, 25 Sep 2017 14:57:17 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:24 -0500 Message-Id: <20170925145526.32690-19-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 25 Sep 2017 14:57:23 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 18/20] qcow2: Switch store_bitmap_data() to byte-based iteration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. iotests 165 was rather weak - on a default 64k-cluster image, where bitmap granularity also defaults to 64k bytes, a single cluster of the bitmap table thus covers (64*1024*8) bits which each cover 64k bytes, or 32G of image space. But the test only uses a 1G image, so it cannot trigger any more than one loop of the code in store_bitmap_data(); and it was writing to the first cluster. In order to test that we are properly aligning which portions of the bitmap are being written to the file, we really want to test a case where the first dirty bit returned by bdrv_dirty_iter_next() is not aligned to the start of a cluster, which we can do by modifying the test to write data that doesn't happen to fall in the first cluster of the image. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: John Snow Reviewed-by: Fam Zheng --- v10: no change, add R-b v9: update iotests to show why aligning down is needed [Kevin], R-b dropped v8: no change v7: rebase to earlier change, make rounding of offset obvious (no semantic change, so R-b kept) [Kevin] v5-v6: no change v4: new patch --- block/qcow2-bitmap.c | 31 ++++++++++++++++--------------- tests/qemu-iotests/165 | 2 +- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 692ce0de88..df957c66d5 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -1072,10 +1072,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, { int ret; BDRVQcow2State *s = bs->opaque; - int64_t sector; - uint64_t limit, sbc; + int64_t offset; + uint64_t limit; uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); - uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE); const char *bm_name = bdrv_dirty_bitmap_name(bitmap); uint8_t *buf = NULL; BdrvDirtyBitmapIter *dbi; @@ -1100,18 +1099,22 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, dbi = bdrv_dirty_iter_new(bitmap); buf = g_malloc(s->cluster_size); limit = bytes_covered_by_bitmap_cluster(s, bitmap); - sbc = limit >> BDRV_SECTOR_BITS; assert(DIV_ROUND_UP(bm_size, limit) == tb_size); - while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) >= 0) { - uint64_t cluster = sector / sbc; + while ((offset = bdrv_dirty_iter_next(dbi)) >= 0) { + uint64_t cluster = offset / limit; uint64_t end, write_size; int64_t off; - sector = cluster * sbc; - end = MIN(bm_sectors, sector + sbc); - write_size = bdrv_dirty_bitmap_serialization_size(bitmap, - sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE); + /* + * We found the first dirty offset, but want to write out the + * entire cluster of the bitmap that includes that offset, + * including any leading zero bits. + */ + offset = QEMU_ALIGN_DOWN(offset, limit); + end = MIN(bm_size, offset + limit); + write_size = bdrv_dirty_bitmap_serialization_size(bitmap, offset, + end - offset); assert(write_size <= s->cluster_size); off = qcow2_alloc_clusters(bs, s->cluster_size); @@ -1123,9 +1126,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, } tb[cluster] = off; - bdrv_dirty_bitmap_serialize_part(bitmap, buf, - sector * BDRV_SECTOR_SIZE, - (end - sector) * BDRV_SECTOR_SIZE); + bdrv_dirty_bitmap_serialize_part(bitmap, buf, offset, end - offset); if (write_size < s->cluster_size) { memset(buf + write_size, 0, s->cluster_size - write_size); } @@ -1143,11 +1144,11 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, goto fail; } - if (end >= bm_sectors) { + if (end >= bm_size) { break; } - bdrv_set_dirty_iter(dbi, end * BDRV_SECTOR_SIZE); + bdrv_set_dirty_iter(dbi, end); } *bitmap_table_size = tb_size; diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165 index 74d7b79a0b..a3932db3de 100755 --- a/tests/qemu-iotests/165 +++ b/tests/qemu-iotests/165 @@ -27,7 +27,7 @@ disk = os.path.join(iotests.test_dir, 'disk') disk_size = 0x40000000 # 1G # regions for qemu_io: (start, count) in bytes -regions1 = ((0, 0x100000), +regions1 = ((0x0fff00, 0x10000), (0x200000, 0x100000)) regions2 = ((0x10000000, 0x20000), From patchwork Mon Sep 25 14:55:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818299 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y17203n0Jz9t67 for ; Tue, 26 Sep 2017 01:15:36 +1000 (AEST) Received: from localhost ([::1]:42932 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV6c-0006Qj-H9 for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:15:34 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54869) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUpC-00077Z-Jd for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUpB-0003wC-Fa for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:34 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48764) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUp8-0003u1-0i; Mon, 25 Sep 2017 10:57:30 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0796E883CA; Mon, 25 Sep 2017 14:57:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 0796E883CA Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FB6617F5F; Mon, 25 Sep 2017 14:57:23 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:25 -0500 Message-Id: <20170925145526.32690-20-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 25 Sep 2017 14:57:29 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 19/20] dirty-bitmap: Switch bdrv_set_dirty() to bytes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , Stefan Hajnoczi , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Both callers already had bytes available, but were scaling to sectors. Move the scaling to internal code. In the case of bdrv_aligned_pwritev(), we are now passing the exact offset rather than a rounded sector-aligned value, but that's okay as long as dirty bitmap widens start/bytes to granularity boundaries. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v4: only context changes v3: rebase to lock context changes, R-b kept v2: no change --- include/block/block_int.h | 2 +- block/io.c | 6 ++---- block/dirty-bitmap.c | 7 ++++--- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index 99abe2ce74..79366b94b5 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -1028,7 +1028,7 @@ void blk_dev_eject_request(BlockBackend *blk, bool force); bool blk_dev_is_tray_open(BlockBackend *blk); bool blk_dev_is_medium_locked(BlockBackend *blk); -void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, int64_t nr_sect); +void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes); bool bdrv_requests_pending(BlockDriverState *bs); void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out); diff --git a/block/io.c b/block/io.c index 4378ae4c7d..8a0cd8835a 100644 --- a/block/io.c +++ b/block/io.c @@ -1334,7 +1334,6 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child, bool waited; int ret; - int64_t start_sector = offset >> BDRV_SECTOR_BITS; int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); uint64_t bytes_remaining = bytes; int max_transfer; @@ -1409,7 +1408,7 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child, bdrv_debug_event(bs, BLKDBG_PWRITEV_DONE); atomic_inc(&bs->write_gen); - bdrv_set_dirty(bs, start_sector, end_sector - start_sector); + bdrv_set_dirty(bs, offset, bytes); stat64_max(&bs->wr_highest_offset, offset + bytes); @@ -2438,8 +2437,7 @@ int coroutine_fn bdrv_co_pdiscard(BlockDriverState *bs, int64_t offset, ret = 0; out: atomic_inc(&bs->write_gen); - bdrv_set_dirty(bs, req.offset >> BDRV_SECTOR_BITS, - req.bytes >> BDRV_SECTOR_BITS); + bdrv_set_dirty(bs, req.offset, req.bytes); tracked_request_end(&req); bdrv_dec_in_flight(bs); return ret; diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 117837b3cc..58a3f330a9 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -628,10 +628,10 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap) hbitmap_deserialize_finish(bitmap->bitmap); } -void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, - int64_t nr_sectors) +void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes) { BdrvDirtyBitmap *bitmap; + int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); if (QLIST_EMPTY(&bs->dirty_bitmaps)) { return; @@ -643,7 +643,8 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, continue; } assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors); + hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, + end_sector - (offset >> BDRV_SECTOR_BITS)); } bdrv_dirty_bitmaps_unlock(bs); } From patchwork Mon Sep 25 14:55:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 818300 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y172W3ZPtz9t67 for ; Tue, 26 Sep 2017 01:16:03 +1000 (AEST) Received: from localhost ([::1]:42938 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwV73-0006t2-Jj for incoming@patchwork.ozlabs.org; Mon, 25 Sep 2017 11:16:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwUpE-00079i-JD for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwUpD-0003x7-4d for qemu-devel@nongnu.org; Mon, 25 Sep 2017 10:57:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55163) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwUp9-0003vB-No; Mon, 25 Sep 2017 10:57:31 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A60AF806B2; Mon, 25 Sep 2017 14:57:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A60AF806B2 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=eblake@redhat.com Received: from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4ACC017F5F; Mon, 25 Sep 2017 14:57:29 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 25 Sep 2017 09:55:26 -0500 Message-Id: <20170925145526.32690-21-eblake@redhat.com> In-Reply-To: <20170925145526.32690-1-eblake@redhat.com> References: <20170925145526.32690-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 25 Sep 2017 14:57:30 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v10 20/20] dirty-bitmap: Convert internal hbitmap size/granularity X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com, qemu-block@nongnu.org, Max Reitz , jsnow@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Now that all callers are using byte-based interfaces, there's no reason for our internal hbitmap to remain with sector-based granularity. It also simplifies our internal scaling, since we already know that hbitmap widens requests out to granularity boundaries. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Kevin Wolf Reviewed-by: Fam Zheng --- v9: no change v8: rebase to earlier truncate changes (R-b kept) v7: rebase to dirty_iter_next cleanup (no semantic change, R-b kept) v6: no change v5: fix bdrv_dirty_bitmap_truncate [John] v4: rebase to earlier changes, include serialization, R-b dropped v3: no change v2: no change --- block/dirty-bitmap.c | 62 +++++++++++++++------------------------------------- 1 file changed, 18 insertions(+), 44 deletions(-) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 58a3f330a9..bd04e991b1 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -38,7 +38,7 @@ */ struct BdrvDirtyBitmap { QemuMutex *mutex; - HBitmap *bitmap; /* Dirty sector bitmap implementation */ + HBitmap *bitmap; /* Dirty bitmap implementation */ HBitmap *meta; /* Meta dirty bitmap */ BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */ char *name; /* Optional non-empty unique ID */ @@ -130,12 +130,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, } bitmap = g_new0(BdrvDirtyBitmap, 1); bitmap->mutex = &bs->dirty_bitmap_mutex; - /* - * TODO - let hbitmap track full granularity. For now, it is tracking - * only sector granularity, as a shortcut for our iterators. - */ - bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap_size, BDRV_SECTOR_SIZE), - ctz32(granularity) - BDRV_SECTOR_BITS); + bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(granularity)); bitmap->size = bitmap_size; bitmap->name = g_strdup(name); bitmap->disabled = false; @@ -312,7 +307,7 @@ void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes) QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) { assert(!bdrv_dirty_bitmap_frozen(bitmap)); assert(!bitmap->active_iterators); - hbitmap_truncate(bitmap->bitmap, DIV_ROUND_UP(bytes, BDRV_SECTOR_SIZE)); + hbitmap_truncate(bitmap->bitmap, bytes); bitmap->size = bytes; } bdrv_dirty_bitmaps_unlock(bs); @@ -442,7 +437,7 @@ bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t offset) { if (bitmap) { - return hbitmap_get(bitmap->bitmap, offset >> BDRV_SECTOR_BITS); + return hbitmap_get(bitmap->bitmap, offset); } else { return false; } @@ -470,7 +465,7 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs) uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap) { - return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap); + return 1U << hbitmap_granularity(bitmap->bitmap); } BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap) @@ -503,20 +498,16 @@ void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter) int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter) { - int64_t ret = hbitmap_iter_next(&iter->hbi); - return ret < 0 ? -1 : ret * BDRV_SECTOR_SIZE; + return hbitmap_iter_next(&iter->hbi); } /* Called within bdrv_dirty_bitmap_lock..unlock */ void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, int64_t offset, int64_t bytes) { - int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); - assert(bdrv_dirty_bitmap_enabled(bitmap)); assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, - end_sector - (offset >> BDRV_SECTOR_BITS)); + hbitmap_set(bitmap->bitmap, offset, bytes); } void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, @@ -531,12 +522,9 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap, void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap, int64_t offset, int64_t bytes) { - int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); - assert(bdrv_dirty_bitmap_enabled(bitmap)); assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_reset(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, - end_sector - (offset >> BDRV_SECTOR_BITS)); + hbitmap_reset(bitmap->bitmap, offset, bytes); } void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap, @@ -556,8 +544,7 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out) hbitmap_reset_all(bitmap->bitmap); } else { HBitmap *backup = bitmap->bitmap; - bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap->size, - BDRV_SECTOR_SIZE), + bitmap->bitmap = hbitmap_alloc(bitmap->size, hbitmap_granularity(backup)); *out = backup; } @@ -576,51 +563,40 @@ void bdrv_undo_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *in) uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap, uint64_t offset, uint64_t bytes) { - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - return hbitmap_serialization_size(bitmap->bitmap, - offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS); + return hbitmap_serialization_size(bitmap->bitmap, offset, bytes); } uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap) { - return hbitmap_serialization_align(bitmap->bitmap) * BDRV_SECTOR_SIZE; + return hbitmap_serialization_align(bitmap->bitmap); } void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap, uint8_t *buf, uint64_t offset, uint64_t bytes) { - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - hbitmap_serialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS); + hbitmap_serialize_part(bitmap->bitmap, buf, offset, bytes); } void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap, uint8_t *buf, uint64_t offset, uint64_t bytes, bool finish) { - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - hbitmap_deserialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, finish); + hbitmap_deserialize_part(bitmap->bitmap, buf, offset, bytes, finish); } void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap, uint64_t offset, uint64_t bytes, bool finish) { - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - hbitmap_deserialize_zeroes(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, finish); + hbitmap_deserialize_zeroes(bitmap->bitmap, offset, bytes, finish); } void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap, uint64_t offset, uint64_t bytes, bool finish) { - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - hbitmap_deserialize_ones(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, finish); + hbitmap_deserialize_ones(bitmap->bitmap, offset, bytes, finish); } void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap) @@ -631,7 +607,6 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap) void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes) { BdrvDirtyBitmap *bitmap; - int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE); if (QLIST_EMPTY(&bs->dirty_bitmaps)) { return; @@ -643,8 +618,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes) continue; } assert(!bdrv_dirty_bitmap_readonly(bitmap)); - hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS, - end_sector - (offset >> BDRV_SECTOR_BITS)); + hbitmap_set(bitmap->bitmap, offset, bytes); } bdrv_dirty_bitmaps_unlock(bs); } @@ -654,12 +628,12 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes) */ void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset) { - hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset >> BDRV_SECTOR_BITS); + hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset); } int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap) { - return hbitmap_count(bitmap->bitmap) << BDRV_SECTOR_BITS; + return hbitmap_count(bitmap->bitmap); } int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)