{"id":815835,"url":"http://patchwork.ozlabs.org/api/patches/815835/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/patch/20170919201910.25656-19-eblake@redhat.com/","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":"","list_archive_url":"","list_archive_url_format":"","commit_url_format":""},"msgid":"<20170919201910.25656-19-eblake@redhat.com>","list_archive_url":null,"date":"2017-09-19T20:19:08","name":"[v9,18/20] qcow2: Switch store_bitmap_data() to byte-based iteration","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"f4664c66106ff810a8de8b25bb86ddcc540a72ca","submitter":{"id":6591,"url":"http://patchwork.ozlabs.org/api/people/6591/?format=json","name":"Eric Blake","email":"eblake@redhat.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/patch/20170919201910.25656-19-eblake@redhat.com/mbox/","series":[{"id":3964,"url":"http://patchwork.ozlabs.org/api/series/3964/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/list/?series=3964","date":"2017-09-19T20:18:54","name":"make dirty-bitmap byte-based","version":9,"mbox":"http://patchwork.ozlabs.org/series/3964/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/815835/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/815835/checks/","tags":{},"related":[],"headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=eblake@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xxZMr2D45z9s81\n\tfor ;\n\tWed, 20 Sep 2017 06:33:44 +1000 (AEST)","from localhost ([::1]:45277 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1duPDC-0000fq-An\n\tfor incoming@patchwork.ozlabs.org; Tue, 19 Sep 2017 16:33:42 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:43950)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1duP1D-0006rF-RW\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 16:21:21 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1duP1C-0005t4-RQ\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 16:21:19 -0400","from mx1.redhat.com ([209.132.183.28]:52042)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1duP19-0005q8-So; Tue, 19 Sep 2017 16:21:16 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id F0831C0578F3;\n\tTue, 19 Sep 2017 20:21:14 +0000 (UTC)","from red.redhat.com (ovpn-124-97.rdu2.redhat.com [10.10.124.97])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 8B3AD17518;\n\tTue, 19 Sep 2017 20:21:13 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com F0831C0578F3","From":"Eric Blake ","To":"qemu-devel@nongnu.org","Date":"Tue, 19 Sep 2017 15:19:08 -0500","Message-Id":"<20170919201910.25656-19-eblake@redhat.com>","In-Reply-To":"<20170919201910.25656-1-eblake@redhat.com>","References":"<20170919201910.25656-1-eblake@redhat.com>","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tTue, 19 Sep 2017 20:21:15 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"[Qemu-devel] [PATCH v9 18/20] qcow2: Switch store_bitmap_data() to\n\tbyte-based iteration","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"kwolf@redhat.com, vsementsov@virtuozzo.com, famz@redhat.com,\n\tqemu-block@nongnu.org, Max Reitz , jsnow@redhat.com","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"},"content":"Now that we have adjusted the majority of the calls this function\nmakes to be byte-based, it is easier to read the code if it makes\npasses over the image using bytes rather than sectors.\n\niotests 165 was rather weak - on a default 64k-cluster image, where\nbitmap granularity also defaults to 64k bytes, a single cluster of\nthe bitmap table thus covers (64*1024*8) bits which each cover 64k\nbytes, or 32G of image space. But the test only uses a 1G image,\nso it cannot trigger any more than one loop of the code in\nstore_bitmap_data(); and it was writing to the first cluster. In\norder to test that we are properly aligning which portions of the\nbitmap are being written to the file, we really want to test a case\nwhere the first dirty bit returned by bdrv_dirty_iter_next() is not\naligned to the start of a cluster, which we can do by modifying the\ntest to write data that doesn't happen to fall in the first cluster\nof the image.\n\nSigned-off-by: Eric Blake \n\n---\nv9: update iotests to show why aligning down is needed [Kevin], R-b dropped\nv8: no change\nv7: rebase to earlier change, make rounding of offset obvious (no semantic\nchange, so R-b kept) [Kevin]\nv5-v6: no change\nv4: new patch\n---\n block/qcow2-bitmap.c | 31 ++++++++++++++++---------------\n tests/qemu-iotests/165 | 2 +-\n 2 files changed, 17 insertions(+), 16 deletions(-)","diff":"diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c\nindex 692ce0de88..df957c66d5 100644\n--- a/block/qcow2-bitmap.c\n+++ b/block/qcow2-bitmap.c\n@@ -1072,10 +1072,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,\n {\n int ret;\n BDRVQcow2State *s = bs->opaque;\n- int64_t sector;\n- uint64_t limit, sbc;\n+ int64_t offset;\n+ uint64_t limit;\n uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);\n- uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);\n const char *bm_name = bdrv_dirty_bitmap_name(bitmap);\n uint8_t *buf = NULL;\n BdrvDirtyBitmapIter *dbi;\n@@ -1100,18 +1099,22 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,\n dbi = bdrv_dirty_iter_new(bitmap);\n buf = g_malloc(s->cluster_size);\n limit = bytes_covered_by_bitmap_cluster(s, bitmap);\n- sbc = limit >> BDRV_SECTOR_BITS;\n assert(DIV_ROUND_UP(bm_size, limit) == tb_size);\n\n- while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) >= 0) {\n- uint64_t cluster = sector / sbc;\n+ while ((offset = bdrv_dirty_iter_next(dbi)) >= 0) {\n+ uint64_t cluster = offset / limit;\n uint64_t end, write_size;\n int64_t off;\n\n- sector = cluster * sbc;\n- end = MIN(bm_sectors, sector + sbc);\n- write_size = bdrv_dirty_bitmap_serialization_size(bitmap,\n- sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE);\n+ /*\n+ * We found the first dirty offset, but want to write out the\n+ * entire cluster of the bitmap that includes that offset,\n+ * including any leading zero bits.\n+ */\n+ offset = QEMU_ALIGN_DOWN(offset, limit);\n+ end = MIN(bm_size, offset + limit);\n+ write_size = bdrv_dirty_bitmap_serialization_size(bitmap, offset,\n+ end - offset);\n assert(write_size <= s->cluster_size);\n\n off = qcow2_alloc_clusters(bs, s->cluster_size);\n@@ -1123,9 +1126,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,\n }\n tb[cluster] = off;\n\n- bdrv_dirty_bitmap_serialize_part(bitmap, buf,\n- sector * BDRV_SECTOR_SIZE,\n- (end - sector) * BDRV_SECTOR_SIZE);\n+ bdrv_dirty_bitmap_serialize_part(bitmap, buf, offset, end - offset);\n if (write_size < s->cluster_size) {\n memset(buf + write_size, 0, s->cluster_size - write_size);\n }\n@@ -1143,11 +1144,11 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,\n goto fail;\n }\n\n- if (end >= bm_sectors) {\n+ if (end >= bm_size) {\n break;\n }\n\n- bdrv_set_dirty_iter(dbi, end * BDRV_SECTOR_SIZE);\n+ bdrv_set_dirty_iter(dbi, end);\n }\n\n *bitmap_table_size = tb_size;\ndiff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165\nindex 74d7b79a0b..a3932db3de 100755\n--- a/tests/qemu-iotests/165\n+++ b/tests/qemu-iotests/165\n@@ -27,7 +27,7 @@ disk = os.path.join(iotests.test_dir, 'disk')\n disk_size = 0x40000000 # 1G\n\n # regions for qemu_io: (start, count) in bytes\n-regions1 = ((0, 0x100000),\n+regions1 = ((0x0fff00, 0x10000),\n (0x200000, 0x100000))\n\n regions2 = ((0x10000000, 0x20000),\n","prefixes":["v9","18/20"]}