[{"id":1776613,"web_url":"http://patchwork.ozlabs.org/comment/1776613/","msgid":"<65959795-4b92-a744-fab7-9f214ebdb13d@redhat.com>","list_archive_url":null,"date":"2017-09-27T20:54:55","subject":"Re: [Qemu-devel] [PATCH v4 14/23] qemu-img: Speed up compare on\n\tpre-allocated larger file","submitter":{"id":64343,"url":"http://patchwork.ozlabs.org/api/people/64343/","name":"John Snow","email":"jsnow@redhat.com"},"content":"On 09/13/2017 12:03 PM, Eric Blake wrote:\n> Compare the following images with all-zero contents:\n> $ truncate --size 1M A\n> $ qemu-img create -f qcow2 -o preallocation=off B 1G\n> $ qemu-img create -f qcow2 -o preallocation=metadata C 1G\n> \n> On my machine, the difference is noticeable for pre-patch speeds,\n> with more than an order of magnitude in difference caused by the\n> choice of preallocation in the qcow2 file:\n> \n> $ time ./qemu-img compare -f raw -F qcow2 A B\n> Warning: Image size mismatch!\n> Images are identical.\n> \n> real\t0m0.014s\n> user\t0m0.007s\n> sys\t0m0.007s\n> \n> $ time ./qemu-img compare -f raw -F qcow2 A C\n> Warning: Image size mismatch!\n> Images are identical.\n> \n> real\t0m0.341s\n> user\t0m0.144s\n> sys\t0m0.188s\n> \n> Why? Because bdrv_is_allocated() returns false for image B but\n> true for image C, throwing away the fact that both images know\n> via lseek(SEEK_HOLE) that the entire image still reads as zero.\n> From there, qemu-img ends up calling bdrv_pread() for every byte\n> of the tail, instead of quickly looking for the next allocation.\n> The solution: use block_status instead of is_allocated, giving:\n> \n> $ time ./qemu-img compare -f raw -F qcow2 A C\n> Warning: Image size mismatch!\n> Images are identical.\n> \n> real\t0m0.014s\n> user\t0m0.011s\n> sys\t0m0.003s\n> \n> which is on par with the speeds for no pre-allocation.\n> \n> Signed-off-by: Eric Blake \n\nMakes good sense to me.\n\nReviewed-by: John Snow ","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=jsnow@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3y2VTV1TVkz9t6B\n\tfor ;\n\tThu, 28 Sep 2017 06:55:41 +1000 (AEST)","from localhost ([::1]:56313 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dxJMo-0006aY-Ua\n\tfor incoming@patchwork.ozlabs.org; Wed, 27 Sep 2017 16:55:38 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:35067)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dxJML-0006Z6-To\n\tfor qemu-devel@nongnu.org; Wed, 27 Sep 2017 16:55:10 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dxJML-0003G2-43\n\tfor qemu-devel@nongnu.org; Wed, 27 Sep 2017 16:55:10 -0400","from mx1.redhat.com ([209.132.183.28]:40012)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1dxJME-0003DA-AP; Wed, 27 Sep 2017 16:55:02 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 9657732DDE4;\n\tWed, 27 Sep 2017 20:55:00 +0000 (UTC)","from [10.10.121.69] (ovpn-121-69.rdu2.redhat.com [10.10.121.69])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id C0A848F3C2;\n\tWed, 27 Sep 2017 20:54:55 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 9657732DDE4","To":"Eric Blake , qemu-devel@nongnu.org","References":"<20170913160333.23622-1-eblake@redhat.com>\n\t<20170913160333.23622-15-eblake@redhat.com>","From":"John Snow ","Message-ID":"<65959795-4b92-a744-fab7-9f214ebdb13d@redhat.com>","Date":"Wed, 27 Sep 2017 16:54:55 -0400","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170913160333.23622-15-eblake@redhat.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-US","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.38]);\n\tWed, 27 Sep 2017 20:55:00 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [PATCH v4 14/23] qemu-img: Speed up compare on\n\tpre-allocated larger file","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"kwolf@redhat.com, famz@redhat.com, qemu-block@nongnu.org,\n\tMax Reitz ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1778805,"web_url":"http://patchwork.ozlabs.org/comment/1778805/","msgid":"","list_archive_url":null,"date":"2017-10-03T09:32:14","subject":"Re: [Qemu-devel] [PATCH v4 14/23] qemu-img: Speed up compare on\n\tpre-allocated larger file","submitter":{"id":66592,"url":"http://patchwork.ozlabs.org/api/people/66592/","name":"Vladimir Sementsov-Ogievskiy","email":"vsementsov@virtuozzo.com"},"content":"13.09.2017 19:03, Eric Blake wrote:\n> Compare the following images with all-zero contents:\n> $ truncate --size 1M A\n> $ qemu-img create -f qcow2 -o preallocation=off B 1G\n> $ qemu-img create -f qcow2 -o preallocation=metadata C 1G\n>\n> On my machine, the difference is noticeable for pre-patch speeds,\n> with more than an order of magnitude in difference caused by the\n> choice of preallocation in the qcow2 file:\n>\n> $ time ./qemu-img compare -f raw -F qcow2 A B\n> Warning: Image size mismatch!\n> Images are identical.\n>\n> real\t0m0.014s\n> user\t0m0.007s\n> sys\t0m0.007s\n>\n> $ time ./qemu-img compare -f raw -F qcow2 A C\n> Warning: Image size mismatch!\n> Images are identical.\n>\n> real\t0m0.341s\n> user\t0m0.144s\n> sys\t0m0.188s\n>\n> Why? Because bdrv_is_allocated() returns false for image B but\n> true for image C, throwing away the fact that both images know\n> via lseek(SEEK_HOLE) that the entire image still reads as zero.\n> From there, qemu-img ends up calling bdrv_pread() for every byte\n> of the tail, instead of quickly looking for the next allocation.\n> The solution: use block_status instead of is_allocated, giving:\n>\n> $ time ./qemu-img compare -f raw -F qcow2 A C\n> Warning: Image size mismatch!\n> Images are identical.\n>\n> real\t0m0.014s\n> user\t0m0.011s\n> sys\t0m0.003s\n>\n> which is on par with the speeds for no pre-allocation.\n>\n> Signed-off-by: Eric Blake \n>\n> ---\n> v3: new patch\n> ---\n> qemu-img.c | 8 ++++----\n> 1 file changed, 4 insertions(+), 4 deletions(-)\n>\n> diff --git a/qemu-img.c b/qemu-img.c\n> index f8423e9b3f..f5ab29d176 100644\n> --- a/qemu-img.c\n> +++ b/qemu-img.c\n> @@ -1477,11 +1477,11 @@ static int img_compare(int argc, char **argv)\n> while (sector_num < progress_base) {\n> int64_t count;\n>\n> - ret = bdrv_is_allocated_above(blk_bs(blk_over), NULL,\n> + ret = bdrv_block_status_above(blk_bs(blk_over), NULL,\n> sector_num * BDRV_SECTOR_SIZE,\n> (progress_base - sector_num) *\n> BDRV_SECTOR_SIZE,\n> - &count);\n> + &count, NULL);\n> if (ret < 0) {\n> ret = 3;\n> error_report(\"Sector allocation test failed for %s\",\n> @@ -1489,11 +1489,11 @@ static int img_compare(int argc, char **argv)\n> goto out;\n>\n> }\n> - /* TODO relax this once bdrv_is_allocated_above does not enforce\n> + /* TODO relax this once bdrv_block_status_above does not enforce\n> * sector alignment */\n> assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE));\n> nb_sectors = count >> BDRV_SECTOR_BITS;\n> - if (ret) {\n> + if (ret & BDRV_BLOCK_ALLOCATED && !(ret & BDRV_BLOCK_ZERO)) {\n> nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS);\n> ret = check_empty_sectors(blk_over, sector_num, nb_sectors,\n> filename_over, buf1, quiet);\n\nReviewed-by: Vladimir Sementsov-Ogievskiy ","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3y5v2q2tWwz9t44\n\tfor ;\n\tTue, 3 Oct 2017 20:32:51 +1100 (AEDT)","from localhost ([::1]:57333 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dzJZJ-0004ww-FU\n\tfor incoming@patchwork.ozlabs.org; Tue, 03 Oct 2017 05:32:49 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:47839)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dzJYw-0004wp-QX\n\tfor qemu-devel@nongnu.org; Tue, 03 Oct 2017 05:32:31 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dzJYt-0002GH-Jr\n\tfor qemu-devel@nongnu.org; Tue, 03 Oct 2017 05:32:26 -0400","from mailhub.sw.ru ([195.214.232.25]:10062 helo=relay.sw.ru)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1dzJYt-00028M-2x\n\tfor qemu-devel@nongnu.org; Tue, 03 Oct 2017 05:32:23 -0400","from [172.16.24.243] (msk-vpn.virtuozzo.com [195.214.232.6])\n\tby relay.sw.ru (8.13.4/8.13.4) with ESMTP id v939WEKS003557;\n\tTue, 3 Oct 2017 12:32:14 +0300 (MSK)"],"To":"Eric Blake , qemu-devel@nongnu.org","References":"<20170913160333.23622-1-eblake@redhat.com>\n\t<20170913160333.23622-15-eblake@redhat.com>","From":"Vladimir Sementsov-Ogievskiy ","Message-ID":"","Date":"Tue, 3 Oct 2017 12:32:14 +0300","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170913160333.23622-15-eblake@redhat.com>","Content-Language":"en-US","X-detected-operating-system":"by eggs.gnu.org: OpenBSD 3.x [fuzzy]","X-Received-From":"195.214.232.25","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"7bit","X-Content-Filtered-By":"Mailman/MimeDel 2.1.21","Subject":"Re: [Qemu-devel] [PATCH v4 14/23] qemu-img: Speed up compare on\n\tpre-allocated larger file","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"kwolf@redhat.com, famz@redhat.com, jsnow@redhat.com,\n\tqemu-block@nongnu.org, Max Reitz ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]