From patchwork Tue Jun 9 20:52:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306275 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=DrCrXYac; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMlq1sWwz9sRh for ; Wed, 10 Jun 2020 06:53:27 +1000 (AEST) Received: from localhost ([::1]:60914 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilFM-0001cy-9V for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:53:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36314) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEr-0001cp-Ed for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:53 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:34445 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEq-0005at-Hw for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qQhZDJFX67O4q0iXndGpODGJi7zttnSdTmpjsoZHdsw=; b=DrCrXYac8o5IgXq5On37l+SCN9jwZmlXpT4VENLIrgb7ru5J+F7yKPCktEibchWl5zkA6U F8gey2ILlHIvXaJyodyI1HHaC9cq8XW0szZmj9aH0KLC0brWE21QOICuy4vLOvDdVZO99C fUBASyFiDSQFsSweodQtYw/aMRFAJUw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-70-oheg6cYTPQGvvq3ZT8Vq8Q-1; Tue, 09 Jun 2020 16:52:48 -0400 X-MC-Unique: oheg6cYTPQGvvq3ZT8Vq8Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E991219200C8; Tue, 9 Jun 2020 20:52:47 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA1F460C80; Tue, 9 Jun 2020 20:52:47 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand Date: Tue, 9 Jun 2020 15:52:31 -0500 Message-Id: <20200609205245.3548257-2-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Vladimir Sementsov-Ogievskiy Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Prefer a consistent naming for the --merge argument. Fixes: 3b51ab4bf Signed-off-by: Eric Blake Message-Id: <20200529144527.1943527-1-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- docs/tools/qemu-img.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 69cd9a30373a..7f0737488ade 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -300,7 +300,7 @@ Command description: ``--disable`` to change *BITMAP* to stop recording future edits. - ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*. + ``--merge`` to merge the contents of the *SOURCE* bitmap into *BITMAP*. Additional options include ``-g`` which sets a non-default *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an From patchwork Tue Jun 9 20:52:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306282 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=eDhckG5j; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMp934gfz9sSJ for ; Wed, 10 Jun 2020 06:55:29 +1000 (AEST) Received: from localhost ([::1]:41038 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilHL-00058q-35 for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:55:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36368) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEw-0001ip-ST for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:58 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:57825 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEu-0005bR-09 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bLALEe20suyTXRiWgricSQvXLZdaHyPd0f+EDscknEY=; b=eDhckG5jfzi3F+fpiJ8s57ADW5iuxf6eFPjkTGCw9dzh/bjkYX68mX7ius6PYgWIEWMHce ROURyucXV3NVvDcLoLC460i7XIkoyxlQQTH3BsrwzaJDBVqJ6fPoJmPEhOlFrJIK/URUvA bT35eUyu2NLqFji1kf2IHXxdRFhtpxg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-89-ymBcn-lKOniPRaLqbrJSKQ-1; Tue, 09 Jun 2020 16:52:49 -0400 X-MC-Unique: ymBcn-lKOniPRaLqbrJSKQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 93C44107ACF9; Tue, 9 Jun 2020 20:52:48 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 257A160C80; Tue, 9 Jun 2020 20:52:48 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 02/15] qcow2.py: python style fixes Date: Tue, 9 Jun 2020 15:52:32 -0500 Message-Id: <20200609205245.3548257-3-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 16:46:48 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Fix flake8 complaints. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200606081806.23897-2-vsementsov@virtuozzo.com> Tested-by: Eric Blake Reviewed-by: Eric Blake [eblake: commit message improved] Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2.py | 95 ++++++++++++++++++++++--------------- 1 file changed, 56 insertions(+), 39 deletions(-) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index 94a07b2f6fa2..d99f4ee3e848 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -4,6 +4,7 @@ import sys import struct import string + class QcowHeaderExtension: def __init__(self, magic, length, data): @@ -11,14 +12,15 @@ class QcowHeaderExtension: padding = 8 - (length % 8) data += b"\0" * padding - self.magic = magic + self.magic = magic self.length = length - self.data = data + self.data = data @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) + class QcowHeader: uint32_t = 'I' @@ -26,27 +28,27 @@ class QcowHeader: fields = [ # Version 2 header fields - [ uint32_t, '%#x', 'magic' ], - [ uint32_t, '%d', 'version' ], - [ uint64_t, '%#x', 'backing_file_offset' ], - [ uint32_t, '%#x', 'backing_file_size' ], - [ uint32_t, '%d', 'cluster_bits' ], - [ uint64_t, '%d', 'size' ], - [ uint32_t, '%d', 'crypt_method' ], - [ uint32_t, '%d', 'l1_size' ], - [ uint64_t, '%#x', 'l1_table_offset' ], - [ uint64_t, '%#x', 'refcount_table_offset' ], - [ uint32_t, '%d', 'refcount_table_clusters' ], - [ uint32_t, '%d', 'nb_snapshots' ], - [ uint64_t, '%#x', 'snapshot_offset' ], + [uint32_t, '%#x', 'magic'], + [uint32_t, '%d', 'version'], + [uint64_t, '%#x', 'backing_file_offset'], + [uint32_t, '%#x', 'backing_file_size'], + [uint32_t, '%d', 'cluster_bits'], + [uint64_t, '%d', 'size'], + [uint32_t, '%d', 'crypt_method'], + [uint32_t, '%d', 'l1_size'], + [uint64_t, '%#x', 'l1_table_offset'], + [uint64_t, '%#x', 'refcount_table_offset'], + [uint32_t, '%d', 'refcount_table_clusters'], + [uint32_t, '%d', 'nb_snapshots'], + [uint64_t, '%#x', 'snapshot_offset'], # Version 3 header fields - [ uint64_t, 'mask', 'incompatible_features' ], - [ uint64_t, 'mask', 'compatible_features' ], - [ uint64_t, 'mask', 'autoclear_features' ], - [ uint32_t, '%d', 'refcount_order' ], - [ uint32_t, '%d', 'header_length' ], - ]; + [uint64_t, 'mask', 'incompatible_features'], + [uint64_t, 'mask', 'compatible_features'], + [uint64_t, 'mask', 'autoclear_features'], + [uint32_t, '%d', 'refcount_order'], + [uint32_t, '%d', 'header_length'], + ] fmt = '>' + ''.join(field[0] for field in fields) @@ -59,7 +61,7 @@ class QcowHeader: header = struct.unpack(QcowHeader.fmt, buf) self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) + for i, field in enumerate(QcowHeader.fields)) self.set_defaults() self.cluster_size = 1 << self.cluster_bits @@ -96,7 +98,8 @@ class QcowHeader: else: padded = (length + 7) & ~7 data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, data)) + self.extensions.append(QcowHeaderExtension(magic, length, + data)) def update_extensions(self, fd): @@ -108,14 +111,13 @@ class QcowHeader: fd.write(buf) fd.write(ex.data) - if self.backing_file != None: + if self.backing_file is not None: self.backing_file_offset = fd.tell() fd.write(self.backing_file) if fd.tell() > self.cluster_size: raise Exception("I think I just broke the image...") - def update(self, fd): header_bytes = self.header_length @@ -163,19 +165,21 @@ def cmd_dump_header(fd): h.dump() h.dump_extensions() + def cmd_dump_header_exts(fd): h = QcowHeader(fd) h.dump_extensions() + def cmd_set_header(fd, name, value): try: value = int(value, 0) - except: + except ValueError: print("'%s' is not a valid number" % value) sys.exit(1) fields = (field[2] for field in QcowHeader.fields) - if not name in fields: + if name not in fields: print("'%s' is not a known header field" % name) sys.exit(1) @@ -183,25 +187,29 @@ def cmd_set_header(fd, name, value): h.__dict__[name] = value h.update(fd) + def cmd_add_header_ext(fd, magic, data): try: magic = int(magic, 0) - except: + except ValueError: print("'%s' is not a valid magic number" % magic) sys.exit(1) h = QcowHeader(fd) - h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii'))) + h.extensions.append(QcowHeaderExtension.create(magic, + data.encode('ascii'))) h.update(fd) + def cmd_add_header_ext_stdio(fd, magic): data = sys.stdin.read() cmd_add_header_ext(fd, magic, data) + def cmd_del_header_ext(fd, magic): try: magic = int(magic, 0) - except: + except ValueError: print("'%s' is not a valid magic number" % magic) sys.exit(1) @@ -219,12 +227,13 @@ def cmd_del_header_ext(fd, magic): h.update(fd) + def cmd_set_feature_bit(fd, group, bit): try: bit = int(bit, 0) if bit < 0 or bit >= 64: raise ValueError - except: + except ValueError: print("'%s' is not a valid bit number in range [0, 64)" % bit) sys.exit(1) @@ -236,21 +245,27 @@ def cmd_set_feature_bit(fd, group, bit): elif group == 'autoclear': h.autoclear_features |= 1 << bit else: - print("'%s' is not a valid group, try 'incompatible', 'compatible', or 'autoclear'" % group) + print("'%s' is not a valid group, try " + "'incompatible', 'compatible', or 'autoclear'" % group) sys.exit(1) h.update(fd) + cmds = [ - [ 'dump-header', cmd_dump_header, 0, 'Dump image header and header extensions' ], - [ 'dump-header-exts', cmd_dump_header_exts, 0, 'Dump image header extensions' ], - [ 'set-header', cmd_set_header, 2, 'Set a field in the header'], - [ 'add-header-ext', cmd_add_header_ext, 2, 'Add a header extension' ], - [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ], - [ 'del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension' ], - [ 'set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'], + ['dump-header', cmd_dump_header, 0, + 'Dump image header and header extensions'], + ['dump-header-exts', cmd_dump_header_exts, 0, + 'Dump image header extensions'], + ['set-header', cmd_set_header, 2, 'Set a field in the header'], + ['add-header-ext', cmd_add_header_ext, 2, 'Add a header extension'], + ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1, + 'Add a header extension, data from stdin'], + ['del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension'], + ['set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'], ] + def main(filename, cmd, args): fd = open(filename, "r+b") try: @@ -267,6 +282,7 @@ def main(filename, cmd, args): finally: fd.close() + def usage(): print("Usage: %s [, ...]" % sys.argv[0]) print("") @@ -274,6 +290,7 @@ def usage(): for name, handler, num_args, desc in cmds: print(" %-20s - %s" % (name, desc)) + if __name__ == '__main__': if len(sys.argv) < 3: usage() From patchwork Tue Jun 9 20:52:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306277 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EK24ItYP; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMlv09R5z9sRh for ; Wed, 10 Jun 2020 06:53:31 +1000 (AEST) Received: from localhost ([::1]:32820 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilFQ-0001hw-L0 for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:53:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36336) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEu-0001eg-5g for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:56 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:58863 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEs-0005bA-Us for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735974; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LWiyzYXvX9jef3nlwu93X70gd/uz2nbcGCmCvQ5AeNY=; b=EK24ItYPdQbUYDs/38+Lgtwu+Ta9nVD3cUl976qFimeyxkDpfwGKsXeuhVRGmb4tK324Xc YhVDT3swusSmOQk3WnGRu/wsYkHGGmhS466HD4x96+I1JPYQ04OYVEf0QmS7xtvDH7xUfi /qE8okFSw1rbo9SdzZMj6l+HJVl6gIs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-61-NLma_j2ROYedU1PcWoUmog-1; Tue, 09 Jun 2020 16:52:50 -0400 X-MC-Unique: NLma_j2ROYedU1PcWoUmog-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3889919200C4; Tue, 9 Jun 2020 20:52:49 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id BF80C6116D; Tue, 9 Jun 2020 20:52:48 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 03/15] qcow2.py: add licensing blurb Date: Tue, 9 Jun 2020 15:52:33 -0500 Message-Id: <20200609205245.3548257-4-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 16:46:48 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add classic heading, which is missing here. Keep copyright place empty, prior authors may add a line later. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200606081806.23897-3-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake [eblake: tweak commit message] Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index d99f4ee3e848..2da434a013ef 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -1,4 +1,20 @@ #!/usr/bin/env python3 +# +# Manipulations with qcow2 image +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# import sys import struct From patchwork Tue Jun 9 20:52:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306283 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LrjoJdSj; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMpH0NwPz9sRh for ; Wed, 10 Jun 2020 06:55:35 +1000 (AEST) Received: from localhost ([::1]:41450 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilHQ-0005Im-MG for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:55:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36352) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEv-0001gX-O5 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:57 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:49864 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEu-0005ba-6c for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FC9Ai7LaHjc2u5ALIBTpHFl/XEM6Scc3lWiNTmCw0Yw=; b=LrjoJdSjczZ5ggnBcVqYvAXnC7B4ar5Y5VLeseu+PnNYGgTWgk+jI+9uJM4zgFJSd4tafw YFP+k6yP0F0e3StawdIDqN5tXAu3ianxqd7NtR06OfmTmiC+31eVqOAh+25/q7dpQKiF3T PKm6zoOlijGZLqOpSzDe/BNpAv/P2kk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-504-Vdep5_0aMqOIcpAaSxyXhQ-1; Tue, 09 Jun 2020 16:52:50 -0400 X-MC-Unique: Vdep5_0aMqOIcpAaSxyXhQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D8B7E1005510; Tue, 9 Jun 2020 20:52:49 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6B1466116D; Tue, 9 Jun 2020 20:52:49 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 04/15] qcow2.py: move qcow2 format classes to separate module Date: Tue, 9 Jun 2020 15:52:34 -0500 Message-Id: <20200609205245.3548257-5-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to enhance qcow2 format parsing by adding more structure classes. Let's split format parsing from utility code. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200606081806.23897-4-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2.py | 160 +------------------------- tests/qemu-iotests/qcow2_format.py | 173 +++++++++++++++++++++++++++++ 2 files changed, 177 insertions(+), 156 deletions(-) create mode 100644 tests/qemu-iotests/qcow2_format.py diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index 2da434a013ef..e968869ea6b2 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -17,163 +17,11 @@ # import sys -import struct -import string - -class QcowHeaderExtension: - - def __init__(self, magic, length, data): - if length % 8 != 0: - padding = 8 - (length % 8) - data += b"\0" * padding - - self.magic = magic - self.length = length - self.data = data - - @classmethod - def create(cls, magic, data): - return QcowHeaderExtension(magic, len(data), data) - - -class QcowHeader: - - uint32_t = 'I' - uint64_t = 'Q' - - fields = [ - # Version 2 header fields - [uint32_t, '%#x', 'magic'], - [uint32_t, '%d', 'version'], - [uint64_t, '%#x', 'backing_file_offset'], - [uint32_t, '%#x', 'backing_file_size'], - [uint32_t, '%d', 'cluster_bits'], - [uint64_t, '%d', 'size'], - [uint32_t, '%d', 'crypt_method'], - [uint32_t, '%d', 'l1_size'], - [uint64_t, '%#x', 'l1_table_offset'], - [uint64_t, '%#x', 'refcount_table_offset'], - [uint32_t, '%d', 'refcount_table_clusters'], - [uint32_t, '%d', 'nb_snapshots'], - [uint64_t, '%#x', 'snapshot_offset'], - - # Version 3 header fields - [uint64_t, 'mask', 'incompatible_features'], - [uint64_t, 'mask', 'compatible_features'], - [uint64_t, 'mask', 'autoclear_features'], - [uint32_t, '%d', 'refcount_order'], - [uint32_t, '%d', 'header_length'], - ] - - fmt = '>' + ''.join(field[0] for field in fields) - - def __init__(self, fd): - - buf_size = struct.calcsize(QcowHeader.fmt) - - fd.seek(0) - buf = fd.read(buf_size) - - header = struct.unpack(QcowHeader.fmt, buf) - self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) - - self.set_defaults() - self.cluster_size = 1 << self.cluster_bits - - fd.seek(self.header_length) - self.load_extensions(fd) - - if self.backing_file_offset: - fd.seek(self.backing_file_offset) - self.backing_file = fd.read(self.backing_file_size) - else: - self.backing_file = None - - def set_defaults(self): - if self.version == 2: - self.incompatible_features = 0 - self.compatible_features = 0 - self.autoclear_features = 0 - self.refcount_order = 4 - self.header_length = 72 - - def load_extensions(self, fd): - self.extensions = [] - - if self.backing_file_offset != 0: - end = min(self.cluster_size, self.backing_file_offset) - else: - end = self.cluster_size - - while fd.tell() < end: - (magic, length) = struct.unpack('>II', fd.read(8)) - if magic == 0: - break - else: - padded = (length + 7) & ~7 - data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, - data)) - - def update_extensions(self, fd): - - fd.seek(self.header_length) - extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, b"")) - for ex in extensions: - buf = struct.pack('>II', ex.magic, ex.length) - fd.write(buf) - fd.write(ex.data) - - if self.backing_file is not None: - self.backing_file_offset = fd.tell() - fd.write(self.backing_file) - - if fd.tell() > self.cluster_size: - raise Exception("I think I just broke the image...") - - def update(self, fd): - header_bytes = self.header_length - - self.update_extensions(fd) - - fd.seek(0) - header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields) - buf = struct.pack(QcowHeader.fmt, *header) - buf = buf[0:header_bytes-1] - fd.write(buf) - - def dump(self): - for f in QcowHeader.fields: - value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: - value_str = f[1] % value - - print("%-25s" % f[2], value_str) - print("") - - def dump_extensions(self): - for ex in self.extensions: - - data = ex.data[:ex.length] - if all(c in string.printable.encode('ascii') for c in data): - data = "'%s'" % data.decode('ascii') - else: - data = "" - - print("Header extension:") - print("%-25s %#x" % ("magic", ex.magic)) - print("%-25s %d" % ("length", ex.length)) - print("%-25s %s" % ("data", data)) - print("") +from qcow2_format import ( + QcowHeader, + QcowHeaderExtension +) def cmd_dump_header(fd): diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py new file mode 100644 index 000000000000..0e517c1bbc5a --- /dev/null +++ b/tests/qemu-iotests/qcow2_format.py @@ -0,0 +1,173 @@ +# Library for manipulations with qcow2 image +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import struct +import string + + +class QcowHeaderExtension: + + def __init__(self, magic, length, data): + if length % 8 != 0: + padding = 8 - (length % 8) + data += b"\0" * padding + + self.magic = magic + self.length = length + self.data = data + + @classmethod + def create(cls, magic, data): + return QcowHeaderExtension(magic, len(data), data) + + +class QcowHeader: + + uint32_t = 'I' + uint64_t = 'Q' + + fields = [ + # Version 2 header fields + [uint32_t, '%#x', 'magic'], + [uint32_t, '%d', 'version'], + [uint64_t, '%#x', 'backing_file_offset'], + [uint32_t, '%#x', 'backing_file_size'], + [uint32_t, '%d', 'cluster_bits'], + [uint64_t, '%d', 'size'], + [uint32_t, '%d', 'crypt_method'], + [uint32_t, '%d', 'l1_size'], + [uint64_t, '%#x', 'l1_table_offset'], + [uint64_t, '%#x', 'refcount_table_offset'], + [uint32_t, '%d', 'refcount_table_clusters'], + [uint32_t, '%d', 'nb_snapshots'], + [uint64_t, '%#x', 'snapshot_offset'], + + # Version 3 header fields + [uint64_t, 'mask', 'incompatible_features'], + [uint64_t, 'mask', 'compatible_features'], + [uint64_t, 'mask', 'autoclear_features'], + [uint32_t, '%d', 'refcount_order'], + [uint32_t, '%d', 'header_length'], + ] + + fmt = '>' + ''.join(field[0] for field in fields) + + def __init__(self, fd): + + buf_size = struct.calcsize(QcowHeader.fmt) + + fd.seek(0) + buf = fd.read(buf_size) + + header = struct.unpack(QcowHeader.fmt, buf) + self.__dict__ = dict((field[2], header[i]) + for i, field in enumerate(QcowHeader.fields)) + + self.set_defaults() + self.cluster_size = 1 << self.cluster_bits + + fd.seek(self.header_length) + self.load_extensions(fd) + + if self.backing_file_offset: + fd.seek(self.backing_file_offset) + self.backing_file = fd.read(self.backing_file_size) + else: + self.backing_file = None + + def set_defaults(self): + if self.version == 2: + self.incompatible_features = 0 + self.compatible_features = 0 + self.autoclear_features = 0 + self.refcount_order = 4 + self.header_length = 72 + + def load_extensions(self, fd): + self.extensions = [] + + if self.backing_file_offset != 0: + end = min(self.cluster_size, self.backing_file_offset) + else: + end = self.cluster_size + + while fd.tell() < end: + (magic, length) = struct.unpack('>II', fd.read(8)) + if magic == 0: + break + else: + padded = (length + 7) & ~7 + data = fd.read(padded) + self.extensions.append(QcowHeaderExtension(magic, length, + data)) + + def update_extensions(self, fd): + + fd.seek(self.header_length) + extensions = self.extensions + extensions.append(QcowHeaderExtension(0, 0, b"")) + for ex in extensions: + buf = struct.pack('>II', ex.magic, ex.length) + fd.write(buf) + fd.write(ex.data) + + if self.backing_file is not None: + self.backing_file_offset = fd.tell() + fd.write(self.backing_file) + + if fd.tell() > self.cluster_size: + raise Exception("I think I just broke the image...") + + def update(self, fd): + header_bytes = self.header_length + + self.update_extensions(fd) + + fd.seek(0) + header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields) + buf = struct.pack(QcowHeader.fmt, *header) + buf = buf[0:header_bytes-1] + fd.write(buf) + + def dump(self): + for f in QcowHeader.fields: + value = self.__dict__[f[2]] + if f[1] == 'mask': + bits = [] + for bit in range(64): + if value & (1 << bit): + bits.append(bit) + value_str = str(bits) + else: + value_str = f[1] % value + + print("%-25s" % f[2], value_str) + print("") + + def dump_extensions(self): + for ex in self.extensions: + + data = ex.data[:ex.length] + if all(c in string.printable.encode('ascii') for c in data): + data = "'%s'" % data.decode('ascii') + else: + data = "" + + print("Header extension:") + print("%-25s %#x" % ("magic", ex.magic)) + print("%-25s %d" % ("length", ex.length)) + print("%-25s %s" % ("data", data)) + print("") From patchwork Tue Jun 9 20:52:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306281 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZVib56X4; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMp86lF3z9sSF for ; Wed, 10 Jun 2020 06:55:28 +1000 (AEST) Received: from localhost ([::1]:40976 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilHK-00057U-DS for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:55:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36328) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEt-0001dy-Hz for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:55 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:58950 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEs-0005bC-Pd for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735974; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yIYNgS++0ipV/7NWq4e6Wzcx8dFefrKfc0qssI+fv9s=; b=ZVib56X4ckJDoMfKhN1yqG5E/eqvn6BawoVB9E/Zi5qIneKSxJ8Xy+xVzONRgRJXB1b5ZA YN3UJZVVKmfndD84HfB9Mi2w+UC4YU/VGCn+VpjHm5/QwKu51rVufPgV11aRC1zL+A4Yuo tjXELA7LA+equtPT88AT4JB+CPwY4wA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-222-yeoWBUMEO6yWMDSmbtehPQ-1; Tue, 09 Jun 2020 16:52:51 -0400 X-MC-Unique: yeoWBUMEO6yWMDSmbtehPQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8F72464AEC; Tue, 9 Jun 2020 20:52:50 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 13E096116D; Tue, 9 Jun 2020 20:52:50 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 05/15] qcow2_format.py: drop new line printing at end of dump() Date: Tue, 9 Jun 2020 15:52:35 -0500 Message-Id: <20200609205245.3548257-6-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:41:53 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy This will simplify further conversion. To compensate, print this empty line directly in cmd_dump_header(). Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-5-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2.py | 1 + tests/qemu-iotests/qcow2_format.py | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index e968869ea6b2..8c187e9a7292 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -27,6 +27,7 @@ from qcow2_format import ( def cmd_dump_header(fd): h = QcowHeader(fd) h.dump() + print() h.dump_extensions() diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 0e517c1bbc5a..2b6c9612ae01 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -155,7 +155,6 @@ class QcowHeader: value_str = f[1] % value print("%-25s" % f[2], value_str) - print("") def dump_extensions(self): for ex in self.extensions: From patchwork Tue Jun 9 20:52:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306291 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hn2NLAhK; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMvt3x2Pz9sRk for ; Wed, 10 Jun 2020 07:00:26 +1000 (AEST) Received: from localhost ([::1]:58806 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilM8-00049v-78 for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 17:00:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36382) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEz-0001ly-6j for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:01 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:55821 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEx-0005ce-2M for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SmaP7JFguXSYIUjpk3oKNGUpu9UHziHLNUscAdqJ86Y=; b=Hn2NLAhKINuTW2OLES3ulTnEXydO8QSEC+ORX6Dx4eWjZliP+D139UD5hwdY2XfC2kVnpK 50LgJXD1j+KO1g+77d6wbwWBrbXlMDs9fIbAT9wWHqWFGn09E8cEidMy1HzkNlsvc+XnIP Dr8jl2w8nkwNqyHzoVlvSfv9k6JtyP8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-152-aLbHigi-P5uHEZcn-u2EEw-1; Tue, 09 Jun 2020 16:52:52 -0400 X-MC-Unique: aLbHigi-P5uHEZcn-u2EEw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 43B006A2BE; Tue, 9 Jun 2020 20:52:51 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id B8A066116D; Tue, 9 Jun 2020 20:52:50 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 06/15] qcow2_format.py: use tuples instead of lists for fields Date: Tue, 9 Jun 2020 15:52:36 -0500 Message-Id: <20200609205245.3548257-7-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy No need in lists: it's a constant variable. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-6-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++--------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 2b6c9612ae01..e2f08ed69194 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -39,29 +39,29 @@ class QcowHeader: uint32_t = 'I' uint64_t = 'Q' - fields = [ + fields = ( # Version 2 header fields - [uint32_t, '%#x', 'magic'], - [uint32_t, '%d', 'version'], - [uint64_t, '%#x', 'backing_file_offset'], - [uint32_t, '%#x', 'backing_file_size'], - [uint32_t, '%d', 'cluster_bits'], - [uint64_t, '%d', 'size'], - [uint32_t, '%d', 'crypt_method'], - [uint32_t, '%d', 'l1_size'], - [uint64_t, '%#x', 'l1_table_offset'], - [uint64_t, '%#x', 'refcount_table_offset'], - [uint32_t, '%d', 'refcount_table_clusters'], - [uint32_t, '%d', 'nb_snapshots'], - [uint64_t, '%#x', 'snapshot_offset'], + (uint32_t, '%#x', 'magic'), + (uint32_t, '%d', 'version'), + (uint64_t, '%#x', 'backing_file_offset'), + (uint32_t, '%#x', 'backing_file_size'), + (uint32_t, '%d', 'cluster_bits'), + (uint64_t, '%d', 'size'), + (uint32_t, '%d', 'crypt_method'), + (uint32_t, '%d', 'l1_size'), + (uint64_t, '%#x', 'l1_table_offset'), + (uint64_t, '%#x', 'refcount_table_offset'), + (uint32_t, '%d', 'refcount_table_clusters'), + (uint32_t, '%d', 'nb_snapshots'), + (uint64_t, '%#x', 'snapshot_offset'), # Version 3 header fields - [uint64_t, 'mask', 'incompatible_features'], - [uint64_t, 'mask', 'compatible_features'], - [uint64_t, 'mask', 'autoclear_features'], - [uint32_t, '%d', 'refcount_order'], - [uint32_t, '%d', 'header_length'], - ] + (uint64_t, 'mask', 'incompatible_features'), + (uint64_t, 'mask', 'compatible_features'), + (uint64_t, 'mask', 'autoclear_features'), + (uint32_t, '%d', 'refcount_order'), + (uint32_t, '%d', 'header_length'), + ) fmt = '>' + ''.join(field[0] for field in fields) From patchwork Tue Jun 9 20:52:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306286 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Ou/j2lQU; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMrp4NZqz9sRk for ; Wed, 10 Jun 2020 06:57:46 +1000 (AEST) Received: from localhost ([::1]:50320 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilJY-0000cs-9O for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:57:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36354) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEv-0001gk-U2 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:57 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:31222 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEu-0005be-Bm for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P+X2GehsEKSsVMlZEA77+G03NgvFhMOMVc9PxzTKk08=; b=Ou/j2lQUC8Rh3sE+7YoyrENTJLHKJsdfsARnXuopU1W7Kk67gHzV6eLFATjTrDfwiybU7i hxy5/s6glCRavsvpeKJsDopZyCaiDGgiWn97ftQRZO0BhMZHQefik4n/PbDNDBMKBTBUQ8 dASrtM5VYlbNZaV0v9DY5bxeorWQSlg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-268-UM8EKLwRMXe9u39n6v5vAw-1; Tue, 09 Jun 2020 16:52:53 -0400 X-MC-Unique: UM8EKLwRMXe9u39n6v5vAw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ECEF51009613; Tue, 9 Jun 2020 20:52:51 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F9EF6116D; Tue, 9 Jun 2020 20:52:51 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 07/15] qcow2_format.py: use modern string formatting Date: Tue, 9 Jun 2020 15:52:37 -0500 Message-Id: <20200609205245.3548257-8-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Use .format and f-strings instead of old %style. Also, the file uses both '' and "" quotes, for consistency let's use '', except for cases when we need '' inside the string (use "" to avoid extra escaping). Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-7-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 54 +++++++++++++++--------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index e2f08ed69194..da66df340876 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -23,7 +23,7 @@ class QcowHeaderExtension: def __init__(self, magic, length, data): if length % 8 != 0: padding = 8 - (length % 8) - data += b"\0" * padding + data += b'\0' * padding self.magic = magic self.length = length @@ -41,26 +41,26 @@ class QcowHeader: fields = ( # Version 2 header fields - (uint32_t, '%#x', 'magic'), - (uint32_t, '%d', 'version'), - (uint64_t, '%#x', 'backing_file_offset'), - (uint32_t, '%#x', 'backing_file_size'), - (uint32_t, '%d', 'cluster_bits'), - (uint64_t, '%d', 'size'), - (uint32_t, '%d', 'crypt_method'), - (uint32_t, '%d', 'l1_size'), - (uint64_t, '%#x', 'l1_table_offset'), - (uint64_t, '%#x', 'refcount_table_offset'), - (uint32_t, '%d', 'refcount_table_clusters'), - (uint32_t, '%d', 'nb_snapshots'), - (uint64_t, '%#x', 'snapshot_offset'), + (uint32_t, '{:#x}', 'magic'), + (uint32_t, '{}', 'version'), + (uint64_t, '{:#x}', 'backing_file_offset'), + (uint32_t, '{:#x}', 'backing_file_size'), + (uint32_t, '{}', 'cluster_bits'), + (uint64_t, '{}', 'size'), + (uint32_t, '{}', 'crypt_method'), + (uint32_t, '{}', 'l1_size'), + (uint64_t, '{:#x}', 'l1_table_offset'), + (uint64_t, '{:#x}', 'refcount_table_offset'), + (uint32_t, '{}', 'refcount_table_clusters'), + (uint32_t, '{}', 'nb_snapshots'), + (uint64_t, '{:#x}', 'snapshot_offset'), # Version 3 header fields (uint64_t, 'mask', 'incompatible_features'), (uint64_t, 'mask', 'compatible_features'), (uint64_t, 'mask', 'autoclear_features'), - (uint32_t, '%d', 'refcount_order'), - (uint32_t, '%d', 'header_length'), + (uint32_t, '{}', 'refcount_order'), + (uint32_t, '{}', 'header_length'), ) fmt = '>' + ''.join(field[0] for field in fields) @@ -118,7 +118,7 @@ class QcowHeader: fd.seek(self.header_length) extensions = self.extensions - extensions.append(QcowHeaderExtension(0, 0, b"")) + extensions.append(QcowHeaderExtension(0, 0, b'')) for ex in extensions: buf = struct.pack('>II', ex.magic, ex.length) fd.write(buf) @@ -129,7 +129,7 @@ class QcowHeader: fd.write(self.backing_file) if fd.tell() > self.cluster_size: - raise Exception("I think I just broke the image...") + raise Exception('I think I just broke the image...') def update(self, fd): header_bytes = self.header_length @@ -152,21 +152,21 @@ class QcowHeader: bits.append(bit) value_str = str(bits) else: - value_str = f[1] % value + value_str = f[1].format(value) - print("%-25s" % f[2], value_str) + print(f'{f[2]:<25} {value_str}') def dump_extensions(self): for ex in self.extensions: data = ex.data[:ex.length] if all(c in string.printable.encode('ascii') for c in data): - data = "'%s'" % data.decode('ascii') + data = f"'{ data.decode('ascii') }'" else: - data = "" + data = '' - print("Header extension:") - print("%-25s %#x" % ("magic", ex.magic)) - print("%-25s %d" % ("length", ex.length)) - print("%-25s %s" % ("data", data)) - print("") + print('Header extension:') + print(f'{"magic":<25} {ex.magic:#x}') + print(f'{"length":<25} {ex.length}') + print(f'{"data":<25} {data}') + print() From patchwork Tue Jun 9 20:52:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306292 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hQs/iJAD; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMx90lzYz9sRk for ; Wed, 10 Jun 2020 07:01:33 +1000 (AEST) Received: from localhost ([::1]:60964 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilNC-0005AP-Pn for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 17:01:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36422) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilF0-0001rE-Mg for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:31418 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEy-0005d7-8V for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9PmOd7xwEkBNsPkTefS2k9IpwawVFjlu2IH+kxt7Nxk=; b=hQs/iJADU/3rt8J0dq3LzkiM5HOvZebRC1/7stU+0PMLErXAUiv4b1S4bmdx45BWhyQfFd K3noMCh8QYwspO+Dh1H0b9un62+ySFn+JesjTzN26XSnan4p3FUpUYTPewIabsP2WGZBgd 6bfng9ShTEE7fyDXB5ETOLRFA8ydwf8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-182-3nvfOYgAMIOH8-ZCVXoV1w-1; Tue, 09 Jun 2020 16:52:53 -0400 X-MC-Unique: 3nvfOYgAMIOH8-ZCVXoV1w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9B2F719200C2; Tue, 9 Jun 2020 20:52:52 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F8B06116D; Tue, 9 Jun 2020 20:52:52 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 08/15] qcow2_format.py: use strings to specify c-type of struct fields Date: Tue, 9 Jun 2020 15:52:38 -0500 Message-Id: <20200609205245.3548257-9-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:41:53 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to move field-parsing to super-class, this will be simpler with simple string specifiers instead of variables. For some reason, python doesn't allow the definition of ctypes variable in the class alongside fields: it would not be available then for use by the 'for' operator. Don't worry: ctypes will be moved to metaclass soon. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-8-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index da66df340876..28f2bfa63bc7 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -34,36 +34,42 @@ class QcowHeaderExtension: return QcowHeaderExtension(magic, len(data), data) +# Mapping from c types to python struct format +ctypes = { + 'u8': 'B', + 'u16': 'H', + 'u32': 'I', + 'u64': 'Q' +} + + class QcowHeader: - uint32_t = 'I' - uint64_t = 'Q' - fields = ( # Version 2 header fields - (uint32_t, '{:#x}', 'magic'), - (uint32_t, '{}', 'version'), - (uint64_t, '{:#x}', 'backing_file_offset'), - (uint32_t, '{:#x}', 'backing_file_size'), - (uint32_t, '{}', 'cluster_bits'), - (uint64_t, '{}', 'size'), - (uint32_t, '{}', 'crypt_method'), - (uint32_t, '{}', 'l1_size'), - (uint64_t, '{:#x}', 'l1_table_offset'), - (uint64_t, '{:#x}', 'refcount_table_offset'), - (uint32_t, '{}', 'refcount_table_clusters'), - (uint32_t, '{}', 'nb_snapshots'), - (uint64_t, '{:#x}', 'snapshot_offset'), + ('u32', '{:#x}', 'magic'), + ('u32', '{}', 'version'), + ('u64', '{:#x}', 'backing_file_offset'), + ('u32', '{:#x}', 'backing_file_size'), + ('u32', '{}', 'cluster_bits'), + ('u64', '{}', 'size'), + ('u32', '{}', 'crypt_method'), + ('u32', '{}', 'l1_size'), + ('u64', '{:#x}', 'l1_table_offset'), + ('u64', '{:#x}', 'refcount_table_offset'), + ('u32', '{}', 'refcount_table_clusters'), + ('u32', '{}', 'nb_snapshots'), + ('u64', '{:#x}', 'snapshot_offset'), # Version 3 header fields - (uint64_t, 'mask', 'incompatible_features'), - (uint64_t, 'mask', 'compatible_features'), - (uint64_t, 'mask', 'autoclear_features'), - (uint32_t, '{}', 'refcount_order'), - (uint32_t, '{}', 'header_length'), + ('u64', 'mask', 'incompatible_features'), + ('u64', 'mask', 'compatible_features'), + ('u64', 'mask', 'autoclear_features'), + ('u32', '{}', 'refcount_order'), + ('u32', '{}', 'header_length'), ) - fmt = '>' + ''.join(field[0] for field in fields) + fmt = '>' + ''.join(ctypes[f[0]] for f in fields) def __init__(self, fd): From patchwork Tue Jun 9 20:52:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306278 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Usu1/nPw; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMmK71mmz9sRh for ; Wed, 10 Jun 2020 06:53:53 +1000 (AEST) Received: from localhost ([::1]:34904 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilFn-0002ee-Mf for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:53:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36366) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEw-0001iC-Gn for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:58 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:35399 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEv-0005c0-AO for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735976; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2gkJQrcmtu07hJLZsxNO23mQmGAJYFCopm0+MuaUKqc=; b=Usu1/nPwyYbGuyytT90DVsmfscEHol8W22WbBHRUJFD1Z2JsXU1kWHoZqFdumcIzPPDku9 2Qv5sALQD/dnOxnvpDL0Rk+7ra+ewIoSgu2pusIRn1GqAhFtrS98KfckGq1qgbnBE8sfsA pIbzVoa38hAUF3vtHk0s6YVZ4xhEdOE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-474-MYBpZcgiMV2PJGRKG4T0cw-1; Tue, 09 Jun 2020 16:52:54 -0400 X-MC-Unique: MYBpZcgiMV2PJGRKG4T0cw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53185835B44; Tue, 9 Jun 2020 20:52:53 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id C959C6116D; Tue, 9 Jun 2020 20:52:52 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 09/15] qcow2_format.py: separate generic functionality of structure classes Date: Tue, 9 Jun 2020 15:52:39 -0500 Message-Id: <20200609205245.3548257-10-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to introduce more Qcow2 structure types, defined like QcowHeader. Move generic functionality into base class to be reused for further structure classes. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-9-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 101 +++++++++++++++++++---------- 1 file changed, 66 insertions(+), 35 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 28f2bfa63bc7..898d388b8adf 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -1,5 +1,7 @@ # Library for manipulations with qcow2 image # +# Copyright (c) 2020 Virtuozzo International GmbH. +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or @@ -18,6 +20,68 @@ import struct import string +class Qcow2StructMeta(type): + + # Mapping from c types to python struct format + ctypes = { + 'u8': 'B', + 'u16': 'H', + 'u32': 'I', + 'u64': 'Q' + } + + def __init__(self, name, bases, attrs): + if 'fields' in attrs: + self.fmt = '>' + ''.join(self.ctypes[f[0]] for f in self.fields) + + +class Qcow2Struct(metaclass=Qcow2StructMeta): + + """Qcow2Struct: base class for qcow2 data structures + + Successors should define fields class variable, which is: list of tuples, + each of three elements: + - c-type (one of 'u8', 'u16', 'u32', 'u64') + - format (format_spec to use with .format() when dump or 'mask' to dump + bitmasks) + - field name + """ + + def __init__(self, fd=None, offset=None, data=None): + """ + Two variants: + 1. Specify data. fd and offset must be None. + 2. Specify fd and offset, data must be None. offset may be omitted + in this case, than current position of fd is used. + """ + if data is None: + assert fd is not None + buf_size = struct.calcsize(self.fmt) + if offset is not None: + fd.seek(offset) + data = fd.read(buf_size) + else: + assert fd is None and offset is None + + values = struct.unpack(self.fmt, data) + self.__dict__ = dict((field[2], values[i]) + for i, field in enumerate(self.fields)) + + def dump(self): + for f in self.fields: + value = self.__dict__[f[2]] + if f[1] == 'mask': + bits = [] + for bit in range(64): + if value & (1 << bit): + bits.append(bit) + value_str = str(bits) + else: + value_str = f[1].format(value) + + print('{:<25} {}'.format(f[2], value_str)) + + class QcowHeaderExtension: def __init__(self, magic, length, data): @@ -34,16 +98,7 @@ class QcowHeaderExtension: return QcowHeaderExtension(magic, len(data), data) -# Mapping from c types to python struct format -ctypes = { - 'u8': 'B', - 'u16': 'H', - 'u32': 'I', - 'u64': 'Q' -} - - -class QcowHeader: +class QcowHeader(Qcow2Struct): fields = ( # Version 2 header fields @@ -69,18 +124,8 @@ class QcowHeader: ('u32', '{}', 'header_length'), ) - fmt = '>' + ''.join(ctypes[f[0]] for f in fields) - def __init__(self, fd): - - buf_size = struct.calcsize(QcowHeader.fmt) - - fd.seek(0) - buf = fd.read(buf_size) - - header = struct.unpack(QcowHeader.fmt, buf) - self.__dict__ = dict((field[2], header[i]) - for i, field in enumerate(QcowHeader.fields)) + super().__init__(fd=fd, offset=0) self.set_defaults() self.cluster_size = 1 << self.cluster_bits @@ -148,20 +193,6 @@ class QcowHeader: buf = buf[0:header_bytes-1] fd.write(buf) - def dump(self): - for f in QcowHeader.fields: - value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: - value_str = f[1].format(value) - - print(f'{f[2]:<25} {value_str}') - def dump_extensions(self): for ex in self.extensions: From patchwork Tue Jun 9 20:52:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306284 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=fdooP4dp; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMpp1tvLz9sRh for ; Wed, 10 Jun 2020 06:56:02 +1000 (AEST) Received: from localhost ([::1]:43382 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilHr-00066V-T6 for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:55:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36376) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEx-0001kE-Hu for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:59 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:55176 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEw-0005cO-Bk for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:52:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735977; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ntvMf01chw3cZA0XjoTedRwr4kDz9k/HwjbN0SfGJB0=; b=fdooP4dp8z//eRBclQRA7gIuf6Y7YmBQWpV6zZA13Zf7Joz2DQasKZLXA4SrCkOIM+EtOy 0PPrkeUsrWJ7lZmEedx8a0n1Q/x53wNlUH/iaf4/NmrJ4pn0fp9ueHqOB6QJB0fXPDZnu8 SPX6xOoNxwZFqbsa49sMpCAA/7EkEPA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-469-CYd-MjRTMAyU9U3hsKUirw-1; Tue, 09 Jun 2020 16:52:55 -0400 X-MC-Unique: CYd-MjRTMAyU9U3hsKUirw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 021F119200C1; Tue, 9 Jun 2020 20:52:54 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 798876116D; Tue, 9 Jun 2020 20:52:53 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 10/15] qcow2_format.py: add field-formatting class Date: Tue, 9 Jun 2020 15:52:40 -0500 Message-Id: <20200609205245.3548257-11-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 16:46:48 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Allow formatter class in structure definition instead of hacking with 'mask'. This will simplify further introduction of new formatters. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-10-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 35 +++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 898d388b8adf..74a82f9263d4 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -20,6 +20,25 @@ import struct import string +class Qcow2Field: + + def __init__(self, value): + self.value = value + + def __str__(self): + return str(self.value) + + +class Flags64(Qcow2Field): + + def __str__(self): + bits = [] + for bit in range(64): + if self.value & (1 << bit): + bits.append(bit) + return str(bits) + + class Qcow2StructMeta(type): # Mapping from c types to python struct format @@ -70,14 +89,10 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): def dump(self): for f in self.fields: value = self.__dict__[f[2]] - if f[1] == 'mask': - bits = [] - for bit in range(64): - if value & (1 << bit): - bits.append(bit) - value_str = str(bits) - else: + if isinstance(f[1], str): value_str = f[1].format(value) + else: + value_str = str(f[1](value)) print('{:<25} {}'.format(f[2], value_str)) @@ -117,9 +132,9 @@ class QcowHeader(Qcow2Struct): ('u64', '{:#x}', 'snapshot_offset'), # Version 3 header fields - ('u64', 'mask', 'incompatible_features'), - ('u64', 'mask', 'compatible_features'), - ('u64', 'mask', 'autoclear_features'), + ('u64', Flags64, 'incompatible_features'), + ('u64', Flags64, 'compatible_features'), + ('u64', Flags64, 'autoclear_features'), ('u32', '{}', 'refcount_order'), ('u32', '{}', 'header_length'), ) From patchwork Tue Jun 9 20:52:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306287 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=RqOZjeeq; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMs44mb8z9sRk for ; Wed, 10 Jun 2020 06:58:00 +1000 (AEST) Received: from localhost ([::1]:51220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilJm-00011N-9W for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:57:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36424) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilF0-0001rY-Q4 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:31486 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEz-0005dd-J3 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735980; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WEkFixcghBG95mVzHuhBJJZD+q/fjGfb5A7T/Fd/47w=; b=RqOZjeeqZYsDCysBBpvu1+bMFzK3Bow1cG1ebKvg5+TLMOOHNqBJOfXpFIuFUmJF1NrqPx ie1mUfklq1PcbPuuxTRDfSmwYCaPTSuXe+Wb3i9lde4a9ZrKX7bSifPx55ht0gcLCKcu3m 8RO592Q/aNqTprDuWCJLCH6I+Cvu2qE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-84-p1FUhIoYND2-yKKr4R44BQ-1; Tue, 09 Jun 2020 16:52:55 -0400 X-MC-Unique: p1FUhIoYND2-yKKr4R44BQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A5F8F19200C3; Tue, 9 Jun 2020 20:52:54 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 29BB56116D; Tue, 9 Jun 2020 20:52:54 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 11/15] qcow2_format.py: QcowHeaderExtension: add dump method Date: Tue, 9 Jun 2020 15:52:41 -0500 Message-Id: <20200609205245.3548257-12-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Obviously, for-loop body in dump_extensions should be the dump method of extension. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-11-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 74a82f9263d4..d4ad5431b296 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -108,6 +108,17 @@ class QcowHeaderExtension: self.length = length self.data = data + def dump(self): + data = self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data): + data = f"'{ data.decode('ascii') }'" + else: + data = '' + + print(f'{"magic":<25} {self.magic:#x}') + print(f'{"length":<25} {self.length}') + print(f'{"data":<25} {data}') + @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) @@ -210,15 +221,6 @@ class QcowHeader(Qcow2Struct): def dump_extensions(self): for ex in self.extensions: - - data = ex.data[:ex.length] - if all(c in string.printable.encode('ascii') for c in data): - data = f"'{ data.decode('ascii') }'" - else: - data = '' - print('Header extension:') - print(f'{"magic":<25} {ex.magic:#x}') - print(f'{"length":<25} {ex.length}') - print(f'{"data":<25} {data}') + ex.dump() print() From patchwork Tue Jun 9 20:52:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306288 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=NrfPwt5B; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMsQ3GHBz9sRh for ; Wed, 10 Jun 2020 06:58:18 +1000 (AEST) Received: from localhost ([::1]:52402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilK4-0001UO-1w for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:58:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36390) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilEy-0001nN-Oj for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:00 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:44726 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEx-0005cu-FW for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xT7Dl2Np4lOx6956wQcDiocA0YR43F2Wm7BjuOHNaGo=; b=NrfPwt5BkCmfKj5dTmk2+Got1gQtRDEHzZV7xZ7WiZLZ0iPiG4v0S4EzJt6+3P64hTjKx2 Z/QQmSaJ29mJ03H96MoaT/oRV/bnjm1mlxUQkpNwTQleN53CUDxZQhgPovxZaVtvhsFuqP OdV7Di0bqPwpHUEjvd5KjURMXJWD/dI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-307-hK0iZE50NXuPlW9nJy6RyA-1; Tue, 09 Jun 2020 16:52:56 -0400 X-MC-Unique: hK0iZE50NXuPlW9nJy6RyA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 554AA8015CE; Tue, 9 Jun 2020 20:52:55 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id CD94F60C80; Tue, 9 Jun 2020 20:52:54 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 12/15] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct Date: Tue, 9 Jun 2020 15:52:42 -0500 Message-Id: <20200609205245.3548257-13-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 16:46:48 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Only two fields we can parse by generic code, but that is better than nothing. Keep further refactoring of variable-length fields for another day. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-12-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 53 +++++++++++++++++++++--------- 1 file changed, 37 insertions(+), 16 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index d4ad5431b296..32371e42da4e 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -97,16 +97,41 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): print('{:<25} {}'.format(f[2], value_str)) -class QcowHeaderExtension: +class QcowHeaderExtension(Qcow2Struct): - def __init__(self, magic, length, data): - if length % 8 != 0: - padding = 8 - (length % 8) - data += b'\0' * padding + fields = ( + ('u32', '{:#x}', 'magic'), + ('u32', '{}', 'length') + # length bytes of data follows + # then padding to next multiply of 8 + ) - self.magic = magic - self.length = length - self.data = data + def __init__(self, magic=None, length=None, data=None, fd=None): + """ + Support both loading from fd and creation from user data. + For fd-based creation current position in a file will be used to read + the data. + + This should be somehow refactored and functionality should be moved to + superclass (to allow creation of any qcow2 struct), but then, fields + of variable length (data here) should be supported in base class + somehow. So, it's a TODO. We'll see how to properly refactor this when + we have more qcow2 structures. + """ + if fd is None: + assert all(v is not None for v in (magic, length, data)) + self.magic = magic + self.length = length + if length % 8 != 0: + padding = 8 - (length % 8) + data += b'\0' * padding + self.data = data + else: + assert all(v is None for v in (magic, length, data)) + super().__init__(fd=fd) + padded = (self.length + 7) & ~7 + self.data = fd.read(padded) + assert self.data is not None def dump(self): data = self.data[:self.length] @@ -115,8 +140,7 @@ class QcowHeaderExtension: else: data = '' - print(f'{"magic":<25} {self.magic:#x}') - print(f'{"length":<25} {self.length}') + super().dump() print(f'{"data":<25} {data}') @classmethod @@ -182,14 +206,11 @@ class QcowHeader(Qcow2Struct): end = self.cluster_size while fd.tell() < end: - (magic, length) = struct.unpack('>II', fd.read(8)) - if magic == 0: + ext = QcowHeaderExtension(fd=fd) + if ext.magic == 0: break else: - padded = (length + 7) & ~7 - data = fd.read(padded) - self.extensions.append(QcowHeaderExtension(magic, length, - data)) + self.extensions.append(ext) def update_extensions(self, fd): From patchwork Tue Jun 9 20:52:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306296 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QHIZiVHs; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hN0b6Fcsz9sRh for ; Wed, 10 Jun 2020 07:04:31 +1000 (AEST) Received: from localhost ([::1]:41022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilQ4-0000GF-6b for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 17:04:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36436) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilF2-0001vM-97 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:04 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:21692 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilF0-0005du-Ct for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aURE4gYYAYgiIkkGHgjWzEuuUfdhbMunL1QWTJ63vro=; b=QHIZiVHsOFSGQXsWutVrOxkvZcpD9X+vYeZVSX461VwGmCaoE3XHEENdTEt9Bu7wxoefhj ntEdOThFi8SlILPqj53+MAMIbNsadXxeNtlOTGGj9LpoZ1Xsyv+OACc3SZkEcHKrCboZ5Z sCUZT+HFyi8dkut9RSSMDsl58otqIn8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-191-phPRY5acP6u2AzWpf8CTcw-1; Tue, 09 Jun 2020 16:52:57 -0400 X-MC-Unique: phPRY5acP6u2AzWpf8CTcw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0D70519200C4; Tue, 9 Jun 2020 20:52:56 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8591D61983; Tue, 9 Jun 2020 20:52:55 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 13/15] qcow2: QcowHeaderExtension print names for extension magics Date: Tue, 9 Jun 2020 15:52:43 -0500 Message-Id: <20200609205245.3548257-14-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 01:38:49 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Suggested-by: Andrey Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200606081806.23897-13-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- tests/qemu-iotests/031.out | 22 +++++++++++----------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++- 4 files changed, 36 insertions(+), 21 deletions(-) diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index 5a4beda6a23c..4b21d6a9ba18 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -25,7 +25,7 @@ refcount_order 4 header_length 72 Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' @@ -53,7 +53,7 @@ refcount_order 4 header_length 72 Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' @@ -81,12 +81,12 @@ refcount_order 4 header_length 72 Header extension: -magic 0xe2792aca +magic 0xe2792aca (Backing format) length 11 data 'host_device' Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' @@ -116,12 +116,12 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' @@ -149,12 +149,12 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' @@ -182,17 +182,17 @@ refcount_order 4 header_length 112 Header extension: -magic 0xe2792aca +magic 0xe2792aca (Backing format) length 11 data 'host_device' Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data Header extension: -magic 0x12345678 +magic 0x12345678 () length 31 data 'This is a test header extension' diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index e409acf60e2b..a9bed828e525 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -25,7 +25,7 @@ incompatible_features [] compatible_features [] autoclear_features [63] Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -37,7 +37,7 @@ incompatible_features [] compatible_features [] autoclear_features [] Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index a51ad1b5ba52..2f03cf045cce 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -25,7 +25,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -83,7 +83,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -139,7 +139,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -194,7 +194,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -263,7 +263,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -325,7 +325,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data @@ -354,7 +354,7 @@ refcount_order 4 header_length 112 Header extension: -magic 0x6803f857 +magic 0x6803f857 (Feature table) length 336 data diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 32371e42da4e..40b5bf467b24 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -39,6 +39,12 @@ class Flags64(Qcow2Field): return str(bits) +class Enum(Qcow2Field): + + def __str__(self): + return f'{self.value:#x} ({self.mapping.get(self.value, "")})' + + class Qcow2StructMeta(type): # Mapping from c types to python struct format @@ -99,8 +105,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): class QcowHeaderExtension(Qcow2Struct): + class Magic(Enum): + mapping = { + 0xe2792aca: 'Backing format', + 0x6803f857: 'Feature table', + 0x0537be77: 'Crypto header', + 0x23852875: 'Bitmaps', + 0x44415441: 'Data file' + } + fields = ( - ('u32', '{:#x}', 'magic'), + ('u32', Magic, 'magic'), ('u32', '{}', 'length') # length bytes of data follows # then padding to next multiply of 8 From patchwork Tue Jun 9 20:52:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306293 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Msv7N1Og; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMx85rwrz9sRh for ; Wed, 10 Jun 2020 07:01:32 +1000 (AEST) Received: from localhost ([::1]:32826 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilNC-0005Cg-Eu for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 17:01:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36434) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilF1-0001uQ-Sw for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:03 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:20377 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilF0-0005e8-Qq for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vd5iKc9l3vMpfq7EOYBUtsqLeuqfAegPZm3hExgksoQ=; b=Msv7N1Ogg9dBGOyaymXMBvgXeGwgT545VVITTZOTcwtPtnNehA40qrx02RAH5edPTxAmsz KMP+uvp/0vAIeJB0ikAVOermYj2O3enCNH9pjYc/99JoMc6IUErFESvTcyDGEWc2jx++zQ 4EGOT1Vfatxp2AdoYBcInxoM72NBD/s= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-330-R2Q4JwMCNHaOSunymNcBpA-1; Tue, 09 Jun 2020 16:52:57 -0400 X-MC-Unique: R2Q4JwMCNHaOSunymNcBpA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BBF898F4E5; Tue, 9 Jun 2020 20:52:56 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3FCE860C80; Tue, 9 Jun 2020 20:52:56 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 14/15] qcow2_format.py: dump bitmaps header extension Date: Tue, 9 Jun 2020 15:52:44 -0500 Message-Id: <20200609205245.3548257-15-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 02:44:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add class for bitmap extension and dump its fields. Further work is to dump bitmap directory. Test new functionality inside 291 iotest. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Message-Id: <20200606081806.23897-14-vsementsov@virtuozzo.com> [eblake: fix iotest output] Signed-off-by: Eric Blake --- tests/qemu-iotests/291 | 4 +++ tests/qemu-iotests/291.out | 33 +++++++++++++++++++++++ tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++------- 3 files changed, 70 insertions(+), 9 deletions(-) diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291 index 3ca83b9cd1f7..e0cffc7cb119 100755 --- a/tests/qemu-iotests/291 +++ b/tests/qemu-iotests/291 @@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io $QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1 $QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2 $QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +echo "Check resulting qcow2 header extensions:" +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts echo echo "=== Bitmap preservation not possible to non-qcow2 ===" @@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 $QEMU_IMG bitmap --remove --image-opts \ driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +echo "Check resulting qcow2 header extensions:" +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts echo echo "=== Check bitmap contents ===" diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out index 8c62017567e9..ccfcdc5e35ce 100644 --- a/tests/qemu-iotests/291.out +++ b/tests/qemu-iotests/291.out @@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 1048576/1048576 bytes at offset 2097152 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Check resulting qcow2 header extensions: +Header extension: +magic 0xe2792aca (Backing format) +length 5 +data 'qcow2' + +Header extension: +magic 0x6803f857 (Feature table) +length 336 +data + +Header extension: +magic 0x23852875 (Bitmaps) +length 24 +nb_bitmaps 2 +reserved32 0 +bitmap_directory_size 0x40 +bitmap_directory_offset 0x510000 + === Bitmap preservation not possible to non-qcow2 === @@ -65,6 +84,20 @@ Format specific information: granularity: 65536 refcount bits: 16 corrupt: false +Check resulting qcow2 header extensions: +Header extension: +magic 0x6803f857 (Feature table) +length 336 +data + +Header extension: +magic 0x23852875 (Bitmaps) +length 24 +nb_bitmaps 3 +reserved32 0 +bitmap_directory_size 0x60 +bitmap_directory_offset 0x520000 + === Check bitmap contents === diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 40b5bf467b24..0f65fd161d5b 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -103,6 +103,19 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): print('{:<25} {}'.format(f[2], value_str)) +class Qcow2BitmapExt(Qcow2Struct): + + fields = ( + ('u32', '{}', 'nb_bitmaps'), + ('u32', '{}', 'reserved32'), + ('u64', '{:#x}', 'bitmap_directory_size'), + ('u64', '{:#x}', 'bitmap_directory_offset') + ) + + +QCOW2_EXT_MAGIC_BITMAPS = 0x23852875 + + class QcowHeaderExtension(Qcow2Struct): class Magic(Enum): @@ -110,7 +123,7 @@ class QcowHeaderExtension(Qcow2Struct): 0xe2792aca: 'Backing format', 0x6803f857: 'Feature table', 0x0537be77: 'Crypto header', - 0x23852875: 'Bitmaps', + QCOW2_EXT_MAGIC_BITMAPS: 'Bitmaps', 0x44415441: 'Data file' } @@ -130,8 +143,11 @@ class QcowHeaderExtension(Qcow2Struct): This should be somehow refactored and functionality should be moved to superclass (to allow creation of any qcow2 struct), but then, fields of variable length (data here) should be supported in base class - somehow. So, it's a TODO. We'll see how to properly refactor this when - we have more qcow2 structures. + somehow. Note also, that we probably want to parse different + extensions. Should they be subclasses of this class, or how to do it + better? Should it be something like QAPI union with discriminator field + (magic here). So, it's a TODO. We'll see how to properly refactor this + when we have more qcow2 structures. """ if fd is None: assert all(v is not None for v in (magic, length, data)) @@ -148,15 +164,23 @@ class QcowHeaderExtension(Qcow2Struct): self.data = fd.read(padded) assert self.data is not None - def dump(self): - data = self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data): - data = f"'{ data.decode('ascii') }'" + if self.magic == QCOW2_EXT_MAGIC_BITMAPS: + self.obj = Qcow2BitmapExt(data=self.data) else: - data = '' + self.obj = None + def dump(self): super().dump() - print(f'{"data":<25} {data}') + + if self.obj is None: + data = self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data): + data = f"'{ data.decode('ascii') }'" + else: + data = '' + print(f'{"data":<25} {data}') + else: + self.obj.dump() @classmethod def create(cls, magic, data): From patchwork Tue Jun 9 20:52:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 1306285 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=DbH9RubC; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49hMrp3BSRz9sRh for ; Wed, 10 Jun 2020 06:57:46 +1000 (AEST) Received: from localhost ([::1]:50262 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jilJY-0000bb-3y for incoming@patchwork.ozlabs.org; Tue, 09 Jun 2020 16:57:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36414) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jilF0-0001qq-IJ for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:46807 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jilEz-0005dZ-D5 for qemu-devel@nongnu.org; Tue, 09 Jun 2020 16:53:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591735980; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fp46CHSLKGKWkUBAgicrH6jIF36/amsY7o6T2XaaBiQ=; b=DbH9RubCndeXdinIHASKEsxCJ6WnhV+6Jl+S3uvGLjQkoDPQZdEVGPv+qTxyvu+43kcnsn AowhNxwzxi8r8AEZibkW2/Pu+U8RswvCpCtslruLb7hWX6kAOcNgV0g9S1Af5VosZ7Rtxg 54jLubZJl9apVyLF5T9/ewNt6wTk3XA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-357-8uIUbHaCM3mSl36h7M9iFA-1; Tue, 09 Jun 2020 16:52:58 -0400 X-MC-Unique: 8uIUbHaCM3mSl36h7M9iFA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5597019200C1; Tue, 9 Jun 2020 20:52:57 +0000 (UTC) Received: from blue.redhat.com (ovpn-113-22.phx2.redhat.com [10.3.113.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA32A6116D; Tue, 9 Jun 2020 20:52:56 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 15/15] iotests: Fix 291 across more file systems Date: Tue, 9 Jun 2020 15:52:45 -0500 Message-Id: <20200609205245.3548257-16-eblake@redhat.com> In-Reply-To: <20200609205245.3548257-1-eblake@redhat.com> References: <20200609205245.3548257-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/09 16:46:48 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Depending on the granularity of holes and amount of metadata consumed by a file, the 'disk size:' number of 'qemu-img info' is not reliable. Adjust our test to use a different set of filters to avoid spurious failures. Reported-by: Kevin Wolf Fixes: cf2d1203dc Signed-off-by: Eric Blake Message-Id: <20200608195629.3299649-1-eblake@redhat.com> Acked-by: Kevin Wolf [eblake: fix merge conflict] Signed-off-by: Eric Blake --- tests/qemu-iotests/291 | 4 ++-- tests/qemu-iotests/291.out | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291 index e0cffc7cb119..404f8521f74a 100755 --- a/tests/qemu-iotests/291 +++ b/tests/qemu-iotests/291 @@ -79,7 +79,7 @@ echo # Only bitmaps from the active layer are copied $QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" -$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +_img_info --format-specific # But we can also merge in bitmaps from other layers. This test is a bit # contrived to cover more code paths, in reality, you could merge directly # into b0 without going through tmp @@ -89,7 +89,7 @@ $QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 $QEMU_IMG bitmap --remove --image-opts \ driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp -$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +_img_info --format-specific echo "Check resulting qcow2 header extensions:" $PYTHON qcow2.py "$TEST_IMG" dump-header-exts diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out index ccfcdc5e35ce..08bfaaaa6b7d 100644 --- a/tests/qemu-iotests/291.out +++ b/tests/qemu-iotests/291.out @@ -43,7 +43,7 @@ qemu-img: Format driver 'raw' does not support bitmaps image: TEST_DIR/t.IMGFMT file format: IMGFMT virtual size: 10 MiB (10485760 bytes) -disk size: 4.39 MiB +cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib @@ -63,7 +63,7 @@ Format specific information: image: TEST_DIR/t.IMGFMT file format: IMGFMT virtual size: 10 MiB (10485760 bytes) -disk size: 4.48 MiB +cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib