From patchwork Mon Aug 7 19:40:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: dverma X-Patchwork-Id: 798839 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xR7Jx632Pz9s76 for ; Tue, 8 Aug 2017 05:44:33 +1000 (AEST) Received: from localhost ([::1]:39075 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1denx1-0002jI-Pd for incoming@patchwork.ozlabs.org; Mon, 07 Aug 2017 15:44:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46914) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dentP-0000nS-Ta for qemu-devel@nongnu.org; Mon, 07 Aug 2017 15:40:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dentK-0006uz-Cw for qemu-devel@nongnu.org; Mon, 07 Aug 2017 15:40:47 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50686) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dentK-0006sR-3c for qemu-devel@nongnu.org; Mon, 07 Aug 2017 15:40:42 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 16AF98125F for ; Mon, 7 Aug 2017 19:40:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 16AF98125F Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dverma@redhat.com Received: from dverma.bos.com (dhcp-17-188.bos.redhat.com [10.18.17.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id 82B9E18039; Mon, 7 Aug 2017 19:40:40 +0000 (UTC) From: dverma To: qemu-devel@nongnu.org Date: Mon, 7 Aug 2017 15:40:06 -0400 Message-Id: <1502134806-13669-4-git-send-email-dverma@redhat.com> In-Reply-To: <1502134806-13669-1-git-send-email-dverma@redhat.com> References: <1502134806-13669-1-git-send-email-dverma@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 07 Aug 2017 19:40:41 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/3] Add new functions for whitelisting and their calls X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dverma , dgilbert@redhat.com, jen@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The 'check_updated_properties' function keeps track of properties that were added/removed from fields across qemu versions. The 'check_updated_sizes' function reduces false positives generated especially while testing backward migration by keeping a list of common size/version changes. The 'check_new_sections' function is used to check for sections that got deprecated or were introduced in different versions of qemu and will show as false positives while testing forward migration. Improved the variable names and added multiple blank newlines to keep Python PEP8 warning away. Signed-off-by: Deepak Verma --- scripts/vmstate-static-checker.py | 226 ++++++++++++++++++++++++++++++-------- 1 file changed, 180 insertions(+), 46 deletions(-) diff --git a/scripts/vmstate-static-checker.py b/scripts/vmstate-static-checker.py index ae41e44..77e63f1 100755 --- a/scripts/vmstate-static-checker.py +++ b/scripts/vmstate-static-checker.py @@ -40,6 +40,107 @@ def bump_taint(): if taint < 255: taint = taint + 1 +# Sections gain/lose new fields with time. +# These are not name changes thats handled by another list. +# These will be 'missing' or 'not found' in different versions of qemu + + +def check_updated_properties(src_desc, field): + src_desc = str(src_desc) + field = str(field) + updated_property = { + 'ICH9LPC': ['ICH9LPC/smi_feat'], + 'ide_bus/error': ['retry_sector_num', 'retry_nsector', 'retry_unit'], + 'e1000': ['e1000/full_mac_state'], + 'ich9_pm': ['ich9_pm/tco', 'ich9_pm/cpuhp'] + } + + if src_desc in updated_property and field in updated_property[src_desc]: + return True + + return False + + +# A lot of errors are generated due to differences in sizes some of which are false positives. This list +# is used to save those common changes +def check_updated_sizes(field, old_size, new_size): + new_sizes_list = { + 'tally_counters.TxOk': [8, 64], + 'intel-iommu': [0, 1], + 'iommu-intel': [0, 1] + } + + if field not in new_sizes_list: + return False + + if (old_size in new_sizes_list[field] and new_size in new_sizes_list[field]): + return True + + return False + + +# With time new sections/hardwares supported and old ones are depreciated on +# chipsets. +# There is no separate list for new or dead sections as it's relative to which +# qemu version you compare too. +# Update this list with such sections. +# some items in this list might overlap with changed sections names. +def check_new_sections(sec): + new_sections_list = [ + 'virtio-balloon-device', + 'virtio-rng-device', + 'virtio-scsi-device', + 'virtio-blk-device', + 'virtio-serial-device', + 'virtio-net-device', + 'vhost-vsock-device', + 'virtio-input-host-device', + 'virtio-input-hid-device', + 'virtio-mouse-device', + 'virtio-keyboard-device', + 'virtio-vga', + 'virtio-input-device', + 'virtio-gpu-device', + 'virtio-tablet-device', + 'isa-pcspk', + 'qemu-xhci', + 'base-xhci', + 'vmgenid', + 'intel-iommu', + 'i8257', + 'i82801b11-bridge', + 'ivshmem', + 'ivshmem-doorbell', + 'ivshmem-plain', + 'usb-storage-device', + 'usb-storage-dev', + 'pci-qxl', + 'pci-uhci-usb', + 'pci-piix3', + 'pci-vga', + 'pci-bridge-seat', + 'pcie-root-port', + 'fw_cfg_io', + 'fw_cfg_mem', + 'exynos4210-ehci-usb', + 'sysbus-ehci-usb', + 'tegra2-ehci-usb', + 'kvm-apic', + 'fusbh200-ehci-usb', + 'apic', + 'apic-common', + 'xlnx,ps7-usb', + 'e1000e', + 'e1000-82544gc', + 'e1000-82545em'] + + if sec in new_sections_list: + return True + + return False + +# Fields might change name with time across qemu versions. + def check_fields_match(name, s_field, d_field): if s_field == d_field: @@ -286,12 +387,22 @@ def check_fields(src_fields, dest_fields, desc, sec): unused_count = s_item["size"] - d_item["size"] continue - print('Section "' + sec + '", ' - 'Description "' + desc + '": ' - 'expected field "' + s_item["field"] + '", ' - 'got "' + d_item["field"] + '"; skipping rest') - bump_taint() - break + # commit 20daa90a20d, extra field 'config' was added in newer releases + # there will be a mismatch in the number of fields of irq_state and config + # it's a known false positive so skip it + if (desc in ["PCIDevice", "PCIEDevice"]): + if((s_item["field"] in ["irq_state", "config"]) and + (d_item["field"] in ["irq_state", "config"])): + break + + # some fields are new some dead, but are not errors. + if not check_fields_match(desc, s_item["field"], d_item["field"]): + print('Section "' + sec + '", ' + 'Description "' + desc + '": ' + 'expected field "' + s_item["field"] + '", ' + 'got "' + d_item["field"] + '"; skipping rest') + bump_taint() + break check_version(s_item, d_item, sec, desc) @@ -312,7 +423,8 @@ def check_subsections(src_sub, dest_sub, desc, sec): found = True check_descriptions(s_item, d_item, sec) - if not found: + # check the updated properties list before throwing error + if not found and (not check_updated_properties(desc, s_item["name"])): print('Section "' + sec + '", ' 'Description "' + desc + '": ' 'Subsection "' + s_item["name"] + '" not found') @@ -343,51 +455,59 @@ def check_descriptions(src_desc, dest_desc, sec): bump_taint() return - for f in src_desc: - if not f in dest_desc: - print('Section "' + sec + '" ' - 'Description "' + src_desc["name"] + '": ' - 'Entry "' + field + '" missing') - bump_taint() - continue + for field in src_desc: + if field not in dest_desc: + # check the updated list of changed properties before throwing error + if check_updated_properties(src_desc["name"], field): + continue + else: + print('Section "' + sec + '" ' + 'Description "' + src_desc["name"] + '": ' + 'Entry "' + field + '" missing') + bump_taint() + continue - if f == 'Fields': - check_fields(src_desc[f], dest_desc[f], src_desc["name"], sec) + if field == 'Fields': + check_fields(src_desc[field], dest_desc[field], src_desc["name"], sec) - if f == 'Subsections': - check_subsections(src_desc[f], dest_desc[f], src_desc["name"], sec) + if field == 'Subsections': + check_subsections(src_desc[field], dest_desc[field], src_desc["name"], sec) -def check_version(s, d, sec, desc=None): - if s["version_id"] > d["version_id"]: - print "Section \"" + sec + "\"", - if desc: - print "Description \"" + desc + "\":", - print "version error:", s["version_id"], ">", d["version_id"] - bump_taint() +def check_version(src_ver, dest_ver, sec, desc=None): + if src_ver["version_id"] > dest_ver["version_id"]: + if not check_updated_sizes(sec, src_ver["version_id"], dest_ver["version_id"]): + print ('Section "' + sec + '"'), + if desc: + print ('Description "' + desc + '":'), + print ('version error: ' + str(src_ver["version_id"]) + ' > ' + str(dest_ver["version_id"])) + bump_taint() if "minimum_version_id" not in dest_ver: return - if s["version_id"] < d["minimum_version_id"]: - print "Section \"" + sec + "\"", - if desc: - print('Description "' + desc + '": ' + - 'minimum version error: ' + str(src_ver["version_id"]) + ' < ' + - str(dest_ver["minimum_version_id"])) - bump_taint() + if src_ver["version_id"] < dest_ver["minimum_version_id"]: + if not check_updated_sizes(sec, src_ver["version_id"], dest_ver["minimum_version_id"]): + print ('Section "' + sec + '"'), + if desc: + print('Description "' + desc + '": ' + + 'minimum version error: ' + str(src_ver["version_id"]) + ' < ' + + str(dest_ver["minimum_version_id"])) + bump_taint() -def check_size(s, d, sec, desc=None, field=None): - if s["size"] != d["size"]: - print "Section \"" + sec + "\"", - if desc: - print "Description \"" + desc + "\"", - if field: - print "Field \"" + field + "\"", - print "size mismatch:", s["size"], ",", d["size"] - bump_taint() +def check_size(src, dest, sec, desc=None, field=None): + if src["size"] != dest["size"]: + # check updated sizes list before throwing error + if not check_updated_sizes(field, src["size"], dest["size"]): + print ('Section "' + sec + '"'), + if desc: + print ('Description "' + desc + '"'), + if field: + print ('Field "' + field + '"'), + print ('size mismatch: ' + str(src["size"]) + ' , ' + str(dest["size"])) + bump_taint() def check_machine_type(src, dest): @@ -428,13 +548,27 @@ def main(): for sec in src_data: dest_sec = sec if dest_sec not in dest_data: - # Either the section name got changed, or the section - # doesn't exist in dest. + # Either the section name got changed, or + # the section doesn't exist in dest or + # section was newly supported. dest_sec = get_changed_sec_name(sec) - if dest_sec not in dest_data: - print('Section "' + sec + '" does not exist in dest') - bump_taint() + if dest_sec == "": + if not check_new_sections(sec): + # section not in newly supported list and not in dest + print('Section "' + sec + '" does not exist in dest') + bump_taint() continue + else: + # section name changed + if dest_sec not in dest_data: + # new name not found in dest. + if check_new_sections(dest_sec): + continue + else: + # new name not in dest and not newly supported + print('Section "' + sec + '" does not exist in dest') + bump_taint() + continue s = src_data[sec] d = dest_data[dest_sec]