From patchwork Fri Dec 21 12:36:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 1017495 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="cXm8c1Yx"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43Lp7s23vjz9sDP for ; Fri, 21 Dec 2018 23:38:17 +1100 (AEDT) Received: from localhost ([::1]:45227 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaK4E-0003Zd-Q8 for incoming@patchwork.ozlabs.org; Fri, 21 Dec 2018 07:38:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41118) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaK2f-0002k8-27 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaK2Z-0003VK-5i for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:35 -0500 Received: from mail-wr1-x42b.google.com ([2a00:1450:4864:20::42b]:42333) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gaK2X-0003R9-7U for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:30 -0500 Received: by mail-wr1-x42b.google.com with SMTP id q18so5096182wrx.9 for ; Fri, 21 Dec 2018 04:36:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=AFGimPKHIbcCKVmKtIBgY6jxoqNSp7gGbsAx4cSo6x4=; b=cXm8c1YxSTgtNpGsGkZSi/03y1hRjHcNUDnvIf9d2mG/KlBPixMDnVYfcmRWrTOewc EuV3oMUC1utbVZPCOC6y2Y4qTNpwjpotlS1EivSvVUN7mfP3byovaJXoQyqcH+R6DEjc lxmaCkr6q/Ef6kVSaq2NOP/IAf4ImGFIXArEac02+av9Za0w7mXOaXSialN1yeAgLS2m dVrroIZpDaGcULjwzfYkPuK1aynUOU7w8usssLrGp6LnVkwYyypZtAr+MYRgaKfMzFgq /aZtDEhswgSEYHYDuoM6ItayAQNDGteWudCvdcvr4dPtplSF+FmZ/5PA32zGxfbKNM4a pwAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=AFGimPKHIbcCKVmKtIBgY6jxoqNSp7gGbsAx4cSo6x4=; b=QCyN2/0Qk0p3y2T7kPvA+DLRkoklcIvU97rW2H+QN4fOWw8yhSSJJHr1hVl1lQmEQ6 BHtU2XDWJAPweqwPmG3bs5x+WRZdU5GFPkNDntZLr1X6idLwzODx9BPZnMN4rmSvHQ4c jv1GJ8lcbTrsLFh6MsIt2ttc35eK9MqgOi4JoJ2jbRrptpyVh7bjam1YFJ05r7lyOb8E fqgsQeGuTMCj4TcaxjMqqv4ljWfd2i/+ourtVORN/CZcOl4sC7uBtdTZYKTp7b/xMZ+V aqrz6TVfn7W6bPhvcuGsLISMCwAngf//V5FKqws7fGgOLGwbpxfqr4rLAmaIePqQRsi5 2idw== X-Gm-Message-State: AJcUukcYU7m9Otn6cnv/5sFtV8Ho0IQOEcsAoXiwUCOzcnPxCQTL8/GW DrsQVSi19J/sopj0jHd48vJQbJSb X-Google-Smtp-Source: ALg8bN4VZY5CX0PbxkAWOGIBF+tyE4ljd+q8o3oHtxxyd6F3z2EaitLcSsPwKNSA8g4J0Ill8HxZTA== X-Received: by 2002:adf:fd87:: with SMTP id d7mr2385015wrr.74.1545395786363; Fri, 21 Dec 2018 04:36:26 -0800 (PST) Received: from 640k.lan ([93.56.166.5]) by smtp.gmail.com with ESMTPSA id k7sm11217839wrl.51.2018.12.21.04.36.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Dec 2018 04:36:25 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Fri, 21 Dec 2018 13:36:22 +0100 Message-Id: <1545395783-39040-2-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1545395783-39040-1-git-send-email-pbonzini@redhat.com> References: <1545395783-39040-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::42b Subject: [Qemu-devel] [PULL 22/35] test: execute g_test_run when tests are skipped X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Sometimes a test's main() function recognizes that the environment does not support the test, and therefore exits. In this case, we still should run g_test_run() so that a TAP harness will print the test plan ("1..0") and the test will be marked as skipped. Signed-off-by: Paolo Bonzini Message-Id: <1543513531-1151-2-git-send-email-pbonzini@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Thomas Huth Signed-off-by: Paolo Bonzini --- tests/cdrom-test.c | 2 +- tests/ivshmem-test.c | 5 ++--- tests/migration-test.c | 8 ++++---- tests/test-crypto-pbkdf.c | 3 ++- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/cdrom-test.c b/tests/cdrom-test.c index 9b43dc9..14bd981 100644 --- a/tests/cdrom-test.c +++ b/tests/cdrom-test.c @@ -169,7 +169,7 @@ int main(int argc, char **argv) if (exec_genisoimg(genisocheck)) { /* genisoimage not available - so can't run tests */ - return 0; + return g_test_run(); } ret = prepare_image(arch, isoimage); diff --git a/tests/ivshmem-test.c b/tests/ivshmem-test.c index 089e268..fe5eb30 100644 --- a/tests/ivshmem-test.c +++ b/tests/ivshmem-test.c @@ -492,7 +492,7 @@ int main(int argc, char **argv) /* shm */ tmpshm = mktempshm(TMPSHMSIZE, &fd); if (!tmpshm) { - return 0; + goto out; } tmpshmem = mmap(0, TMPSHMSIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); g_assert(tmpshmem != MAP_FAILED); @@ -514,9 +514,8 @@ int main(int argc, char **argv) } } +out: ret = g_test_run(); - cleanup(); - return ret; } diff --git a/tests/migration-test.c b/tests/migration-test.c index 06ca506..8352612 100644 --- a/tests/migration-test.c +++ b/tests/migration-test.c @@ -789,7 +789,7 @@ int main(int argc, char **argv) g_test_init(&argc, &argv, NULL); if (!ufd_version_check()) { - return 0; + return g_test_run(); } /* @@ -800,7 +800,7 @@ int main(int argc, char **argv) if (g_str_equal(qtest_get_arch(), "ppc64") && access("/sys/module/kvm_hv", F_OK)) { g_test_message("Skipping test: kvm_hv not available"); - return 0; + return g_test_run(); } /* @@ -811,11 +811,11 @@ int main(int argc, char **argv) #if defined(HOST_S390X) if (access("/dev/kvm", R_OK | W_OK)) { g_test_message("Skipping test: kvm not available"); - return 0; + return g_test_run(); } #else g_test_message("Skipping test: Need s390x host to work properly"); - return 0; + return g_test_run(); #endif } diff --git a/tests/test-crypto-pbkdf.c b/tests/test-crypto-pbkdf.c index d937aff..85ed1f9 100644 --- a/tests/test-crypto-pbkdf.c +++ b/tests/test-crypto-pbkdf.c @@ -440,6 +440,7 @@ int main(int argc, char **argv) #else int main(int argc, char **argv) { - return 0; + g_test_init(&argc, &argv, NULL); + return g_test_run(); } #endif From patchwork Fri Dec 21 12:36:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 1017494 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="bQZdAvAg"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43Lp6Z522zz9sDr for ; Fri, 21 Dec 2018 23:37:10 +1100 (AEDT) Received: from localhost ([::1]:45212 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaK39-0002lA-P3 for incoming@patchwork.ozlabs.org; Fri, 21 Dec 2018 07:37:07 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41121) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaK2f-0002kC-3w for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaK2b-0003X9-4Q for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:35 -0500 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:37692) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gaK2Z-0003SX-7B for qemu-devel@nongnu.org; Fri, 21 Dec 2018 07:36:33 -0500 Received: by mail-wr1-x442.google.com with SMTP id s12so5100829wrt.4 for ; Fri, 21 Dec 2018 04:36:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=vq9JUkh/+TLTjb/f1GiJVOo6Ci5cv1+4zKRsIeiTjEg=; b=bQZdAvAgdGLql3uGY9bXpDFBweH/hqpkO2AFKY79d11hD99y0vw7j7JEWBE1gga0rl bFIcRTtnsWbTNhB1MxWBhhhaSvCoNdk2sk/LWalLDvz7JLSdEANIbeJLvpp8/f04dsWy kaxIqF6ZQZ7QQklkjM5dMZGWibAGwTMJ/y6VQ315Edzo7Xy7xOTYtX+m9hftw2bnjPzy CXDYf/DTwehXLxvzkGQT2k0/N+t2WgH12X7DJq8I2IFq+8gBmBLaqQRbS4k4iQeuikIw yQxErKKU6RDD40XD/YUBirdLiwEWx1Y6a2w+tV6cwMppn9cC3On/AIFZ42cS/RrFmpe+ gKXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=vq9JUkh/+TLTjb/f1GiJVOo6Ci5cv1+4zKRsIeiTjEg=; b=kDYMiQnxjcgNNgk48IJqVqwD/N1AEZGhycMDP10c3bxADJTGIYGW7+yP319ls2wgfp oP3B7C/5NPA03G5gbES0LuEb260YObrcfUlfqiZqYG0VUHaIFPucz3zDrxGousrU4uE4 0sqp6rs9YNpTOJ6zFkIBv/q/mrq/yYEyVAkY7XqE13NOBMDhe4MDqs1mVbFt7MfPYEJr plhtgPcq+JaPmblfqMcw/W4QoZdR/J9kKDGNUz0Y1s+YRKm9Th6Obm9cehP4XQaZxgaG ChmQPbDx9FigjpDS+eM6+Pv4YMhd7Kem8X0BZZNptvKIk11JRSPM3mGIDVDex/JmFUgf FYYA== X-Gm-Message-State: AJcUukeDpL2P2gvMEkElMYPHyhFPUTzS2NNzRvSZIpw0qVQLuCRiFRMm 6IBixauUVjlobHCsyGw4nMzaCfCF X-Google-Smtp-Source: ALg8bN5d/f/vOvfLe6/OPJjDpgtgHTd8isExvIOXsst0nuYAZj4ssWwvQAmLk4xRCV5NfpQELwb5dw== X-Received: by 2002:adf:f101:: with SMTP id r1mr2514844wro.32.1545395787362; Fri, 21 Dec 2018 04:36:27 -0800 (PST) Received: from 640k.lan ([93.56.166.5]) by smtp.gmail.com with ESMTPSA id k7sm11217839wrl.51.2018.12.21.04.36.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Dec 2018 04:36:26 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Fri, 21 Dec 2018 13:36:23 +0100 Message-Id: <1545395783-39040-3-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1545395783-39040-1-git-send-email-pbonzini@redhat.com> References: <1545395783-39040-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PULL 23/35] test: replace gtester with a TAP driver X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" gtester is deprecated by upstream glib (see for example the announcement at https://blog.gtk.org/2018/07/11/news-from-glib-2-58/) and it does not support tests that call g_test_skip in some glib stable releases. glib suggests instead using Automake's TAP support, which gtest itself supports since version 2.38 (QEMU's minimum requirement is 2.40). We do not support Automake, but we can use Automake's code to beautify the TAP output. I chose to use the Perl copy rather than the shell/awk one, with some changes so that it can accept TAP through stdin, in order to reuse Perl's TAP parsing package. This also avoids duplicating the parser between tap-driver.pl and tap-merge.pl. Signed-off-by: Paolo Bonzini Message-Id: <1543513531-1151-3-git-send-email-pbonzini@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Paolo Bonzini --- rules.mak | 4 +- scripts/gtester-cat | 26 -- scripts/tap-driver.pl | 378 +++++++++++++++++++++++++++ scripts/tap-merge.pl | 110 ++++++++ tests/Makefile.include | 74 ++++-- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/debian-amd64.docker | 1 + tests/docker/dockerfiles/debian-ports.docker | 1 + tests/docker/dockerfiles/debian-sid.docker | 1 + tests/docker/dockerfiles/debian8.docker | 1 + tests/docker/dockerfiles/debian9.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/ubuntu.docker | 1 + 13 files changed, 548 insertions(+), 52 deletions(-) delete mode 100755 scripts/gtester-cat create mode 100755 scripts/tap-driver.pl create mode 100755 scripts/tap-merge.pl diff --git a/rules.mak b/rules.mak index bbb2667..86e033d 100644 --- a/rules.mak +++ b/rules.mak @@ -132,7 +132,9 @@ modules: # otherwise print the 'quiet' output in the format " NAME args to print" # NAME should be a short name of the command, 7 letters or fewer. # If called with only a single argument, will print nothing in quiet mode. -quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) +quiet-command-run = $(if $(V),,$(if $2,printf " %-7s %s\n" $2 $3 && ))$1 +quiet-@ = $(if $(V),,@) +quiet-command = $(quiet-@)$(call quiet-command-run,$1,$2,$3) # cc-option # Usage: CFLAGS+=$(call cc-option, -falign-functions=0, -malign-functions=0) diff --git a/scripts/gtester-cat b/scripts/gtester-cat deleted file mode 100755 index 061a952..0000000 --- a/scripts/gtester-cat +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/sh -# -# Copyright IBM, Corp. 2012 -# -# Authors: -# Anthony Liguori -# -# This work is licensed under the terms of the GNU GPLv2 or later. -# See the COPYING file in the top-level directory. - -cat < - - - qemu - 0.0 - rev - -EOF - -sed \ - -e '/$/d' \ - -e '//,/<\/info>/d' \ - -e '$b' \ - -e '/^<\/gtester>$/d' "$@" diff --git a/scripts/tap-driver.pl b/scripts/tap-driver.pl new file mode 100755 index 0000000..5e59b5d --- /dev/null +++ b/scripts/tap-driver.pl @@ -0,0 +1,378 @@ +#! /usr/bin/env perl +# Copyright (C) 2011-2013 Free Software Foundation, Inc. +# Copyright (C) 2018 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# As a special exception to the GNU General Public License, if you +# distribute this file as part of a program that contains a +# configuration script generated by Autoconf, you may include it under +# the same distribution terms that you use for the rest of that program. + +# ---------------------------------- # +# Imports, static data, and setup. # +# ---------------------------------- # + +use warnings FATAL => 'all'; +use strict; +use Getopt::Long (); +use TAP::Parser; +use Term::ANSIColor qw(:constants); + +my $ME = "tap-driver.pl"; +my $VERSION = "2018-11-30"; + +my $USAGE = <<'END'; +Usage: + tap-driver [--test-name=TEST] [--color={always|never|auto}] + [--verbose] [--show-failures-only] +END + +my $HELP = "$ME: TAP-aware test driver for QEMU testsuite harness." . + "\n" . $USAGE; + +# It's important that NO_PLAN evaluates "false" as a boolean. +use constant NO_PLAN => 0; +use constant EARLY_PLAN => 1; +use constant LATE_PLAN => 2; + +use constant DIAG_STRING => "#"; + +# ------------------- # +# Global variables. # +# ------------------- # + +my $testno = 0; # Number of test results seen so far. +my $bailed_out = 0; # Whether a "Bail out!" directive has been seen. +my $failed = 0; # Final exit code + +# Whether the TAP plan has been seen or not, and if yes, which kind +# it is ("early" is seen before any test result, "late" otherwise). +my $plan_seen = NO_PLAN; + +# ----------------- # +# Option parsing. # +# ----------------- # + +my %cfg = ( + "color" => 0, + "verbose" => 0, + "show-failures-only" => 0, +); + +my $color = "auto"; +my $test_name = undef; + +# Perl's Getopt::Long allows options to take optional arguments after a space. +# Prevent --color by itself from consuming other arguments +foreach (@ARGV) { + if ($_ eq "--color" || $_ eq "-color") { + $_ = "--color=$color"; + } +} + +Getopt::Long::GetOptions + ( + 'help' => sub { print $HELP; exit 0; }, + 'version' => sub { print "$ME $VERSION\n"; exit 0; }, + 'test-name=s' => \$test_name, + 'color=s' => \$color, + 'show-failures-only' => sub { $cfg{"show-failures-only"} = 1; }, + 'verbose' => sub { $cfg{"verbose"} = 1; }, + ) or exit 1; + +if ($color =~ /^always$/i) { + $cfg{'color'} = 1; +} elsif ($color =~ /^never$/i) { + $cfg{'color'} = 0; +} elsif ($color =~ /^auto$/i) { + $cfg{'color'} = (-t STDOUT); +} else { + die "Invalid color mode: $color\n"; +} + +# ------------- # +# Prototypes. # +# ------------- # + +sub colored ($$); +sub decorate_result ($); +sub extract_tap_comment ($); +sub handle_tap_bailout ($); +sub handle_tap_plan ($); +sub handle_tap_result ($); +sub is_null_string ($); +sub main (); +sub report ($;$); +sub stringify_result_obj ($); +sub testsuite_error ($); + +# -------------- # +# Subroutines. # +# -------------- # + +# If the given string is undefined or empty, return true, otherwise +# return false. This function is useful to avoid pitfalls like: +# if ($message) { print "$message\n"; } +# which wouldn't print anything if $message is the literal "0". +sub is_null_string ($) +{ + my $str = shift; + return ! (defined $str and length $str); +} + +sub stringify_result_obj ($) +{ + my $result_obj = shift; + if ($result_obj->is_unplanned || $result_obj->number != $testno) + { + return "ERROR"; + } + elsif ($plan_seen == LATE_PLAN) + { + return "ERROR"; + } + elsif (!$result_obj->directive) + { + return $result_obj->is_ok ? "PASS" : "FAIL"; + } + elsif ($result_obj->has_todo) + { + return $result_obj->is_actual_ok ? "XPASS" : "XFAIL"; + } + elsif ($result_obj->has_skip) + { + return $result_obj->is_ok ? "SKIP" : "FAIL"; + } + die "$ME: INTERNAL ERROR"; # NOTREACHED +} + +sub colored ($$) +{ + my ($color_string, $text) = @_; + return $color_string . $text . RESET; +} + +sub decorate_result ($) +{ + my $result = shift; + return $result unless $cfg{"color"}; + my %color_for_result = + ( + "ERROR" => BOLD.MAGENTA, + "PASS" => GREEN, + "XPASS" => BOLD.YELLOW, + "FAIL" => BOLD.RED, + "XFAIL" => YELLOW, + "SKIP" => BLUE, + ); + if (my $color = $color_for_result{$result}) + { + return colored ($color, $result); + } + else + { + return $result; # Don't colorize unknown stuff. + } +} + +sub report ($;$) +{ + my ($msg, $result, $explanation) = (undef, @_); + if ($result =~ /^(?:X?(?:PASS|FAIL)|SKIP|ERROR)/) + { + # Output on console might be colorized. + $msg = decorate_result($result); + if ($result =~ /^(?:PASS|XFAIL|SKIP)/) + { + return if $cfg{"show-failures-only"}; + } + else + { + $failed = 1; + } + } + elsif ($result eq "#") + { + $msg = " "; + } + else + { + die "$ME: INTERNAL ERROR"; # NOTREACHED + } + $msg .= " $explanation" if defined $explanation; + print $msg . "\n"; +} + +sub testsuite_error ($) +{ + report "ERROR", "- $_[0]"; +} + +sub handle_tap_result ($) +{ + $testno++; + my $result_obj = shift; + + my $test_result = stringify_result_obj $result_obj; + my $string = $result_obj->number; + + my $description = $result_obj->description; + $string .= " $test_name" unless is_null_string $test_name; + $string .= " $description" unless is_null_string $description; + + if ($plan_seen == LATE_PLAN) + { + $string .= " # AFTER LATE PLAN"; + } + elsif ($result_obj->is_unplanned) + { + $string .= " # UNPLANNED"; + } + elsif ($result_obj->number != $testno) + { + $string .= " # OUT-OF-ORDER (expecting $testno)"; + } + elsif (my $directive = $result_obj->directive) + { + $string .= " # $directive"; + my $explanation = $result_obj->explanation; + $string .= " $explanation" + unless is_null_string $explanation; + } + + report $test_result, $string; +} + +sub handle_tap_plan ($) +{ + my $plan = shift; + if ($plan_seen) + { + # Error, only one plan per stream is acceptable. + testsuite_error "multiple test plans"; + return; + } + # The TAP plan can come before or after *all* the TAP results; we speak + # respectively of an "early" or a "late" plan. If we see the plan line + # after at least one TAP result has been seen, assume we have a late + # plan; in this case, any further test result seen after the plan will + # be flagged as an error. + $plan_seen = ($testno >= 1 ? LATE_PLAN : EARLY_PLAN); + # If $testno > 0, we have an error ("too many tests run") that will be + # automatically dealt with later, so don't worry about it here. If + # $plan_seen is true, we have an error due to a repeated plan, and that + # has already been dealt with above. Otherwise, we have a valid "plan + # with SKIP" specification, and should report it as a particular kind + # of SKIP result. + if ($plan->directive && $testno == 0) + { + my $explanation = is_null_string ($plan->explanation) ? + undef : "- " . $plan->explanation; + report "SKIP", $explanation; + } +} + +sub handle_tap_bailout ($) +{ + my ($bailout, $msg) = ($_[0], "Bail out!"); + $bailed_out = 1; + $msg .= " " . $bailout->explanation + unless is_null_string $bailout->explanation; + testsuite_error $msg; +} + +sub extract_tap_comment ($) +{ + my $line = shift; + if (index ($line, DIAG_STRING) == 0) + { + # Strip leading `DIAG_STRING' from `$line'. + $line = substr ($line, length (DIAG_STRING)); + # And strip any leading and trailing whitespace left. + $line =~ s/(?:^\s*|\s*$)//g; + # Return what is left (if any). + return $line; + } + return ""; +} + +sub main () +{ + my $iterator = TAP::Parser::Iterator::Stream->new(\*STDIN); + my $parser = TAP::Parser->new ({iterator => $iterator }); + + while (defined (my $cur = $parser->next)) + { + # Parsing of TAP input should stop after a "Bail out!" directive. + next if $bailed_out; + + if ($cur->is_plan) + { + handle_tap_plan ($cur); + } + elsif ($cur->is_test) + { + handle_tap_result ($cur); + } + elsif ($cur->is_bailout) + { + handle_tap_bailout ($cur); + } + elsif ($cfg{"verbose"}) + { + my $comment = extract_tap_comment ($cur->raw); + report "#", "$comment" if length $comment; + } + } + # A "Bail out!" directive should cause us to ignore any following TAP + # error. + if (!$bailed_out) + { + if (!$plan_seen) + { + testsuite_error "missing test plan"; + } + elsif ($parser->tests_planned != $parser->tests_run) + { + my ($planned, $run) = ($parser->tests_planned, $parser->tests_run); + my $bad_amount = $run > $planned ? "many" : "few"; + testsuite_error (sprintf "too %s tests run (expected %d, got %d)", + $bad_amount, $planned, $run); + } + } +} + +# ----------- # +# Main code. # +# ----------- # + +main; +exit($failed); + +# Local Variables: +# perl-indent-level: 2 +# perl-continued-statement-offset: 2 +# perl-continued-brace-offset: 0 +# perl-brace-offset: 0 +# perl-brace-imaginary-offset: 0 +# perl-label-offset: -2 +# cperl-indent-level: 2 +# cperl-brace-offset: 0 +# cperl-continued-brace-offset: 0 +# cperl-label-offset: -2 +# cperl-extra-newline-before-brace: t +# cperl-merge-trailing-else: nil +# cperl-continued-statement-offset: 2 +# End: diff --git a/scripts/tap-merge.pl b/scripts/tap-merge.pl new file mode 100755 index 0000000..59e3fa5 --- /dev/null +++ b/scripts/tap-merge.pl @@ -0,0 +1,110 @@ +#! /usr/bin/env perl +# Copyright (C) 2018 Red Hat, Inc. +# +# Author: Paolo Bonzini +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# ---------------------------------- # +# Imports, static data, and setup. # +# ---------------------------------- # + +use warnings FATAL => 'all'; +use strict; +use Getopt::Long (); +use TAP::Parser; + +my $ME = "tap-merge.pl"; +my $VERSION = "2018-11-30"; + +my $HELP = "$ME: merge multiple TAP inputs from stdin."; + +use constant DIAG_STRING => "#"; + +# ----------------- # +# Option parsing. # +# ----------------- # + +Getopt::Long::GetOptions + ( + 'help' => sub { print $HELP; exit 0; }, + 'version' => sub { print "$ME $VERSION\n"; exit 0; }, + ); + +# -------------- # +# Subroutines. # +# -------------- # + +sub main () +{ + my $iterator = TAP::Parser::Iterator::Stream->new(\*STDIN); + my $parser = TAP::Parser->new ({iterator => $iterator }); + my $testno = 0; # Number of test results seen so far. + my $bailed_out = 0; # Whether a "Bail out!" directive has been seen. + + while (defined (my $cur = $parser->next)) + { + if ($cur->is_bailout) + { + $bailed_out = 1; + print DIAG_STRING . " " . $cur->as_string . "\n"; + next; + } + elsif ($cur->is_plan) + { + $bailed_out = 0; + next; + } + elsif ($cur->is_test) + { + $bailed_out = 0 if $cur->number == 1; + $testno++; + $cur = TAP::Parser::Result::Test->new({ + ok => $cur->ok, + test_num => $testno, + directive => $cur->directive, + explanation => $cur->explanation, + description => $cur->description + }); + } + elsif ($cur->is_version) + { + next if $testno > 0; + } + print $cur->as_string . "\n" unless $bailed_out; + } + print "1..$testno\n"; +} + +# ----------- # +# Main code. # +# ----------- # + +main; + +# Local Variables: +# perl-indent-level: 2 +# perl-continued-statement-offset: 2 +# perl-continued-brace-offset: 0 +# perl-brace-offset: 0 +# perl-brace-imaginary-offset: 0 +# perl-label-offset: -2 +# cperl-indent-level: 2 +# cperl-brace-offset: 0 +# cperl-continued-brace-offset: 0 +# cperl-label-offset: -2 +# cperl-extra-newline-before-brace: t +# cperl-merge-trailing-else: nil +# cperl-continued-statement-offset: 2 +# End: diff --git a/tests/Makefile.include b/tests/Makefile.include index 3f5a1d0..aacd070 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -808,41 +808,67 @@ tests/test-qga$(EXESUF): qemu-ga$(EXESUF) tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y) SPEED = quick -GTESTER_OPTIONS = -k $(if $(V),--verbose,-q) -GCOV_OPTIONS = -n $(if $(V),-f,) # gtester tests, possibly with verbose output +# do_test_tap runs all tests, even if some of them fail, while do_test_human +# stops at the first failure unless -k is given on the command line + +define do_test_human_k + $(quiet-@)rc=0; $(foreach COMMAND, $1, \ + $(call quiet-command-run, \ + export MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} $2; \ + $(COMMAND) -m=$(SPEED) -k --tap \ + | ./scripts/tap-driver.pl --test-name="$(notdir $(COMMAND))" $(if $(V),, --show-failures-only) \ + || rc=$$?;, "TEST", "$@: $(COMMAND)")) exit $$rc +endef +define do_test_human_no_k + $(foreach COMMAND, $1, \ + $(call quiet-command, \ + MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} $2 \ + $(COMMAND) -m=$(SPEED) -k --tap \ + | ./scripts/tap-driver.pl --test-name="$(notdir $(COMMAND))" $(if $(V),, --show-failures-only), \ + "TEST", "$@: $(COMMAND)") +) +endef +do_test_human = \ + $(if $(findstring k, $(MAKEFLAGS)), $(do_test_human_k), $(do_test_human_no_k)) + +define do_test_tap + $(call quiet-command, \ + { export MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} $2; \ + $(foreach COMMAND, $1, \ + $(COMMAND) -m=$(SPEED) -k --tap | sed "s/^[a-z][a-z]* [0-9]* /&$(notdir $(COMMAND)) /" || true; ) } \ + | ./scripts/tap-merge.pl | tee "$@" \ + | ./scripts/tap-driver.pl $(if $(V),, --show-failures-only), \ + "TAP","$@") +endef .PHONY: $(patsubst %, check-qtest-%, $(QTEST_TARGETS)) $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-softmmu $(check-qtest-y) - $(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \ - QTEST_QEMU_IMG=qemu-img$(EXESUF) \ - MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \ - gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@") + $(call do_test_human,$(check-qtest-$*-y) $(check-qtest-generic-y), \ + QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \ + QTEST_QEMU_IMG=qemu-img$(EXESUF)) -.PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)) -$(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: % - $(call quiet-command, \ - MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \ - gtester $(GTESTER_OPTIONS) -m=$(SPEED) $*,"GTESTER","$*") +check-unit: $(check-unit-y) + $(call do_test_human, $^) -# gtester tests with XML output +check-speed: $(check-speed-y) + $(call do_test_human, $^) -$(patsubst %, check-report-qtest-%.xml, $(QTEST_TARGETS)): check-report-qtest-%.xml: $(check-qtest-y) - $(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \ - QTEST_QEMU_IMG=qemu-img$(EXESUF) \ - gtester -q $(GTESTER_OPTIONS) -o $@ -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@") +# gtester tests with TAP output -check-report-unit.xml: $(check-unit-y) - $(call quiet-command,gtester -q $(GTESTER_OPTIONS) -o $@ -m=$(SPEED) $^,"GTESTER","$@") +$(patsubst %, check-report-qtest-%.tap, $(QTEST_TARGETS)): check-report-qtest-%.tap: $(check-qtest-y) + $(call do_test_tap, $(check-qtest-$*-y) $(check-qtest-generic-y), \ + QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \ + QTEST_QEMU_IMG=qemu-img$(EXESUF)) -# Reports and overall runs +check-report-unit.tap: $(check-unit-y) + $(call do_test_tap,$^) -check-report.xml: $(patsubst %,check-report-qtest-%.xml, $(QTEST_TARGETS)) check-report-unit.xml - $(call quiet-command,$(SRC_PATH)/scripts/gtester-cat $^ > $@,"GEN","$@") +# Reports and overall runs -check-report.html: check-report.xml - $(call quiet-command,gtester-report $< > $@,"GEN","$@") +check-report.tap: $(patsubst %,check-report-qtest-%.tap, $(QTEST_TARGETS)) check-report-unit.tap + $(call quiet-command,./scripts/tap-merge.py $^ > $@,"GEN","$@") # Per guest TCG tests @@ -957,8 +983,6 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) .PHONY: check-qapi-schema check-qtest check-unit check check-clean check-qapi-schema: $(patsubst %,check-%, $(check-qapi-schema-y)) check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) -check-unit: $(patsubst %,check-%, $(check-unit-y)) -check-speed: $(patsubst %,check-%, $(check-speed-y)) check-block: $(patsubst %,check-%, $(check-block-y)) check: check-qapi-schema check-unit check-qtest check-decodetree check-clean: diff --git a/tests/docker/dockerfiles/centos7.docker b/tests/docker/dockerfiles/centos7.docker index 0a04bfb..e0f18f5 100644 --- a/tests/docker/dockerfiles/centos7.docker +++ b/tests/docker/dockerfiles/centos7.docker @@ -22,6 +22,7 @@ ENV PACKAGES \ mesa-libEGL-devel \ mesa-libgbm-devel \ nettle-devel \ + perl-Test-Harness \ pixman-devel \ SDL-devel \ spice-glib-devel \ diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker index 24b113b..c66e341 100644 --- a/tests/docker/dockerfiles/debian-amd64.docker +++ b/tests/docker/dockerfiles/debian-amd64.docker @@ -16,6 +16,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ liblzo2-dev \ librdmacm-dev \ libsnappy-dev \ + libtest-harness-perl \ libvte-dev # virgl diff --git a/tests/docker/dockerfiles/debian-ports.docker b/tests/docker/dockerfiles/debian-ports.docker index e05a9a9..514ab53 100644 --- a/tests/docker/dockerfiles/debian-ports.docker +++ b/tests/docker/dockerfiles/debian-ports.docker @@ -29,6 +29,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ flex \ gettext \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker index 9a3d168..b30cbe7 100644 --- a/tests/docker/dockerfiles/debian-sid.docker +++ b/tests/docker/dockerfiles/debian-sid.docker @@ -26,6 +26,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ ca-certificates \ flex \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/debian8.docker b/tests/docker/dockerfiles/debian8.docker index 5294563..cdc3f11 100644 --- a/tests/docker/dockerfiles/debian8.docker +++ b/tests/docker/dockerfiles/debian8.docker @@ -29,6 +29,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ gettext \ git \ gnupg \ + libtest-harness-perl \ pkg-config \ python-minimal diff --git a/tests/docker/dockerfiles/debian9.docker b/tests/docker/dockerfiles/debian9.docker index 154ae2a..9561d4f 100644 --- a/tests/docker/dockerfiles/debian9.docker +++ b/tests/docker/dockerfiles/debian9.docker @@ -24,6 +24,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ flex \ gettext \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 0c4eb9e..1d0e3dc 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -70,6 +70,7 @@ ENV PACKAGES \ nss-devel \ numactl-devel \ perl \ + perl-Test-Harness \ pixman-devel \ python3 \ PyYAML \ diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker index 36e2b17..229add5 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -45,6 +45,7 @@ ENV PACKAGES flex bison \ libspice-protocol-dev \ libspice-server-dev \ libssh2-1-dev \ + libtest-harness-perl \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \