From patchwork Sun Sep 19 21:14:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1529882 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=BRCXypUK; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4HCL7g24CNz9sW4 for ; Mon, 20 Sep 2021 07:15:27 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id C61DF8324B; Sun, 19 Sep 2021 23:15:18 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="BRCXypUK"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 3EB2181BC8; Sun, 19 Sep 2021 23:15:03 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 6020181BF4 for ; Sun, 19 Sep 2021 23:14:57 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-ot1-x336.google.com with SMTP id i3-20020a056830210300b0051af5666070so20886457otc.4 for ; Sun, 19 Sep 2021 14:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I10kgmCeNxLsF3Nl3mbmBjjTlCQYMcZOxR4mzTG3H0c=; b=BRCXypUKBch4AVLY1fSGErGfKGNUm4flqLZ2V6XVf7i2NcUNHLs1BysQdUS3DnRcpL Y0qzIqW+sXzAPakwuqLJ+/WD9Mi7tGn05QwCdWOK8DvnrTcQ3ONUOfZmW2uisbZSdOox sk3QRn05nhR8xRzKGza4RzkF4rD7yBVS/Jlbk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I10kgmCeNxLsF3Nl3mbmBjjTlCQYMcZOxR4mzTG3H0c=; b=0vU/WhAKzVtGxR9Xi0G0XqG+et0goY7HudiOMTHW6OaJPzLJ1vbPrUj9nX5H1lqc7n z4CKMW/lNNb7/jJ3V9KmMl17+T34XVcHqCNQqQPuVvXbmIAq5mIGkotLTNZINMGJnb41 8MzTS654Y3vALGnBfXYhYGkZ80imOqHkJQt4c9R7alpe2K0MGfoYumBkm+RZj3bYPu9x vDUfAi5ab6MTCZyLX8OGG003QMVDPJIhdHQNNWEDdbnDGlsUtTSdO3rrD2DOrahXO2RF a8fuZDqRI//Kkkbwm6Er3SAc6L+u1oZVR7mhI05Id2EugHIitiS5xT2WRyykSOdeNiSZ PogQ== X-Gm-Message-State: AOAM530ahYC1H3CHfvx8Z9XKuzbu2bJgctJrBYX4yY45MucMs3KyzwEG 24+IXBKRoKlyq3dkpEB5KydW9TQVTaZ4Zg== X-Google-Smtp-Source: ABdhPJzwFtzK7FI7VuoucORRJ7SGmQWBIOvXFmLjxZ9pXTRHWaZZVXPK5bfX0EqPvUKWA6z3MDI8Yg== X-Received: by 2002:a05:6830:2f5:: with SMTP id r21mr18478651ote.24.1632086095643; Sun, 19 Sep 2021 14:14:55 -0700 (PDT) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id b11sm3082118ooi.0.2021.09.19.14.14.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Sep 2021 14:14:55 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Stephen Warren , Stephen Warren , Simon Glass Subject: [RESEND PATCH v2 1/4] test: Allow vboot tests to run in parallel Date: Sun, 19 Sep 2021 15:14:48 -0600 Message-Id: <20210919211451.3825645-2-sjg@chromium.org> X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog In-Reply-To: <20210919211451.3825645-1-sjg@chromium.org> References: <20210919211451.3825645-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean Update the tests to use separate working directories, so we can run them in parallel. It also makes it possible to see the individual output files after the tests have completed. Signed-off-by: Simon Glass --- (no changes since v1) test/py/tests/test_vboot.py | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index 6dff6779d17..095e00cce36 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -24,6 +24,7 @@ For configuration verification: Tests run with both SHA1 and SHA256 hashing. """ +import os import shutil import struct import pytest @@ -34,16 +35,16 @@ import vboot_evil # Only run the full suite on a few combinations, since it doesn't add any more # test coverage. TESTDATA = [ - ['sha1', '', None, False, True], - ['sha1', '', '-E -p 0x10000', False, False], - ['sha1', '-pss', None, False, False], - ['sha1', '-pss', '-E -p 0x10000', False, False], - ['sha256', '', None, False, False], - ['sha256', '', '-E -p 0x10000', False, False], - ['sha256', '-pss', None, False, False], - ['sha256', '-pss', '-E -p 0x10000', False, False], - ['sha256', '-pss', None, True, False], - ['sha256', '-pss', '-E -p 0x10000', True, True], + ['sha1-basic', 'sha1', '', None, False, True], + ['sha1-pad', 'sha1', '', '-E -p 0x10000', False, False], + ['sha1-pss', 'sha1', '-pss', None, False, False], + ['sha1-pss-pad', 'sha1', '-pss', '-E -p 0x10000', False, False], + ['sha256-basic', 'sha256', '', None, False, False], + ['sha256-pad', 'sha256', '', '-E -p 0x10000', False, False], + ['sha256-pss', 'sha256', '-pss', None, False, False], + ['sha256-pss-pad', 'sha256', '-pss', '-E -p 0x10000', False, False], + ['sha256-pss-required', 'sha256', '-pss', None, True, False], + ['sha256-pss-pad-required', 'sha256', '-pss', '-E -p 0x10000', True, True], ] @pytest.mark.boardspec('sandbox') @@ -52,9 +53,9 @@ TESTDATA = [ @pytest.mark.requiredtool('fdtget') @pytest.mark.requiredtool('fdtput') @pytest.mark.requiredtool('openssl') -@pytest.mark.parametrize("sha_algo,padding,sign_options,required,full_test", +@pytest.mark.parametrize("name,sha_algo,padding,sign_options,required,full_test", TESTDATA) -def test_vboot(u_boot_console, sha_algo, padding, sign_options, required, +def test_vboot(u_boot_console, name, sha_algo, padding, sign_options, required, full_test): """Test verified boot signing with mkimage and verification with 'bootm'. @@ -365,7 +366,9 @@ def test_vboot(u_boot_console, sha_algo, padding, sign_options, required, run_bootm(sha_algo, 'multi required key', '', False) cons = u_boot_console - tmpdir = cons.config.result_dir + '/' + tmpdir = os.path.join(cons.config.result_dir, name) + '/' + if not os.path.exists(tmpdir): + os.mkdir(tmpdir) datadir = cons.config.source_dir + '/test/py/tests/vboot/' fit = '%stest.fit' % tmpdir mkimage = cons.config.build_dir + '/tools/mkimage' From patchwork Sun Sep 19 21:14:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1529883 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=DszpolAw; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4HCL7w0VBFz9sW4 for ; Mon, 20 Sep 2021 07:15:40 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 5504283223; Sun, 19 Sep 2021 23:15:26 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="DszpolAw"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 3A72D8322E; Sun, 19 Sep 2021 23:15:11 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 7EAE782DA1 for ; Sun, 19 Sep 2021 23:14:58 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-ot1-x334.google.com with SMTP id 97-20020a9d006a000000b00545420bff9eso13950684ota.8 for ; Sun, 19 Sep 2021 14:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3RMuou2z54ncgVGP1YZ/uRSQR0wTi/F9Vl9JMiqNElQ=; b=DszpolAwBAyDGcxbFYUgMZtz66+yDhjhlUtr1eUh4yGccF6D3DeAH1gf2e09+X6iu2 rzNoUe8k6F3V3Q8BCOtg6QoCxC1yqt1a92vL4jwGfmDbC7c0SzJamxh76j1c4P1+2muF hvV5ZPCkePp9nUrxOjTMRpyT1w3kPA3P3UbwU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3RMuou2z54ncgVGP1YZ/uRSQR0wTi/F9Vl9JMiqNElQ=; b=HHTmGNy33dLVgBLAAt9aQ9PConSiDbZ+72e7auupXNkNDfLNUd9r4+mNxIvKQnsT61 wpGWUqCIbihYymwx/ejfquT9PtYl9YqsO42wu5/rcG46xmnbK6aHvasGKipti+jNON5J CKWNpv5aEpcxthe8oqBBeT73WXxsSOd5+E0CsgpBhSdaD+LHKJp+00jLrD5FYdasuuFN Juf3BIgVisuWcq2XpGH3PmaVDDKVsIsCKIFdqbc/mWMaQySnEaRKZr/2au+sRQI/WqTE WARGLkqV3ahDi4IM3vZEH4PHZ54ILL3CXYK9sqVEPEDoAA7iepfC9AL9sdxppQ/PrJ9e nLFA== X-Gm-Message-State: AOAM532hdzYL8voMc2nIvTVsTdq5ycOGM13u2DJvoK9W2oIxtfGd6u+f S4T749+I2tg+r29c7xwiBJNnl5+IcwSoqQ== X-Google-Smtp-Source: ABdhPJynNHz5pWxex5n28TEsMLiZwaYdyMLvKPBvsIHdFfmA7SqB1KKS2djwQ+U2NxlnH1KhpG3h8A== X-Received: by 2002:a05:6830:3113:: with SMTP id b19mr15442217ots.189.1632086096844; Sun, 19 Sep 2021 14:14:56 -0700 (PDT) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id b11sm3082118ooi.0.2021.09.19.14.14.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Sep 2021 14:14:56 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Stephen Warren , Stephen Warren , Simon Glass Subject: [RESEND PATCH v2 2/4] test: Allow hush tests to run in parallel Date: Sun, 19 Sep 2021 15:14:49 -0600 Message-Id: <20210919211451.3825645-3-sjg@chromium.org> X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog In-Reply-To: <20210919211451.3825645-1-sjg@chromium.org> References: <20210919211451.3825645-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean The -z tests don't really need to be part of the main set. Separate them out so we can drop the test setup/cleans functions and thus run all tests in parallel. Signed-off-by: Simon Glass --- (no changes since v1) test/py/tests/test_hush_if_test.py | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/test/py/tests/test_hush_if_test.py b/test/py/tests/test_hush_if_test.py index d117921a6ac..37c1608bb22 100644 --- a/test/py/tests/test_hush_if_test.py +++ b/test/py/tests/test_hush_if_test.py @@ -119,11 +119,6 @@ subtests = ( ('test ! ! aaa != aaa -o ! ! bbb = bbb', True), ('test ! ! aaa = aaa -o ! ! bbb != bbb', True), ('test ! ! aaa = aaa -o ! ! bbb = bbb', True), - - # -z operator. - - ('test -z "$ut_var_nonexistent"', True), - ('test -z "$ut_var_exists"', False), ) def exec_hush_if(u_boot_console, expr, result): @@ -141,12 +136,6 @@ def exec_hush_if(u_boot_console, expr, result): response = u_boot_console.run_command(cmd) assert response.strip() == str(result).lower() -def test_hush_if_test_setup(u_boot_console): - """Set up environment variables used during the "if" tests.""" - - u_boot_console.run_command('setenv ut_var_nonexistent') - u_boot_console.run_command('setenv ut_var_exists 1') - @pytest.mark.buildconfigspec('cmd_echo') @pytest.mark.parametrize('expr,result', subtests) def test_hush_if_test(u_boot_console, expr, result): @@ -154,9 +143,12 @@ def test_hush_if_test(u_boot_console, expr, result): exec_hush_if(u_boot_console, expr, result) -def test_hush_if_test_teardown(u_boot_console): - """Clean up environment variables used during the "if" tests.""" - +def test_hush_z(u_boot_console): + """Test the -z operator""" + u_boot_console.run_command('setenv ut_var_nonexistent') + u_boot_console.run_command('setenv ut_var_exists 1') + exec_hush_if(u_boot_console, 'test -z "$ut_var_nonexistent"', True) + exec_hush_if(u_boot_console, 'test -z "$ut_var_exists"', False) u_boot_console.run_command('setenv ut_var_exists') # We might test this on real filesystems via UMS, DFU, 'save', etc. From patchwork Sun Sep 19 21:14:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1529885 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=J4jCMLaC; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4HCL8J6czrz9sW4 for ; Mon, 20 Sep 2021 07:16:00 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id CB44983281; Sun, 19 Sep 2021 23:15:41 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="J4jCMLaC"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 0154982DA1; Sun, 19 Sep 2021 23:15:14 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-oo1-xc35.google.com (mail-oo1-xc35.google.com [IPv6:2607:f8b0:4864:20::c35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id D1D328320F for ; Sun, 19 Sep 2021 23:14:59 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-oo1-xc35.google.com with SMTP id p22-20020a4a8156000000b002a8c9ea1858so1236003oog.11 for ; Sun, 19 Sep 2021 14:14:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LmIEfY9gj7PgeHDhvDOuozGu86aIiapJT1NsoAc70E0=; b=J4jCMLaC2GXtnt9bUmDBNk6Lmg7zjrcKhH9/2aZzwR/hf8iK4HUhtvpNV8bYgXG6sJ I6ITpH7zjS/JmGQkkNIXaPYjSbd4YTsHexADhKi+tXnMxurtnvzp/vhrvdSxoHhPJmPg 3MjspURmmhg6wZD17iARxkJ/8VeAtTvEAbbbE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LmIEfY9gj7PgeHDhvDOuozGu86aIiapJT1NsoAc70E0=; b=V6JVq/Dj5cUKELAB6x+q3C+yhq6Wb7bdpR+rNAD61f/R9PmLtwhTqiFAIQtIS//Ikt PaZmHZjt5tbc+hu25rr6cGrfEX63rKbVBrbQ+8bGbKQGtzTk6G6FmbqLM0YybLxp7AH6 DhZyNN2/2JiB6LDw1TwmE1mEQ89LPBpgzQ9IVIgeIXnTiMXitiulTFd6fObGExlEI8EQ w32KK+p4CHVoIvEQ/ZsTikNXyyVeSfq0kd7xad4yCpUZXCbGb+3+Jgfkuj9k1WkaID5Z 2LtOnphEjqkZQTi7VmyK+bw7Kg3yrqpqYA7ZAUd1ltp0VXW6fOFiQDvaiUdGvRmtYpbG F0Sg== X-Gm-Message-State: AOAM531p+lOTVvqeyCx5MKe0yt3tQaitEwQ/v5OU5EMVeKisx1iR6t2K YjXTCid5viuERYp3QxYbTR2gldDiEXWyfw== X-Google-Smtp-Source: ABdhPJySPoLGYNa11a6IMcA/PKl08OlPEtAvFsZVAjOXs7l5HxUXNGWWCotS80H05ZGKz9kIvdxg2Q== X-Received: by 2002:a4a:6f4a:: with SMTP id i10mr17304479oof.64.1632086097876; Sun, 19 Sep 2021 14:14:57 -0700 (PDT) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id b11sm3082118ooi.0.2021.09.19.14.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Sep 2021 14:14:57 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Stephen Warren , Stephen Warren , Simon Glass Subject: [RESEND PATCH v2 3/4] test: Allow tpm2 tests to run in parallel Date: Sun, 19 Sep 2021 15:14:50 -0600 Message-Id: <20210919211451.3825645-4-sjg@chromium.org> X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog In-Reply-To: <20210919211451.3825645-1-sjg@chromium.org> References: <20210919211451.3825645-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean These tests currently run in a particular sequence, with some of them depending on the actions of earlier tests. Add a check for sandbox and reset to a known state at the start of each test, so that all tests can run in parallel. Signed-off-by: Simon Glass --- (no changes since v1) test/py/tests/test_tpm2.py | 52 +++++++++++++++++++++++++++++++++----- 1 file changed, 46 insertions(+), 6 deletions(-) diff --git a/test/py/tests/test_tpm2.py b/test/py/tests/test_tpm2.py index ac04f7191ec..64cf8cc5959 100644 --- a/test/py/tests/test_tpm2.py +++ b/test/py/tests/test_tpm2.py @@ -40,10 +40,14 @@ def force_init(u_boot_console, force=False): u_boot_console.run_command('tpm2 clear TPM2_RH_PLATFORM') u_boot_console.run_command('echo --- end of init ---') +def is_sandbox(cons): + # Array slice removes leading/trailing quotes. + sys_arch = cons.config.buildconfig.get('config_sys_arch', '"sandbox"')[1:-1] + return sys_arch == 'sandbox' + @pytest.mark.buildconfigspec('cmd_tpm_v2') def test_tpm2_init(u_boot_console): """Init the software stack to use TPMv2 commands.""" - u_boot_console.run_command('tpm2 init') output = u_boot_console.run_command('echo $?') assert output.endswith('0') @@ -54,17 +58,43 @@ def test_tpm2_startup(u_boot_console): Initiate the TPM internal state machine. """ + u_boot_console.run_command('tpm2 startup TPM2_SU_CLEAR') + output = u_boot_console.run_command('echo $?') + assert output.endswith('0') + +def tpm2_sandbox_init(u_boot_console): + """Put sandbox back into a known state so we can run a test + + This allows all tests to run in parallel, since no test depends on another. + """ + u_boot_console.restart_uboot() + u_boot_console.run_command('tpm2 init') + output = u_boot_console.run_command('echo $?') + assert output.endswith('0') u_boot_console.run_command('tpm2 startup TPM2_SU_CLEAR') output = u_boot_console.run_command('echo $?') assert output.endswith('0') + u_boot_console.run_command('tpm2 self_test full') + output = u_boot_console.run_command('echo $?') + assert output.endswith('0') + @pytest.mark.buildconfigspec('cmd_tpm_v2') -def test_tpm2_self_test_full(u_boot_console): +def test_tpm2_sandbox_self_test_full(u_boot_console): """Execute a TPM2_SelfTest (full) command. Ask the TPM to perform all self tests to also enable full capabilities. """ + if is_sandbox(u_boot_console): + u_boot_console.restart_uboot() + u_boot_console.run_command('tpm2 init') + output = u_boot_console.run_command('echo $?') + assert output.endswith('0') + + u_boot_console.run_command('tpm2 startup TPM2_SU_CLEAR') + output = u_boot_console.run_command('echo $?') + assert output.endswith('0') u_boot_console.run_command('tpm2 self_test full') output = u_boot_console.run_command('echo $?') @@ -77,7 +107,8 @@ def test_tpm2_continue_self_test(u_boot_console): Ask the TPM to finish its self tests (alternative to the full test) in order to enter a fully operational state. """ - + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) u_boot_console.run_command('tpm2 self_test continue') output = u_boot_console.run_command('echo $?') assert output.endswith('0') @@ -94,6 +125,8 @@ def test_tpm2_clear(u_boot_console): not have a password set, otherwise this test will fail. ENDORSEMENT and PLATFORM hierarchies are also available. """ + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) u_boot_console.run_command('tpm2 clear TPM2_RH_LOCKOUT') output = u_boot_console.run_command('echo $?') @@ -112,7 +145,8 @@ def test_tpm2_change_auth(u_boot_console): Use the LOCKOUT hierarchy for this. ENDORSEMENT and PLATFORM hierarchies are also available. """ - + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) force_init(u_boot_console) u_boot_console.run_command('tpm2 change_auth TPM2_RH_LOCKOUT unicorn') @@ -136,6 +170,8 @@ def test_tpm2_get_capability(u_boot_console): There is no expected default values because it would depend on the chip used. We can still save them in order to check they have changed later. """ + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) force_init(u_boot_console) ram = u_boot_utils.find_ram_base(u_boot_console) @@ -158,7 +194,8 @@ def test_tpm2_dam_parameters(u_boot_console): the authentication, otherwise the lockout will be engaged after the first failed authentication attempt. """ - + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) force_init(u_boot_console) ram = u_boot_utils.find_ram_base(u_boot_console) @@ -181,6 +218,8 @@ def test_tpm2_pcr_read(u_boot_console): Perform a PCR read of the 0th PCR. Must be zero. """ + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) force_init(u_boot_console) ram = u_boot_utils.find_ram_base(u_boot_console) @@ -208,7 +247,8 @@ def test_tpm2_pcr_extend(u_boot_console): No authentication mechanism is used here, not protecting against packet replay, yet. """ - + if is_sandbox(u_boot_console): + tpm2_sandbox_init(u_boot_console) force_init(u_boot_console) ram = u_boot_utils.find_ram_base(u_boot_console) From patchwork Sun Sep 19 21:14:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1529884 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=l+7p11Tp; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4HCL862H4pz9sW4 for ; Mon, 20 Sep 2021 07:15:50 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E5B9983273; Sun, 19 Sep 2021 23:15:33 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="l+7p11Tp"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id D2C808325E; Sun, 19 Sep 2021 23:15:10 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id EB0D98322E for ; Sun, 19 Sep 2021 23:15:00 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-oi1-x236.google.com with SMTP id w206so10428932oiw.4 for ; Sun, 19 Sep 2021 14:15:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aPDTWhVKSxdEVzfRLGLY/HufJRlHv8pc5JKBGUGMMIk=; b=l+7p11TpeHSt+R9YkCA9vulPUcby1k57g7Ms+a1UT/JA4+OZtzNTWBWA4ERaxw5b8b fQ+uty5jaUgPKczSMDCZpJz3nHDUM2fjUZ7hfJcCLZ5MRQdwP+5bB22/gGOm9X1XFZ8R OmNAalpzCqimib9AeOwSwerE1CfEx8BZaDTCI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aPDTWhVKSxdEVzfRLGLY/HufJRlHv8pc5JKBGUGMMIk=; b=ya9WqAx+lW3+QolQEZcCJv5kXjg7lFal7R5Ix0tc87340oLe7WXZpBkPFp6PyZ0P/f crdn2cK8NU19hUrtG0u+obQd7FE8gqjGVJUkcXrP5T+PT0eWiNWDPuCfRF4UKOiqbMVo 5lJRIC4DhGwkaMTxv+6ZUp9XifJKdcY+cyMN76D3AOxMJxeXi9uZNpB+paA/yeVt6M9E cZmF3jKaI2sBYpe9Z99Tcy1fGT0m1Gpk1P1szU+lokRC8kZGv1gkVoGYb7JBVLfh1GVp eJVUOnJgvNLzTmL07xG9bfk/pDqk9KNfZlx5VNKr4q//7GeNIqmsp4L10DUZ6WExqcNz dJZg== X-Gm-Message-State: AOAM533IsnJpZ6sU/arI2w+uF+2iEkBaDlLPw3ipj/ABudUzf6b4LBgM RwzOa/IwyLbX0idLsm9Zc4AcW26CvtCpuA== X-Google-Smtp-Source: ABdhPJxVYCml6K6G0/giz6Dj6haeKfL2T07hhKmGl7u+3eoJwegX/pWqaSSZb71o6asn4xesWomOBQ== X-Received: by 2002:aca:c2c6:: with SMTP id s189mr19692951oif.123.1632086099182; Sun, 19 Sep 2021 14:14:59 -0700 (PDT) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id b11sm3082118ooi.0.2021.09.19.14.14.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Sep 2021 14:14:58 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Stephen Warren , Stephen Warren , Simon Glass Subject: [RESEND PATCH v2 4/4] doc: test: Explain how to run pytests in parallel Date: Sun, 19 Sep 2021 15:14:51 -0600 Message-Id: <20210919151448.RESEND.v2.4.I4b868f615036d4b8bc4d9f6a8555db49be9f6009@changeid> X-Mailer: git-send-email 2.33.0.464.g1972c5931b-goog In-Reply-To: <20210919211451.3825645-1-sjg@chromium.org> References: <20210919211451.3825645-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean Add documentation for this so people can try it out. At present it does not fully work. Signed-off-by: Simon Glass --- Changes in v2: - Add documentation for how to run parallel tests doc/develop/py_testing.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst index c4cecc0a01b..4f1e1f66e76 100644 --- a/doc/develop/py_testing.rst +++ b/doc/develop/py_testing.rst @@ -103,6 +103,36 @@ will be written to `${build_dir}/test-log.html`. This is best viewed in a web browser, but may be read directly as plain text, perhaps with the aid of the `html2text` utility. +Running tests in parallel +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Note: This does not fully work yet and is documented only so you can try to +fix the problems. + +First install support for parallel tests:: + + pip3 install pytest-xdist + +Then build sandbox in a suitable build directory. It is not possible to use +the --build flag with xdist. + +Finally, run the tests in parallel using the -n flag:: + + # build sandbox first, in a suitable build directory. It is not possible + # to use the --build flag with -n + test/py/test.py -B sandbox --build-dir /tmp/b/sandbox -q -k 'not slow' -n32 + +At least the following non-slow tests are known to fail: + +- test_fit_ecdsa +- test_bind_unbind_with_uclass +- ut_dm_spi_flash +- test_gpt_rename_partition +- test_gpt_swap_partitions +- test_pinmux_status +- test_sqfs_load + + Testing under a debugger ~~~~~~~~~~~~~~~~~~~~~~~~