From patchwork Fri Nov 2 04:12:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 992176 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="uYx+x4EE"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42mTGK0wLvzB4R1 for ; Fri, 2 Nov 2018 15:13:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id DFDD787D96; Fri, 2 Nov 2018 04:13:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dPS2j9vCzTkY; Fri, 2 Nov 2018 04:13:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 621DF87CC9; Fri, 2 Nov 2018 04:13:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 2948F1C1694 for ; Fri, 2 Nov 2018 04:13:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 21B4285BA3 for ; Fri, 2 Nov 2018 04:13:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IcwygR0vA2GZ for ; Fri, 2 Nov 2018 04:13:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id 65BCC86C80 for ; Fri, 2 Nov 2018 04:13:38 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id o125so1157933qkf.3 for ; Thu, 01 Nov 2018 21:13:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=yKG/a7/+MqRYj+/dSgaucedM5IizibutR6SB8B3w0gI=; b=uYx+x4EESXRyZ5FCWsLF0C9E/5qN9G0MhPO2PDuv7GhXYH7M2/8gzmGcvnz6DkITl4 9cPWLcVy+zoQjTIE+YYDJTERPeV8KAMQiGacJzIzGspT93xX7FE87CR2woNJhCjvDwaN PhKPuTIN/EOpMXxAnUHXtT+PmYO1R/E11r/GUhOhn8LSmOjfZAJRbDb9goaO5soWHXLU h8NA5Asvd5SjhEZS1Nk4LXP+4fWyFf3EFMWA77w0gCtv9+g0WCWYxN/OaorZHqkWWGyQ Ecc1rib+kS2mbmQO6ci0OxvlMD9rDX1TUibRDK8dxqfOj4CfQOLBI6Zx/N70rTv3vNC/ Ty8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=yKG/a7/+MqRYj+/dSgaucedM5IizibutR6SB8B3w0gI=; b=F+NEm73v3558cr68FpIxAJ4sGsT5DgyyicdGkxtomQqDgy/IKH8pOt6+LToUQ3ZOq0 Vo1p4X+3RIcvaSz8hwPKIye/68pYV7IhOHuJDMdnHcedHbWce5z6Ot9mN6jiNr6y9Fev 84Hjfyj0ncUJPfwJ4SMMzR6+0qnQv3PWaaT/MGIcGhiPnZcD/cb6jTrbCHMk/Lndqmyc o1Ya5qiwxlsGcn0Ove7yzwhKbN14U3ogmbkZCJpydanEQ4tF52r3AqCsGlen+mf4Trov u7O+WQMUUpdsbhzjiv4Yaau8Buhnbaicr5ZncTkGxnkcDkmAGpF8BVkVHscugJY5dLvY kYMQ== X-Gm-Message-State: AGRZ1gI/0HkYwmvs7nZs10212YPMYnmwHMyq3U/0o1sDZaAGx8Bd/KPB pyZHa1ZGu2QpP1xMo+Ycqa9X6h3n X-Google-Smtp-Source: AJdET5fQ9vncgpZiK1zmYB7UqPTPBW1GlPQGebyOefjispN/UHxwBf2NUE4X+2wdA4kaJH4EVNTOVw== X-Received: by 2002:a37:5946:: with SMTP id n67-v6mr9082761qkb.202.1541132017245; Thu, 01 Nov 2018 21:13:37 -0700 (PDT) Received: from ultri5.home ([179.181.159.8]) by smtp.gmail.com with ESMTPSA id 83sm32849688qkz.73.2018.11.01.21.13.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 21:13:36 -0700 (PDT) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Fri, 2 Nov 2018 01:12:33 -0300 Message-Id: <20181102041241.28910-5-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181102041241.28910-1-ricardo.martincoski@gmail.com> References: <20181102041241.28910-1-ricardo.martincoski@gmail.com> In-Reply-To: <20181016004230.10393-1-ricardo.martincoski@gmail.com> References: <20181016004230.10393-1-ricardo.martincoski@gmail.com> Subject: [Buildroot] [PATCH v2 04/12] support/testing: create default test case for python packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Asaf Kahlon , Ricardo Martincoski MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Test cases for python packages are very similar among each other: run a simple script in the target that minimally tests the package. So create a new helper class named TestPythonPackageBase that holds all the logic to run a script on the target. This new class is not a subclass of unittest.TestCase and therefore nose2 ignores it, avoiding to create a bogus test case. TestPythonPackageBase adds in build time one or more sample scripts to be run on the target. The test case for the python package must explicitly list them in the "sample_scripts" property. The test case then automatically logins to the target, checks the scripts are really in the rootfs (it calls "md5sum" instead of "ls" or "test" in an attempt to make the logfile more friendly, since someone analysing a failure can easily check the expected script was executed) and then calls the python interpreter passing the sample script as parameter. An optional property "timeout" exists for the case the sample script needs more time to run than the default timeout from the test infra (currently 5 seconds). A simple test case for a package that only supports Python 2 will look like this: |from tests.package.test_python import TestPythonPackageBase, TestPythonBase2 | | |class TestPython2(TestPythonPackageBase, TestPythonBase2): | config_package = \ | """ | BR2_PACKAGE_PYTHON_=y | """ | sample_scripts = ["tests/package/sample_python_.py"] | timeout = 15 When the python package supports both Python 2 and Python 3 a helper class can be used to hold the common properties: |class TestPython(TestPythonPackageBase): | config_package = \ ... |class TestPythonPy2(TestPython, TestPythonBase2): ... |class TestPythonPy3(TestPython, TestPythonBase3): A trick is used in order to allow this new class to change the defconfig used in the build of the image for the testcase: this class' __init__ method calls the __init__ method from a parent class that in turn inherits from unittest.TestCase. This would be a normal usage ... if this class actually inherited from another class! This is done to make nose2 to ignore this class when looking for test cases. And it works because all classes that inherit from it also are subclasses of unittest.TestCase. An alternative solution to this would be to create yet another help class in test_python.py that doesn't inherit from unittest.TestCase to hold only the test_run method and make every test case for python package (not the helper class, i.e. TestPythonArgh) to inherit from this class too. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle Cc: Asaf Kahlon Cc: Thomas Petazzoni Cc: Yegor Yefremov --- NOTE: example of the alternative solution without the mentioned trick: https://gitlab.com/RicardoMartincoski/buildroot/commit/ea7177c7a233a32129230189f609ecd36853d7e2 Changes v1 -> v2: - do not reuse TestPython2 and TestPython3 to for two entirely separate things: (Thomas) - As a base class for testing individual Python packages; - As test cases for the Python interpreter itself; - use a better class hierarchy (Thomas). I did this in various patches (before and after this one) trying to make the review easier. - with the new class hierarchy the trick that allows the defconfig to be changed by each test case without explicitly naming the class that contains the base defconfig is moved from every test case to the new base class. v1: http://patchwork.ozlabs.org/patch/984425/ --- .../package/copy-sample-script-to-target.sh | 7 ++++ support/testing/tests/package/test_python.py | 38 +++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh diff --git a/support/testing/tests/package/copy-sample-script-to-target.sh b/support/testing/tests/package/copy-sample-script-to-target.sh new file mode 100755 index 0000000000..6448a80d6d --- /dev/null +++ b/support/testing/tests/package/copy-sample-script-to-target.sh @@ -0,0 +1,7 @@ +#!/bin/sh +set -e + +shift +for file in "$@"; do + cp -f "${file}" "${TARGET_DIR}/root/" +done diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py index e4679233df..bc2e14a1ed 100644 --- a/support/testing/tests/package/test_python.py +++ b/support/testing/tests/package/test_python.py @@ -73,3 +73,41 @@ class TestPython2(TestPythonInterpreter, TestPythonBase2): class TestPython3(TestPythonInterpreter, TestPythonBase3): version_string = "Python 3" + + +class TestPythonPackageBase(): + config_package = None + config_sample_scripts = \ + """ + BR2_ROOTFS_POST_BUILD_SCRIPT="{}" + BR2_ROOTFS_POST_SCRIPT_ARGS="{}" + """.format(infra.filepath("tests/package/copy-sample-script-to-target.sh"), + "{sample_scripts}") + sample_scripts = None + timeout = -1 + + def __init__(self, names): + # every class that inherits from this one will inherit from TestPythonBase too + super(TestPythonBase, self).__init__(names) + if self.config_package: + self.config += self.config_package + if self.sample_scripts: + scripts = [infra.filepath(s) for s in self.sample_scripts] + self.config += self.config_sample_scripts.format(sample_scripts=" ".join(scripts)) + + def check_sample_scripts_exist(self): + scripts = [os.path.basename(s) for s in self.sample_scripts] + cmd = "md5sum " + " ".join(scripts) + _, exit_code = self.emulator.run(cmd) + self.assertEqual(exit_code, 0) + + def run_sample_scripts(self): + for script in self.sample_scripts: + cmd = self.interpreter + " " + os.path.basename(script) + _, exit_code = self.emulator.run(cmd, timeout=self.timeout) + self.assertEqual(exit_code, 0) + + def test_run(self): + self.login() + self.check_sample_scripts_exist() + self.run_sample_scripts()