From patchwork Sun Dec 26 18:49:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 1573257 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=G4SaJVAd; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMVHR5J6zz9t1r for ; Mon, 27 Dec 2021 05:50:43 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id DBE3A814B4; Sun, 26 Dec 2021 18:50:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IpIXVsdjIpfd; Sun, 26 Dec 2021 18:50:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp1.osuosl.org (Postfix) with ESMTP id 3465081495; Sun, 26 Dec 2021 18:50:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 4E0461BF391 for ; Sun, 26 Dec 2021 18:50:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3C1BB60E0B for ; Sun, 26 Dec 2021 18:50:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mb1VRy5NABWf for ; Sun, 26 Dec 2021 18:50:15 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3197660E0A for ; Sun, 26 Dec 2021 18:50:14 +0000 (UTC) Received: by mail-qt1-x82d.google.com with SMTP id bp39so11622891qtb.6 for ; Sun, 26 Dec 2021 10:50:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lOU4wlND2qyCJMFMiALWDpi2fW5jGX6NJbpRPcI8gnM=; b=G4SaJVAdIVm9nr/Ig7aMmj+xKH/jUdYamZYiOFyk0r9TV5S2R11E0LLgiWhrnw/jvC gMHqt6nTtTvncVcfwoTeqEgZ5/foQfiWTxS7lBhT3zV8Ml22ASxtMZqc+I3weWLurlwF lXNEzhWr9JQ/aMVBVb0V3EffE4XnYpgQJCvnhrI+5xRsZ1MZUJ+FJbaGoNEuYDb9wLOE 6z4cckSnawvPS0duXWZxeLp8JF/6ILpIA6eJxbPo5OYMPRLX8CTyo48LSm5Gp8MoG881 P6j3adDCNyti4JybQHsO6YQ32X5sA+JgHFhh40lGy9WWw7ivpxFR4a3uYefWCoE7vBig qBAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lOU4wlND2qyCJMFMiALWDpi2fW5jGX6NJbpRPcI8gnM=; b=LOf5UeQhoBObE9CkljwQ8bfE9k3IA9Yn47EMumesF2yBtazt08BjifcoQoeNeJ4t5C mXv53OiqToezVqIV3/6ES16N19UPFRTnA+ENkh1GtYQWwqAdEHxkSWjS4UAdkYyC+qeo tURveQ61i0qVtTP2swU+X1XpHnrnUMMgEkKYorN+mXBlXDu9t+ssLVQTkE7Qu9ZCZ9hb ZjHTQHaVBSP/e5yGU/fdT6Daopd3TYeA0JXhFAc9W46eMLVAtUmp+4QM4W7QCvnV4gcm 1YxV0TBT50HR9SR3j142XFIq+YvRF/R09DSxP+8ybaC10W3oS8D2SzIwBpi6ndl6rz9T glCg== X-Gm-Message-State: AOAM530qH03qmbfdjnmYo4UaCchiUmgk07HoFIwL1ZsVDcUOmI37nMuw M169qhbGbgiTqshL/XKM5zJOewulc5g= X-Google-Smtp-Source: ABdhPJy9UqX21IwTFWbo+eRyN0yQ1kgC0jWeURaL13mxDakxudcMPLNxcCfb4hutzKu61vhA6cBx+Q== X-Received: by 2002:ac8:5fcc:: with SMTP id k12mr12874799qta.217.1640544612486; Sun, 26 Dec 2021 10:50:12 -0800 (PST) Received: from localhost.localdomain ([179.232.77.5]) by smtp.gmail.com with ESMTPSA id j22sm12349717qko.68.2021.12.26.10.50.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Dec 2021 10:50:12 -0800 (PST) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 26 Dec 2021 15:49:15 -0300 Message-Id: <20211226184919.2753591-2-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> References: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 1/5] utils/check-package: prepare to run external tools X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" Some file formats have well-established syntax checkers. One example of this is the tool 'shellcheck' that can analyse shell scripts for common mistakes. There is no reason to reimplement such tools in check-package, when we can just call them. Add the ability to check-package to call external tools that will run once for each file to be analysed. For simplicity, when the tool generated one or more warnings, count it as a single warning from check-package, that can display something like this: |$ ./utils/check-package package/unscd/S46unscd |package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings |25 lines processed |1 warnings generated |$ ./utils/check-package -vvvvvvvvvvvvvvvv package/unscd/S46unscd |package/unscd/S46unscd:0: run 'shellcheck' and fix the warnings |In package/unscd/S46unscd line 9: | printf "Starting ${NAME}: " | ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo". |In package/unscd/S46unscd line 11: | [ $? -eq 0 ] && echo "OK" || echo "FAIL" | ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?. |In package/unscd/S46unscd line 14: | printf "Stopping ${NAME}: " | ^------------------^ SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo". |In package/unscd/S46unscd line 16: | [ $? -eq 0 ] && echo "OK" || echo "FAIL" | ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?. |For more information: | https://www.shellcheck.net/wiki/SC2059 -- Don't use variables in the printf... | https://www.shellcheck.net/wiki/SC2181 -- Check exit code directly with e.g... |25 lines processed |1 warnings generated In this first commit, add only the ability for check-package to call external tools and not an example of such tool, as adding each tool to call may need update to the docker image and can lead to it's own discussion on how to implement. Signed-off-by: Ricardo Martincoski --- Please read the cover letter before applying. --- utils/check-package | 34 ++++++++++++++++++++++++++++------ utils/checkpackagelib/base.py | 8 ++++++++ 2 files changed, 36 insertions(+), 6 deletions(-) diff --git a/utils/check-package b/utils/check-package index a959fef079..5fb430902d 100755 --- a/utils/check-package +++ b/utils/check-package @@ -8,6 +8,7 @@ import re import six import sys +import checkpackagelib.base import checkpackagelib.lib_config import checkpackagelib.lib_hash import checkpackagelib.lib_mk @@ -87,9 +88,7 @@ def get_lib_from_filename(fname): return None -def is_a_check_function(m): - if not inspect.isclass(m): - return False +def common_inspect_rules(m): # do not call the base class if m.__name__.startswith("_"): return False @@ -100,6 +99,22 @@ def is_a_check_function(m): return True +def is_a_check_function(m): + if not inspect.isclass(m): + return False + if not issubclass(m, checkpackagelib.base._CheckFunction): + return False + return common_inspect_rules(m) + + +def is_external_tool(m): + if not inspect.isclass(m): + return False + if not issubclass(m, checkpackagelib.base._Tool): + return False + return common_inspect_rules(m) + + def print_warnings(warnings): # Avoid the need to use 'return []' at the end of every check function. if warnings is None: @@ -121,14 +136,16 @@ def check_file_using_lib(fname): if flags.verbose >= VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES: print("{}: ignored".format(fname)) return nwarnings, nlines - classes = inspect.getmembers(lib, is_a_check_function) + internal_functions = inspect.getmembers(lib, is_a_check_function) + external_tools = inspect.getmembers(lib, is_external_tool) + all_checks = internal_functions + external_tools if flags.dry_run: - functions_to_run = [c[0] for c in classes] + functions_to_run = [c[0] for c in all_checks] print("{}: would run: {}".format(fname, functions_to_run)) return nwarnings, nlines - objects = [c[1](fname, flags.manual_url) for c in classes] + objects = [c[1](fname, flags.manual_url) for c in internal_functions] for cf in objects: nwarnings += print_warnings(cf.before()) @@ -148,6 +165,11 @@ def check_file_using_lib(fname): for cf in objects: nwarnings += print_warnings(cf.after()) + tools = [c[1](fname) for c in external_tools] + + for tool in tools: + nwarnings += print_warnings(tool.run()) + return nwarnings, nlines diff --git a/utils/checkpackagelib/base.py b/utils/checkpackagelib/base.py index 9544a64e5a..73da925a03 100644 --- a/utils/checkpackagelib/base.py +++ b/utils/checkpackagelib/base.py @@ -16,3 +16,11 @@ class _CheckFunction(object): def after(self): pass + + +class _Tool(object): + def __init__(self, filename): + self.filename = filename + + def run(self): + pass From patchwork Sun Dec 26 18:49:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 1573256 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=onHpAaBV; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMVHF2R2zz9t1r for ; Mon, 27 Dec 2021 05:50:33 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E79AE60E11; Sun, 26 Dec 2021 18:50:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WXuQPsaJPhGq; Sun, 26 Dec 2021 18:50:29 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 36AD260E0B; Sun, 26 Dec 2021 18:50:28 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 337E31BF391 for ; Sun, 26 Dec 2021 18:50:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 22D7260E0D for ; Sun, 26 Dec 2021 18:50:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u5MkkVsybvkU for ; Sun, 26 Dec 2021 18:50:15 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by smtp3.osuosl.org (Postfix) with ESMTPS id 5F4BE60E0B for ; Sun, 26 Dec 2021 18:50:15 +0000 (UTC) Received: by mail-qk1-x731.google.com with SMTP id f138so12770334qke.10 for ; Sun, 26 Dec 2021 10:50:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QrA9gq5B5Q2f9iZYWGyVwtycJejz0ZUmQpZakOwyUmY=; b=onHpAaBVoZhQd972sHHGH2ENPR56vWcVzyUjEZTzCCqMNsCrVBRe1yU7hpTPMA+SLK lLJwL9+cMMOtnZ1JyT2cUb8rOPXJ+vA8dJB02ibVLuPFeIzi6+NlZ5LRSy2dUOe0FyEH pHGSz+RXPfowQBhCU65qgvGNurDFXVte4aRVzg2WK/axHnm977bmFJpZXomuG83v0/2d EGTUMNyUnLwcENNZ9WmFSNFPAiWAO64xkuRVt+50e37rHno8dbzG5W7DYqpoy223o5CU zXkt7cIams3Kx+5yMoXcQi5hMU0p2uIgfnrO6jhW9+nQC8JvuGOl3HT64bJfDAERUWQA CpMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QrA9gq5B5Q2f9iZYWGyVwtycJejz0ZUmQpZakOwyUmY=; b=WPyXnRpPoPTZIyfMOxy6lbgVaJylOx08Ems8VgzJotksQo1xoNNek6QHzNlX6Fg4Sy t0qNhEHwj0HchZaaL8S6JF6+m4qLaFzdFx1GU5loLDQ9yWp5DEKAqMhZFoFMztpjBr/9 mlJe6Z1cTmIPgaMQRXlSgGiNl0e4mUUMyf9pJuhHrwrRiJTe6ALXegWM1g7j/cNdY7Il Dm/4NAJWpf5oAr0mEElUtSvUhBqqOJp3tBUQUVoC/0K5HrdMh1XFKv5lHgpBNK2P4kLS U/2W65ijKUg1ujJX/WcRvCPxNHXWNf0slajtCEh+s6CP6ONXwj7hs/HDZ2CALvarwjvr peiw== X-Gm-Message-State: AOAM533Ihri5U1Sjn3Th8RHlhplLe+b3qO9bX7Q+RmDyNPEmuT0gFqzR FAO+3rhYqja+gHtv1drpeTTYYYeqD/4= X-Google-Smtp-Source: ABdhPJyyyIPl1HbUeqtwD/pKpBOhGoipZzytbRcxKSYUV3NV+woVx2JUCUK6sdNFm/Xhl73CJmxX6g== X-Received: by 2002:a05:620a:21d1:: with SMTP id h17mr10339255qka.495.1640544614021; Sun, 26 Dec 2021 10:50:14 -0800 (PST) Received: from localhost.localdomain ([179.232.77.5]) by smtp.gmail.com with ESMTPSA id j22sm12349717qko.68.2021.12.26.10.50.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Dec 2021 10:50:13 -0800 (PST) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 26 Dec 2021 15:49:16 -0300 Message-Id: <20211226184919.2753591-3-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> References: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 2/5] utils/checkpackagelib: warn about executable files X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" Currently there are no .mk, Config.in, .patch or .hash files with executable permissions in the tree. But we don't want to have that. So warn when a file checked by check-package has executable permission. This check will be reused when testing SysV init scripts in the tree. Signed-off-by: Ricardo Martincoski --- Please read the cover letter before applying. --- utils/checkpackagelib/lib_config.py | 1 + utils/checkpackagelib/lib_hash.py | 1 + utils/checkpackagelib/lib_mk.py | 1 + utils/checkpackagelib/lib_patch.py | 1 + utils/checkpackagelib/test_tool.py | 41 +++++++++++++++++++++++++++++ utils/checkpackagelib/tool.py | 8 ++++++ 6 files changed, 53 insertions(+) create mode 100644 utils/checkpackagelib/test_tool.py create mode 100644 utils/checkpackagelib/tool.py diff --git a/utils/checkpackagelib/lib_config.py b/utils/checkpackagelib/lib_config.py index c348eec399..b05831f2c3 100644 --- a/utils/checkpackagelib/lib_config.py +++ b/utils/checkpackagelib/lib_config.py @@ -10,6 +10,7 @@ from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 from checkpackagelib.lib import EmptyLastLine # noqa: F401 from checkpackagelib.lib import NewlineAtEof # noqa: F401 from checkpackagelib.lib import TrailingSpace # noqa: F401 +from checkpackagelib.tool import NotExecutable # noqa: F401 def _empty_or_comment(text): diff --git a/utils/checkpackagelib/lib_hash.py b/utils/checkpackagelib/lib_hash.py index 3e381119a5..c67046e16d 100644 --- a/utils/checkpackagelib/lib_hash.py +++ b/utils/checkpackagelib/lib_hash.py @@ -10,6 +10,7 @@ from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 from checkpackagelib.lib import EmptyLastLine # noqa: F401 from checkpackagelib.lib import NewlineAtEof # noqa: F401 from checkpackagelib.lib import TrailingSpace # noqa: F401 +from checkpackagelib.tool import NotExecutable # noqa: F401 def _empty_line_or_comment(text): diff --git a/utils/checkpackagelib/lib_mk.py b/utils/checkpackagelib/lib_mk.py index 88e4b15c7c..153754b6f1 100644 --- a/utils/checkpackagelib/lib_mk.py +++ b/utils/checkpackagelib/lib_mk.py @@ -13,6 +13,7 @@ from checkpackagelib.lib import EmptyLastLine # noqa: F401 from checkpackagelib.lib import NewlineAtEof # noqa: F401 from checkpackagelib.lib import TrailingSpace # noqa: F401 from checkpackagelib.lib import Utf8Characters # noqa: F401 +from checkpackagelib.tool import NotExecutable # noqa: F401 # used in more than one check start_conditional = ["ifdef", "ifeq", "ifndef", "ifneq"] diff --git a/utils/checkpackagelib/lib_patch.py b/utils/checkpackagelib/lib_patch.py index e4e914b7f0..caee36158f 100644 --- a/utils/checkpackagelib/lib_patch.py +++ b/utils/checkpackagelib/lib_patch.py @@ -8,6 +8,7 @@ import re from checkpackagelib.base import _CheckFunction from checkpackagelib.lib import NewlineAtEof # noqa: F401 +from checkpackagelib.tool import NotExecutable # noqa: F401 class ApplyOrder(_CheckFunction): diff --git a/utils/checkpackagelib/test_tool.py b/utils/checkpackagelib/test_tool.py new file mode 100644 index 0000000000..e9e0838103 --- /dev/null +++ b/utils/checkpackagelib/test_tool.py @@ -0,0 +1,41 @@ +import os +import pytest +import re +import tempfile +import checkpackagelib.tool as m + +workdir = os.path.join(tempfile.mkdtemp(suffix='-checkpackagelib-test-tool')) +workdir_regex = re.compile(r'/tmp/tmp[^/]*-checkpackagelib-test-tool') + + +def check_file(tool, filename, string, permissions=None): + script = os.path.join(workdir, filename) + with open(script, 'wb') as f: + f.write(string.encode()) + if permissions: + os.chmod(script, permissions) + obj = tool(script) + result = obj.run() + if result is None: + return [] + return [workdir_regex.sub('dir', r) for r in result] + + +NotExecutable = [ + ('664', + 'package.mk', + 0o664, + '', + []), + ('775', + 'package.mk', + 0o775, + '', + ["dir/package.mk:0: This file does not need to be executable"]), + ] + + +@pytest.mark.parametrize('testname,filename,permissions,string,expected', NotExecutable) +def test_NotExecutable(testname, filename, permissions, string, expected): + warnings = check_file(m.NotExecutable, filename, string, permissions) + assert warnings == expected diff --git a/utils/checkpackagelib/tool.py b/utils/checkpackagelib/tool.py new file mode 100644 index 0000000000..f2007be1ff --- /dev/null +++ b/utils/checkpackagelib/tool.py @@ -0,0 +1,8 @@ +import os +from checkpackagelib.base import _Tool + + +class NotExecutable(_Tool): + def run(self): + if os.access(self.filename, os.X_OK): + return ["{}:0: This file does not need to be executable".format(self.filename)] From patchwork Sun Dec 26 18:49:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 1573258 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=R4QAxHHt; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMVHh4v9dz9t1r for ; Mon, 27 Dec 2021 05:50:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3265C60E0B; Sun, 26 Dec 2021 18:50:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4amqTRKZCUgc; Sun, 26 Dec 2021 18:50:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 2AA5360E11; Sun, 26 Dec 2021 18:50:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id A765B1BF391 for ; Sun, 26 Dec 2021 18:50:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 8F87D4034A for ; Sun, 26 Dec 2021 18:50:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 23PP-YR52f3G for ; Sun, 26 Dec 2021 18:50:17 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1D28940235 for ; Sun, 26 Dec 2021 18:50:17 +0000 (UTC) Received: by mail-qv1-xf33.google.com with SMTP id fo11so12240153qvb.4 for ; Sun, 26 Dec 2021 10:50:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HgdfkuWY39wv/P0yeJfcLFVXpzcUQK8N3ZiS9WU0iEI=; b=R4QAxHHt/feG01Z6AGvetCs65hDnU2srL2ch1mMu60rCP9FgTQNgvq4Xm5SVBKMfLF WzAi3T8mpQS3fjBg27Pdd42XL3QQflfcid/a7zBH+ZvI5qSnahS+FTRfP97J6zTbmvts c/abrlQ1Xbiuf0VGLLjcv5eKsAfTxTOuqwg5hayiwM+OLRS/QBvD9KiWWjdyQWYstFBs 9lHDhQdkcExBrdE9o3zbFKc+EO5Aa2HAFgbkcdR/94aVd9zdRN/GN6iiRFMcyiu4bWbY dewvRCbg/unOWPa4uoSkYP3Cnl6bAm295o3fFdaPrfrXA+DmlMDwfG4vxYp7TDiRnVtW 1WWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HgdfkuWY39wv/P0yeJfcLFVXpzcUQK8N3ZiS9WU0iEI=; b=jBZqYoEuTtt7Xm+yreRGkO1FiYpETwMYuh09KORCbkvONoWSVO23R4iSKo+xcsqA5z m0TFyOd2ViSewuWjBcfuQfRTiRhnSZfGglcc6KaOeJrlMBGBIlb68lQNPJfwg4iatmH/ Ni2mAo/+c9ZEj9attbFsnmmGcEZj1KB9Xhe6zIpb/B7rrnOQeTz8UoMgwYD0/+stHL7E qLr7wnkk2dUG2V2IY2rt/Xpav6FUdRQGlLl9/t3k217bzwwYjLGkGAvWsswnP6EKR7PV I4NvqU+WZcuq47Bvud6Kld4AaTDxMo9KRyO/ryjkedsgG6zs3mNZFb/NZVh2eERJqzSI /BVQ== X-Gm-Message-State: AOAM532KAu0nriA7mgXOGWff8KUkP2lnxHNU0d2WsbiAhjyAyncEuGrl TnEd1ZuHZnsXQsKqiiG16WycWN+WSKQ= X-Google-Smtp-Source: ABdhPJzML6Ewoyg0oYOAHBOELd4MdkIv4T1f1ZuIYygmnTYFPc9PAjPNjDau+WsN0WvBHubip17EnQ== X-Received: by 2002:a05:6214:2aab:: with SMTP id js11mr12518671qvb.54.1640544615923; Sun, 26 Dec 2021 10:50:15 -0800 (PST) Received: from localhost.localdomain ([179.232.77.5]) by smtp.gmail.com with ESMTPSA id j22sm12349717qko.68.2021.12.26.10.50.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Dec 2021 10:50:15 -0800 (PST) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 26 Dec 2021 15:49:17 -0300 Message-Id: <20211226184919.2753591-4-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> References: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 3/5] utils/checkpackagelib/lib_sysv: check SysV init scripts X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" Enable the common checks: - consecutive empty lines - empty last line - missing new line at end of file - trailing space - warn for executable files, with the hint to instead use '$(INSTALL) -D -m 0755' in the .mk file Check indent with tabs: - add a simple check function to warn only when the indent is done using spaces or a mix of tabs and spaces. It does not check indenting levels, but it already makes the review easier, since it diferentiates spaces and tabs. Check variables: - check DAEMON is defined - when DAEMON is defined, check the filename is in the form S01daemon - when PIDFILE is defined, expect it to be in /var/run and defined using $DAEMON. Signed-off-by: Ricardo Martincoski --- Please read the cover letter before applying. --- utils/check-package | 5 + utils/checkpackagelib/base.py | 3 + utils/checkpackagelib/lib_sysv.py | 66 +++++++++++++ utils/checkpackagelib/test_lib_sysv.py | 131 +++++++++++++++++++++++++ utils/checkpackagelib/test_tool.py | 25 +++++ utils/checkpackagelib/tool.py | 2 +- 6 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 utils/checkpackagelib/lib_sysv.py create mode 100644 utils/checkpackagelib/test_lib_sysv.py diff --git a/utils/check-package b/utils/check-package index 5fb430902d..f64daed84c 100755 --- a/utils/check-package +++ b/utils/check-package @@ -13,6 +13,7 @@ import checkpackagelib.lib_config import checkpackagelib.lib_hash import checkpackagelib.lib_mk import checkpackagelib.lib_patch +import checkpackagelib.lib_sysv VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES = 3 flags = None # Command line arguments. @@ -66,6 +67,8 @@ DO_NOT_CHECK_INTREE = re.compile(r"|".join([ r"toolchain/toolchain-external/pkg-toolchain-external\.mk$", ])) +SYSV_INIT_SCRIPT_FILENAME = re.compile(r"/S\d\d[^/]+$") + def get_lib_from_filename(fname): if flags.intree_only: @@ -85,6 +88,8 @@ def get_lib_from_filename(fname): return checkpackagelib.lib_mk if fname.endswith(".patch"): return checkpackagelib.lib_patch + if SYSV_INIT_SCRIPT_FILENAME.search(fname): + return checkpackagelib.lib_sysv return None diff --git a/utils/checkpackagelib/base.py b/utils/checkpackagelib/base.py index 73da925a03..f666e4110b 100644 --- a/utils/checkpackagelib/base.py +++ b/utils/checkpackagelib/base.py @@ -24,3 +24,6 @@ class _Tool(object): def run(self): pass + + def hint(self): + return "" diff --git a/utils/checkpackagelib/lib_sysv.py b/utils/checkpackagelib/lib_sysv.py new file mode 100644 index 0000000000..77e6283b25 --- /dev/null +++ b/utils/checkpackagelib/lib_sysv.py @@ -0,0 +1,66 @@ +import os +import re + +from checkpackagelib.base import _CheckFunction +from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 +from checkpackagelib.lib import EmptyLastLine # noqa: F401 +from checkpackagelib.lib import NewlineAtEof # noqa: F401 +from checkpackagelib.lib import TrailingSpace # noqa: F401 +from checkpackagelib.tool import NotExecutable as NotExecutable_base + + +class Indent(_CheckFunction): + INDENTED_WITH_SPACES = re.compile(r"^[\t]* ") + + def check_line(self, lineno, text): + if self.INDENTED_WITH_SPACES.search(text.rstrip()): + return ["{}:{}: should be indented with tabs, see package/busybox/S01syslogd".format(self.filename, lineno), + text] + + +class NotExecutable(NotExecutable_base): + def hint(self): + return ", just make sure you use '$(INSTALL) -D -m 0755' in the .mk file" + + +# avoid check-package running both the parent and child classes. NOTE: NotExecutable_base.__name__ returns 'NotExecutable' +del NotExecutable_base + + +class Variables(_CheckFunction): + DAEMON_VAR = re.compile(r"^DAEMON=[\"']{0,1}([^\"']*)[\"']{0,1}") + PIDFILE_PATTERN = re.compile(r"/var/run/(\$DAEMON|\$\{DAEMON\}).pid") + PIDFILE_VAR = re.compile(r"^PIDFILE=[\"']{0,1}([^\"']*)[\"']{0,1}") + + def before(self): + self.name = None + + def check_line(self, lineno, text): + name_found = self.DAEMON_VAR.search(text.rstrip()) + if name_found: + if self.name: + return ["{}:{}: DAEMON variable redefined, see package/busybox/S01syslogd".format(self.filename, lineno), + text] + self.name = name_found.group(1) + if '/' in self.name: + self.name = os.path.basename(self.name) # to be used in after() to check the expected filename + return ["{}:{}: Do not include path in DAEMON, see package/busybox/S01syslogd".format(self.filename, lineno), + text, + 'DAEMON="{}"'.format(self.name)] + return + + pidfile_found = self.PIDFILE_VAR.search(text.rstrip()) + if pidfile_found: + pidfile = pidfile_found.group(1) + if not self.PIDFILE_PATTERN.match(pidfile): + return ["{}:{}: For PIDFILE use the same pattern found in package/busybox/S01syslogd".format(self.filename, lineno), + text, + 'PIDFILE="/var/run/$DAEMON.pid"'] + + def after(self): + if self.name is None: + return ["{}:0: DAEMON variable not defined, see package/busybox/S01syslogd".format(self.filename)] + expected_filename = re.compile(r"S\d\d{}$".format(self.name)) + if not expected_filename.match(os.path.basename(self.filename)): + return ["{}:0: filename should be S, see package/busybox/S01syslogd".format(self.filename), + "expecting S{}".format(self.name)] diff --git a/utils/checkpackagelib/test_lib_sysv.py b/utils/checkpackagelib/test_lib_sysv.py new file mode 100644 index 0000000000..290539c0e4 --- /dev/null +++ b/utils/checkpackagelib/test_lib_sysv.py @@ -0,0 +1,131 @@ +import os +import pytest +import re +import tempfile +import checkpackagelib.test_util as util +import checkpackagelib.lib_sysv as m +from checkpackagelib.test_tool import check_file as tool_check_file + +workdir = os.path.join(tempfile.mkdtemp(suffix='-checkpackagelib-test-sysv')) +workdir_regex = re.compile(r'/tmp/tmp[^/]*-checkpackagelib-test-sysv') + + +Indent = [ + ('empty file', + 'any', + '', + []), + ('empty line', + 'any', + '\n', + []), + ('ignore whitespace', + 'any', + ' \n', + []), + ('spaces', + 'any', + 'case "$1" in\n' + ' start)', + [['any:2: should be indented with tabs, see package/busybox/S01syslogd', + ' start)']]), + ('tab', + 'any', + 'case "$1" in\n' + '\tstart)', + []), + ('tabs and spaces', + 'any', + 'case "$1" in\n' + '\t start)', + [['any:2: should be indented with tabs, see package/busybox/S01syslogd', + '\t start)']]), + ('spaces and tabs', + 'any', + 'case "$1" in\n' + ' \tstart)', + [['any:2: should be indented with tabs, see package/busybox/S01syslogd', + ' \tstart)']]), + ] + + +@pytest.mark.parametrize('testname,filename,string,expected', Indent) +def test_Indent(testname, filename, string, expected): + warnings = util.check_file(m.Indent, filename, string) + assert warnings == expected + + +NotExecutable = [ + ('SysV', + 'sh-shebang.sh', + 0o775, + '#!/bin/sh', + ["dir/sh-shebang.sh:0: This file does not need to be executable," + " just make sure you use '$(INSTALL) -D -m 0755' in the .mk file"]), + ] + + +@pytest.mark.parametrize('testname,filename,permissions,string,expected', NotExecutable) +def test_NotExecutable(testname, filename, permissions, string, expected): + warnings = tool_check_file(m.NotExecutable, filename, string, permissions) + assert warnings == expected + + +Variables = [ + ('empty file', + 'any', + '', + [['any:0: DAEMON variable not defined, see package/busybox/S01syslogd']]), + ('daemon and pidfile ok', + 'package/busybox/S01syslogd', + 'DAEMON="syslogd"\n' + 'PIDFILE="/var/run/$DAEMON.pid"\n', + []), + ('wrong filename', + 'package/busybox/S01syslog', + 'DAEMON="syslogd"\n' + 'PIDFILE="/var/run/${DAEMON}.pid"\n', + [['package/busybox/S01syslog:0: filename should be S, see package/busybox/S01syslogd', + 'expecting Ssyslogd']]), + ('no pidfile ok', + 'S99something', + 'DAEMON="something"\n', + []), + ('hardcoded pidfile', + 'S99something', + 'DAEMON="something"\n' + 'PIDFILE="/var/run/something.pid"\n', + [['S99something:2: For PIDFILE use the same pattern found in package/busybox/S01syslogd', + 'PIDFILE="/var/run/something.pid"\n', + 'PIDFILE="/var/run/$DAEMON.pid"']]), + ('redefined daemon', + 'S50any', + 'DAEMON="any"\n' + 'DAEMON="other"\n', + [['S50any:2: DAEMON variable redefined, see package/busybox/S01syslogd', + 'DAEMON="other"\n']]), + ('daemon name with dash', + 'S82cups-browsed', + 'DAEMON="cups-browsed"', + []), + ('daemon with path', + 'S50avahi-daemon', + 'DAEMON=/usr/sbin/avahi-daemon', + [['S50avahi-daemon:1: Do not include path in DAEMON, see package/busybox/S01syslogd', + 'DAEMON=/usr/sbin/avahi-daemon', + 'DAEMON="avahi-daemon"']]), + ('daemon with path and wrong filename', + 'S50avahi', + 'DAEMON=/usr/sbin/avahi-daemon', + [['S50avahi:1: Do not include path in DAEMON, see package/busybox/S01syslogd', + 'DAEMON=/usr/sbin/avahi-daemon', + 'DAEMON="avahi-daemon"'], + ['S50avahi:0: filename should be S, see package/busybox/S01syslogd', + 'expecting Savahi-daemon']]), + ] + + +@pytest.mark.parametrize('testname,filename,string,expected', Variables) +def test_Variables(testname, filename, string, expected): + warnings = util.check_file(m.Variables, filename, string) + assert warnings == expected diff --git a/utils/checkpackagelib/test_tool.py b/utils/checkpackagelib/test_tool.py index e9e0838103..aab7cb51c9 100644 --- a/utils/checkpackagelib/test_tool.py +++ b/utils/checkpackagelib/test_tool.py @@ -39,3 +39,28 @@ NotExecutable = [ def test_NotExecutable(testname, filename, permissions, string, expected): warnings = check_file(m.NotExecutable, filename, string, permissions) assert warnings == expected + + +NotExecutable_hint = [ + ('no hint', + "", + 'sh-shebang.sh', + 0o775, + '#!/bin/sh', + ["dir/sh-shebang.sh:0: This file does not need to be executable"]), + ('hint', + ", very special hint", + 'sh-shebang.sh', + 0o775, + '#!/bin/sh', + ["dir/sh-shebang.sh:0: This file does not need to be executable, very special hint"]), + ] + + +@pytest.mark.parametrize('testname,hint,filename,permissions,string,expected', NotExecutable_hint) +def test_NotExecutable_hint(testname, hint, filename, permissions, string, expected): + class NotExecutable(m.NotExecutable): + def hint(self): + return hint + warnings = check_file(NotExecutable, filename, string, permissions) + assert warnings == expected diff --git a/utils/checkpackagelib/tool.py b/utils/checkpackagelib/tool.py index f2007be1ff..e931272554 100644 --- a/utils/checkpackagelib/tool.py +++ b/utils/checkpackagelib/tool.py @@ -5,4 +5,4 @@ from checkpackagelib.base import _Tool class NotExecutable(_Tool): def run(self): if os.access(self.filename, os.X_OK): - return ["{}:0: This file does not need to be executable".format(self.filename)] + return ["{}:0: This file does not need to be executable{}".format(self.filename, self.hint())] From patchwork Sun Dec 26 18:49:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 1573259 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=cIVb7vWS; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMVHx13mXz9t1r for ; Mon, 27 Dec 2021 05:51:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id EDA6A400CB; Sun, 26 Dec 2021 18:51:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p7IZsvvb77Bf; Sun, 26 Dec 2021 18:51:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 5D15F400DC; Sun, 26 Dec 2021 18:51:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 4A37F1BF391 for ; Sun, 26 Dec 2021 18:50:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 3855E400A2 for ; Sun, 26 Dec 2021 18:50:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hl4-OJ6vwkvx for ; Sun, 26 Dec 2021 18:50:18 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by smtp4.osuosl.org (Postfix) with ESMTPS id 981674008D for ; Sun, 26 Dec 2021 18:50:18 +0000 (UTC) Received: by mail-qk1-x735.google.com with SMTP id e25so7083740qkl.12 for ; Sun, 26 Dec 2021 10:50:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yNHfNn/uAEZyGNgvfF77L0s9+E91UMrh31IFxlq4zgo=; b=cIVb7vWSg6u0eO27pjgRHIydzWjsEUrqTFI0l2h1jLF6ZcvvPLY6WX5upAIde3WHSJ xmRljB9nxn9j2mklUEIzCVsFEAO+RDtHkiefZolA3qEhpXd/gSm+LBihsm6qlwFBb2LF bh6muo7BzahGenaiYUEYX6sZcIHmZYZ8BAbW2rmAIPSzjRRUxKfyjsJ+IN8YkY0uT3M/ dTkHZqjsfKSelGiQG0i4qClI4rF5gBBWyLwMUj6ppesQAeM8teBKoPTn8uYNEScu+cQ+ 5yHwsOrUTZOtCkIggkYJLGuMe3oUqxvlXDvlilO8pGr7L/vUk33gAgwD18hxoa2Jw5G7 Ik2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yNHfNn/uAEZyGNgvfF77L0s9+E91UMrh31IFxlq4zgo=; b=M0pgw2/1VM4Ma3XE4WLWo3RBs83bioW4ccUMoBBf4hgIVPQ+qw/Yfp7ht1WlAWkNBb FXzOZYumEYdB/mAR61n+vgqaLHpQ/7gUUAgFgq8Cqz1zRcIjYyEjaukxjn//m6Gh2YMW EsXuAQM9TnVUF4s/X/Gp4TEyT/k3FqZZxId5DN0h5I521WAe2I7WQtXIXmdjaQ878W+t S5/jnnVOI1+/U3dYJQtQ7to94oj7yZdMckpyss35dMzM9HGfv8Fa4LNPHbTqOrScs03Q qR5ZnwiD7XMMbJ1+iKm6dEKtrDMhGnMkddYezN1jPxXy5sPxDwE3UDwl7MniQLELZT7J mtMg== X-Gm-Message-State: AOAM532dtVBmTvDxnaoyFl9s2qx6pWsLtDKc+F1BzNQn0RXDjuLAMqgx X0H89dGGitrMovgJciLdtiUWPKE6/kE= X-Google-Smtp-Source: ABdhPJwjQgUrsg8SR8c028ANLLJMvDQoFTiO65K5mqL7To+xfsBdEu4amrb1k11dkYpuyts+UqWG8A== X-Received: by 2002:a37:42d0:: with SMTP id p199mr10145886qka.523.1640544617459; Sun, 26 Dec 2021 10:50:17 -0800 (PST) Received: from localhost.localdomain ([179.232.77.5]) by smtp.gmail.com with ESMTPSA id j22sm12349717qko.68.2021.12.26.10.50.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Dec 2021 10:50:17 -0800 (PST) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 26 Dec 2021 15:49:18 -0300 Message-Id: <20211226184919.2753591-5-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> References: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 4/5] support/docker: add shellcheck X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" Allow developers to run check-package for init scripts, that call shellcheck, without having to install the tool. Since the docker have a fixed version of the tool, there will be no difference between runs in different machines. One can call: $ utils/docker-run utils/check-package package/package/S* $ utils/docker-run shellcheck package/package/S* This change also allows to eventually run check-package for init scripts in the GitLab CI. Signed-off-by: Ricardo Martincoski Acked-by: Romain Naour --- Please read the cover letter before applying. --- support/docker/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile index 792e70e292..2b14f0f7a7 100644 --- a/support/docker/Dockerfile +++ b/support/docker/Dockerfile @@ -44,6 +44,7 @@ RUN apt-get install -y --no-install-recommends \ qemu-system-arm \ qemu-system-x86 \ rsync \ + shellcheck \ subversion \ unzip \ wget \ From patchwork Sun Dec 26 18:49:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 1573260 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=UpXksEHM; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMVJ86xmmz9t1r for ; Mon, 27 Dec 2021 05:51:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5353B400A2; Sun, 26 Dec 2021 18:51:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UZeUmPJoPAOR; Sun, 26 Dec 2021 18:51:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 5456940319; Sun, 26 Dec 2021 18:51:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 3CCB81BF391 for ; Sun, 26 Dec 2021 18:50:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 372BE400A2 for ; Sun, 26 Dec 2021 18:50:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iU3enTXXs-Hd for ; Sun, 26 Dec 2021 18:50:20 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by smtp4.osuosl.org (Postfix) with ESMTPS id 30D934008D for ; Sun, 26 Dec 2021 18:50:19 +0000 (UTC) Received: by mail-qv1-xf35.google.com with SMTP id kj16so12257533qvb.2 for ; Sun, 26 Dec 2021 10:50:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9SbSoISW1t9JVlQNrERd6V0Q0nr69DsIuXXXA6/DWcc=; b=UpXksEHMP399In94OXjgFNyhDOVWqiDjmYixCQYLxgj2a6+xTELQW6VvQF69blUbv8 0Ek4S87UUh4oDxEFFRrR+tG+ZcPMuxw4HyL4Jl9KCyo6eQc5v6y7FrslbdnTzGAhGKe6 SF7tKC+oyzC1RsaAoM2DPlExsfa0VI9LCJOsRBtMy7nlqQnAm/xFLvFl3y5w64OGR0FY jQyvcztIpmIICfVYwgIkQopqo3NhbrXB9EkmFOCApuddZkO/e+JW/dSBJpO3F562myYS wOZJld4dlHA0K1ZBMxZI1s1b7ot1Zlsfdcse9xNrMBQBo6l9fZ6RHNp+iACfS0Q+6oo2 ZKtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9SbSoISW1t9JVlQNrERd6V0Q0nr69DsIuXXXA6/DWcc=; b=UW5fAQGjRBV3+YjjkrpyKakoCdbtJCEvMMZWZ6iqGt+NnXshswL3y1X+djC4CMQbpk MguWJOxFsXQ/VXPJg4J5FTvipByG7Q9gLth5vQrU280BQymcC2oSF6BkMe5VTYzUYB1E mNTwFmbjH0Qr+/3P0LGwOA5Z2YjFCEJ4o4XT1E56C0g2m9eEW2NU+DfZy9DNz7q56drn SXz2tO0JsREa7vd7hBWUFz4iBzIlhEudt5LZ9MCftXlUPMyFB+/fGGdIWCXUwRWmfNQC v0ZihAK2oFgASgc3yGFByu8N8Bm6j3Fz/kBMcE7o8lSffHBSCd+kMutdwUCYTW+bgzXf zvOA== X-Gm-Message-State: AOAM531S+QsJAsP4r1sW0ynsA9+AgkSmzldrlysLs7f1JMn3YyGIvOLT VbfMV87SUujqsZ76/tNzGGtiyYib6Xk= X-Google-Smtp-Source: ABdhPJzatpqsyjgFQ2z+2wFkmLkGfEHcaS2ReelZb6GtFL4UqrxuMFxoGiFt8s1BBa7cp5Lvy/v10g== X-Received: by 2002:a05:6214:e63:: with SMTP id jz3mr12719400qvb.65.1640544618964; Sun, 26 Dec 2021 10:50:18 -0800 (PST) Received: from localhost.localdomain ([179.232.77.5]) by smtp.gmail.com with ESMTPSA id j22sm12349717qko.68.2021.12.26.10.50.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Dec 2021 10:50:18 -0800 (PST) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 26 Dec 2021 15:49:19 -0300 Message-Id: <20211226184919.2753591-6-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> References: <20211226184919.2753591-1-ricardo.martincoski@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 5/5] utils/checkpackagelib/lib_sysv: run shellcheck X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" For simplicity, when shellcheck returns one or more warnings, count it as a single check-package warning. The developer can get the full output either by running shellcheck directly or by calling check-package with enough -v. Examples: |$ ./utils/docker-run utils/check-package --include Shellcheck package/polkit/S50polkit |package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings |51 lines processed |1 warnings generated |$ ./utils/docker-run utils/check-package --include Shellcheck -vvvvvvvvvvv package/polkit/S50polkit |package/polkit/S50polkit:0: run 'shellcheck' and fix the warnings |In package/polkit/S50polkit line 43: |< tab >start|stop|restart|reload) | ^----^ SC2221: This pattern always overrides a later one on line 45. |In package/polkit/S50polkit line 45: |< tab >reload) | ^----^ SC2222: This pattern never matches because of a previous pattern on line 43. |For more information: | https://www.shellcheck.net/wiki/SC2221 -- This pattern always overrides a l... | https://www.shellcheck.net/wiki/SC2222 -- This pattern never matches becaus... |51 lines processed |1 warnings generated NOTICE: shellcheck results depends on the version of the tool. This is why the examples above run inside the docker image. Signed-off-by: Ricardo Martincoski --- Please read the cover letter before applying. --- utils/checkpackagelib/lib_sysv.py | 1 + utils/checkpackagelib/test_tool.py | 46 ++++++++++++++++++++++++++++++ utils/checkpackagelib/tool.py | 15 ++++++++++ 3 files changed, 62 insertions(+) diff --git a/utils/checkpackagelib/lib_sysv.py b/utils/checkpackagelib/lib_sysv.py index 77e6283b25..786510badf 100644 --- a/utils/checkpackagelib/lib_sysv.py +++ b/utils/checkpackagelib/lib_sysv.py @@ -7,6 +7,7 @@ from checkpackagelib.lib import EmptyLastLine # noqa: F401 from checkpackagelib.lib import NewlineAtEof # noqa: F401 from checkpackagelib.lib import TrailingSpace # noqa: F401 from checkpackagelib.tool import NotExecutable as NotExecutable_base +from checkpackagelib.tool import Shellcheck # noqa: F401 class Indent(_CheckFunction): diff --git a/utils/checkpackagelib/test_tool.py b/utils/checkpackagelib/test_tool.py index aab7cb51c9..c5c1d48dfe 100644 --- a/utils/checkpackagelib/test_tool.py +++ b/utils/checkpackagelib/test_tool.py @@ -64,3 +64,49 @@ def test_NotExecutable_hint(testname, hint, filename, permissions, string, expec return hint warnings = check_file(NotExecutable, filename, string, permissions) assert warnings == expected + + +Shellcheck = [ + ('missing shebang', + 'empty.sh', + '', + ["dir/empty.sh:0: run 'shellcheck' and fix the warnings", + "In dir/empty.sh line 1:", + "^-- SC2148: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.", + "For more information:", + " https://www.shellcheck.net/wiki/SC2148 -- Tips depend on target shell and y..."]), + ('sh shebang', + 'sh-shebang.sh', + '#!/bin/sh', + []), + ('bash shebang', + 'bash-shebang.sh', + '#!/bin/bash', + []), + ('2 warnings', + 'unused.sh', + 'unused=""', + ["dir/unused.sh:0: run 'shellcheck' and fix the warnings", + "In dir/unused.sh line 1:", + 'unused=""', + "^-- SC2148: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.", + "^----^ SC2034: unused appears unused. Verify use (or export if used externally).", + "For more information:", + " https://www.shellcheck.net/wiki/SC2148 -- Tips depend on target shell and y...", + " https://www.shellcheck.net/wiki/SC2034 -- unused appears unused. Verify use..."]), + ('tab', + 'tab.sh', + '\t#!/bin/sh', + ["dir/tab.sh:0: run 'shellcheck' and fix the warnings", + "In dir/tab.sh line 1:", + '\t#!/bin/sh', + "^-- SC1114: Remove leading spaces before the shebang.", + "For more information:", + " https://www.shellcheck.net/wiki/SC1114 -- Remove leading spaces before the ..."]), + ] + + +@pytest.mark.parametrize('testname,filename,string,expected', Shellcheck) +def test_Shellcheck(testname, filename, string, expected): + warnings = check_file(m.Shellcheck, filename, string) + assert warnings == expected diff --git a/utils/checkpackagelib/tool.py b/utils/checkpackagelib/tool.py index e931272554..47f9634486 100644 --- a/utils/checkpackagelib/tool.py +++ b/utils/checkpackagelib/tool.py @@ -1,4 +1,5 @@ import os +import subprocess from checkpackagelib.base import _Tool @@ -6,3 +7,17 @@ class NotExecutable(_Tool): def run(self): if os.access(self.filename, os.X_OK): return ["{}:0: This file does not need to be executable{}".format(self.filename, self.hint())] + + +class Shellcheck(_Tool): + def run(self): + cmd = ['shellcheck', self.filename] + try: + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout = p.communicate()[0] + processed_output = [str(line.decode().rstrip()) for line in stdout.splitlines() if line] + if p.returncode == 0: + return + return ["{}:0: run 'shellcheck' and fix the warnings".format(self.filename)] + processed_output + except FileNotFoundError: + return ["{}:0: failed to call 'shellcheck'".format(self.filename)]