From patchwork Fri Feb 24 11:51:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neha Malcom Francis X-Patchwork-Id: 1747432 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=ti.com header.i=@ti.com header.a=rsa-sha256 header.s=ti-com-17Q1 header.b=lUWogKIW; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4PNSsl0ZXwz240f for ; Fri, 24 Feb 2023 22:51:37 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E741A85B10; Fri, 24 Feb 2023 12:51:27 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=ti.com header.i=@ti.com header.b="lUWogKIW"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 8B3F985AF7; Fri, 24 Feb 2023 12:51:25 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS, SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from fllv0015.ext.ti.com (fllv0015.ext.ti.com [198.47.19.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 87B0385707 for ; Fri, 24 Feb 2023 12:51:19 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=n-francis@ti.com Received: from lelv0265.itg.ti.com ([10.180.67.224]) by fllv0015.ext.ti.com (8.15.2/8.15.2) with ESMTP id 31OBp5nZ044006; Fri, 24 Feb 2023 05:51:05 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1677239465; bh=jvc/iacgv7mksnyQrIH73iS2wXucBOoLudSFWu4Blw4=; h=From:To:CC:Subject:Date; b=lUWogKIW9gOYdW8/2W3FF2d2S2HOqBBrgq5nWrXur25gGppsJ6Eehm93q/vTXBZwL Jo/q3vmC7flgW+uuMh76oGRtJUMMO0a4HwMXNBYii+X1zOE7a1PqT51lF3nbJ7fZIZ 5EO6KKfIaUjaYxVA3zIG1uq6uHUKHNZYocnr4Q8U= Received: from DLEE114.ent.ti.com (dlee114.ent.ti.com [157.170.170.25]) by lelv0265.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 31OBp54E027001 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 24 Feb 2023 05:51:05 -0600 Received: from DLEE115.ent.ti.com (157.170.170.26) by DLEE114.ent.ti.com (157.170.170.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.16; Fri, 24 Feb 2023 05:51:05 -0600 Received: from lelv0326.itg.ti.com (10.180.67.84) by DLEE115.ent.ti.com (157.170.170.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.16 via Frontend Transport; Fri, 24 Feb 2023 05:51:04 -0600 Received: from ula0497641.dhcp.ti.com (ileaxei01-snat2.itg.ti.com [10.180.69.6]) by lelv0326.itg.ti.com (8.15.2/8.15.2) with ESMTP id 31OBp1Em004638; Fri, 24 Feb 2023 05:51:02 -0600 From: Neha Malcom Francis To: , , , , CC: , , Subject: [PATCH v3] binman: bintool: Add support for tool directories Date: Fri, 24 Feb 2023 17:21:01 +0530 Message-ID: <20230224115101.563729-1-n-francis@ti.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean Currently, bintool supports external compilable tools as single executable files. Adding support for git repos that can be used to run non-compilable scripting tools that cannot otherwise be present in binman. Signed-off-by: Neha Malcom Francis --- Changes in v3: - moved back to using DOWNLOAD_DIR as community is making relevant changes - extended coverage for bintool_test.py - added function comment for new parameter Changes in v2: - added parameter to obtain path to download the directory optionally, enables flexibility to avoid using DOWNLOAD_DESTDIR - added test to bintool_test.py - s/FETCH_NO_BUILD/FETCH_SOURCE - code reformatting tools/binman/bintool.py | 47 +++++++++++++++++++++++++++++----- tools/binman/bintool_test.py | 43 +++++++++++++++++++++++++++++++ tools/binman/btool/_testing.py | 4 +++ tools/patman/tools.py | 2 +- 4 files changed, 88 insertions(+), 8 deletions(-) diff --git a/tools/binman/bintool.py b/tools/binman/bintool.py index 8fda13ff01..33f563c46f 100644 --- a/tools/binman/bintool.py +++ b/tools/binman/bintool.py @@ -32,12 +32,13 @@ FORMAT = '%-16.16s %-12.12s %-26.26s %s' modules = {} # Possible ways of fetching a tool (FETCH_COUNT is number of ways) -FETCH_ANY, FETCH_BIN, FETCH_BUILD, FETCH_COUNT = range(4) +FETCH_ANY, FETCH_BIN, FETCH_BUILD, FETCH_SOURCE, FETCH_COUNT = range(5) FETCH_NAMES = { FETCH_ANY: 'any method', FETCH_BIN: 'binary download', - FETCH_BUILD: 'build from source' + FETCH_BUILD: 'build from source', + FETCH_SOURCE: 'download source without building' } # Status of tool fetching @@ -201,12 +202,13 @@ class Bintool: print(f'- trying method: {FETCH_NAMES[try_method]}') result = try_fetch(try_method) if result: + method = try_method break else: result = try_fetch(method) if not result: return FAIL - if result is not True: + if result is not True and method != FETCH_SOURCE: fname, tmpdir = result dest = os.path.join(DOWNLOAD_DESTDIR, self.name) print(f"- writing to '{dest}'") @@ -261,7 +263,7 @@ class Bintool: show_status(col.RED, 'Failures', status[FAIL]) return not status[FAIL] - def run_cmd_result(self, *args, binary=False, raise_on_error=True): + def run_cmd_result(self, *args, binary=False, raise_on_error=True, add_name=True): """Run the bintool using command-line arguments Args: @@ -270,6 +272,7 @@ class Bintool: binary (bool): True to return output as bytes instead of str raise_on_error (bool): True to raise a ValueError exception if the tool returns a non-zero return code + add_name (bool): True to add bintool name to the beginning of command Returns: CommandResult: Resulting output from the bintool, or None if the @@ -278,7 +281,10 @@ class Bintool: if self.name in self.missing_list: return None name = os.path.expanduser(self.name) # Expand paths containing ~ - all_args = (name,) + args + if add_name: + all_args = (name,) + args + else: + all_args = args env = tools.get_env_with_path() tout.detail(f"bintool: {' '.join(all_args)}") result = command.run_pipe( @@ -304,18 +310,19 @@ class Bintool: tout.debug(result.stderr) return result - def run_cmd(self, *args, binary=False): + def run_cmd(self, *args, binary=False, add_name=True): """Run the bintool using command-line arguments Args: args (list of str): Arguments to provide, in addition to the bintool name binary (bool): True to return output as bytes instead of str + add_name (bool): True to add bintool name to the beginning of command Returns: str or bytes: Resulting stdout from the bintool """ - result = self.run_cmd_result(*args, binary=binary) + result = self.run_cmd_result(*args, binary=binary, add_name=add_name) if result: return result.stdout @@ -354,6 +361,32 @@ class Bintool: return None return fname, tmpdir + @classmethod + def fetch_from_git(cls, git_repo, name): + """Fetch a bintool git repo + + This clones the repo and returns + + Args: + git_repo (str): URL of git repo + name (str): Bintool name assigned as tool directory name + + Returns: + str: Directory of fetched repo + or None on error + """ + dir = os.path.join(DOWNLOAD_DESTDIR, name) + if os.path.exists(dir): + print(f"- {dir} repo already exists") + return None + os.mkdir(dir) + print(f"- clone git repo '{git_repo}' to '{dir}'") + tools.run('git', 'clone', '--depth', '1', git_repo, dir) + if len(os.listdir(dir)) == 0: + print(f"- '{dir}' repo is empty") + return None + return dir + @classmethod def fetch_from_url(cls, url): """Fetch a bintool from a URL diff --git a/tools/binman/bintool_test.py b/tools/binman/bintool_test.py index 7efb8391db..e391744289 100644 --- a/tools/binman/bintool_test.py +++ b/tools/binman/bintool_test.py @@ -258,6 +258,36 @@ class TestBintool(unittest.TestCase): fname = os.path.join(self._indir, '_testing') return fname if write_file else self.fname, stdout.getvalue() + def check_fetch_source_method(self, write_file): + """Check output from fetching using SOURCE method + + Args: + write_file (bool): True to write to output directory + + Returns: + tuple: + str: Filename of directory created + str: Contents of stdout + """ + def fake_run(*cmd): + if cmd[0] == 'git': + # See Bintool.fetch_from_git() + tmpdir = cmd[5] + self.fname = os.path.join(tmpdir, 'gitfile') + if write_file: + tools.write_file(self.fname, b'hello') + + btest = Bintool.create('_testing') + col = terminal.Color() + self.dirname = None + with unittest.mock.patch.object(bintool, 'DOWNLOAD_DESTDIR', + self._indir): + with unittest.mock.patch.object(tools, 'run', side_effect=fake_run): + with test_util.capture_sys_output() as (stdout, _): + btest.fetch_tool(bintool.FETCH_SOURCE, col, False) + dirname = os.path.join(self._indir, 'mygit') + return self.fname if write_file else dirname, stdout.getvalue() + def test_build_method(self): """Test fetching using the build method""" fname, stdout = self.check_build_method(write_file=True) @@ -270,6 +300,17 @@ class TestBintool(unittest.TestCase): self.assertFalse(os.path.exists(fname)) self.assertIn(f"File '{fname}' was not produced", stdout) + def test_fetch_source_method(self): + """Test fetching using the source method""" + dirname, stdout = self.check_fetch_source_method(write_file=True) + self.assertTrue(os.path.exists(dirname)) + + def test_fetch_source_method_fail(self): + """Test fetching using the source method when directory is empty""" + dirname, stdout = self.check_fetch_source_method(write_file=False) + self.assertEqual(len(os.listdir(dirname)), 0) + self.assertIn(f"'{dirname}' repo is empty", stdout) + def test_install(self): """Test fetching using the install method""" btest = Bintool.create('_testing') @@ -347,6 +388,8 @@ class TestBintool(unittest.TestCase): btool = Bintool.create('_testing') result = btool.run_cmd_result('fred') self.assertIsNone(result) + result = btool.run_cmd_result('daphne', add_name=False) + self.assertIsNone(result) if __name__ == "__main__": diff --git a/tools/binman/btool/_testing.py b/tools/binman/btool/_testing.py index 4005e8a8a5..b291d35c58 100644 --- a/tools/binman/btool/_testing.py +++ b/tools/binman/btool/_testing.py @@ -5,6 +5,8 @@ This is not a real bintool, just one used for testing""" +import tempfile + from binman import bintool # pylint: disable=C0103 @@ -33,4 +35,6 @@ class Bintool_testing(bintool.Bintool): return self.fetch_from_drive('junk') if method == bintool.FETCH_BUILD: return self.build_from_git('url', 'target', 'pathname') + if method == bintool.FETCH_SOURCE: + return self.fetch_from_git('giturl', 'mygit') return None diff --git a/tools/patman/tools.py b/tools/patman/tools.py index 2ac814d476..b69a651eab 100644 --- a/tools/patman/tools.py +++ b/tools/patman/tools.py @@ -397,7 +397,7 @@ def tool_find(name): paths += tool_search_paths for path in paths: fname = os.path.join(path, name) - if os.path.isfile(fname) and os.access(fname, os.X_OK): + if (os.path.isfile(fname) or os.path.isdir(fname)) and os.access(fname, os.X_OK): return fname def run(name, *args, **kwargs):