From patchwork Mon Mar 8 13:45:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1449086 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=mXRmhjMc; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DvKNz0MG5z9sW1 for ; Tue, 9 Mar 2021 00:45:54 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C78D1400A7; Mon, 8 Mar 2021 13:45:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MKCHmImDwUEO; Mon, 8 Mar 2021 13:45:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 03B06400BE; Mon, 8 Mar 2021 13:45:50 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id A3A681BF312 for ; Mon, 8 Mar 2021 13:45:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 91ADC835D3 for ; Mon, 8 Mar 2021 13:45:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vZqQ_ho_Yuz5 for ; Mon, 8 Mar 2021 13:45:48 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by smtp1.osuosl.org (Postfix) with ESMTPS id 43DB1830E3 for ; Mon, 8 Mar 2021 13:45:48 +0000 (UTC) Received: by mail-ej1-x634.google.com with SMTP id lr13so20524397ejb.8 for ; Mon, 08 Mar 2021 05:45:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=PmLqvhQFjHyh4bswVuePMPsLjqisb5aSaipESvKyakI=; b=mXRmhjMcYDFI1b1fEMBwDbqQ39OamP2V4YLK6tQG/RHLs4CdS3wDotsNxKKv4NkB1D 6qBx8c0+0qyakcU1HA3CiHBSYCb82q3eXUAR0d14taSWaMUparaVnp4c7KUQQRQ0CghY mjusoQN7bZDr+5RU2bi5tvAJd9UW1pK37MlUKmCojRUvewRbMiOryS07xOqg38SJ8Bjk /stRewAjLCKps1+9Sjz1bXk4ohPR+REm4jBWYoL1yyFqF0VyHzXU5YZcb3IAm0NNVZUD G3Pa4xk9xeKk+Gjuw8a95MiSJ/EEXxPbA8JJGJ/KEDwBej6KGdK0uQBmxHPVqGi1fGdA AoYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=PmLqvhQFjHyh4bswVuePMPsLjqisb5aSaipESvKyakI=; b=hH5SQKoPzZaPA4s4vQMOq1sAVxHXOmpX+Nfkst40gH5jNAw+4vpkCsckQiPerwsEF6 a84q7jKjnKQZ8l/E27cV90YjYflKVZHfjiDe7jJ6f51uvDKj5FOq6sxHfRGao+9S0sjn x9SN9dXR8Vgbv0pngMfTZ56Ws459CVcZWBZrnFbgBNMJoeb+k6xsCnd/PND3TBJ+QkXQ 7bx70djrxrG1o6zTQ2S3odJY7e3D+2SOYJEUNlc7QGD4Cpea4Ln1PmZJyz5vWZtzUG0w PWOolFnL7sb+8u2JOJ+KIy8BlTU3TQZv4Fto7/ROY5HQRNFOTRmZ5R+1P/JgqiANxHbA n2QA== X-Gm-Message-State: AOAM532JZqqKgDSvWldyyrghGKgLAWNysXGvjbvWe1+9q6wf7sfsRhGV 2ovfUrbLJ2XZtynRjmreaycZhgo/fBH2UApG X-Google-Smtp-Source: ABdhPJz/9XB4mGm51T9n7eZDhIc9N36aiK9cYSTjWkJcy99ji1B6QQX4Y2OH3tCqav3pLrI+LdzkDg== X-Received: by 2002:a17:906:c099:: with SMTP id f25mr15357633ejz.141.1615211146419; Mon, 08 Mar 2021 05:45:46 -0800 (PST) Received: from localhost (ptr-5gw9txf6g5tndjqqqz.18120a2.ip6.access.telenet.be. [2a02:1810:510:3800:1b15:18e4:57d1:eb]) by smtp.gmail.com with ESMTPSA id w24sm7232763edt.44.2021.03.08.05.45.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Mar 2021 05:45:46 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Mon, 8 Mar 2021 14:45:39 +0100 Message-Id: <20210308134542.22324-1-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Subject: [Buildroot] [PATCH 1/2] utils/scanpypi: add setup.py script directory as sys.path[0] X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Even though the directory containing a package's setup.py was added to sys.path, some setup.py implementations rely on the fact that it is placed in sys.path[0]. An example package is 'cram' which failed to be added with scanpypi: Traceback (most recent call last): File "utils/scanpypi", line 756, in main() File "utils/scanpypi", line 703, in main package.load_setup() File "utils/scanpypi", line 303, in load_setup setup = imp.load_module('setup', s_file, s_path, s_desc) File "/usr/lib/python3.8/imp.py", line 234, in load_module return load_source(name, filename, file) File "/usr/lib/python3.8/imp.py", line 171, in load_source module = _load(spec) File "", line 702, in _load File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 44, in long_description=long_description(), File "/tmp/scanpypi-2pzc5wb_/python-cram/cram-0.7/setup.py", line 20, in long_description return open(os.path.join(sys.path[0], 'README.rst')).read() FileNotFoundError: [Errno 2] No such file or directory: '.../buildroot/utils/README.rst' The corresponding code from cram's setup.py is: def long_description(): """Get the long description from the README""" return open(os.path.join(sys.path[0], 'README.rst')).read() Indeed, the Python documentation says: https://docs.python.org/3.8/library/sys.html#sys.path "... As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. ..." Fix this by inserting explicitly at index 0 instead of appending to sys.path. Signed-off-by: Thomas De Schampheleire --- utils/scanpypi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/scanpypi b/utils/scanpypi index 47c7c00c60..24c64dddde 100755 --- a/utils/scanpypi +++ b/utils/scanpypi @@ -297,7 +297,7 @@ class BuildrootPackage(): """ current_dir = os.getcwd() os.chdir(self.tmp_extract) - sys.path.append(self.tmp_extract) + sys.path.insert(0,self.tmp_extract) s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract]) setup = imp.load_module('setup', s_file, s_path, s_desc) if self.metadata_name in self.setup_args: From patchwork Mon Mar 8 13:45:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1449087 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=IiajsImf; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DvKPD3Njsz9sW1 for ; Tue, 9 Mar 2021 00:46:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id E16194A332; Mon, 8 Mar 2021 13:46:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BUNJ_Bpcu4k3; Mon, 8 Mar 2021 13:46:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id AC4BE4A43F; Mon, 8 Mar 2021 13:46:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 0B0601BF312 for ; Mon, 8 Mar 2021 13:45:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id EEE63400A7 for ; Mon, 8 Mar 2021 13:45:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4hLwuENVKaPD for ; Mon, 8 Mar 2021 13:45:50 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by smtp2.osuosl.org (Postfix) with ESMTPS id 839A7400A4 for ; Mon, 8 Mar 2021 13:45:50 +0000 (UTC) Received: by mail-ej1-x62d.google.com with SMTP id hs11so20508282ejc.1 for ; Mon, 08 Mar 2021 05:45:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gX3cHgLfeQdKFmXDHyI7Vex1MWJeRLwv5DI637EtqXo=; b=IiajsImfvzJUIKYv+nwZA2FkmVHM29Xgi+peg1Luec/pRUc6i1nCceDgeHkkY9A5ze jebswYJZjC/j5HnOvi9uxk3MAHgtJC12O55l+fyyANRaBRflF+w5BWwicVDDppZjIgPl WOKD/+rwqO3s3AAKDV0PkQBsY/adUTcyzQASbDAmB0bVxoXmJ5V8KjAKK7ftg+WOLOJO Wx97gMYeNBtDM1HQCcwb1+xOuNvJf3djlPtIPNXseIq5H+QLByjd8yBJOAk3r2JavMiw 1ZPXZQfd4bTrgf1vBkoWy2hpQU0R9EgYhsydHetLVJcfD1vmq3rOz3b2O7umtGzK88kp ccxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gX3cHgLfeQdKFmXDHyI7Vex1MWJeRLwv5DI637EtqXo=; b=ExKq7HcTghDSz2npN6CtnsOMeT6mYNeXky90C49KphAqLVey9ncEGB14Kboo1RvP9a k3jDTBWsZPv8UZk05GV2+y71tOagF503ZBNThsRHp7PhO1Z8yILUhW0T4ICUuiSYt4Np yj4JeX/oPu373F4EseCRn3PBXC34K8FxDiBy7k5PSyanq4GsfK7aWW8qZc58Aqc89yk+ GA5lH59SoZK/sVqy4ILYMuNpaKOjMbWPWv02pTOsg4ypo0mQTa/ZK9dYyjEjmvYk4u1Y ys7pVwyjqCdsxM8n0/rcOy5DryZaYeZi0epGF1PsBJzG95LfTdZ/dgibwXwemBf5CHxp tofQ== X-Gm-Message-State: AOAM532yjzkqFcibvrAaS125/siHx5jxjpPo72C/LvWEH1k0MMQrL6IT I35MPJpq2YBE72JSCItCfcDVRHF0iSJkP9pA X-Google-Smtp-Source: ABdhPJwoNVNSW67Y/pUs+z6LOLj5E3V0sumsyiB8ueC/8uoj5Dd4bZ3pPdhCnmydjPcKJTCWPAcBZg== X-Received: by 2002:a17:906:3899:: with SMTP id q25mr14897644ejd.157.1615211148561; Mon, 08 Mar 2021 05:45:48 -0800 (PST) Received: from localhost (ptr-5gw9txf6g5tndjqqqz.18120a2.ip6.access.telenet.be. [2a02:1810:510:3800:1b15:18e4:57d1:eb]) by smtp.gmail.com with ESMTPSA id s6sm6531189ejx.83.2021.03.08.05.45.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Mar 2021 05:45:48 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Mon, 8 Mar 2021 14:45:40 +0100 Message-Id: <20210308134542.22324-2-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210308134542.22324-1-patrickdepinguin@gmail.com> References: <20210308134542.22324-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 2/2] utils/scanpypi: allow installation of commands without 'main' method X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire In case the setup.py file of a python package does not directly call the 'setup' method, utils/scanpypi was hoping there be a 'main' function which would do the work, normally called via a construct like: if __name__ == '__main__': main() However, this construct is nonstandard, and there are packages in PyPI which call 'setup()' directly from the 'if' statement, without a main() method. But scanpypi does not actually need to make such assumption: when loading the module, it can decide the name to be '__main__', just as if setup.py would be loaded interactively. Additionally, remove some logic seemingly related to the previous trick of calling 'main'. There should not be a problem in keeping already loaded modules in sys.modules, as this is the purpose of sys.modules. Signed-off-by: Thomas De Schampheleire --- utils/scanpypi | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/utils/scanpypi b/utils/scanpypi index 24c64dddde..1d1f25f596 100755 --- a/utils/scanpypi +++ b/utils/scanpypi @@ -299,7 +299,7 @@ class BuildrootPackage(): os.chdir(self.tmp_extract) sys.path.insert(0,self.tmp_extract) s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract]) - setup = imp.load_module('setup', s_file, s_path, s_desc) + setup = imp.load_module('__main__', s_file, s_path, s_desc) if self.metadata_name in self.setup_args: pass elif self.metadata_name.replace('_', '-') in self.setup_args: @@ -309,19 +309,10 @@ class BuildrootPackage(): try: self.setup_metadata = self.setup_args[self.metadata_name] except KeyError: - # This means setup was not called which most likely mean that it is - # called through the if __name__ == '__main__' directive. - # In this case, we can only pray that it is called through a - # function called main() in setup.py. - setup.main() # Will raise AttributeError if not found - self.setup_metadata = self.setup_args[self.metadata_name] - # Here we must remove the module the hard way. - # We must do this because of a very specific case: if a package calls - # setup from the __main__ but does not come with a 'main()' function, - # for some reason setup.main() will successfully call the main - # function of a previous package... - sys.modules.pop('setup', None) - del setup + # This means setup was not called + print('ERROR: Could not determine package metadata for {pkg}.\n' + .format(pkg=self.real_name)) + raise os.chdir(current_dir) sys.path.remove(self.tmp_extract) @@ -706,7 +697,7 @@ def main(): else: raise continue - except AttributeError as error: + except (AttributeError, KeyError) as error: print('Error: Could not install package {pkg}: {error}'.format( pkg=package.real_name, error=error)) continue