From patchwork Tue Sep 8 08:10:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 1359568 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=6wind.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind.com header.i=@6wind.com header.a=rsa-sha256 header.s=google header.b=Nj7NQMUq; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BlyX64Lbwz9sRK for ; Tue, 8 Sep 2020 18:11:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E403E87245; Tue, 8 Sep 2020 08:10:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MxklH19WodLc; Tue, 8 Sep 2020 08:10:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 38B528722E; Tue, 8 Sep 2020 08:10:56 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id BF6E11BF2B8 for ; Tue, 8 Sep 2020 08:10:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id A772387225 for ; Tue, 8 Sep 2020 08:10:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZpibEbmj1Oko for ; Tue, 8 Sep 2020 08:10:48 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by hemlock.osuosl.org (Postfix) with ESMTPS id 72AB187223 for ; Tue, 8 Sep 2020 08:10:48 +0000 (UTC) Received: by mail-wr1-f44.google.com with SMTP id z1so18123070wrt.3 for ; Tue, 08 Sep 2020 01:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aeBTckoO1F23D8l/VQ4rXc3VrKmQOdUnoCqwmkYhL+4=; b=Nj7NQMUqaMP7ghfV4e+ob4aXegcK5ESLNzEGDq4qiaNbgs8xZSIl5TnFFezcxDNIiF Z0k8vwL8ucyx7Rj5iYfHRc95vZrvOR7JKi3BoUQFVYtoF773tKQR3GHJkVb72HbY/V6B zXCf8lYihj+yeErMXBA6nYAy0iHCr5GxIXF/R5kJDiDmAIT4hFIC7KADYPsPqVj/INFM i+kChzGcxA71PEoF6fJ0rLSOaqUjw87bsHELVWhQciocjV8Qx91BhKVJNTpCxLjn5FAe IPISjVpbhxBQqucCDLDKTvAwRa+iqVLoqQz+djLDXYGxzrBSU0eQWMg5hM5fvUg5TALH jy6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aeBTckoO1F23D8l/VQ4rXc3VrKmQOdUnoCqwmkYhL+4=; b=kH7lv/STVAMaah/jE3R872e/PA1vX6C2+aA/lfspS7jYflGbxfePK+8GB/izlBZxSJ Nacluhw4Qj4WnVd1tHirrz1w7NK6xUZB1uBwNp+kEPHj4nUhRceJRBuRVzBoWrSmKr6S oM9JzXpp+18l94CkNdgmkka/0G4PryyA9y/Ro7MbDFwQ3zR9d8QtwCSPl8BLN89yA6zF IDirPaJNAZrFxzWYwc47R0JNLmwUGGqF2nL7zgfZfk2sEClxm2XVCgsT0Ws3vqbzpS58 2u6cHFeOlhCL6Zd1/X3xrGFG+wbwSi1/UGXZvUsZblAvlEwVgaMBGytxLGPqYlkwD8Vq KNyg== X-Gm-Message-State: AOAM530rz+4y2oV9GUVs5TMG0jmzj4/EMYDoevjPn0y6kZeqKWOlEzC+ bMn0wC1JrUWx0iiZcOSo7TyCQPPNc/atvmX9 X-Google-Smtp-Source: ABdhPJzN6yjj20kORslJMVflWqMdwJSMhXG+dqB9D9ICwXBaSbKr+Syrb1LqU5FjzG+2zM75YKQ3fA== X-Received: by 2002:adf:a1d6:: with SMTP id v22mr26216866wrv.185.1599552646532; Tue, 08 Sep 2020 01:10:46 -0700 (PDT) Received: from tom.dev.6wind.com. (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id e18sm32273109wrx.50.2020.09.08.01.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 01:10:46 -0700 (PDT) From: Robin Jarry To: buildroot@buildroot.org Date: Tue, 8 Sep 2020 10:10:23 +0200 Message-Id: <20200908081026.4701-2-robin.jarry@6wind.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200908081026.4701-1-robin.jarry@6wind.com> References: <20200904112908.21686-1-julien.floret@6wind.com> <20200908081026.4701-1-robin.jarry@6wind.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/4] pycompile: add main entry point X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E . MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Only run code when the script is executed directly (not imported). Factorize command description by using the script's __doc__ variable. Fix typo in --force help message. Signed-off-by: Robin Jarry --- support/scripts/pycompile.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py index 9192a7016a78..fb1a12b2b782 100644 --- a/support/scripts/pycompile.py +++ b/support/scripts/pycompile.py @@ -58,12 +58,19 @@ class ReportProblem(int): return not self == other -parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.') -parser.add_argument("target", metavar='DIRECTORY', - help='Directory to scan') -parser.add_argument("--force", action='store_true', - help="Force compilation even if alread compiled") +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("target", metavar="TARGET", + help="Directory to scan") + parser.add_argument("--force", action="store_true", + help="Force compilation even if already compiled") -args = parser.parse_args() + args = parser.parse_args() -compileall.compile_dir(args.target, force=args.force, quiet=ReportProblem()) + compileall.compile_dir(args.target, force=args.force, quiet=ReportProblem()) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Tue Sep 8 08:10:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 1359569 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=6wind.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind.com header.i=@6wind.com header.a=rsa-sha256 header.s=google header.b=J1ZCGND8; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BlyX773tDz9sSJ for ; Tue, 8 Sep 2020 18:11:03 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 48A4F8722F; Tue, 8 Sep 2020 08:11:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZZQgNUP96nKm; Tue, 8 Sep 2020 08:10:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id F419A87239; Tue, 8 Sep 2020 08:10:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 6E7661BF2B8 for ; Tue, 8 Sep 2020 08:10:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6AC8886442 for ; Tue, 8 Sep 2020 08:10:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vJ473gmWm6vN for ; Tue, 8 Sep 2020 08:10:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 39FA585F3E for ; Tue, 8 Sep 2020 08:10:51 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id q9so16355501wmj.2 for ; Tue, 08 Sep 2020 01:10:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ep3xhBk+m4sL+Tico1nGQXlGW+S26uleAfPVZobb3LE=; b=J1ZCGND8eap3oSMTv/Gi4xePMltaTmMVSniXfGR50B3MSoXCRW9lI9Hy+YoWbWObza BpQgp9dicREPQ5zDPH3mMcbz+eeIvUXhOrehJYlrQ31sSx+LGddtM+XOcXm3CmH0Y+6P m3ZQ0QEly3p+dI25B3Fygd/KEyYeNGKpqTjiH9/BTxbdc5XNeYh9w4nhJXnC/5dQd1Zu AWgxuhcL5DrJl1l0ugxvF6vYHiRRw5v5EoGOtO/j2scWd5u039iUPG5jwqnetaICabIR VucTh7A9aH8up12qqs+EyLKCIBOyw1AWFTr/fwJnjOUgjr1DiTsMEaDtKbFor6N0l6q1 oqYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ep3xhBk+m4sL+Tico1nGQXlGW+S26uleAfPVZobb3LE=; b=GK2GKuGkYibbYXYlWMi1OEWVYiar7QMmCNnB80LA3NL1SDVS2SQ1FcuNqmzComqyGK +7g/iH1KDzLGhgZi2yqKWRLZLeoIOliO2YxaM0r6HQNt52bRmbheKTfT+CjoWz1Uguhg YtsMOn2vwCDMV4e1bwFsM6v+w9eHqQodCK7w0gLs8aXItVncz0uNEpiBGPO5Tl2fVcdL BahQcVd2tTBeuB4g83Rx1ZLP7SP7g17TKJ5JsaBAYwKLcu0D7yQdFe5fEJYAHIbOO3je o9+L/xblfpxnlzg5l+v29jfct6NFumgJ6yZ/ZZSKbYSEBL6yOi0XCHNrMt6mApICIF+r wNlQ== X-Gm-Message-State: AOAM530QcXHENHA/Nx7tyL+OQ1pxbvlCNSax0tblMPAL4ylHpNZ2M7Nd agTB0WbuR46uOzubSRUMoYqimsMIZ2JV5Ay3 X-Google-Smtp-Source: ABdhPJwy7x8ynPmY1d09PHRPBwT1jo3hSsYUAG9dx4e5IVupC6hiBZrtSKEDCxAgLHVDa7nmpCVBZQ== X-Received: by 2002:a7b:c059:: with SMTP id u25mr3070260wmc.103.1599552649158; Tue, 08 Sep 2020 01:10:49 -0700 (PDT) Received: from tom.dev.6wind.com. (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id e18sm32273109wrx.50.2020.09.08.01.10.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 01:10:48 -0700 (PDT) From: Robin Jarry To: buildroot@buildroot.org Date: Tue, 8 Sep 2020 10:10:24 +0200 Message-Id: <20200908081026.4701-3-robin.jarry@6wind.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200908081026.4701-1-robin.jarry@6wind.com> References: <20200904112908.21686-1-julien.floret@6wind.com> <20200908081026.4701-1-robin.jarry@6wind.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 2/4] pycompile: sort imports X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E . MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Sort imports alphabetically. Signed-off-by: Robin Jarry --- support/scripts/pycompile.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py index fb1a12b2b782..b713fe19323c 100644 --- a/support/scripts/pycompile.py +++ b/support/scripts/pycompile.py @@ -6,11 +6,13 @@ when a python byte code generation failed. Inspired from: http://stackoverflow.com/questions/615632/how-to-detect-errors-from-compileall-compile-dir ''' + from __future__ import print_function -import sys -import py_compile -import compileall + import argparse +import compileall +import py_compile +import sys def check_for_errors(comparison): From patchwork Tue Sep 8 08:10:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 1359571 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=6wind.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind.com header.i=@6wind.com header.a=rsa-sha256 header.s=google header.b=P0niiDrt; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BlyXF6S3Nz9sSJ for ; Tue, 8 Sep 2020 18:11:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9662F272F7; Tue, 8 Sep 2020 08:11:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VKb3ZaRC7z5i; Tue, 8 Sep 2020 08:10:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id CEFD3203E8; Tue, 8 Sep 2020 08:10:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6C1FA1BF2B8 for ; Tue, 8 Sep 2020 08:10:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 66FCD84B89 for ; Tue, 8 Sep 2020 08:10:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5LPIS1+TgxI5 for ; Tue, 8 Sep 2020 08:10:53 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by whitealder.osuosl.org (Postfix) with ESMTPS id 0432684AC4 for ; Tue, 8 Sep 2020 08:10:52 +0000 (UTC) Received: by mail-wm1-f48.google.com with SMTP id e17so16343200wme.0 for ; Tue, 08 Sep 2020 01:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UzjOCK0nUSIaZSRzHvOdsbR3tX4scP1gSwENMDa9vzM=; b=P0niiDrtgUvozFwqPi6Hryk2kN54z3WTfN1UDflVddR3Tw4oSQNhfozcF3V6Wq1xEx DrrneZsrKYlCooSxxsMzWhR/grJNd2BsdrqW8MYMWnztZygsUOHeDQkSXL7Zh8y31RwU Nda51AuPIupjebd+H134Ss9qT8ScmGgBBmgkyZsiJbOZFcuxGBWrlPl2kTf3Np28XLn+ WqjwNIcetvI8OHKnF1XAx4ZXRvzD/vcmw9+TrRAKPSYXKh5q9AsAD9mh3Z2xkR1dGZHh ayL30BlbGyBXFcKWWx5MqAJMNpMv/JICG4ALzDxFvQAxCODY36x7J2uvUfJu6prvSaEo koLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UzjOCK0nUSIaZSRzHvOdsbR3tX4scP1gSwENMDa9vzM=; b=ZDgz6qnbmygFHueCw0rqF7ikzQwnRFQChQR4xSvIRKUwHjwyUP20i6HwKH62liZYbE jcWp362c9imJVIYtm/ksX6Xz6EQqGovhD/HHY0yT8X3GE/7JN0Lb4lKY5s6AOABybZCi zmNHEP5f61sHMUnT+8AnaNxc8R5MyGJXU3cptoag3P8Tety/5L5vmNxbC24l/pZ9ZOq9 kOvSUWjAE3F4k+ifplsqTP9GNvHJO4hAmIrC7CwZBdcp2WVOeas9Ks0eoFFXH2VSjatg T58CdwyL65v7Xb2V1KNWOrNZXMltlw/LSnliHU8WRb8azRw2xOvbJDYJXmSP6vWSHD6n nSKA== X-Gm-Message-State: AOAM530wGB8zPlYsxBgLa3oN8jCTod4f593z00nl0dnw2R0yaQ9wovqc iz1/Z0xuxjXjSLWLfoxM189KtRTmTCwQjUds X-Google-Smtp-Source: ABdhPJxdWKl/KQ90mbELnAWHYu73UvfUNTgrgU3l44JqofUA/QLjKUJ5w6LKXcIBwOeLTcb7IrZF6g== X-Received: by 2002:a1c:5f41:: with SMTP id t62mr3083939wmb.22.1599552650883; Tue, 08 Sep 2020 01:10:50 -0700 (PDT) Received: from tom.dev.6wind.com. (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id e18sm32273109wrx.50.2020.09.08.01.10.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 01:10:50 -0700 (PDT) From: Robin Jarry To: buildroot@buildroot.org Date: Tue, 8 Sep 2020 10:10:25 +0200 Message-Id: <20200908081026.4701-4-robin.jarry@6wind.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200908081026.4701-1-robin.jarry@6wind.com> References: <20200904112908.21686-1-julien.floret@6wind.com> <20200908081026.4701-1-robin.jarry@6wind.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/4] pycompile: fix .pyc original source file paths X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E . MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When generating a .pyc file, the original .py source file path is encoded in it. It is used for various purposes: traceback generation, .pyc file comparison with its .py source, and code inspection. By default, the source path used when invoking compileall is encoded in the .pyc file. Since we use paths relative to TARGET_DIR, we end up with paths that are only valid when relative to '/' encoded in the installed .pyc files on the target. This breaks code inspection at runtime since the original source path will be invalid unless the code is executed from '/'. Unfortunately, compileall cannot be forced to use the proper path. It was not written with cross-compilation usage in mind. Rework the script to call py_compile.compile() with pertinent options: - The script is now called with $(TARGET_DIR) as first argument. This is the future runtime /. - All other (non-optional) arguments are folders in which all "importable" .py files will be compiled to .pyc. - Using the $(TARGET_DIR), the future runtime path of each .py file is computed and encoded into the compiled .pyc. No need to change directory before running the script anymore. The trickery used to handle error reporting was only applicable with compileall. Since we implement our own "compileall", error reporting becomes trivial. Signed-off-by: Julien Floret Signed-off-by: Robin Jarry --- package/python/python.mk | 5 +- package/python3/python3.mk | 5 +- support/scripts/pycompile.py | 122 ++++++++++++++++++++--------------- 3 files changed, 76 insertions(+), 56 deletions(-) diff --git a/package/python/python.mk b/package/python/python.mk index ccaaadd012a5..7000658330e8 100644 --- a/package/python/python.mk +++ b/package/python/python.mk @@ -260,10 +260,11 @@ endif define PYTHON_CREATE_PYC_FILES $(PYTHON_FIX_TIME) PYTHONPATH="$(PYTHON_PATH)" \ - cd $(TARGET_DIR) && $(HOST_DIR)/bin/python$(PYTHON_VERSION_MAJOR) \ + $(HOST_DIR)/bin/python$(PYTHON_VERSION_MAJOR) \ $(TOPDIR)/support/scripts/pycompile.py \ $(if $(BR2_REPRODUCIBLE),--force) \ - usr/lib/python$(PYTHON_VERSION_MAJOR) + $(TARGET_DIR) \ + $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y) diff --git a/package/python3/python3.mk b/package/python3/python3.mk index 31e7ca3d3af3..e8c200b2081e 100644 --- a/package/python3/python3.mk +++ b/package/python3/python3.mk @@ -277,10 +277,11 @@ endif define PYTHON3_CREATE_PYC_FILES $(PYTHON3_FIX_TIME) PYTHONPATH="$(PYTHON3_PATH)" \ - cd $(TARGET_DIR) && $(HOST_DIR)/bin/python$(PYTHON3_VERSION_MAJOR) \ + $(HOST_DIR)/bin/python$(PYTHON3_VERSION_MAJOR) \ $(TOPDIR)/support/scripts/pycompile.py \ $(if $(BR2_REPRODUCIBLE),--force) \ - usr/lib/python$(PYTHON3_VERSION_MAJOR) + $(TARGET_DIR) \ + $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y) diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py index b713fe19323c..c60d4e13f3b4 100644 --- a/support/scripts/pycompile.py +++ b/support/scripts/pycompile.py @@ -1,75 +1,93 @@ #!/usr/bin/env python -'''Wrapper for python2 and python3 around compileall to raise exception -when a python byte code generation failed. - -Inspired from: - http://stackoverflow.com/questions/615632/how-to-detect-errors-from-compileall-compile-dir -''' +""" +Byte compile all .py files from a target dir and encode a proper source runtime +path into them. +""" from __future__ import print_function import argparse -import compileall +import importlib +import os import py_compile +import re +import struct import sys -def check_for_errors(comparison): - '''Wrap comparison operator with code checking for PyCompileError. - If PyCompileError was raised, re-raise it again to abort execution, - otherwise perform comparison as expected. - ''' - def operator(self, other): - exc_type, value, traceback = sys.exc_info() - if exc_type is not None and issubclass(exc_type, - py_compile.PyCompileError): - print("Cannot compile %s" % value.file) - raise value - - return comparison(self, other) - - return operator - - -class ReportProblem(int): - '''Class that pretends to be an int() object but implements all of its - comparison operators such that it'd detect being called in - PyCompileError handling context and abort execution - ''' - VALUE = 1 - - def __new__(cls, *args, **kwargs): - return int.__new__(cls, ReportProblem.VALUE, **kwargs) - - @check_for_errors - def __lt__(self, other): - return ReportProblem.VALUE < other - - @check_for_errors - def __eq__(self, other): - return ReportProblem.VALUE == other - - def __ge__(self, other): - return not self < other - - def __gt__(self, other): - return not self < other and not self == other - - def __ne__(self, other): - return not self == other +def compile_one(host_path, target_root, force=False): + """ + Compile a .py file into a .pyc file located next to it. + + :arg host_path: + Absolute path to the file to compile on the host running the build. + :arg target_root: + Absolute path of the target dir (i.e. the future /). + :arg force: + Recompile even if already up-to-date. + """ + if os.path.islink(host_path): + return # only compile real files + + if not re.match(r"^[_A-Za-z][_A-Za-z0-9]+\.py$", + os.path.basename(host_path)): + return # only compile "importable" python modules + + if not force: + # inspired from compileall.compile_file in the standard library + try: + with open(host_path + "c", "rb") as f: + header = f.read(12) + expect = struct.pack("<4sll", importlib.util.MAGIC_NUMBER, + 0, int(os.stat(host_path).st_mtime)) + if header == expect: + return # .pyc file already up to date. + except OSError: + pass # .pyc file does not exist + + # determine the runtime path of the file (i.e.: relative path to target + # root dir prepended with "/"). + runtime_path = os.path.join("/", os.path.relpath(host_path, target_root)) + + # will raise an error if the file cannot be compiled + py_compile.compile(host_path, cfile=host_path + "c", + dfile=runtime_path, doraise=True) def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("target", metavar="TARGET", - help="Directory to scan") + help="Path on the build host that will be / at runtime") + parser.add_argument("dirs", metavar="PATH", nargs="+", + help="Folder located inside TARGET to scan and compile") parser.add_argument("--force", action="store_true", help="Force compilation even if already compiled") args = parser.parse_args() - compileall.compile_dir(args.target, force=args.force, quiet=ReportProblem()) + if not os.path.isdir(args.target): + parser.error("TARGET: no such directory: %r" % args.target) + + # only work with absolute paths + args.target = os.path.abspath(args.target) + + try: + # Make sure that all scanned dirs do exist and that they are located + # *inside* the target dir. + for d in args.dirs: + if not os.path.isdir(d): + parser.error("PATH: no such directory: %r" % d) + d = os.path.abspath(d) + if ".." in os.path.relpath(d, args.target): + parser.error("PATH: not inside TARGET dir: %r" % d) + for parent, _, files in os.walk(d): + for f in files: + compile_one(os.path.join(parent, f), args.target, args.force) + + except Exception as e: + print("error: %s" % e, file=sys.stderr) + return 1 return 0 From patchwork Tue Sep 8 08:10:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 1359570 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=6wind.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind.com header.i=@6wind.com header.a=rsa-sha256 header.s=google header.b=aIhKiVjF; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BlyXC3cdLz9sSJ for ; Tue, 8 Sep 2020 18:11:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2F4E587256; Tue, 8 Sep 2020 08:11:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oTBF+zmE6EvL; Tue, 8 Sep 2020 08:11:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A23E88724D; Tue, 8 Sep 2020 08:11:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 5B1501BF2B8 for ; Tue, 8 Sep 2020 08:10:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 56DBA84B89 for ; Tue, 8 Sep 2020 08:10:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pK+GMJN380LU for ; Tue, 8 Sep 2020 08:10:54 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by whitealder.osuosl.org (Postfix) with ESMTPS id 7158784AC4 for ; Tue, 8 Sep 2020 08:10:54 +0000 (UTC) Received: by mail-wm1-f42.google.com with SMTP id l9so16198068wme.3 for ; Tue, 08 Sep 2020 01:10:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5VCpCv+mqWgEwGNCcHePuOukzPv5DwoZieCOlkm0DKs=; b=aIhKiVjFLKv9klh2/zqy13PtHLFeb5BvXc5ix5UxAVMy1pbR2z3cL8HY+dIjRRWjbb lkpeocasq+xqki6+fB61bQr42pzqNfsFqgyN0SbLNrK+gqd38UHxysXyymEnTrojnPul TFW+l5f+GqADeYJ9ZE5Nx+Q2gJeCS/RD+krXDR8h69Inu1j3EVnuFb7d2oE1kk0M0kRm toZPDoy1o2y0qGT8OJsLEFiedynj2ekMJRAe0V3eK/Momdnul/nAOrWRPr6kJ4ZaC9pp Ys07xfLEIVWIAhArw2P9JUZneluhrFKp82hQl/b4Cwh3eDz5alxOz3j3TkiiEkHQ49wI er5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5VCpCv+mqWgEwGNCcHePuOukzPv5DwoZieCOlkm0DKs=; b=mYJ/ILsYfrdQ5Me13kwDGDyN7WNtEzCTV9n4OgJ6g+MmqpCQ8kGXu+1pJsr7oxTtu7 7JrEiECHhm8UqqiLy9ESzEtgAFU5oBGcYlOdpDJtDIcvQAoE/B4B28kP6C40U15OnM/3 TdQsoXawu1qKqJhBs3lvZ1IwQOyq4ug2fRy9rtyvXKbdEyqa4szsVZYi0mydcIpL4IKs 4lw5ZHC/4usfGLnFsRmXNsV5T8PBQu2kXGB4BpJfwJ9NuT1d9OpDIxgQ8UjGzdbORdXE 0SJR5MUnnd1/xx7poS4mogDDwCyOI1PdGh21crePNXxwhU1XbH3+ozeG9P61sILRqa5U cOGQ== X-Gm-Message-State: AOAM5304Qtx607ishh8Kh6MU3ihK5vQ1O29NcAsXx3XwNEBksY3zPn3a dkL7Qt6FA8zYQFQw0cGsMQvlhN4lBlfa/8bw X-Google-Smtp-Source: ABdhPJxDNVO9sE8f8RqcVNTSwlx4dvhlMAiYy2ao3+hux84mkMyNm6bzXef3dSuN7XgIqbqzWdVseQ== X-Received: by 2002:a7b:c384:: with SMTP id s4mr2999082wmj.138.1599552652549; Tue, 08 Sep 2020 01:10:52 -0700 (PDT) Received: from tom.dev.6wind.com. (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id e18sm32273109wrx.50.2020.09.08.01.10.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 01:10:52 -0700 (PDT) From: Robin Jarry To: buildroot@buildroot.org Date: Tue, 8 Sep 2020 10:10:26 +0200 Message-Id: <20200908081026.4701-5-robin.jarry@6wind.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200908081026.4701-1-robin.jarry@6wind.com> References: <20200904112908.21686-1-julien.floret@6wind.com> <20200908081026.4701-1-robin.jarry@6wind.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 4/4] pycompile: add --verbose option X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E . MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Add a new option that prints the (runtime) path of compiled .py files when VERBOSE=1 is set. Signed-off-by: Robin Jarry --- package/python/python.mk | 1 + package/python3/python3.mk | 1 + support/scripts/pycompile.py | 11 +++++++++-- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/package/python/python.mk b/package/python/python.mk index 7000658330e8..50e77db74628 100644 --- a/package/python/python.mk +++ b/package/python/python.mk @@ -263,6 +263,7 @@ define PYTHON_CREATE_PYC_FILES $(HOST_DIR)/bin/python$(PYTHON_VERSION_MAJOR) \ $(TOPDIR)/support/scripts/pycompile.py \ $(if $(BR2_REPRODUCIBLE),--force) \ + $(if $(VERBOSE),--verbose) \ $(TARGET_DIR) \ $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) endef diff --git a/package/python3/python3.mk b/package/python3/python3.mk index e8c200b2081e..1ef757c6b66f 100644 --- a/package/python3/python3.mk +++ b/package/python3/python3.mk @@ -280,6 +280,7 @@ define PYTHON3_CREATE_PYC_FILES $(HOST_DIR)/bin/python$(PYTHON3_VERSION_MAJOR) \ $(TOPDIR)/support/scripts/pycompile.py \ $(if $(BR2_REPRODUCIBLE),--force) \ + $(if $(VERBOSE),--verbose) \ $(TARGET_DIR) \ $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) endef diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py index c60d4e13f3b4..cb2ba7f5016e 100644 --- a/support/scripts/pycompile.py +++ b/support/scripts/pycompile.py @@ -16,7 +16,7 @@ import struct import sys -def compile_one(host_path, target_root, force=False): +def compile_one(host_path, target_root, force=False, verbose=False): """ Compile a .py file into a .pyc file located next to it. @@ -26,6 +26,8 @@ def compile_one(host_path, target_root, force=False): Absolute path of the target dir (i.e. the future /). :arg force: Recompile even if already up-to-date. + :arg verbose: + Print compiled file paths. """ if os.path.islink(host_path): return # only compile real files @@ -49,6 +51,8 @@ def compile_one(host_path, target_root, force=False): # determine the runtime path of the file (i.e.: relative path to target # root dir prepended with "/"). runtime_path = os.path.join("/", os.path.relpath(host_path, target_root)) + if verbose: + print(" PYC %s" % (runtime_path,)) # will raise an error if the file cannot be compiled py_compile.compile(host_path, cfile=host_path + "c", @@ -63,6 +67,8 @@ def main(): help="Folder located inside TARGET to scan and compile") parser.add_argument("--force", action="store_true", help="Force compilation even if already compiled") + parser.add_argument("--verbose", action="store_true", + help="Print compiled files") args = parser.parse_args() @@ -83,7 +89,8 @@ def main(): parser.error("PATH: not inside TARGET dir: %r" % d) for parent, _, files in os.walk(d): for f in files: - compile_one(os.path.join(parent, f), args.target, args.force) + compile_one(os.path.join(parent, f), args.target, + args.force, args.verbose) except Exception as e: print("error: %s" % e, file=sys.stderr)