From patchwork Fri Jul 19 14:35:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134120 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtqB25dnz9s00 for ; Sat, 20 Jul 2019 00:36:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id E2BCC87A50; Fri, 19 Jul 2019 14:36:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N4fEltUHLduj; Fri, 19 Jul 2019 14:36:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 9471087A73; Fri, 19 Jul 2019 14:36:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id AD85A1BF9BF for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 942C920380 for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W9704nvipc8M for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by silver.osuosl.org (Postfix) with ESMTPS id B06BE20477 for ; Fri, 19 Jul 2019 14:36:09 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 2728624000D; Fri, 19 Jul 2019 14:36:07 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:52 +0200 Message-Id: <20190719143556.14907-2-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/5] support/scripts/pkg-stats: Use the 'multiprocess' fork instead of 'multiprocessing' X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This fork has a better serialization process whereas 'multiprocessing' uses the half-broken pickle. This allow to use any kind of object for process-pool related function. In particular this change allow lambdas to be used as asynchronous function in 'Pool.apply_async'. Signed-off-by: Victor Huesca --- support/scripts/pkg-stats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 7dcb2b4cbd..77819c4804 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -28,7 +28,7 @@ import json import certifi from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError -from multiprocessing import Pool +from multiprocess import Pool INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile(r"\s*https?://\S*\s*$") From patchwork Fri Jul 19 14:35:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134117 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtq45nZVz9s00 for ; Sat, 20 Jul 2019 00:36:16 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2512A87A79; Fri, 19 Jul 2019 14:36:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UP2CiOqaocwU; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id B8F9487A47; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 1422F1BF48C for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 117FF87A3D for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jY19BD6zL5Rj for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by whitealder.osuosl.org (Postfix) with ESMTPS id 79A4987A47 for ; Fri, 19 Jul 2019 14:36:09 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id B32CA240011; Fri, 19 Jul 2019 14:36:07 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:53 +0200 Message-Id: <20190719143556.14907-3-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 2/5] support/scripts/pkg-stats: retrieve packages latest version using processes X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The major bottleneck in pkg-stats is the time spent waiting for answer from distant servers. Two functions involve such communications with remote servers are: - 'check_package_urls' which check that package website are up, it is efficient do to the use of process-pools thanks to Matt Weber. - 'check_package_latest_version' which fetch the latest package version from release-monitoring, it uses a http-pool but run sequentially. This patch extends the use of process-pools to 'check_latest_version'. This implementation rely on the apply_async's callback to allow per-package progress feedback. To simplify this feedback creation, this patch introduce the following functions: - 'apply_async': this function simply wrap the Pool's method of the same in order to pass additional arguments to the callback. In particular it is used to print the package name in the feedback message. - 'progress_callback': this function ease the definition of "progress feedback function": it create a callable that will keep track of how many time it has been called and print a custom message. Also change the behaviour of print for python 2 to be a function instead of a statement, allowing to use it in lambdas. Runtimes for this function are ~3m vs ~25m for the linear version. Tested on an i7 7500U (2/4 cores/threads @3.5GHz) with 15ms ping. Note: There have already been work trying to parallelize this function using threads but there were a failure on some configurations [1]. This implementation rely on a dedicated module already in use on this script, so it's unlikely to see failure with this version. [1] http://lists.busybox.net/pipermail/buildroot/2018-March/215368.html Signed-off-by: Victor Huesca Reviewed-by: Matt Weber --- support/scripts/pkg-stats | 64 +++++++++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 77819c4804..08730b8d43 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -16,6 +16,7 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +from __future__ import print_function import argparse import datetime import fnmatch @@ -159,6 +160,37 @@ class Package: (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) +class progress_callback: + def __init__(self, progress_fn, start=0, end=100): + ''' + Create a callback 'function' which purpose is to display a progress message. + + :param progress_fn: must take at least 2 arguments representing the current step + and the 'end' step. + :param start: First step. + :param end: Last step. + ''' + self._progress_fn = progress_fn + self._cpt = start + self._end = end + + def __call__(self, *args): + ''' + Calls progress_fn. + ''' + self._progress_fn(self._cpt, self._end, *args) + self._cpt += 1 + + +def apply_async(pool, func, args=(), kwds={}, callback=None, cb_args=(), cb_kwds={}): + ''' + Wrapper around `pool.apply_async()` to allow passing arguments to the callback + ''' + _func = lambda: func(*args, **kwds) + _cb = lambda res: callback(res, *cb_args, **cb_kwds) + return pool.apply_async(_func, callback=_cb) + + def get_pkglist(npackages, package_list): """ Builds the list of Buildroot packages, returning a list of Package @@ -345,6 +377,14 @@ def release_monitoring_get_latest_version_by_guess(pool, name): return (RM_API_STATUS_NOT_FOUND, None, None) +def check_package_latest_version_worker(pool, name): + """Wrapper to try both by name then by guess""" + res = release_monitoring_get_latest_version_by_distro(pool, name) + if res[0] == RM_API_STATUS_NOT_FOUND: + res = release_monitoring_get_latest_version_by_guess(pool, name) + return res + + def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects @@ -360,18 +400,18 @@ def check_package_latest_version(packages): - id: string containing the id of the project corresponding to this package, as known by release-monitoring.org """ - pool = HTTPSConnectionPool('release-monitoring.org', port=443, - cert_reqs='CERT_REQUIRED', ca_certs=certifi.where(), - timeout=30) - count = 0 - for pkg in packages: - v = release_monitoring_get_latest_version_by_distro(pool, pkg.name) - if v[0] == RM_API_STATUS_NOT_FOUND: - v = release_monitoring_get_latest_version_by_guess(pool, pkg.name) - - pkg.latest_version = v - print("[%d/%d] Package %s" % (count, len(packages), pkg.name)) - count += 1 + http_pool = HTTPSConnectionPool('release-monitoring.org', port=443, + cert_reqs='CERT_REQUIRED', ca_certs=certifi.where(), + timeout=30) + worker_pool = Pool(processes=64) + cb = progress_callback( + lambda i, n, (status, ver, id), name: + print("[%d/%d] (version) Package %s: %s" % (i, n, name, id)), + 1, len(packages)) + results = [apply_async(worker_pool, check_package_latest_version_worker, (http_pool, pkg.name), + callback=cb, cb_args=(pkg.name,)) for pkg in packages] + for pkg, r in zip(packages, results): + pkg.latest_version = r.get() def calculate_stats(packages): From patchwork Fri Jul 19 14:35:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134119 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtq72gbMz9s3Z for ; Sat, 20 Jul 2019 00:36:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 1393520380; Fri, 19 Jul 2019 14:36:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9f5qx6W1AUCY; Fri, 19 Jul 2019 14:36:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id AA17F20477; Fri, 19 Jul 2019 14:36:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id AB1F51BF48C for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A84E486B3A for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PZ4Ylh0PqX9R for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 29DE7866A9 for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 2BAF6240012; Fri, 19 Jul 2019 14:36:08 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:54 +0200 Message-Id: <20190719143556.14907-4-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/5] support/scripts/pkg-stats: add current progress in 'check_url_status' X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The 'check_url_status' function can take a few minutes to process depending on how many packages are involved. The current implementation uses process-pool to speed-up the completion time but does not allow to trace which package as complete nor to track the overall progress with for example: '[42/2243] Package jpeg' This patch adds a progress feedback to 'check_url_status' to report the current package and the overall progression. It rely on the pool's callback and follow the same scheme as 'check_package_latest_version'. This patch also remove the unnecessary 'url_worker' Package's attribute in favor of a local list of workers. This imply removing this field form the excluded fields from 'dump_json'. Signed-off-by: Victor Huesca --- support/scripts/pkg-stats | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 08730b8d43..8b59cd1e76 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -57,7 +57,6 @@ class Package: self.current_version = None self.url = None self.url_status = None - self.url_worker = None self.latest_version = (RM_API_STATUS_ERROR, None, None) def pkgvar(self): @@ -319,7 +318,7 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value -def check_url_status_worker(url, url_status): +def check_url_status(url, url_status): if url_status != "Missing" and url_status != "No Config.in": try: url_status_code = requests.head(url, timeout=30).status_code @@ -332,11 +331,12 @@ def check_url_status_worker(url, url_status): def check_package_urls(packages): - Package.pool = Pool(processes=64) - for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) - for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pool = Pool(processes=64) + cb = progress_callback(lambda i, n, res, name: print("[%d/%d] (url) Package %s: %s" % (i, n, name, res)), 1, len(packages)) + results = [apply_async(pool, check_url_status, (pkg.url, pkg.url_status), + callback=cb, cb_args=(pkg.name,)) for pkg in packages] + for pkg, r in zip(packages, results): + pkg.url_status = r.get() def release_monitoring_get_latest_version_by_distro(pool, name): @@ -735,7 +735,7 @@ def dump_html(packages, stats, date, commit, output): def dump_json(packages, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date - excluded_fields = ['url_worker', 'name'] + excluded_fields = ['name'] pkgs = { pkg.name: { k: v From patchwork Fri Jul 19 14:35:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134121 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtqF6zmmz9s3Z for ; Sat, 20 Jul 2019 00:36:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 78BE987A74; Fri, 19 Jul 2019 14:36:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JiNXmLSrt9rw; Fri, 19 Jul 2019 14:36:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 40B9C87AC5; Fri, 19 Jul 2019 14:36:21 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id A90CC1BF9BF for ; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id A5D3A8832D for ; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J6MSG4Dd6+ED for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by hemlock.osuosl.org (Postfix) with ESMTPS id DFBD38832C for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id B2206240004; Fri, 19 Jul 2019 14:36:08 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:55 +0200 Message-Id: <20190719143556.14907-5-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 4/5] support/scripts/pkg-stats: improve 'package_init_make_info' X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The pkg-stats calls 3 times `make` to get a bunch of variables. These variables could have been obtained in only one call. This patch replace the three calls by one and adjust the parsing logic accordingly. Note: An other (better?) option suggested by Arnout would be to run `make show-info` that produces a json with the necessary variables. This would avoid the duplicated effort done in pkg-stats and pkg-utils and allow to add other infos to pkg-stats like dependencies, reversed dependencies or if the package is virtual. In order to use this method, the following changes are required in pkg-generic's show-info. - include license_files; - have an option to run it on *all* packages, not just the selected ones. This patch take the simplest approach of only factorize the make calls as it require less changes. Signed-off-by: Victor Huesca --- support/scripts/pkg-stats | 83 +++++++++++++-------------------------- 1 file changed, 27 insertions(+), 56 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8b59cd1e76..5b27a806ef 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -252,70 +252,41 @@ def get_pkglist(npackages, package_list): def package_init_make_info(): - # Licenses - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_LICENSE"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - # Strip _LICENSE - pkgvar = pkgvar[:-8] - - # If value is "unknown", no license details available - if value == "unknown": - continue - Package.all_licenses.append(pkgvar) - - # License files - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_LICENSE_FILES"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): - continue - - # Strip _LICENSE_FILES - pkgvar = pkgvar[:-14] - - Package.all_license_files.append(pkgvar) - - # Version - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_VERSION"]) + # Fetch all variables at once + variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars", + "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"]) + variable_list = variables.splitlines() # We process first the host package VERSION, and then the target # package VERSION. This means that if a package exists in both - # target and host variants, with different version numbers - # (unlikely), we'll report the target version number. - version_list = o.splitlines() - version_list = [x for x in version_list if x.startswith("HOST_")] + \ - [x for x in version_list if not x.startswith("HOST_")] - for l in version_list: + # target and host variants, with different values (eg. version + # numbers (unlikely)), we'll report the target one. + variable_list = [x[5:] for x in variable_list if x.startswith("HOST_")] + \ + [x for x in variable_list if not x.startswith("HOST_")] + + for l in variable_list: # Get variable name and value pkgvar, value = l.split("=") - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_DL_VERSION"): - continue + # Strip the suffix according to the variable + if pkgvar.endswith("_LICENSE"): + # If value is "unknown", no license details available + if value == "unknown": + continue + pkgvar = pkgvar[:-8] + Package.all_licenses.append(pkgvar) - # Strip _VERSION - pkgvar = pkgvar[:-8] + elif pkgvar.endswith("_LICENSE_FILES") : + if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): + continue + pkgvar = pkgvar[:-14] + Package.all_license_files.append(pkgvar) - Package.all_versions[pkgvar] = value + elif pkgvar.endswith("_VERSION"): + if pkgvar.endswith("_DL_VERSION"): + continue + pkgvar = pkgvar[:-8] + Package.all_versions[pkgvar] = value def check_url_status(url, url_status): From patchwork Fri Jul 19 14:35:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134118 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtq662msz9s00 for ; Sat, 20 Jul 2019 00:36:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 637FB86059; Fri, 19 Jul 2019 14:36:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m7SPx628CeME; Fri, 19 Jul 2019 14:36:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A8F9E85EE8; Fri, 19 Jul 2019 14:36:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 2E3DD1BF48C for ; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2BCF086B3A for ; Fri, 19 Jul 2019 14:36:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xiadlmmiWZ3c for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 39B6686B39 for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 5B1ED240016; Fri, 19 Jul 2019 14:36:09 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:56 +0200 Message-Id: <20190719143556.14907-6-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 5/5] support/scripts/pkg-stats: add option to reduce verbosity X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Having the output flooded by status for all packages may be unnecessary. This patch add a basic log level and set all print outside of the main to have lower level. Running this script with '-q' only prints the main thread while it default prints all messages. Signed-off-by: Victor Huesca --- support/scripts/pkg-stats | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 5b27a806ef..dede8ced9d 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -303,7 +303,7 @@ def check_url_status(url, url_status): def check_package_urls(packages): pool = Pool(processes=64) - cb = progress_callback(lambda i, n, res, name: print("[%d/%d] (url) Package %s: %s" % (i, n, name, res)), 1, len(packages)) + cb = progress_callback(lambda i, n, res, name: log(1, "[%d/%d] (url) Package %s: %s" % (i, n, name, res)), 1, len(packages)) results = [apply_async(pool, check_url_status, (pkg.url, pkg.url_status), callback=cb, cb_args=(pkg.name,)) for pkg in packages] for pkg, r in zip(packages, results): @@ -377,7 +377,7 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) cb = progress_callback( lambda i, n, (status, ver, id), name: - print("[%d/%d] (version) Package %s: %s" % (i, n, name, id)), + log(1, "[%d/%d] (version) Package %s: %s" % (i, n, name, id)), 1, len(packages)) results = [apply_async(worker_pool, check_package_latest_version_worker, (http_pool, pkg.name), callback=cb, cb_args=(pkg.name,)) for pkg in packages] @@ -745,14 +745,26 @@ def parse_args(): help='Number of packages') packages.add_argument('-p', dest='packages', action='store', help='List of packages (comma separated)') + parser.add_argument('-q', dest='quiet', action='count', + help='Avoid flooding, use -qq for being even quieter') args = parser.parse_args() if not args.html and not args.json: parser.error('at least one of --html or --json (or both) is required') return args +def log(lvl, *args, **kwargs): + ''' + Print message depending on the log level. + Lower level means only print on more versbose log.LEVEL. + ''' + if lvl > log.LEVEL: + print(*args, **kwargs) + + def __main__(): args = parse_args() + log.LEVEL = args.quiet if args.packages: package_list = args.packages.split(",") else: @@ -760,11 +772,11 @@ def __main__(): date = datetime.datetime.utcnow() commit = subprocess.check_output(['git', 'log', 'master', '-n', '1', '--pretty=format:%H']).splitlines()[0] - print("Build package list ...") + log(2, "Build package list ...") packages = get_pkglist(args.npackages, package_list) - print("Getting package make info ...") + log(2, "Getting package make info ...") package_init_make_info() - print("Getting package details ...") + log(2, "Getting package details ...") for pkg in packages: pkg.set_infra() pkg.set_license() @@ -773,17 +785,17 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() - print("Checking URL status") + log(2, "Checking URL status ...") check_package_urls(packages) - print("Getting latest versions ...") + log(2, "Getting latest versions ...") check_package_latest_version(packages) - print("Calculate stats") + log(2, "Calculate stats") stats = calculate_stats(packages) if args.html: - print("Write HTML") + log(2, "Write HTML") dump_html(packages, stats, date, commit, args.html) if args.json: - print("Write JSON") + log(2, "Write JSON") dump_json(packages, stats, date, commit, args.json)