From patchwork Fri Jul 19 14:35:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1134119 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45qtq72gbMz9s3Z for ; Sat, 20 Jul 2019 00:36:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 1393520380; Fri, 19 Jul 2019 14:36:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9f5qx6W1AUCY; Fri, 19 Jul 2019 14:36:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id AA17F20477; Fri, 19 Jul 2019 14:36:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id AB1F51BF48C for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A84E486B3A for ; Fri, 19 Jul 2019 14:36:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PZ4Ylh0PqX9R for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 29DE7866A9 for ; Fri, 19 Jul 2019 14:36:10 +0000 (UTC) Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 2BAF6240012; Fri, 19 Jul 2019 14:36:08 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Fri, 19 Jul 2019 16:35:54 +0200 Message-Id: <20190719143556.14907-4-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190719143556.14907-1-victor.huesca@bootlin.com> References: <20190719143556.14907-1-victor.huesca@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/5] support/scripts/pkg-stats: add current progress in 'check_url_status' X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The 'check_url_status' function can take a few minutes to process depending on how many packages are involved. The current implementation uses process-pool to speed-up the completion time but does not allow to trace which package as complete nor to track the overall progress with for example: '[42/2243] Package jpeg' This patch adds a progress feedback to 'check_url_status' to report the current package and the overall progression. It rely on the pool's callback and follow the same scheme as 'check_package_latest_version'. This patch also remove the unnecessary 'url_worker' Package's attribute in favor of a local list of workers. This imply removing this field form the excluded fields from 'dump_json'. Signed-off-by: Victor Huesca --- support/scripts/pkg-stats | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 08730b8d43..8b59cd1e76 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -57,7 +57,6 @@ class Package: self.current_version = None self.url = None self.url_status = None - self.url_worker = None self.latest_version = (RM_API_STATUS_ERROR, None, None) def pkgvar(self): @@ -319,7 +318,7 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value -def check_url_status_worker(url, url_status): +def check_url_status(url, url_status): if url_status != "Missing" and url_status != "No Config.in": try: url_status_code = requests.head(url, timeout=30).status_code @@ -332,11 +331,12 @@ def check_url_status_worker(url, url_status): def check_package_urls(packages): - Package.pool = Pool(processes=64) - for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) - for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pool = Pool(processes=64) + cb = progress_callback(lambda i, n, res, name: print("[%d/%d] (url) Package %s: %s" % (i, n, name, res)), 1, len(packages)) + results = [apply_async(pool, check_url_status, (pkg.url, pkg.url_status), + callback=cb, cb_args=(pkg.name,)) for pkg in packages] + for pkg, r in zip(packages, results): + pkg.url_status = r.get() def release_monitoring_get_latest_version_by_distro(pool, name): @@ -735,7 +735,7 @@ def dump_html(packages, stats, date, commit, output): def dump_json(packages, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date - excluded_fields = ['url_worker', 'name'] + excluded_fields = ['name'] pkgs = { pkg.name: { k: v