From patchwork Tue Aug 4 19:52:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1341003 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BLlmN2Shnz9sRN for ; Wed, 5 Aug 2020 05:53:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D63AC8638F; Tue, 4 Aug 2020 19:53:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7rJlWN99f61K; Tue, 4 Aug 2020 19:53:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id F3D14862AB; Tue, 4 Aug 2020 19:53:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 512681BF341 for ; Tue, 4 Aug 2020 19:52:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4E2C987F8E for ; Tue, 4 Aug 2020 19:52:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LwJlwyr09yED for ; Tue, 4 Aug 2020 19:52:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id 0715987F9F for ; Tue, 4 Aug 2020 19:52:56 +0000 (UTC) X-Originating-IP: 83.193.91.77 Received: from localhost (lfbn-bay-1-248-77.w83-193.abo.wanadoo.fr [83.193.91.77]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id D93C6E0002; Tue, 4 Aug 2020 19:52:54 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Tue, 4 Aug 2020 21:52:48 +0200 Message-Id: <20200804195248.1238754-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200804195248.1238754-1-thomas.petazzoni@bootlin.com> References: <20200804195248.1238754-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/3] support/scripts/pkg-stats: show progress of upstream URL and latest version X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit slightly improves the output of pkg-stats by showing the progress of the upstream URL checks and latest version retrieval, on a package basis: Checking URL status [0001/0062] curlpp [0002/0062] cmocka [0003/0062] snappy [0004/0062] nload [...] [0060/0062] librtas [0061/0062] libsilk [0062/0062] jhead Getting latest versions ... [0001/0064] libglob [0002/0064] perl-http-daemon [0003/0064] shadowsocks-libev [...] [0061/0064] lua-flu [0062/0064] python-aiohttp-security [0063/0064] ljlinenoise [0064/0064] matchbox-lib Note that the above sample was run on 64 packages. Only 62 packages appear for the URL status check, because packages that do not have any URL in their Config.in file, or don't have any Config.in file at all, are not checked and therefore not accounted. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 30aca89336..570bbe833f 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -498,20 +498,31 @@ def package_init_make_info(): Package.all_ignored_cves[pkgvar] = value.split() -async def check_url_status(session, pkg, retry=True): +check_url_count = 0 + + +async def check_url_status(session, pkg, npkgs, retry=True): + global check_url_count + try: async with session.get(pkg.url) as resp: if resp.status >= 400: pkg.status['url'] = ("error", "invalid {}".format(resp.status)) + check_url_count += 1 + print("[%04d/%04d] %s" % (check_url_count, npkgs, pkg.name)) return except (aiohttp.ClientError, asyncio.exceptions.TimeoutError): if retry: - return await check_url_status(session, pkg, retry=False) + return await check_url_status(session, pkg, npkgs, retry=False) else: pkg.status['url'] = ("error", "invalid (err)") + check_url_count += 1 + print("[%04d/%04d] %s" % (check_url_count, npkgs, pkg.name)) return pkg.status['url'] = ("ok", "valid") + check_url_count += 1 + print("[%04d/%04d] %s" % (check_url_count, npkgs, pkg.name)) async def check_package_urls(packages): @@ -520,7 +531,7 @@ async def check_package_urls(packages): async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess: packages = [p for p in packages if p.status['url'][0] == 'ok'] for pkg in packages: - tasks.append(check_url_status(sess, pkg)) + tasks.append(check_url_status(sess, pkg, len(packages))) await asyncio.wait(tasks) @@ -592,17 +603,27 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True): return False -async def check_package_latest_version_get(session, pkg): +check_latest_count = 0 + + +async def check_package_latest_version_get(session, pkg, npkgs): + global check_latest_count if await check_package_get_latest_version_by_distro(session, pkg): + check_latest_count += 1 + print("[%04d/%04d] %s" % (check_latest_count, npkgs, pkg.name)) return if await check_package_get_latest_version_by_guess(session, pkg): + check_latest_count += 1 + print("[%04d/%04d] %s" % (check_latest_count, npkgs, pkg.name)) return check_package_latest_version_set_status(pkg, RM_API_STATUS_NOT_FOUND, None, None) + check_latest_count += 1 + print("[%04d/%04d] %s" % (check_latest_count, npkgs, pkg.name)) async def check_package_latest_version(packages): @@ -629,7 +650,7 @@ async def check_package_latest_version(packages): async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess: packages = [p for p in packages if p.has_valid_infra] for pkg in packages: - tasks.append(check_package_latest_version_get(sess, pkg)) + tasks.append(check_package_latest_version_get(sess, pkg, len(packages))) await asyncio.wait(tasks)