From patchwork Tue Aug 4 19:52:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1341002 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BLlmG5vgcz9sT6 for ; Wed, 5 Aug 2020 05:53:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D2C59875EE; Tue, 4 Aug 2020 19:52:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dydM3xxCCuCP; Tue, 4 Aug 2020 19:52:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 87593875FD; Tue, 4 Aug 2020 19:52:56 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id E66D11BF341 for ; Tue, 4 Aug 2020 19:52:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D5BF1875FD for ; Tue, 4 Aug 2020 19:52:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RlqZ7wwgMu6l for ; Tue, 4 Aug 2020 19:52:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by whitealder.osuosl.org (Postfix) with ESMTPS id 2A488875EE for ; Tue, 4 Aug 2020 19:52:53 +0000 (UTC) X-Originating-IP: 83.193.91.77 Received: from localhost (lfbn-bay-1-248-77.w83-193.abo.wanadoo.fr [83.193.91.77]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 457C91BF203; Tue, 4 Aug 2020 19:52:50 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Tue, 4 Aug 2020 21:52:45 +0200 Message-Id: <20200804195248.1238754-1-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 0/3] Use aiohttp in pkg-stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Hello, I started investigating why pkg-stats was sometimes stuck on the server running it on a daily basis to populate autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The subprocesses started by "multiprocessing" to retrieve the latest upstream version from release-monitoring.org were stuck holding a lock. Without providing a definitive conclusion, some preliminary research showed that multiprocessing can be tricky and cause some issues with locks. Discussing this with Titouan, he suggested to use aiohttp instead of multiprocessing. And indeed, it makes a lot of sense to use this popular asynchronous HTTP library. This patch series switches the latest version retrieval and the upstream URL checking to aiohttp, and as a bonus adds some logging to show the progress of the retrieval, as it can be quite long. Changes since v1: - Pass trust_env=True when creating the aiohttp.ClientSession() so that HTTP proxy environment variables are taken into account. Suggested by Matt Weber. - Fix bogus indentation that was breaking the logic of the latest version retrieval. Thanks, Thomas Thomas Petazzoni (3): support/scripts/pkg-stats: use aiohttp for latest version retrieval support/scripts/pkg-stats: use aiohttp for upstream URL checking support/scripts/pkg-stats: show progress of upstream URL and latest version support/scripts/pkg-stats | 206 ++++++++++++++++++++++---------------- 1 file changed, 122 insertions(+), 84 deletions(-)