From patchwork Sun Jan 5 09:23:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217719 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="V1IC+2jX"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrG0gW7z9sPJ for ; Sun, 5 Jan 2020 20:23:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 573A2860D5; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wH2U3ekCjEqQ; Sun, 5 Jan 2020 09:23:55 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 335CB85DF5; Sun, 5 Jan 2020 09:23:55 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 078ED1BF285 for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0376385A82 for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lc-RxYglT1vo for ; Sun, 5 Jan 2020 09:23:52 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 07D8F85A76 for ; Sun, 5 Jan 2020 09:23:52 +0000 (UTC) Received: by mail-wm1-f54.google.com with SMTP id d73so12034001wmd.1 for ; Sun, 05 Jan 2020 01:23:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jt8p4gWANbXe3c6C2WnPNjrz01ZgzLbeFJmp0Z8vyzk=; b=V1IC+2jXKzGSjL92FDI+lsQcK+52rk2+27XbFNjThVylX8KHoPz53q5fkWK7t0T/SJ sV6c9oHKwGAK085/SvDKyTaiyAVX727zlg77AvtGTvISK6FRnPvIGwQmbcy+TKcjEExx oq9CewRNTW7M98VgvZaixegtfwsQsU5bl9l/QrNxpOtoNP35OWVueQMI82mOze0gzc4I /Sof9/DafzJu9cPmaFqM4NULWCr+8v3/w4GMW/PnBlpaq4Yz2pn/MWxprDbOw7gjg9HN 5qF7yc/4zPVTGE3b90x/hYTP42CFp/rVuEgWel0WEJGsA/yD6TeljOE1kR9rP7BjyQUP x1LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jt8p4gWANbXe3c6C2WnPNjrz01ZgzLbeFJmp0Z8vyzk=; b=JuY6laDXIPRb5K+Yjre3NwmEV6Y5MjwiEK3DF4f/cTsxcuV4Tx1BBgrQaZaGW/ZQEF O3h6RGcX+ExojH0FbBHudal8WrOxSJwLluWahC2XZq3kirqp0thpHZjTzF4Yg0EW4H2G 85VVZcae/OfzbA6ThgC4oOOJ1vKf5wj4mbFYaXJ2Mis/SwezL3Y1P/dqQNKAFkKfLbhl 90Ls5MvMj3/k5bqQt6BpDVhgiUq+nh/aEub9/vKkzIiouJNs1Q+w3XFQDIauXNbcisuW 6LGNvH6UGmHWdFRIODUsSytTxARPXmh6sQM+w4EezS22Xyydxzc32qaZOkKZRad3I8sH 8ZWQ== X-Gm-Message-State: APjAAAWKFUkiIIi76VIR3T2TO1fILaYQPUKp60X34wkla1G6xznAKu5E zSyxMIs5uD40RP2R4DDijIe82BRL X-Google-Smtp-Source: APXvYqyNUemdnPWC/o2N+R1auRZ8ATHGoL+PfxYRWNYQnxkoSkdmmWZtadN3Z0E7gB8FcVnyTXAWkw== X-Received: by 2002:a1c:1b41:: with SMTP id b62mr28147669wmb.53.1578216229945; Sun, 05 Jan 2020 01:23:49 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:49 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:23 +0100 Message-Id: <20200105092329.6288-2-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/8] support/scripts/pkg-stats: store latest version in a dict X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Heiko Thiery This patch changes the name and the variable from latest_verion of type list to rm_status of type dict. This is for better readability/usability of the data. With this the json output is more descriptive. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index e477828f7b..8c64993aaf 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -61,7 +61,7 @@ class Package: self.url = None self.url_status = None self.url_worker = None - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.rm_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -331,11 +331,10 @@ def check_package_latest_version_worker(name): def check_package_latest_version(packages): """ - Fills in the .latest_version field of all Package objects + Fills in the .rm_version field of all Package objects + + This field is a dict and has the following keys: - This field has a special format: - (status, version, id) - with: - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND @@ -351,7 +350,9 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.rm_version['status'] = r[0] + pkg.rm_version['version'] = r[1] + pkg.rm_version['id'] = r[2] del http_pool @@ -379,13 +380,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.rm_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.rm_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.rm_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 @@ -548,29 +549,29 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.rm_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.rm_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "Error" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.rm_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "Not found" else: - if pkg.latest_version[1] is None: + if pkg.rm_version['version'] is None: latest_version_text = "Found, but no version" else: latest_version_text = "%s" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.rm_version['id'], str(pkg.rm_version['version'])) latest_version_text += "
" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.rm_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by distro" else: latest_version_text += "found by guess" From patchwork Sun Jan 5 09:23:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217723 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="UpqLmnEn"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrP1Y8tz9sPK for ; Sun, 5 Jan 2020 20:24:04 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6DBB78614D; Sun, 5 Jan 2020 09:24:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M-bwAdIRLYKB; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 362F386018; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id DCA251BF285 for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D9C3985DF5 for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2OaeZjfrlj4d for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by whitealder.osuosl.org (Postfix) with ESMTPS id F05C082157 for ; Sun, 5 Jan 2020 09:23:52 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id a5so12050308wmb.0 for ; Sun, 05 Jan 2020 01:23:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KneWRHoItYYI4O+sCLv5LUpnH1ODYGQJoMUeZWYEk0c=; b=UpqLmnEnitrysyr4N0GoHrJvHbVos/D05yrFptuYeDgStOU9Td9OcLqXEvmn8UcU7p 2DACx9mEOJo2KBjcae9NkiKTdB8+fGAk3MZYOCX9grK5nv1YiHsn0ceMvpfNNp9wxJjl uO5JdF8ee6Nr/XdHCeu4eVOxvpCvAsZC3cEc1m46F2KnbX+TdTtkZg+iR/E1gYq2NrXL GmVznry/tvlZgOINmviFItsln8yXxJGy6SIpB9M06biy7vIURnAok5SS+7Wx3vk8E+Et u+AfSGpFRUe3LpsGFeuPBBN/tZvOehHHNkUTvOPn5r2P0zb/c/b52wnZtlk35nZYpawF VYgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KneWRHoItYYI4O+sCLv5LUpnH1ODYGQJoMUeZWYEk0c=; b=dtGe5ucAvkjGuwrlM9QDemG+E0S4GAffJqnwENYbBAbwN6NWQ9Tigb90AW2h6uH4hu 60/ZF5dBKkIcjD/6FP9zPrnEOcpQ79GFr830maZ/o/Lt4pySQGMn27TxWOaREKxuQlMd 5o4jJSEYKUPg886hg59RLW3Od6951ohNG3Zsu0MkfC1xl4s0k4lumvFh5aNqo1HPQ07N 5JZe0orY3iC/JA/ssoWUQYqAawNecgAscXmjX9NS1nfoWveJ4+w+pc7y4NAxPcanKM0d xtItLILOEpyNPWwIBUdxYBtTcNHb7pMZ2d6UOBeQN057SNxmpU5iehGEAMgAq+yTLSpO wbDA== X-Gm-Message-State: APjAAAX1VGgHoZr7uvAqnXG+ieEpS9ZeJ90ydAmb17m2MKo7Zzgolowp 43Lwp8iqYdV3vD7S/xXngT5qdqho X-Google-Smtp-Source: APXvYqxK2NFHiYebQj4XsX/Tj33z3Omc6vj4FD6wKqo7gfYeERk+1MSq7REOVGnRPbZmo31BBk6MLA== X-Received: by 2002:a05:600c:21c6:: with SMTP id x6mr27532738wmj.177.1578216231053; Sun, 05 Jan 2020 01:23:51 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:50 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:24 +0100 Message-Id: <20200105092329.6288-3-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 2/8] support/scripts/pkg-stats: store patch files for the package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Heiko Thiery Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8c64993aaf..6e250556e9 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -56,6 +56,7 @@ class Package: self.has_license_files = False self.has_hash = False self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None @@ -121,10 +122,10 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + self.patch_count = len(self.patch_files) def set_current_version(self): """ From patchwork Sun Jan 5 09:23:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217721 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="oqRdCnTb"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrM43Wwz9sPK for ; Sun, 5 Jan 2020 20:24:03 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A982B85C00; Sun, 5 Jan 2020 09:23:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PRG_YnOxbY1B; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id A372285AE9; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C24021BF285 for ; Sun, 5 Jan 2020 09:23:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B95BB8789C for ; Sun, 5 Jan 2020 09:23:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5rP+Zpaoy5kH for ; Sun, 5 Jan 2020 09:23:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by hemlock.osuosl.org (Postfix) with ESMTPS id E692B8776C for ; Sun, 5 Jan 2020 09:23:53 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id a5so12050317wmb.0 for ; Sun, 05 Jan 2020 01:23:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T5l0kvhe+DugL/PNnS5Ow4ntZBALmjAGsxHnyB8LFRU=; b=oqRdCnTb//ACojhyjgtr0jorD2JqXWJBbp+vx0KV9aLYz4jlALbtQmZWTNgryrD0vF tA9gL0TpXOzekn5uvrJSQqJeEDFw0btibokuVLxYXVmXQeYenuScUGuTqn2YJ7nudKRL Cn0mG5vYSCE4bqbXdcZ3oPaBmE6dXCw8mmAoO6O43/ySTJ195QGbdJ7ACPt4o019OxtY SNs4oJn4FLdDiR8UwlJGEICtRH4y28o2ZBJqxKzB0juRi8limBYz+qV8wBmcJfJ2WYRe Cfk1r4PoBF+ouzHDunG/zcF7o7SCoYqlEltgpVCKOSQHKqoyQ48t+ghssfKbxMNGFYYd 0jjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T5l0kvhe+DugL/PNnS5Ow4ntZBALmjAGsxHnyB8LFRU=; b=ci7M+IsU74l5Kg64zz0nhwwlMRYm7bulA67ySDJH3UxDkhRqxU7Pdjmf/XcnrQc70p J/YduvJgAFIrlDNnJOLgK9+Jljl28SYbKo3OyjFnPwU9Jw7o7KmT9H21rgQoYu8Dgang r3/6kUKpjeRrQWnlR2dNuLlC2LNOhV1CV9bzSTNu2+3iuCeo2g4lpzOJD/41jFUqwhXR 0WiOgRCqG4zZwQkIHxWNIRuDLWqEPeycejWByD5Mh9YrAWsn4q0wlM9kSQbMEiGaQAsH sqPtnA+A9lll3lX9VsH4fAVHXhHv9zzLXV+4k70MF0tTw/N3Qh2xz4x36tcNo2J3V/2i TUSw== X-Gm-Message-State: APjAAAVCXeukdZvBBHnCdJnb7IYW7mXJPkpmzA9Bp4sOaWNJEgxJA8CC +POTN7rOfZZNf1rRsDFaRdAoEl9B X-Google-Smtp-Source: APXvYqyhwifHzLE06tPnMunMaWw/QrjirMl/kd3SW2E3YoWca6XDKsez2MngbZwfxMxWC/m1InmMbw== X-Received: by 2002:a1c:1d8c:: with SMTP id d134mr29662273wmd.16.1578216232051; Sun, 05 Jan 2020 01:23:52 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:51 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:25 +0100 Message-Id: <20200105092329.6288-4-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/8] support/scripts/pkg-stats: set developers info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 6e250556e9..43d14785a1 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -26,10 +26,14 @@ import subprocess import requests # URL checking import json import certifi +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile(r"\s*https?://\S*\s*$") @@ -153,6 +157,15 @@ class Package: self.warnings = int(m.group(1)) return + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + if dev.hasfile(self.path): + self.developers.append((dev.name)) + def __eq__(self, other): return self.path == other.path @@ -733,6 +746,8 @@ def __main__(): 'HEAD']).splitlines()[0] print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -744,6 +759,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...") From patchwork Sun Jan 5 09:23:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217720 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Jnpbw/cc"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrM3jKDz9sPJ for ; Sun, 5 Jan 2020 20:24:03 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1615E878B0; Sun, 5 Jan 2020 09:24:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v8FDv0nWz7lz; Sun, 5 Jan 2020 09:23:58 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 9B3E58789C; Sun, 5 Jan 2020 09:23:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 5BB241BF285 for ; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5622D203EA for ; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6aQI2+-gqiz9 for ; Sun, 5 Jan 2020 09:23:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by silver.osuosl.org (Postfix) with ESMTPS id 36F222047D for ; Sun, 5 Jan 2020 09:23:55 +0000 (UTC) Received: by mail-wm1-f46.google.com with SMTP id p9so12042492wmc.2 for ; Sun, 05 Jan 2020 01:23:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5IatiG69HNhmVWHRacCwq8uBd5h4e5WSCNMYg19DdyQ=; b=Jnpbw/cc3d6f1MiePEoLMc5XLfB4q4Svvu9LXysUfQy/eTFR9ghZsvaUF3ro4nF+Hi wxebV/AImK4GVTYEDSttI+a1rz1LuguLFmXEG+b4HbyxFOpyIoNDhUuDq2EdrgFepPxR DTZAjWzP7fIqibnZAA7P/EkwewgromT9cG8H0abBvsAwcUY7+T1eNwxE52F3p/zcWw4U m8T72/7RYAB055DKIKiXLsJoPeqH1Wtma0RXm2L3MeLgvd9YDXjgdlhvAh3k5nV+N22a 0qERgEQmdWA5R3B3/880J9TvUmBI+WDjWoXGvAuXqFr+812+fPhb4k7NFkMfn7NM1Yv5 VXEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5IatiG69HNhmVWHRacCwq8uBd5h4e5WSCNMYg19DdyQ=; b=K7z9g3sz3K397QsUKJUJqk/1WKHo3v3LC4rgLdv5BwpNlqog0i0LvfYdhjwyi+A0g4 dG3gy5JdX8AZmTvE63PUoNvh63m3tdzUf/nep6dG4HfXxXaKH3KltZBS/mfU/WkIf6Gr Yyg63F6ihCUirJm7N5I/sRzexVvqxhddMDV0My94z9Xka/9wkog0s+gi3QW+ygILpQ5Q ur++CNYKNG+k/gQdo3jjWxR49StEog5pcVyCT+YMQL/BHyg3L9W1HLzWUnDmERkhXi8v gMiGKUU5D0SWj7UdONWX77iHIZHkjn7xCNDK2PE1rykTGYm4x83FnN/3JC78Zfami7mJ Rf0A== X-Gm-Message-State: APjAAAXN9is0mR3VLU2/rroiezb8D+BA0dBVBdVSmGKqyd2sIX9xrf8H 4PdUS07ODn55yrncVSia+vumyZou X-Google-Smtp-Source: APXvYqxuRu4EMb88Uc/3laN2ZIcLH6WLH/DF0bTL1C/RIYz2aPzl1N/lotN2F3Arzsxf2k/zXCKC8A== X-Received: by 2002:a7b:c084:: with SMTP id r4mr27564251wmh.99.1578216233237; Sun, 05 Jan 2020 01:23:53 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:52 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:26 +0100 Message-Id: <20200105092329.6288-5-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 4/8] support/scripts/pkg-stats: store licences of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 43d14785a1..c10eb79a08 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -48,7 +48,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() @@ -56,6 +56,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False @@ -112,6 +113,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True @@ -261,7 +263,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): From patchwork Sun Jan 5 09:23:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217722 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="obDaCDZ/"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrP0BHXz9sPJ for ; Sun, 5 Jan 2020 20:24:05 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5EFD185CD8; Sun, 5 Jan 2020 09:24:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xLaHQfdsWStr; Sun, 5 Jan 2020 09:24:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 322DE85B0A; Sun, 5 Jan 2020 09:24:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 97E231BF285 for ; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 94E3D85A76 for ; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SNOt_U4SJJ8g for ; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 53ECC85AE9 for ; Sun, 5 Jan 2020 09:23:56 +0000 (UTC) Received: by mail-wm1-f45.google.com with SMTP id p17so12324322wmb.0 for ; Sun, 05 Jan 2020 01:23:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J3fwq+4GJ8qv6n2lqdsBnaMGnq+JZmDAJ54KbOoDdas=; b=obDaCDZ/7MT9PApuJ/aEgtZLmQeCiuMc0OT23JqbTXt5VSVZAYx4L4h3DMYq/M02CR tVBDiQRmN9ST6Q5ypgYZxK4av83bDGMF/lebOC07sioHTvR24Sbt2cqVQetsTWjGbBeA ge/VV8aSkb8tV3Y6nltguiohBOGkLmUycfPDuWQgY/lBEtlpPYE/rxAWQ9fb2enwKKNg PyzDAZXpc9kDrLlcWE9zzJNsR4k3gpWtn0cBW9dyfUX7LVO9EzjTCsEQA0qovopwRNZg 3bXlj3p/ewZiQ6JQhojf85Da44VIO/B7F8UOj958ND0rra+nSHNQg4wE501+4HQLM/8L +E4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J3fwq+4GJ8qv6n2lqdsBnaMGnq+JZmDAJ54KbOoDdas=; b=UeFdJDT7TpMv3gT9RaGNxuKozIg9yoAx/TJMmosJJCNdaWh8jlbtG53t1UGNlg0qOj 3ppyqehG3OTvKCdxGR34Tu9eYxo5HF95K8H7/nO9ICdswtKRFnfEfxAlSWtX20yCDH4o sC8V0DvKlbVmmG2uLR0eT/VshKbXH6X0YH+R96ywn2x/Zl/Vit+hCbGGSHMeD1j8PBb9 b81WD1P8R6wXXySOAqZ/hHg9q5h0MjYKxdyz/I7uxtnyASrKLTgoarjtLA1b9xjEeZry sDuRbKdvG2gh/kUYcIPfxRjmEVRUtnyDtc+8nZReIhMGYKp6BZ8FSaVs16c0XDCIJG8T XHRA== X-Gm-Message-State: APjAAAWVy60zIA+ghRV67ObOrdp3TRayKanAAjRZ0DbwWA94/rEpg7gG TAqRlo8APYXlfOPOxSQDXPp6gT8T X-Google-Smtp-Source: APXvYqzr7zi5cOtkjIftfyYYKmns8Nf8vwP8qMKvQr86m5PBXpCSt7BfAswYrZsRZg8oGU3o2/BGwA== X-Received: by 2002:a05:600c:30a:: with SMTP id q10mr27697432wmd.84.1578216234502; Sun, 05 Jan 2020 01:23:54 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:53 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:27 +0100 Message-Id: <20200105092329.6288-6-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 5/8] support/scripts/pkg-stats: add package status X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Unify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message. With that status information the following variables are replaced: has_license, has_license_files, has_hash, url_status Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 104 +++++++++++++++++++++++++------------- 1 file changed, 69 insertions(+), 35 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index c10eb79a08..ace092cf3b 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -57,17 +57,14 @@ class Package: self.path = path self.infras = None self.license = None - self.has_license = False - self.has_license_files = False - self.has_hash = False self.patch_count = 0 self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.rm_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -76,17 +73,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("warning", "missing") fp.close() def set_infra(self): @@ -108,31 +105,43 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("ok", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') self.patch_count = len(self.patch_files) + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field @@ -143,10 +152,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": @@ -157,17 +167,28 @@ class Package: m = re.match("^([0-9]*) warnings generated", line) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = list() + self.status['developers'] = ("warning", "no developers") for dev in developers: if dev.hasfile(self.path): self.developers.append((dev.name)) + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path @@ -176,7 +197,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) def get_pkglist(npackages, package_list): @@ -279,23 +300,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): Package.pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) def release_monitoring_get_latest_version_by_distro(pool, name): @@ -369,6 +390,18 @@ def check_package_latest_version(packages): pkg.rm_version['status'] = r[0] pkg.rm_version['version'] = r[1] pkg.rm_version['id'] = r[2] + + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', 'RM API error') + elif pkg.rm_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', 'RM package not found') + + if pkg.rm_version['version'] is None: + pkg.status['version'] = ('warning', 'no upstream version available') + elif pkg.rm_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', 'package needs update') + else: + pkg.status['version'] = ('ok', 'up-to-date') del http_pool @@ -384,15 +417,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1 @@ -532,30 +565,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20: @@ -606,12 +639,13 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] == "warning": + td_class.append("missing_url") + elif pkg.status['url'][0] == "error": td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): td_class.append("invalid_url") - url_str = "%s" % (pkg.url, pkg.url_status) + url_str = "%s" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "Link" % pkg.url From patchwork Sun Jan 5 09:23:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217725 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ujZX1au2"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrX4XQvz9sPK for ; Sun, 5 Jan 2020 20:24:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 425C287BC2; Sun, 5 Jan 2020 09:24:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jbnN3w3TYWnV; Sun, 5 Jan 2020 09:24:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A67A88789D; Sun, 5 Jan 2020 09:24:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 583171BF285 for ; Sun, 5 Jan 2020 09:24:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 4E5AF86378 for ; Sun, 5 Jan 2020 09:24:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TO+l0TtiSbfP for ; Sun, 5 Jan 2020 09:24:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id 62C798610F for ; Sun, 5 Jan 2020 09:23:57 +0000 (UTC) Received: by mail-wr1-f47.google.com with SMTP id t2so46397011wrr.1 for ; Sun, 05 Jan 2020 01:23:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9Cc6hKulUBw4Mv93bFabPms3/fXTchUrHajrleRX1QA=; b=ujZX1au2qkMYsSi6jACsaj9LuVYlzSr47wUntTWxD026tJ3knG3iFgnsHtaprvzv+O A7A3r/mxgJDGothiJSWXvmv5fTiRx4re+anXmEPGqp3FT9DyxpQ9j5EcCN8h0RCU1A1S MRTQZ9Vb6Xq4hZ0rW1+2TgapHJHV+nzt+wNbcncQy20ZWNxE/qYpBfZP5bz9P9mlaal2 H17taxGf2Z1ex30AJ1WS0WizsCcGZv8xrBytsrlrXdoT7Sqpw9xJvg3eaN7GsaNVBaO2 K/ZCIDGV+mTmNUPnp08FrP0/6z3siaZVrHvnTmzNHNFd8G8y6L8S7s9ufnMNY8DXdDmS rAYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9Cc6hKulUBw4Mv93bFabPms3/fXTchUrHajrleRX1QA=; b=ii4T5s56qt72gtzqfZqiXka3nrTuhDUcM8VwtICTp3Op8L2WUGoMTsmSY4rXjvwHtN 4egQK0XzC1Kkf5x3igrmfNDlXM0jEhG+l2UU2amwNbMLiOGsthsxdQ9mV0dMhoJ0A8Nu G7khrT4gip6ej9GPaeKno6g0IAWkBoTM9ahKAS/uSJomENOzTSwX/TZ4bVuRbh5fj5ca ILlUOxEfb1BWXMsvY2HjKOrzF6t49b/O6h/cEEZidZEz4Vyo0T6gvzWUpw5Q00z4IP8I XhWh3r21QhkuzXJGZqqET0JX8N2atYvz/fWHwei2kfdqTw3pzghEB2RzbsHvxkmvphTi KrRw== X-Gm-Message-State: APjAAAXEemPpQGwXK+GyQCU2v9vt822LW12nKfonXjCBEQXLML1DzO3R keWrx7KgGKOs4oNZgprzMw9MSR9s X-Google-Smtp-Source: APXvYqyI8Um4DX/hJhNLGtwTpCOaOcs+qzK/vBLKnjd4TBQsePxivqg3kkFmD+3LWhOdI9KLQdupPg== X-Received: by 2002:a5d:6703:: with SMTP id o3mr55038802wru.235.1578216235423; Sun, 05 Jan 2020 01:23:55 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:54 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:28 +0100 Message-Id: <20200105092329.6288-7-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 6/8] support/scripts/pkg-stats: add package count to stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index ace092cf3b..8ec3204d9a 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -407,6 +407,7 @@ def check_package_latest_version(packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the From patchwork Sun Jan 5 09:23:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217724 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FUzuSgel"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrX19g0z9sPJ for ; Sun, 5 Jan 2020 20:24:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 20DF085F7E; Sun, 5 Jan 2020 09:24:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PguD43MjQzOY; Sun, 5 Jan 2020 09:24:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 365C485C5E; Sun, 5 Jan 2020 09:24:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 98AF01BF285 for ; Sun, 5 Jan 2020 09:24:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8CC8B2047D for ; Sun, 5 Jan 2020 09:24:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fBnRPO7+SfEe for ; Sun, 5 Jan 2020 09:23:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by silver.osuosl.org (Postfix) with ESMTPS id 44759203EA for ; Sun, 5 Jan 2020 09:23:58 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id z7so46311681wrl.13 for ; Sun, 05 Jan 2020 01:23:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XRRM0OJrlU4r7urwXH0RUdtJgFbd/Gc6vWkLfzsKmwk=; b=FUzuSgeljZrFAUVkMv+S/McqBPDcB78Cde3eISyTvywUBOBRn/KZYdPNuRMYcp+bN0 e1neQBJIudV12ra4RL9Mu4LEKboRagEKUdLCkgABAhRuKE/QuUYFYLo838+BBpKlIluE 0e0uQ3lgs/4WDwXPE0nhaQJBsHXA+HDaZtbaUDhxIf2qsnnj7GU/RVzPw7wvo562krDG S61POghnryjcZwELq7P6IiDQXioFeHEUkyBu+nh0qoEFsPRZ6Sr1mu8cOpd0KzLVB7+O CQSbP5dkpfAukOu9xXgB2xVIykbpZgIrKcf0CpKFJu3GAUpcnS/GK4kGV0amfFIVbL1C lXVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XRRM0OJrlU4r7urwXH0RUdtJgFbd/Gc6vWkLfzsKmwk=; b=rDj9B3rsm9lTjxsw2w1yvhWmdhqle9J/l1cOITLb+BXI+2BU+H5M+bCp6GOoLEXjRK tHPcccIIWr5jmy88WQiprWUs5D43QlunAZ1yS2BpnVoYrQ5jWRnV00/g7nkQjGFstptN sPqbEzAck4gT2dzwcj0uGwSa2q8ZN+n5ilSc9/Rmk0x5Jk1jlflAyv+Ne435hNOhXSXq +1dy9hXE8qTNLH/GQ1bX3p1PIXcT8UIwNH7jbsa4R7cyBmAo2oAE8ZboMSffvZ2jqZl0 7OBa7XvPpRDuaDR8xhqTq5OveWekvdfp/CyQB0D5fThSr+2jvfPs+gDkqxGrLUCXWxv0 14rA== X-Gm-Message-State: APjAAAWZk6h1l+lgmCs43iFWjgudgAEn70OmwtLuHzxSbhWAHXwtCVjM V6K0zIQD+mkpijbl99i4g/koknoM X-Google-Smtp-Source: APXvYqwxwnsJeIwn5oiOEJXYvaHVMuoCYaJo3QGGLopdCY3c5qsqwuO3REMrKPjmB2GGyBGwcVqrsw== X-Received: by 2002:adf:f789:: with SMTP id q9mr99632525wrp.103.1578216236471; Sun, 05 Jan 2020 01:23:56 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:55 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:29 +0100 Message-Id: <20200105092329.6288-8-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 7/8] support/scripts/pkg-stats: store pkg dir path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This value can be used for later processing. In the buildroot-stats application this is used to create links pointing to the git repo of buildroot. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8ec3204d9a..5f61a095b6 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -55,6 +55,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.patch_count = 0 From patchwork Sun Jan 5 09:23:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217726 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VYT7WC9I"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47rCrY0hZdz9sRR for ; Sun, 5 Jan 2020 20:24:13 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B70C385B0A; Sun, 5 Jan 2020 09:24:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bRHNEIe5IW-e; Sun, 5 Jan 2020 09:24:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2CE0F85D08; Sun, 5 Jan 2020 09:24:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 60FFB1BF285 for ; Sun, 5 Jan 2020 09:24:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5CCE12047F for ; Sun, 5 Jan 2020 09:24:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iIs9GNHW+y78 for ; Sun, 5 Jan 2020 09:23:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by silver.osuosl.org (Postfix) with ESMTPS id 1B73020467 for ; Sun, 5 Jan 2020 09:23:59 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id d73so12034133wmd.1 for ; Sun, 05 Jan 2020 01:23:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CXUcqC7DPYvPTF6iS58JTIe7O/YqoZ2JE5G5qeJLHU0=; b=VYT7WC9I4v7hXrqaHsJqDXpV6idWP6NkEA87IjL4+CRJRUrnyNBd4wBxKDqd6Mw6Yn mXj3Ahep+5JHtmchwwRg98ZTMoG1THWmNLKBXw87YtoYIbQfeeE6qTSZiFskTTRohMVq OvT2gAaVVEBgvdlTJxc3zisiWSUuziVu0AYb7NY1ve2e2nNPCzb9f10Bwt9M/qNKvP2C uVXl/h1NWV+H0FPEI8PcsXdKwN6C/1O/7cXaN5HNEv04MSNg6d74Wq9oHsDGOnX624EK YNEs0orfGrxFAWiiEJfuRU3Jz7BX4jlT7H166vq5UYW2U3itirj7lM3dW4KCEIGVn9BD 4Msw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CXUcqC7DPYvPTF6iS58JTIe7O/YqoZ2JE5G5qeJLHU0=; b=r7rOlADNF/tM8KGF9P02JmJJ0eMhn8QHkLvyXRWhR5rQheIG6v+Lsx8mLQRGGAxgfX fg815gFK28u4yzyuA5+/uU0VyVT2LS892OTkKSp4diBTtdODf/diPdCP4JVLKYwJcH3G WdErxM7pZIcsjqONY/4V9h37B2PNuKPzv+XhkEhKcjyztDmgll9d5okypY9TfA/ftrpA NSTBHzKaizTXHXyv2MOyikpJ7XpB4E7fv1rRwiEEOxdWydbC7fJWlzs6D8nUBloaNFYU bs31RCk6d6YwJAV4R2myRzF4NE4IOvS/sgPLQcRNH5oLV7E51UxNpej7eSHWpoo1HwuE 7n+w== X-Gm-Message-State: APjAAAXaJ3JK+Or7Bf4VzKQJmPrvJXfUhfUkOlOj3o/iHtekLFOstJwG lunRALA2cHBp2TBnhp8yrGqJY5c+ X-Google-Smtp-Source: APXvYqzYyP8CdSFNcU22iZVXW5/CpZyuPZctrrd9lgdwJ8Ll5LBJ19NZgO45J0f6jEbhoztutfeyog== X-Received: by 2002:a05:600c:2150:: with SMTP id v16mr26566375wml.156.1578216237340; Sun, 05 Jan 2020 01:23:57 -0800 (PST) Received: from t450s.fritz.box (ip5f589927.dynamic.kabel-deutschland.de. [95.88.153.39]) by smtp.gmail.com with ESMTPSA id r62sm19600729wma.32.2020.01.05.01.23.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Jan 2020 01:23:56 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sun, 5 Jan 2020 10:23:30 +0100 Message-Id: <20200105092329.6288-9-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200105092329.6288-1-heiko.thiery@gmail.com> References: <20200105092329.6288-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 8/8] support/scripts/pkg-stats: add defconfig support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Scan configs directory and create Defconfig objects. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 42 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 5f61a095b6..cf5de784cd 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -46,6 +46,33 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + if dev.hasfile(self.path): + self.developers.append(dev.name) + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + defconfigs = list() + files = [f for f in os.listdir('configs')] + for name in files: + d = Defconfig(name[:-10], os.path.join('configs', name)) + defconfigs.append(d) + return defconfigs + class Package: all_licenses = dict() @@ -726,7 +753,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name'] @@ -737,6 +764,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v @@ -747,6 +780,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)} @@ -786,6 +820,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -809,7 +847,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__()