From patchwork Sat Feb 22 08:57:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242457 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=BUYHWHu6; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjGp23Z2z9sPk for ; Sat, 22 Feb 2020 20:10:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id AE62186DBD; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xBrmGkAqpf1S; Sat, 22 Feb 2020 09:10:38 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 3BEFF86DD6; Sat, 22 Feb 2020 09:10:38 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id DA9881BF5D7 for ; Sat, 22 Feb 2020 09:10:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D74C086DD6 for ; Sat, 22 Feb 2020 09:10:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kGepez+ldcg8 for ; Sat, 22 Feb 2020 09:10:33 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by whitealder.osuosl.org (Postfix) with ESMTPS id AA24186DBD for ; Sat, 22 Feb 2020 09:10:32 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id p18so1030919wre.9 for ; Sat, 22 Feb 2020 01:10:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qPQlhlfnPRpK5YkSSaLj9aUnh6gkYIsCZbcu7c8jRwg=; b=BUYHWHu6nZdv+PXFue4YBgEvyTrwcnRbKKAJpoUCgf3bQaIuJ9pRranLVJ/nP4Gdd1 WG9FTA6OhRVo4pgluXhrzZVLJBKjxFOeiwZaerGUnaFuA+0dPJYxhq30xVk5Hy+kl2Ce 4fxsOizZ9nrTHDE+3HCMxaEvEmFLTTRskHw2XPX+OZPQ1zfQkU7GnlZfIVkMLWR1OqgV 7HXGItRo1F8e6OzwfBmHjnlvRuUeMlyaKZbTk6eIWG7Tlu6SmaR0Ep6GRNpfcOj+ihKX f8dTASN4uuXgD/Uir/jAL011CNFQaIjEWdb8JAjzZ+ycnbmmHHx341CTUQoMgBqKK6eP gXLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qPQlhlfnPRpK5YkSSaLj9aUnh6gkYIsCZbcu7c8jRwg=; b=s4wWGPZpJiORvCDOBgr2qlkC4uQ1kIkCYcYmsWaMM2KjXLoQYqiBwtFPg0E99FyPvb 7YYXV+8Q2hy3pP653BkqZNecYxX+JwVk4CTfBcGrUD/s/3rPUB35Muli36QN1mwSD2cK 9zy+3ycSrKC2CK3XQpbIMYVxkoiXLHQAANSqXxOy8QFlWz+7RN52ceVZSU0q21eiuG3S cLa50M/LiQV496rEZDBjWDNfsbQ+hGMFrjkvlJgpf5tjYzuylGyyF4IpmWW4TgOXZUOE R4bLgLxVYySTIZx43E/11l1l0XvIqnCCaXJ5eXRBq118w0M9VxaG7NjF2jG7/LlooeOu 8UtA== X-Gm-Message-State: APjAAAUXxIdvNxpDPLUji394QY6G78FLLkuKwE/A2bWwlfBFB1MZ37bL 7xPaLJskln4XID43kFh0xkbtp3Bj X-Google-Smtp-Source: APXvYqxJgMgTdi4pqbLu8HtoydqZRibb7yQ0POX3Yjp1RHIqPjGStoHj1USw25rwYvvBNq4XUd8Grw== X-Received: by 2002:adf:f403:: with SMTP id g3mr2219216wro.196.1582362630776; Sat, 22 Feb 2020 01:10:30 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:30 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:05 +0100 Message-Id: <20200222085715.23769-2-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 01/12] support/scripts/pkg-stats: store latest version in a dict X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Heiko Thiery This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output. Signed-off-by: Heiko Thiery Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 7721d98459..8cc78f2f66 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -71,7 +71,7 @@ class Package: self.url_status = None self.url_worker = None self.cves = list() - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -460,9 +460,8 @@ def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects - This field has a special format: - (status, version, id) - with: + This field is a dict and has the following keys: + - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND @@ -478,7 +477,9 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.latest_version['status'] = r[0] + pkg.latest_version['version'] = r[1] + pkg.latest_version['id'] = r[2] del http_pool @@ -516,13 +517,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.latest_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.latest_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 @@ -688,29 +689,29 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.latest_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "Error" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "Not found" else: - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: latest_version_text = "Found, but no version" else: latest_version_text = "%s" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.latest_version['id'], str(pkg.latest_version['version'])) latest_version_text += "
" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by distro" else: latest_version_text += "found by guess" From patchwork Sat Feb 22 08:57:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242458 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KEUojvwR; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjGq5CXhz9sRJ for ; Sat, 22 Feb 2020 20:10:43 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C3A6987FD8; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ddrjyFuQD4KQ; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 3052287F5A; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 72A761BF5D7 for ; Sat, 22 Feb 2020 09:10:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6B1B6860E6 for ; Sat, 22 Feb 2020 09:10:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MFqqySUmyNbZ for ; Sat, 22 Feb 2020 09:10:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by fraxinus.osuosl.org (Postfix) with ESMTPS id C488F85F6A for ; Sat, 22 Feb 2020 09:10:33 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id y11so4672848wrt.6 for ; Sat, 22 Feb 2020 01:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x2gcKFxVUTNK5k1JpLGqHaTL+oi1ohwulpeVSB+NryQ=; b=KEUojvwRNwUFvK3VXIYNnRcZjxH2YECltb9VT1Sq6od8xZFhKc9sHneU55/0tqfsu5 VKgyxMHBnf+OWAf7BAFtamxiZY6ml9g6LdZaDRsinQJZ1I904k3KUA5/fiUAApU3L3jQ zVA2Uy193uKl1WXG1UOZtH0Gc5LAREuOkvb1F+ttFL34BtLB+sGFYrVWyDcQ/DpD7Qn4 pPJVvBkhD3LY2LxMdQ+X541rMjiNx0yvmF0M6Xq9k0+QiFqcYUYbuMThhyH7NCsguruV LXXE38wwFj87lcwEPKNDph/+/d2TrpcwCQW+Uo3+xkle7l++JmHT3Swowmt1OHqq2cMn s3Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x2gcKFxVUTNK5k1JpLGqHaTL+oi1ohwulpeVSB+NryQ=; b=XshdDGWCPbhv7oE8Ny3+UoArIt//wttWY5bYAYLOGQr3FIQxdypFK5tzRqVR5WaDrv p1StkBDtdNxUYK0TZB3L2XtE/Ffoy4mKC7vynzdaCrvYHzAHN0pk7h3JM6Nfbwo8EqTX hOfk5inD+49XvX8gBiznKV+OKv+70mIjvJ/x7HP94Gg04a+WCE12z0pQdUvDR8FiQWGL aBPfI/vEzekHufCN/QwLQM4q2uB0q5lPVpO2p6MQW/BuhmgpzkH+Qgo0ddsYtgUYXupd pe9oIYgGV6Bmlf+Qa5Iz7/yhAHqaIJaIfrUCvUOs2Y8pCptnEakzWAi2CSzOCCdNv8SX +7kA== X-Gm-Message-State: APjAAAXW/bkrSISkORT64mUMtYiwwRVT2rHep3NKFuuuF3yC05Lgz+jG FgsQOXfe1nxvpIguON6/CfYOWY7w X-Google-Smtp-Source: APXvYqyMBMTZKmsufZSZwftqifidK+WWT0F8PfsMtCxvDw/F6Fy6ETcdfKkzFgY4kwEGRlB2glY6Dg== X-Received: by 2002:adf:c3d0:: with SMTP id d16mr51363935wrg.376.1582362631890; Sat, 22 Feb 2020 01:10:31 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:31 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:06 +0100 Message-Id: <20200222085715.23769-3-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 02/12] support/scripts/pkg-stats: store patch files for the package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Heiko Thiery Signed-off-by: Heiko Thiery Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8cc78f2f66..4c963cef0f 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -65,6 +65,7 @@ class Package: self.has_license_files = False self.has_hash = False self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None @@ -131,10 +132,10 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + self.patch_count = len(self.patch_files) def set_current_version(self): """ From patchwork Sat Feb 22 08:57:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242460 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=EunBcfFj; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjGv5rxtz9sRJ for ; Sat, 22 Feb 2020 20:10:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id DF68520119; Sat, 22 Feb 2020 09:10:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3eqUyaN-ntmF; Sat, 22 Feb 2020 09:10:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 97CBB203E8; Sat, 22 Feb 2020 09:10:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 838E51BF5D7 for ; Sat, 22 Feb 2020 09:10:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 80A0D86DD6 for ; Sat, 22 Feb 2020 09:10:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IpZTqnJgyoYP for ; Sat, 22 Feb 2020 09:10:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1C42886DBD for ; Sat, 22 Feb 2020 09:10:35 +0000 (UTC) Received: by mail-wr1-f49.google.com with SMTP id p18so1030987wre.9 for ; Sat, 22 Feb 2020 01:10:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5CWy/2+eZGrf+MnvjqsAJRhE0UdAlHsqr+CsWA+9ACY=; b=EunBcfFjiKbX3NYrfNMxIACJXZTQFth48sDkFjOnA5ahrsMimA6DlxnU453yPz0vyL A6KtckE5BaacBs6yEwViBLIf1myUIvcx5hEm7gVxmIDgby35il5desU36k+wALqYQQeQ O9QM4tUq2Z8NMsjMYGFUxIttj8YFE2lCUGnOQxS5YMZ7PYkP94WmP3IqbgGBlMOcXzBB Zfxr1j+CDOcZ7274ck56aMO3NwIbBSJwLPrONyn6fvqwR8Cv9cBTzp4Z9yAiEY61r5MC vlAAzrtZq6H4294S3hR1HRAlMvzEf4MbFX+8cPG0UYRfvc6yej9v8jxy4n22chcMAcHg v6Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5CWy/2+eZGrf+MnvjqsAJRhE0UdAlHsqr+CsWA+9ACY=; b=MF+2Ahm1FWWXbSA2gLB/PRmnQm1X+hYUVr2UCIlOEqClayftrR4TG3jL654Lo/ZZRQ 0YLivnFG/7iAdwrmPF/4mvi0vjGRGiV4/hksRTapAI8FTAoh2ZhF0ZjWCHWtzm/hP0bt PUKln+BLrqCNGYZMxSFKly0AxtKmdw/B/iUYJH3ZxPZi5X9F6H4LovRomN30JeUBWlSQ 1Q9Q2Oc7QviT9yikGZR4c/k2dwp1xR/r/u0N8NaGV2R1Q8ukS4y1u3RlVbaASsbyuTPh pp/CbhuybIUM9QmNL95AgYrLTByrduUkKmrKWbWP+12tZyBJlkDwNLPfF4Ajvc/8YrKt xejA== X-Gm-Message-State: APjAAAVgje1VD2uAeIWt1yF0GI/IME7mW+tJooZI2HdteQRcTM+9JQjJ ADAf1ngvHLhhAn9Wn/CbKPy16sGh X-Google-Smtp-Source: APXvYqz6ugmSUuI9InU8hsLpJeu/Fn5hLWIr8NX23Q1Z4Rz6RV621KRT2ytCekVV10GPEAyU9cf53A== X-Received: by 2002:adf:b351:: with SMTP id k17mr53224903wrd.199.1582362633017; Sat, 22 Feb 2020 01:10:33 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:32 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:07 +0100 Message-Id: <20200222085715.23769-4-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 03/12] support/scripts/pkg-stats: set developers info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Use the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 4c963cef0f..643272e9d2 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -30,10 +30,14 @@ import certifi import distutils.version import time import gzip +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + NVD_START_YEAR = 2002 NVD_JSON_VERSION = "1.0" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION @@ -169,6 +173,15 @@ class Package: """ return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + if dev.hasfile(self.path): + self.developers.append((dev.name)) + def __eq__(self, other): return self.path == other.path @@ -891,6 +904,8 @@ def __main__(): 'HEAD']).splitlines()[0] print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -902,6 +917,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...") From patchwork Sat Feb 22 08:57:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242459 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=cy9IJJEb; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjGt6m0Fz9sPk for ; Sat, 22 Feb 2020 20:10:46 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 3B99787F5A; Sat, 22 Feb 2020 09:10:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KvwlFQnMXPiC; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 7870A87FF7; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id A19271BF5D7 for ; Sat, 22 Feb 2020 09:10:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 9C08B86E33 for ; Sat, 22 Feb 2020 09:10:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oAl0zxbK7swA for ; Sat, 22 Feb 2020 09:10:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by whitealder.osuosl.org (Postfix) with ESMTPS id 5293886DEA for ; Sat, 22 Feb 2020 09:10:36 +0000 (UTC) Received: by mail-wr1-f51.google.com with SMTP id m16so4646873wrx.11 for ; Sat, 22 Feb 2020 01:10:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=87TfTpPePY98SWww2llHX+6LRCYIvl6wxBAqnIqXTgI=; b=cy9IJJEbiFgJzjkEVzQt3h6cPLo5SZnS4uoP2v5aoQwaXcUO+1d1bevf2nxcN8Cul5 wqrXHdoWnf7fAua25NVHSmdeQA06OLKeser3FVyAt2dzW0xJWftU2+BWbWDSjnlfJ8V7 /Dd43JRitHk/h7KpKtQ4jNKczOHkk18TUM8xaesW4fDLBd9AWNWskJqkpsBRBwt15to+ TIA/VarJiq+BJoh5nzDkqc2e2vakrv0LRE3+reXjP2D7OXsQPfSICvCwho+dS+L2RZAS yHYbpdkoU5rYzAkuf9wKhnxhKY2N3GMWbS8t9UNh1dAZkJI3DHpwaW4wVRntGbTcrmwr OMAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=87TfTpPePY98SWww2llHX+6LRCYIvl6wxBAqnIqXTgI=; b=CkcFjDIBiFvlVlzo55itVKP9iwfjJXIlXP/8zGe1U/Qj7BsV5HuKEMd0DD5KOwYwcb 2c7DXoh7RICcEmHPedWwag5qZ0t0YcStrY6DYGEPT6h9E20hQ2dt7Sc3I6M8Zx6PcZ/x NJkw+oP+gWy32mpPXznIyeOM042PffZbAl/PHiODEb3ODwV4fMD1SysNqjZZgvnNrhgz 1/JZIVFnDQWr24qMwLMnwLJfBi0+4t1sWGnNUISB15o9bRYfVaH4UJ8T/GeXvjDACJiW 3eUbsGh7r8PyfMaiBCumuWxSprriCtkgkFGPVzqMUNhf8zHgVmeLm+yuaob7XQhX5rkU Qr7w== X-Gm-Message-State: APjAAAVHbbCepDOgs0DganDlXWgQ+u8HJkeJvmMke60qoVgNSgiowp9o HgI1fXgFYjf9ZKWypqB0kQHLE5Vk X-Google-Smtp-Source: APXvYqzaKpEjni5vms9/saZ71BUJg6pJWygyxM634C3ZLL/nqtfhh90pZArbk5aiqioKGvtS0RHwCg== X-Received: by 2002:adf:fe83:: with SMTP id l3mr56114203wrr.41.1582362634606; Sat, 22 Feb 2020 01:10:34 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:33 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:08 +0100 Message-Id: <20200222085715.23769-5-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 04/12] support/scripts/pkg-stats: store licences of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery Reviewed-by: Titouan Christophe Tested-by: Titouan Christophe --- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 643272e9d2..ebaf04465e 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -56,7 +56,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() all_ignored_cves = dict() @@ -65,6 +65,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False @@ -122,6 +123,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True @@ -384,7 +386,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): From patchwork Sat Feb 22 08:57:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242465 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=PHn3D68D; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjH62hjbz9sPk for ; Sat, 22 Feb 2020 20:10:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C7F9186DEA; Sat, 22 Feb 2020 09:10:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SpQHa3lMBZZU; Sat, 22 Feb 2020 09:10:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 6861B87015; Sat, 22 Feb 2020 09:10:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 37F8B1BF5D7 for ; Sat, 22 Feb 2020 09:10:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8DAAB2010B for ; Sat, 22 Feb 2020 09:10:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CXijIPGZNKLE for ; Sat, 22 Feb 2020 09:10:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by silver.osuosl.org (Postfix) with ESMTPS id 162B320110 for ; Sat, 22 Feb 2020 09:10:38 +0000 (UTC) Received: by mail-wm1-f52.google.com with SMTP id b17so4363048wmb.0 for ; Sat, 22 Feb 2020 01:10:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JEGL/fLtJHAfcs/fCSFhvUu7gzDrPYdUvesRI05mEBU=; b=PHn3D68DXPW3HAmwzcUDNxYRfd9ST1Q0PHLFVs5OtLACb/FMLX5nZlasfODa09btX6 bDJ67YbqRcsgbDltEcdHbycJBk4e52x8To/jAsLF6m8irQqqK3f2Y7DJXNoW2Eo3sPa0 bWkLDMblp7ilyeuVSOY2+vygP1wkDFphuB2lOBRUrzU6/IhSYtabyc1hQaWoee/hcbF8 bqlw5Y3GANYCuXy/yldMbXMWT4LmMtFjvNwEaKcIGRYL1a43ZevtNG9cm4S3kmesqrXm 4tegJ+KWV+e53+TsfDklg1XxnCMZx3EfbhKx0haaqLblAvzIEXaEnC7LX29A3zeIl0XO w+hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JEGL/fLtJHAfcs/fCSFhvUu7gzDrPYdUvesRI05mEBU=; b=lcTZJV5tjImYIvDV2UTFrFrTdHBlIqkLxXlh5oUzaDQilHcx/cZ97ulkMrZSNzPo9m ewzGSJ1WJMN/gO/TODnzV3FLJW1rQ7ROFg1G8fIy5Ep9DL4+ZDZSAOBlxtdIstHBBOEn EUZcJoZQw5nCDhRy36PkIzc2SXA8xC7M9fB8sb3OKk38LT01ptjc8uE+c/GaT09tnnJg /2b1WHUawv1LPa2J4NH/uqEIzRsQ2jYydO6Mo3jsz5SGCGk+7UKC4u/R8Qs7djQJlTdx USVce1uhexL8kdJmTHgqFHaHa56GAu1ZH0lIeHXt/Sh5Xj4TFxRbCoKTVQnMPKs6HqFy 4dBg== X-Gm-Message-State: APjAAAWKwNL8B3M5IT/fzT9x/Kz+Lyh9HOOXvvBYrMPffUaVu6aDq2cu bfmkTyQHQQIjEkWUykmA3xQfVPMj X-Google-Smtp-Source: APXvYqxrKgjmletOwsPgGLDnQj60uNm4kYOl7H88Yi+IGxZVXwDszrfzxY/OrLXnSUA9fL6v3MvnsA== X-Received: by 2002:a05:600c:2c47:: with SMTP id r7mr8931536wmg.123.1582362635762; Sat, 22 Feb 2020 01:10:35 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:35 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:09 +0100 Message-Id: <20200222085715.23769-6-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 05/12] support/scripts/pkg-stats: add package status X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Unify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message. The following checks are performed: - url: status of the URL check - license: status of the license presence check - license-files: status of the license file check - hash: status of the hash file presence check - patches: status of the patches count check - pkg-check: status of the check-package script result - developers: status if a package has developers in the DEVELOPERS file - version: status of the version check With that status information the following variables are replaced: has_license, has_license_files, has_hash, url_status Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 104 +++++++++++++++++++++++++------------- 1 file changed, 69 insertions(+), 35 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index ebaf04465e..7c8cc81cf2 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -66,18 +66,15 @@ class Package: self.path = path self.infras = None self.license = None - self.has_license = False - self.has_license_files = False - self.has_hash = False self.patch_count = 0 self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -86,17 +83,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("warning", "missing") fp.close() def set_infra(self): @@ -118,31 +115,43 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') self.patch_count = len(self.patch_files) + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field @@ -153,10 +162,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": @@ -167,6 +177,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def is_cve_ignored(self, cve): @@ -177,13 +191,20 @@ class Package: def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = list() + self.status['developers'] = ("warning", "no developers") for dev in developers: if dev.hasfile(self.path): self.developers.append((dev.name)) + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path @@ -192,7 +213,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) class CVE: @@ -406,23 +427,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): Package.pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) def release_monitoring_get_latest_version_by_distro(pool, name): @@ -496,6 +517,18 @@ def check_package_latest_version(packages): pkg.latest_version['status'] = r[0] pkg.latest_version['version'] = r[1] pkg.latest_version['id'] = r[2] + + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', 'RM API error') + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', 'RM package not found') + + if pkg.latest_version['version'] is None: + pkg.status['version'] = ('warning', 'no upstream version available') + elif pkg.latest_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', 'package needs update') + else: + pkg.status['version'] = ('ok', 'up-to-date') del http_pool @@ -521,15 +554,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1 @@ -672,30 +705,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20: @@ -746,12 +779,13 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] == "warning": + td_class.append("missing_url") + elif pkg.status['url'][0] == "error": td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): td_class.append("invalid_url") - url_str = "%s" % (pkg.url, pkg.url_status) + url_str = "%s" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "Link" % pkg.url From patchwork Sat Feb 22 08:57:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242461 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=W+U+y6AI; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjGy6Xqkz9sPk for ; Sat, 22 Feb 2020 20:10:50 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id CA0E287FF7; Sat, 22 Feb 2020 09:10:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 01J3Yik0ZDyj; Sat, 22 Feb 2020 09:10:46 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 55E7387FFB; Sat, 22 Feb 2020 09:10:46 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 95A3E1BF5D7 for ; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 91DEE860C0 for ; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sigyzXe3IR1Q for ; Sat, 22 Feb 2020 09:10:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E83F2855CF for ; Sat, 22 Feb 2020 09:10:38 +0000 (UTC) Received: by mail-wr1-f51.google.com with SMTP id w12so4686735wrt.2 for ; Sat, 22 Feb 2020 01:10:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WOWXX4T/BVSfXLjKMW/4bgPD/kC/UFAq8aqbRjPQePs=; b=W+U+y6AIJ6JOkrhG/9OTxrrQMnI4cFeg8nysYIXGjARS38yTEKAqh/ahY/YRjQq4Ir umaa6ZP7eqTMeP3BoLvjRxC+c4o5VJmru/zxjDRq63+WIe6NcZxwG7sxMS62Y7xNPAva KoDsqD8OuZnUasKSKj+52Xd6WBWQW09n10GUZFz/chXqKLMqiTfL8fYNLVSLIO3lvCv5 7nSy8js0uSkOfoAvH4C14NoJZr61mTskXC4SN/MmaEOennOy5ZtJpwjndJWKxBT++bbA aq3RZLneVJH2M6La8+YK1aW6CNHnXUPAwsWCUyxMi83rStJ9ax+h6VyNOpIj5hRjjdAm TknA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WOWXX4T/BVSfXLjKMW/4bgPD/kC/UFAq8aqbRjPQePs=; b=JDTXp2yREKpTvrv3/w8tco1rUpi77VIri10iZ44FF0vsvR+6lvxxEGSOcIddYsCAgT IbRyjvmk8yo01csW6ds3YNjg3B/7oqqb13clxCy3/ISeuxFpzzeg/9fnlKLqjwzSHLte KpZUa+po06/SDNeXeba/cCqkgk69aO4QS0QHqLakc1Ph1J0pAMbt/7D/CWXFc0S+oss0 vr9qaNYcODwgy3K/grXD6ZeQ8wzIuJavi6h6umr1xdfOBiyBu9jTVVtyOG6A9IrudcBz Xf2cdJiBChco5GVtidy83EdHu583TD7xqKicet+21G1C9GWNnjRMy0A0bFOQIL6k8SVG z5bQ== X-Gm-Message-State: APjAAAUQs64V6GJ1sbwZxKRelhis4uwgqrd6bwJdy8BVlt3gAv6yE3jJ qrIg/0CYZsQ7UuE+sjmgFrfnd3l5 X-Google-Smtp-Source: APXvYqz0Bq14R1zpHJ11tm6EVAIzQQ/WaBpxxCLzSeHymKr1Df0VoE48+55+A70mtWsWiGJy/6jcqA== X-Received: by 2002:a5d:6709:: with SMTP id o9mr52556810wru.82.1582362637134; Sat, 22 Feb 2020 01:10:37 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:36 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:10 +0100 Message-Id: <20200222085715.23769-7-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 06/12] support/scripts/pkg-stats: add package count to stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery Reviewed-by: Titouan Christophe Tested-by: Titouan Christophe --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 7c8cc81cf2..90afcc2332 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -544,6 +544,7 @@ def check_package_cves(nvd_path, packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the From patchwork Sat Feb 22 08:57:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242462 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=oIc8Xd4g; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjH06XPBz9sPk for ; Sat, 22 Feb 2020 20:10:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 04D3D20110; Sat, 22 Feb 2020 09:10:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QU9PnDEK-5z2; Sat, 22 Feb 2020 09:10:48 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 05AE9203FB; Sat, 22 Feb 2020 09:10:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id E51BD1BF5D7 for ; Sat, 22 Feb 2020 09:10:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C611C87FF7 for ; Sat, 22 Feb 2020 09:10:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K13rXuhqn2U4 for ; Sat, 22 Feb 2020 09:10:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by hemlock.osuosl.org (Postfix) with ESMTPS id 2814887F5A for ; Sat, 22 Feb 2020 09:10:40 +0000 (UTC) Received: by mail-wr1-f49.google.com with SMTP id m16so4646988wrx.11 for ; Sat, 22 Feb 2020 01:10:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KuQU9jQ8DtZ6BkI+0bJE3+D8wgqp0U7pq3mPVcsIBVA=; b=oIc8Xd4gwKbRE3r/FrGAHDZlQb0y5VUE3d8YovSnsOsLJex4M3Iua5WZot29Y5roj6 qdDSD5WPtrYcUjHQFRaOCm0axgRBKcWUKuipUTP8tvF05Z7BgPFrO+gQdbQd9Zl5RU+O Tkop7QfjCoi5f2HCPglwHSfEhr8AhvjwK8hhbCkbNsaAh9o+Lbo4A0W1W3Gmt7xeJu04 OWxOlHL2xxdpDAYWTrFcLjh737e1EyJktxNnLd5kV+LD2m0a/pfi8zR2G2LIVccZM68O qh1iBqaiS2WPBus7rvjW4ItjgvftKv0i3TZT7yZjhC/aWV6wybLOoZzJPYRi/Rnfd8ic YMtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KuQU9jQ8DtZ6BkI+0bJE3+D8wgqp0U7pq3mPVcsIBVA=; b=nLzSSTO2VGat8X3E+dy/MKWR1qFx+nsGz9KvJJrPoflqpgzagnZJ5T/6XiiIOVu4KU zixO4pn+lE/WlU6ZIZHT7x+BuQZXla08vxUouftAZjNzjo4do4yHuV5jBTBAQkOjPmec LWjHl+PivIg5IDeOqai/ErFMX6VNCsBKQbHFht8+yoNrZNeD9TWITMkg/KChb3zrWz8F tliE0Id1xWykZ8nZtcHs3fvDMc7X8PRsrJgNiafHDE1L9RKofXK7BGBFIiOucZKFnwPD uSkTWrbNgPOzleG0QBX8brm7soo1elAjz9cOD0fUfNGUczkU7yHFdQuNPWHxUM99kIrf vebA== X-Gm-Message-State: APjAAAVd6KrnPq6zgH/rtpHF4q20w1ZTXCJTJU7DCivBGVAY1iols2FK auRR8NbZlFjjRiDxgf0hvnMZnaUx X-Google-Smtp-Source: APXvYqz9e5yO76ZeAg2m9bQBcTbnTUTayWqMGP1LS0wBTciVMrjQinao1Qt98MPCaZuMgJs8fuEoNA== X-Received: by 2002:adf:fa87:: with SMTP id h7mr56506624wrr.172.1582362638028; Sat, 22 Feb 2020 01:10:38 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:37 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:11 +0100 Message-Id: <20200222085715.23769-8-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 07/12] support/scripts/pkg-stats: store pkg dir path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This value can be used for later processing. In the buildroot-stats application this is used to create links pointing to the git repo of buildroot. Signed-off-by: Heiko Thiery Reviewed-by: Titouan Christophe Tested-by: Titouan Christophe --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 90afcc2332..36b33586ef 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -64,6 +64,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.patch_count = 0 From patchwork Sat Feb 22 08:57:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242464 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Ea3OwATi; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjH46K3Lz9sPk for ; Sat, 22 Feb 2020 20:10:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7BFB022274; Sat, 22 Feb 2020 09:10:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6CctyyEOzC7g; Sat, 22 Feb 2020 09:10:50 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id D00D6204A2; Sat, 22 Feb 2020 09:10:49 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 1FEC51BF873 for ; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 1348786E33 for ; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ovewbyqZ274S for ; Sat, 22 Feb 2020 09:10:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id 0A7DB86ECC for ; Sat, 22 Feb 2020 09:10:40 +0000 (UTC) Received: by mail-wr1-f47.google.com with SMTP id z7so4645992wrl.13 for ; Sat, 22 Feb 2020 01:10:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O04DN2UEzXttOvKIhulZy4QsLxXdyxiXOTkfGycFMNE=; b=Ea3OwATiS24loXix6gTOsuJaHSoE2HUtGi0josFaxTMPQeUeU+e3IwVJpsjASN3Gma 3d1bso+FeUJZoMVMPsuMVtRbrjPRBL6koMknYZjK2nqhT7OEGbls0Djltf/MyfA+Wy7W 7atv+sHvWQjkRvdJjhwK8srpRNTnE3Kgr0KomHbOz39/sVitDkCUepk5344zUZAlGBXM EeLLj4lE5J72BQmjR25q1LCyeKHT/PLZ5EgQLabOEUxRIowBOFkNH4nkpx8cPPe4aQKn /WoNGJwqZa70baiqTq51zqIvAP5IXwX3A0yuBuBxad/XuGGM+fyyRDvYqP0SBhqSARMb d59g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O04DN2UEzXttOvKIhulZy4QsLxXdyxiXOTkfGycFMNE=; b=oEi2PMCZB27e56dT8QGtJddPmVCMU1XwbUXhiQ25gR6GUqyTzJLftXpspP1PEqlGhN ieAEB41faIndoCDG4ADwpWagqG8ZWa5om+9QNOTaC02oR2RRepIwDofVqcd7ahuZkglA fj2Ck7GkZgHoJ1Zx9m7deau7949/NR8t6iKhBHgYa4ZuVJALWHX9KDdF2LAmV+mJ+jUy hsFrSsOxDFV8swuy6Ib7AvoqfSiUMfmGB0HsPNEPjGFo14V1Y17joySk2hGVfbn6H8pC SzUeTUC5GymYs3qbH9TYyjqTaTk2rXIzh3SWvMJpFIolUKKcPu4WmYu2gnKfxotz048C F/zg== X-Gm-Message-State: APjAAAXoBPO9NgS6fcQ7WRbx1lZfREy8V1xDgYhmZrMk8MnCSF42fjIZ gT29hgs3h7C+exzgFPVizCUO7RUL X-Google-Smtp-Source: APXvYqx31NzvjbKhZd3r0yuw/8buUyMbHfAMWcLZxetNPWgkIvhKaxItD4WNoXf6ai99kIW3/yTPoQ== X-Received: by 2002:adf:eec3:: with SMTP id a3mr50763305wrp.337.1582362639127; Sat, 22 Feb 2020 01:10:39 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:38 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:12 +0100 Message-Id: <20200222085715.23769-9-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 08/12] support/scripts/pkg-stats: add defconfig support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Scan configs directory and create Defconfig objects. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 42 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 36b33586ef..ae70f90485 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -54,6 +54,33 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + if dev.hasfile(self.path): + self.developers.append(dev.name) + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + defconfigs = list() + files = [f for f in os.listdir('configs')] + for name in files: + d = Defconfig(name[:-10], os.path.join('configs', name)) + defconfigs.append(d) + return defconfigs + class Package: all_licenses = dict() @@ -882,7 +909,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name'] @@ -893,6 +920,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v @@ -903,6 +936,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)} @@ -944,6 +978,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -970,7 +1008,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__() From patchwork Sat Feb 22 08:57:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242463 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=SqvJPqnU; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjH30WFCz9sPk for ; Sat, 22 Feb 2020 20:10:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C796C87FFB; Sat, 22 Feb 2020 09:10:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZY1r3vgDiaXz; Sat, 22 Feb 2020 09:10:52 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 19BE387FFF; Sat, 22 Feb 2020 09:10:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 72BEA1BF5D7 for ; Sat, 22 Feb 2020 09:10:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 67EF386DF4 for ; Sat, 22 Feb 2020 09:10:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l16kzsLCGx6s for ; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id 4116A86E33 for ; Sat, 22 Feb 2020 09:10:42 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id p9so4182584wmc.2 for ; Sat, 22 Feb 2020 01:10:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wmmnRUlunBukdg/3vXQ3I1zLIwh0v+Ow8MGNYW3HGgY=; b=SqvJPqnUNgP4iXvVap+ZbyCe+a6T08r1iQiv47qcCFDML4N2b4MZJ144JHAh0RApAZ 7y3oYegys0+UqQUBuMJQ2hZp3/IxtBnyUwT9rlz8soEme0fGz5Ib7XASWyPbyxeLd0Rs XCA1XxuaVg107FEtvXoompU1yx35tS6a5SN23u6xiWUaNyMEtZABXWyBE3lP44z8nI6q NO4trAFOg/JwHX36Rxs5t+83am/QCBIaaEBkTJkIy+ZYDeXY5N3BUQDzmdiCaDKwJZo+ tYdEdTLRm/QroUuPkas+xVE9vuItHJxeg3SNjlwWpLxGI1L+/eaAVBd36fuoEDcVJXGZ vztA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wmmnRUlunBukdg/3vXQ3I1zLIwh0v+Ow8MGNYW3HGgY=; b=Gyvv7NW33ghuZ23wESnTrgVmLW3UTj4YyVtfVSmc/O9Pcozi4tYPiqzlcjwxYvziOB EV7k5YDc6uxqmfep44IDnzG5CGOzYeyaL9K1CSUpaT2fMaQ7XPvEgkrUEqRhF0oTQXiq CP6f/t9gyuN7kzVs5IHZ1Sf/0R2lnQ3zHE0iecQ8peBaQ+oz0NDAwWRaPjvWWRh6HQAx F/4uI7TfCnvv5ONwJgKd50zHYe1guOVkM4Pish4G3GTqUZeRiJL1Ag29EvOsM/k6EooS CMVXn7YGBP7NmtK9KLr7ILf3Ic4e/dIkxkBy2dqSouQOoy9uKsUocM6BrF9cv1aTt1Fq VluQ== X-Gm-Message-State: APjAAAVrxOYeJzjMQPAiIr6Ljjkh3p4bYDSfMMbpas4sepFMem5i4VBg uodDNPd+RJCT5oKe4M3RpPdVAqni X-Google-Smtp-Source: APXvYqxH8i7Y2AqDRldXuxCSYxy1va6mnA2WdB30wLmuMAMPsI6ViJNmDiuncCVkkmuUmHYJqTREgg== X-Received: by 2002:a7b:c14e:: with SMTP id z14mr8929454wmi.58.1582362640101; Sat, 22 Feb 2020 01:10:40 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:39 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:13 +0100 Message-Id: <20200222085715.23769-10-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 09/12] support/scripts/pkg-stats: add support for license hash check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Store the names of license files and check if they are in the hash file. Signed-off-by: Heiko Thiery Reviewed-by: Titouan Christophe Tested-by: Titouan Christophe --- support/scripts/pkg-stats | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index ae70f90485..e954dd125e 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -84,7 +84,7 @@ def get_defconfig_list(): class Package: all_licenses = dict() - all_license_files = list() + all_license_files = dict() all_versions = dict() all_ignored_cves = dict() @@ -94,6 +94,7 @@ class Package: self.pkg_path = os.path.dirname(path) self.infras = None self.license = None + self.license_files = None self.patch_count = 0 self.patch_files = [] self.warnings = 0 @@ -152,6 +153,7 @@ class Package: self.license = self.all_licenses[var] self.status['license'] = ("ok", "found") if var in self.all_license_files: + self.license_files = self.all_license_files[var].split(' ') self.status['license-files'] = ("ok", "found") def set_hash_info(self): @@ -159,8 +161,18 @@ class Package: Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") + self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): self.status['hash'] = ("ok", "found") + self.status['hash-license'] = ("error", "no license in hash file") + # check if license files are in hash file + if self.license_files is not None: + self.status['hash-license'] = ("ok", "found") + with open(hashpath) as f: + content = f.read() + for license in self.license_files: + if content.find(license) == -1: + self.status['hash-license'] = ("error", "license missing in hash file") else: self.status['hash'] = ("error", "missing") @@ -441,7 +453,7 @@ def package_init_make_info(): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): continue pkgvar = pkgvar[:-14] - Package.all_license_files.append(pkgvar) + Package.all_license_files[pkgvar] = value elif pkgvar.endswith("_VERSION"): if pkgvar.endswith("_DL_VERSION"): From patchwork Sat Feb 22 08:57:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242466 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=UhswK2lH; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjH72B4Cz9sRJ for ; Sat, 22 Feb 2020 20:10:59 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 831538805D; Sat, 22 Feb 2020 09:10:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7PhEp74m2Rhe; Sat, 22 Feb 2020 09:10:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id E37D48807C; Sat, 22 Feb 2020 09:10:55 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 646E71BF5D7 for ; Sat, 22 Feb 2020 09:10:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 18A3E203FB for ; Sat, 22 Feb 2020 09:10:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xcinS-8+isOe for ; Sat, 22 Feb 2020 09:10:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by silver.osuosl.org (Postfix) with ESMTPS id 18AC6204B0 for ; Sat, 22 Feb 2020 09:10:43 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id c84so4351769wme.4 for ; Sat, 22 Feb 2020 01:10:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=REDQAUnBfcamcNtQaCK5yzCOoqaTay3IygzONpxLncs=; b=UhswK2lHnOY02KsTfhgBowy3sg3XIC0E3K3nJ18lUM0AnTpI2FTpEBqYPAQhqAouaV aSt06huUtX7g4KoCCr1ZQuWOZTPKnFl/MpCZXFSguKozg6r7UQDj/EqNbcCB1uyMAR6M E+hNgVrnNAFwtOlJbBxC0UNjfgO7i9gR/hjmjBfpPbIj4Y7pACo0UKLhzs8IHQAdL7hR lhGhNqpwCkuwi4hQB898TjxED9q94ma6iGbaU+9TnBxtOz/ANSDolu+9Eu27LOJAQuBn 3bjou8VxT30HVpQdng1iTB/36U8bnHsxqdVoqb4dmZCzdSMWaCanJICgOMcDKh3QgPo0 EPJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=REDQAUnBfcamcNtQaCK5yzCOoqaTay3IygzONpxLncs=; b=p56TW75NLA0w4hi5ELdf7bY9+RJf8sVYmBsO5H+R3HxP/AL56XPrWhq5ArWJ30l1wL qm9tSNcY5DhwmKC7vNV+DlaXe/ed/uB4kiAi6jVrgyDahad3/1rCz3iqOifXGU2RHiPd JKSLVoP/l47uqoPKaH8KJWXNzJIfLWNO6vEY69mik4DW6gGfND7xSzGs93EDxO+RL1fJ k8+3X+kFe6/X0A/hZ+dGS9PsB5WhUZuoD3SDdWJSVMdrclNTqrBAnEpo7QvcbszMEYMF mYcVW2+ui58iRpzv3soQfe12eoTM0SZlb6ZWINfZ6+85xPjz4qYLHaRF5Uwkti5CLYJx 73Wg== X-Gm-Message-State: APjAAAU+TOiDE3XnqdSP8e6ebvDFG/ELTyeGJSEYcEcNfqom5j9Fx7nC iI0N8Qgp4gCG5JzaE5ayBm7UzvK0 X-Google-Smtp-Source: APXvYqz5MyCiw3lZmw1il8gnJDcrC4FksUjoL0IBen9aQ0o49NUhABX821b1UYrwO+0BaG/YCVXpFg== X-Received: by 2002:a1c:ded7:: with SMTP id v206mr9432042wmg.106.1582362641135; Sat, 22 Feb 2020 01:10:41 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:40 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:14 +0100 Message-Id: <20200222085715.23769-11-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" If there is no infra set or infra is virtual the status is set to 'na'. This is done for the follwing checks: - license - license-files - hash - hash-license - patches - version Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index e954dd125e..be3b6d7e71 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -125,6 +125,14 @@ class Package: self.status['url'] = ("warning", "missing") fp.close() + def is_valid_infra(self): + try: + if self.infras[0][1] == 'virtual': + return False + except IndexError: + return False + return True + def set_infra(self): """ Fills in the .infras field @@ -146,6 +154,11 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ + if self.is_valid_infra() == False: + self.status['license'] = ("na", "no valid package infra") + self.status['license-files'] = ("na", "no valid package infra") + return + var = self.pkgvar() self.status['license'] = ("error", "missing") self.status['license-files'] = ("error", "missing") @@ -160,6 +173,11 @@ class Package: """ Fills in the .status['hash'] field """ + if self.is_valid_infra() == False: + self.status['hash'] = ("na", "no valid package infra") + self.status['hash-license'] = ("na", "no valid package infra") + return + hashpath = self.path.replace(".mk", ".hash") self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): @@ -180,6 +198,10 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ + if self.is_valid_infra() == False: + self.status['patches'] = ("na", "no valid package infra") + return + pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') @@ -558,6 +580,10 @@ def check_package_latest_version(packages): pkg.latest_version['version'] = r[1] pkg.latest_version['id'] = r[2] + if pkg.is_valid_infra() == False: + pkg.status['version'] = ("na", "no valid package infra") + continue + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: pkg.status['version'] = ('warning', 'RM API error') elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: From patchwork Sat Feb 22 08:57:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242467 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=iIuwBV9e; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjHB19f7z9sPk for ; Sat, 22 Feb 2020 20:11:02 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A1F142262E; Sat, 22 Feb 2020 09:10:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F5a98zpknKxf; Sat, 22 Feb 2020 09:10:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 4D4DC20499; Sat, 22 Feb 2020 09:10:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id CEAD01BF5D7 for ; Sat, 22 Feb 2020 09:10:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CBFCA855CF for ; Sat, 22 Feb 2020 09:10:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zI2b8n4Ny4Oh for ; Sat, 22 Feb 2020 09:10:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 242E9855A1 for ; Sat, 22 Feb 2020 09:10:44 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id s10so4177962wmh.3 for ; Sat, 22 Feb 2020 01:10:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+4k25AiIdcA+/zKiV3UsKdWg50MxHk/VT2CjfJafYoc=; b=iIuwBV9egLZ9GgGV3Z3eyeYmpaKgMgS4+QXnymrZiXXVpRePoe2D7TFc1asKRZ5b3L 2va3he9sEGDHHLqjjpLAtSzdWz98h7SBkp6Uc2ASRP6kHgDdHNaCT8FZPEdW9eyP+8dn zLDW8RBWVks8ZlMHJscPkHWnRngEFXdjd5GIQSi5Hafkng6sOKyOlCy2CG9Fv628Dp53 VoWLMMp1UdMuQokzIXF1FqCEgXU7mbyA8+C9EbUx5la7f1Ntjy+7dEr8FiOqWOPmEdY8 Gsey9RIPT4SpNJbGGlJB+Pf5zU75LRIAx+vJ6XUEp9uHkJbWtVFIWXvJU9UATzf9wu66 cmAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+4k25AiIdcA+/zKiV3UsKdWg50MxHk/VT2CjfJafYoc=; b=SdlDrLQB6KW65ohmRE4cWVgQEEkVlIdln8if1VkrPUzTuS4c+UOW69XpszVrXH78Ev Xz9hqAOk6qyyfia77qThlpJuDcOcEIKiDQyWdcLAHJ1UjIYLit6vKrBC80DOt62CfY4v 4/RWBVVz3mb/xsWRZ9VWjgWC7hA264CsNKE25j3catDqNIRzkmUNv3d2WmyEHMIY+AUq PTk6wgeS0BI55lzz7dLDpmwexXZ4fH9oKp5tE4cFDlWGEbvFeupvg6RMVunRW9XoQ/fj 03hHUbTHqwpF9dK7YyTy7JrThfvTvv484HCJ2C2UiuoY7Yl2yWur6d3UeMEtj3UfyTcz t2uw== X-Gm-Message-State: APjAAAULzx81BgK/al6IfWIlU5poFSZ17qzHkt+KzpQIiEsF4NBsXgqf HAN6F+Y2vf2z0FNpj/HqR5v4bZYS X-Google-Smtp-Source: APXvYqyz7J6rvJe8zMNeMkyqPxe6ERJVIH5gWBkwcq4YL10F5+OL0feeQOnq/9qZpr9DfItC8xu+Yw== X-Received: by 2002:a1c:bc46:: with SMTP id m67mr8858820wmf.40.1582362642105; Sat, 22 Feb 2020 01:10:42 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:41 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:15 +0100 Message-Id: <20200222085715.23769-12-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 11/12] support/scripts/pkg-stats: initialize all package status checks X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Also the list of status checks is added to the json output. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index be3b6d7e71..ed22f6b650 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -87,6 +87,11 @@ class Package: all_license_files = dict() all_versions = dict() all_ignored_cves = dict() + # This is the list of all possible checks. Add new checks to this list so + # a tool that post-processeds the json output knows the checks before + # iterating over the packages. + status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license', + 'license-files', 'patches', 'pkg-check', 'url', 'version'] def __init__(self, name, path): self.name = name @@ -103,7 +108,15 @@ class Package: self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.init_check_status() + + def init_check_status(self): + """ + Set the status of all check to 'na'. + """ self.status = {} + for check in self.status_checks: + self.status[check] = ("na", "no status check done") def pkgvar(self): return self.name.upper().replace("-", "_") @@ -975,6 +988,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output): final = {'packages': pkgs, 'stats': statistics, 'defconfigs': defconfigs, + 'package_status_checks': Package.status_checks, 'commit': commit, 'date': str(date)} From patchwork Sat Feb 22 08:57:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1242468 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Hazncsr2; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PjHH3TPYz9sPk for ; Sat, 22 Feb 2020 20:11:07 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 37E42203FB; Sat, 22 Feb 2020 09:11:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V22r-cYd3+Ed; Sat, 22 Feb 2020 09:10:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 3663F204B0; Sat, 22 Feb 2020 09:10:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 258091BF873 for ; Sat, 22 Feb 2020 09:10:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1E38887FFB for ; Sat, 22 Feb 2020 09:10:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0SXEiVW5Pxmv for ; Sat, 22 Feb 2020 09:10:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by hemlock.osuosl.org (Postfix) with ESMTPS id 23CE887FF7 for ; Sat, 22 Feb 2020 09:10:45 +0000 (UTC) Received: by mail-wm1-f42.google.com with SMTP id m10so6870873wmc.0 for ; Sat, 22 Feb 2020 01:10:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BjSV+L/dUHcL0xbNQNgd1I7ypBF2bKE/Fi8V2OmtgRk=; b=Hazncsr2ldr7kPexEAqyvCuwTApFeH/BBNySfTuro9v98IGN+hGB3H0ZMbjmdNsGTt vwdwaApW4FegJ+GkPafJMM29gwoDBl2PyrQDfhCk0s/5BNOyipHD+HpMITLV6EX6s+ps aNGPXNhUT778mEztyxYSCnNG9OUFVPBKlUX58KBfKDNQxgJooKfrM4ck6Upa+KrFOHfa /hp42FVcnSMzUUqSjy2akfmESidbjXP2McMJKm/8W2fhtI+RvylRKJZ59L2MSpXhZj34 9KkECQqvvLXUZcWE5N2lMUk7z2lt8SzjXMgJij6cY1wNWJ8Xj85F5sN1k8AddBIoAPBb J+/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BjSV+L/dUHcL0xbNQNgd1I7ypBF2bKE/Fi8V2OmtgRk=; b=JcKw3Uq+EgAAj8YVqDcQnZXRJU4KJBCnKpUReXUjKZst3sd8HgTHval/1Z2GP2B+40 C+MJHTl1T14o+Ykww6Fea9eWS6R/XoDMgblBAYvKcA/J+rRQrST1BWGdjcZkacZBUEju vv33SJ44VZ6MMfr3LU2R5L6butAobHbdzJXt6bd+R3NCAlxMDgaUYQ6t/P+bUUoW8Caa NYWRAKCXOID+i1sE9lirQYKOhWm4syn+S6ANdalwAw7YsejWKHyevQ2AqTmhkLTfJZW1 9QCk2KWCDoBvb3hQ8nFaqV2uawU122Au/+GRSktD0gLKdV0TvdxhquQItZwZ6MYoHz2s NF0g== X-Gm-Message-State: APjAAAVFA9Wnqjj8aJ15Z99OOS+SBbbkwfks6OyXXxqdGIPv9T7skJaP HBa5kx4jRXbSfH4audUaJJlJY7H8 X-Google-Smtp-Source: APXvYqzCQl1/N5FOhF2fljHB1LqA+bDmNaIMEos1yNB0dcBGYYodccjmZNPDHqBrqmx9dqAxHnAYQw== X-Received: by 2002:a1c:7512:: with SMTP id o18mr4624450wmc.110.1582362643121; Sat, 22 Feb 2020 01:10:43 -0800 (PST) Received: from t450s.fritz.box (ip5f589959.dynamic.kabel-deutschland.de. [95.88.153.89]) by smtp.gmail.com with ESMTPSA id l6sm603970wrb.75.2020.02.22.01.10.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2020 01:10:42 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 22 Feb 2020 09:57:16 +0100 Message-Id: <20200222085715.23769-13-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200222085715.23769-1-heiko.thiery@gmail.com> References: <20200222085715.23769-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 12/12] support/scripts/pkg-stats: add status for cve check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index ed22f6b650..4efff8624f 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -617,8 +617,14 @@ def check_package_cves(nvd_path, packages): for cve in CVE.read_nvd_dir(nvd_path): for pkg_name in cve.pkg_names: - if pkg_name in packages and cve.affects(packages[pkg_name]): - packages[pkg_name].cves.append(cve.identifier) + if pkg_name in packages: + if cve.affects(packages[pkg_name]): + packages[pkg_name].cves.append(cve.identifier) + if len(packages[pkg_name].cves): + packages[pkg_name].status['cve'] = ('error', 'affected by cve') + else: + packages[pkg_name].status['cve'] = ('ok', 'no cve found') + def calculate_stats(packages):