From patchwork Sat Mar 7 07:56:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250839 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=qLlPmuQr; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGz26rXYz9sPR for ; Sat, 7 Mar 2020 18:56:46 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 51C2686EFF; Sat, 7 Mar 2020 07:56:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7O5PLBb8U61k; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7B81686EDD; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 1B0ED1BF42C for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1822387E07 for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gzh7bRrEeXJ4 for ; Sat, 7 Mar 2020 07:56:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by hemlock.osuosl.org (Postfix) with ESMTPS id 872A987D9C for ; Sat, 7 Mar 2020 07:56:40 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id v11so4928160wrm.9 for ; Fri, 06 Mar 2020 23:56:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WJoIOr8zl6kKt8h4OxGy4NyIQVyIm8l0DUARGur6fZc=; b=qLlPmuQr5m3mkkKUP1g7tfEXNzcZg5S9GSge3fK+QybjpfNrjO6HuzTO2/ibwr6QBv 360L167Go6GfEoOJCeo8a1A1TJkzVwc+17JxHw4fpgxpQTYnqdA6KanE7XXrzZhv4oPM xmR8fRUkQkqVpWw65zl1Gn/2vpuKKnr1rlKY24o7ip9VlRE2ndd+CvW6iVH+x66UEdmh hktQcmOmZq0IeJvqSClqS9EURgMYUKoPzkkPB+kxgfxXTqa/+P5JvL9kyWCTm0t0/FCs gItJgeGG8OJpb2eymttesbUmj5pOQeoPedW8fZTg6FMAVPTSM3fgMciUwqW/SxOXIOFx 5QMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WJoIOr8zl6kKt8h4OxGy4NyIQVyIm8l0DUARGur6fZc=; b=phxCeoH33sb3HMLXdAW2l3WVmIxaR0P68H8tSCcOwob8pDFfNz/61JrhwqwIBzflHU eFfFSINt1IYi6dfcxGTmHWRqbSR0WAClhqkXSQS2PiVICyIhhFESwCcUCO88ID+Krs8f GwNa+/tBKxxYMvGK0dSTBQ0QqdG3XTvTH5yhmjPEr9DR9khe0mrHf9mFyqy6d0z/eMTf Qy7u3KB+D+maRFnvXy3BjYSkUkjnaJSuaSKzkLH9p6EYtw19r3f56WugnZs+AeBuMSD2 eFIdXOS5+pdydddrDpgQYsMzaWNGsnxSx/nYGews1BRvkN3ENVC791ifOedBMjbLHJtU QgJQ== X-Gm-Message-State: ANhLgQ0ZsF5iJPZyJ4OG0uYmZ8kR/lBIX7BzJB/gHJnXLqLsW/BN9S64 /ULTcw2kLbeXWGD8bjZ7FIpnLp0j X-Google-Smtp-Source: ADFU+vvSXWJg8yZxkFiQzzMOW/W3JnbFA7O21EcYYS8VbALKTtHK0nMcC7N4xcl33H27I+4O+V9WIA== X-Received: by 2002:adf:df91:: with SMTP id z17mr7987660wrl.97.1583567798479; Fri, 06 Mar 2020 23:56:38 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:38 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:22 +0100 Message-Id: <20200307075633.7514-2-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 01/12] support/scripts/pkg-stats: store latest version in a dict X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8a67e509e1..95ec9e86bd 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -71,7 +71,7 @@ class Package: self.url_status = None self.url_worker = None self.cves = list() - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -462,9 +462,8 @@ def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects - This field has a special format: - (status, version, id) - with: + This field is a dict and has the following keys: + - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND @@ -480,7 +479,7 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) worker_pool.terminate() del http_pool @@ -519,13 +518,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.latest_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.latest_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 @@ -691,29 +690,29 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.latest_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "Error" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "Not found" else: - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: latest_version_text = "Found, but no version" else: latest_version_text = "%s" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.latest_version['id'], str(pkg.latest_version['version'])) latest_version_text += "
" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by distro" else: latest_version_text += "found by guess" From patchwork Sat Mar 7 07:56:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250841 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=FZnpj5/h; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGz73Wy3z9sPk for ; Sat, 7 Mar 2020 18:56:51 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DE33F86ECB; Sat, 7 Mar 2020 07:56:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DwcppMQ2G4-i; Sat, 7 Mar 2020 07:56:45 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id AA06286F02; Sat, 7 Mar 2020 07:56:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 1DBEE1BF867 for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 196A520356 for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B3ymTYbuJue5 for ; Sat, 7 Mar 2020 07:56:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by silver.osuosl.org (Postfix) with ESMTPS id 2B9811FE2F for ; Sat, 7 Mar 2020 07:56:41 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id 6so4817266wmi.5 for ; Fri, 06 Mar 2020 23:56:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LgRSMd2TGaNfC1nbcdijmuj1M9ZxTmtAT02U1yGHSz8=; b=FZnpj5/hrwkAWcONqEXD+G/gPMhTFjh7N+NrlemK0mpEh62LUQm2DUbwImEmPDBIrL b7r7fNZU8ULJ4RdWI88E2Tu0DPYBObY81bZm7k1sa9FuoxhZXB6pFabrvi9rikA+VNQp rMXWpQItZbsSd5yEVvCzoPdYfGw+TGAZy4+p6k643zUs9exQH8Z0e1Pd8W87SuVvNWEA IA1/FXwS/BK91NR6NFXnjc/P5BZ1hqoExizYp+Abpv3VgI2JJMXg3xJ4yOFsYNN9ft2I qbI4cFRk2rkbiEloNrfEME0HELHOt4Xu278tAakYn+VRKpaFuNNh5QGf/nHalz/+T7+j MSlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LgRSMd2TGaNfC1nbcdijmuj1M9ZxTmtAT02U1yGHSz8=; b=mE65Fh79jTZ94TBx/gDwuqzONZX/ZYBbLx2MuSqiFwHJ+CBpZHkUIaVvridc9k/zSX 6WLdgbVXQRMhPtHaQxCH0/n8iRrkBW1TSzVhahoktG8v7vQLRgfTmK72/HTjdmNGM8QE yOeoAmNvh5oysnJBc1bFzHWLuTMu6nHjKpNv1vnomCPbCqlvMiJnRub/Gt+shslQytTV nO4YXXPn4PCyiCZKxSpppYJj4KE18eaiZCgKiGBmxRkZxcNnreDRUOPLRCKMiVKiXXhj mh0bDkGdC/r1fMgnKie29mCG0N3YF3qyPUUCpj9aEl/jjr8AVCZpjhjZv8z4WkWyyfCD KIhg== X-Gm-Message-State: ANhLgQ1mf2ldHmuPNqi02hnpTsOw2ODcXlJFIp6Q9iRqC5DcAF4QiwEH I5Re7HVLWAHBRKLsqR3+X5PTQc7v X-Google-Smtp-Source: ADFU+vub57/XXmWqLoTRlkhUMqQGeVE+q3LsgMvZDY3VRErGRRqv4cIa4e5m+D8kX4SyQtNDp7xasw== X-Received: by 2002:a7b:cd83:: with SMTP id y3mr8394052wmj.176.1583567799455; Fri, 06 Mar 2020 23:56:39 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:39 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:23 +0100 Message-Id: <20200307075633.7514-3-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 02/12] support/scripts/pkg-stats: store patch files for the package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Remove the patch_count attribute and use a class property instead. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 95ec9e86bd..87018eabd3 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -64,7 +64,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False - self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None @@ -93,6 +93,10 @@ class Package: self.url_status = "Missing" fp.close() + @property + def patch_count(self): + return len(self.patch_files) + def set_infra(self): """ Fills in the .infras field @@ -131,10 +135,9 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') def set_current_version(self): """ From patchwork Sat Mar 7 07:56:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250842 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lXDXCwZv; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGz81cpWz9sPK for ; Sat, 7 Mar 2020 18:56:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1F88188221; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6kUUezh0iI49; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 56E7D87E3D; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 6115C1BF42C for ; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5E93B881C8 for ; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cHsvXSY7HzHM for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by hemlock.osuosl.org (Postfix) with ESMTPS id 53E3D87D9C for ; Sat, 7 Mar 2020 07:56:42 +0000 (UTC) Received: by mail-wr1-f47.google.com with SMTP id v2so4920683wrp.12 for ; Fri, 06 Mar 2020 23:56:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qzt9h2wqpeUN1PxkJufSDvUMn+DKhZOMbOjaUNXzVsg=; b=lXDXCwZvpy3x5gFZZnEsdc5BvmJ6HtIm+ZcFoSGXQe2BuwvkLisYt3+Uc6Mwr8jK8h 4iSPKCeygNPqOtYo3zPr1NAxZ+COqHFnIWzPrn5+qNBMczvJMwBRpFVnNLzEI6TKvr7T 9LxHV1csprwgqmW4XZ8XwQiY12ZOjEtn7QnZ0mCVbeVTuCTAtuz/GGt9A3okrT6xBCKQ pozUFquASdieRP9z8znm8Lg9hiSpKU+56Ng25V+CQwUx8xrXaZ8BBoaaBqadFPerO5PN e/T4ZlVcc4Hbilso/jbed2/wMBGtraeDFda9kgZRTaSeG9h1FJkvXNbJyVsRUR7AmbNC unBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qzt9h2wqpeUN1PxkJufSDvUMn+DKhZOMbOjaUNXzVsg=; b=FrRNpAvkerMW3dBxEXUSQnp8m2xAaRJoQeV3Rstv3CHlIn/GoUc5n81+UX65SGR3t9 QQNvF34LZoRTKN4SQ/5twnmyC5WOOETehbRDe5bukAG7Q53+rC2Ap6uW+htMBFh0cqWg tnzKloXip7BEPB5k5Ckb4JLRr2HOaqXV1gcJHRdbXorxECozd3QVak4m0A4g5/ZnuN+g Mg72gBtEVz5sXL3x0wGZpvIilz6W60z4n+iAM5WfYdcN1bokqoVk7E4XC6WrgncX70uL 2gt0UUKX3cKwb4BET1eEULqyYc068Shjit2cpwHepmRn0ZKp194JxUTT02R6H5pd+Bgn EGUw== X-Gm-Message-State: ANhLgQ0amWdAlSfzJC+WtVuJ6odVvQd7k8gUfLQrdBg0RwfMNGF8dioI 5sAL44TXZHljv32JhQlNd1UzxOg7 X-Google-Smtp-Source: ADFU+vuBoBIjGAU+/ZqwkaQKpbobBlhDdiPRrBHmFylyIlV9J8D00YNK3INCLc2s7pSjMOohKNfQDQ== X-Received: by 2002:adf:f641:: with SMTP id x1mr8298458wrp.248.1583567800499; Fri, 06 Mar 2020 23:56:40 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:39 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:24 +0100 Message-Id: <20200307075633.7514-4-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 03/12] support/scripts/pkg-stats: set developers info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Use the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 87018eabd3..e34841fc7c 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -30,10 +30,14 @@ import certifi import distutils.version import time import gzip +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + NVD_START_YEAR = 2002 NVD_JSON_VERSION = "1.0" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION @@ -171,6 +175,16 @@ class Package: """ return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + dev.name + for dev in developers + if dev.hasfile(self.path) + ] + def __eq__(self, other): return self.path == other.path @@ -894,6 +908,8 @@ def __main__(): 'HEAD']).splitlines()[0].decode() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -905,6 +921,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...") From patchwork Sat Mar 7 07:56:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250843 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=X4nDb17O; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzB6C7Xz9sPK for ; Sat, 7 Mar 2020 18:56:54 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 94DCB86EDF; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CtgO2887PVAa; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0874486F06; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 1D2A91BF42C for ; Sat, 7 Mar 2020 07:56:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 154FA20356 for ; Sat, 7 Mar 2020 07:56:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F7WPUm5Q9wDY for ; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by silver.osuosl.org (Postfix) with ESMTPS id 3FBF21FE2F for ; Sat, 7 Mar 2020 07:56:43 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id v4so4958250wrs.8 for ; Fri, 06 Mar 2020 23:56:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NhzGSarS0rCfkoc6jpQXtzpVe7BEkd6piUPFHxxHQbw=; b=X4nDb17OOddoN66YYuiP2q6NE3Qcdy2eyDitgCOojIO0ZUTSzOBiPiu8ieFUPbnRlX 12NtDmwXb7HWxmsbR8ELMzISUKIIC3JOLI499lrVRES7tTSMlRLrGjleiIO6p98p6Tbi hFp1VhWp+xkLBeAW1FouDxVv3awv9sCOJpGLeZA8NvHBLDsBRcjNMmD1WrnVA61ZO9sX UyVYKt4Ewio6tiwQBXZR4kvKzHuAAxcY5CvJrTsxHLF32IE6z8yIaqSedROdI7kMofS5 HT/uuCMI2KWbdS1/umADt+7pclwn7BZo95w8hq6TL3Jb1xhsMonegE/AUeE9kdyYeo6I WlAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NhzGSarS0rCfkoc6jpQXtzpVe7BEkd6piUPFHxxHQbw=; b=j0rEtfIi3PC+jl1ml6v0bTtISa7BASMJwOmnDCTM3X1yO9ZtwKf9ZNMiofLM0y028J UwLOPA2v+v3/SiYvTs3rC/pd35BfqlRZYybyhcnC3qmO++sme8evKKRj254676EgYYl+ rCMCgNE7+IqUdgqiIUg9kRR3dZQoj+VRkmZq9nthEnclFkNK6KjDg1iey+KnJnWQ90hR f/c+Oi15Udy0FjQqMhHzcYIckYhX6iTL7wkqXF+1kDeuHhH+9J8/LlUcQqQyjcXzcGLL TxPtgveZkOnBh5Vxta+Cm8qjqX29AmvLkTju6O3+pAXnWyunNa16KXsEluCY12WJs88+ NZEg== X-Gm-Message-State: ANhLgQ0paEw+TAJcBRKKunwUwj0bSoTywFBhtzThq1puOXhnsxOXK6zc wpKsFFpSodpJ46unWBIuMAFsQlRE X-Google-Smtp-Source: ADFU+vumGRmmybn5sa7DR1fjSTwg3IEdMc7rUb5QTYiEt5RHyZoAloFgiqN1ihFSbpVjjBslrlK41w== X-Received: by 2002:adf:de12:: with SMTP id b18mr8305979wrm.268.1583567801475; Fri, 06 Mar 2020 23:56:41 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:40 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:25 +0100 Message-Id: <20200307075633.7514-5-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 04/12] support/scripts/pkg-stats: store licences of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index e34841fc7c..9529ad5c6a 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -56,7 +56,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() all_ignored_cves = dict() @@ -65,6 +65,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False @@ -125,6 +126,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True @@ -387,7 +389,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): From patchwork Sat Mar 7 07:56:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250845 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XeGdBTHK; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzG4yyWz9sPR for ; Sat, 7 Mar 2020 18:56:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7428686F06; Sat, 7 Mar 2020 07:56:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7TSkZKC5wg7f; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 199B886F53; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id B4A7D1BF42C for ; Sat, 7 Mar 2020 07:56:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id ABB3D8766A for ; Sat, 7 Mar 2020 07:56:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id of7+5fNmVzqS for ; Sat, 7 Mar 2020 07:56:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by whitealder.osuosl.org (Postfix) with ESMTPS id DE9EF87660 for ; Sat, 7 Mar 2020 07:56:44 +0000 (UTC) Received: by mail-wr1-f54.google.com with SMTP id n15so4922909wrw.13 for ; Fri, 06 Mar 2020 23:56:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ckvyv3yqh0N5rfJPUu9QrDEEoEtFc8TDq8wz9JW2Q68=; b=XeGdBTHKkxPcrnclHZ2X9SbyddAG2maKfE+RCdZ5GfqUMSHvWmiiwymDNIAzTQ2rtV MLLSux3jXEnZ5FPkLuVf+04qHZjzQxLlPvKY5tp2oYuT5WDFxt+i4IR7dMWVDprazEZd PG9JNySpNO6C7gqsHSHjhDK5EnPGXbWQxDOhNgIGCgp96y4X68QKBCGTmofwXsn4P3sB H2pU9YwaUu2qV3kw+wIUFTgrUYI1BshiUiJlFNrwk0JpsPoj8ClDcwPLFQCFhgrb/kzz fxPfsJmYcKOhZXZJy/lDJT38FIug8VRj5XMvfKaM7Rj6JNJezZr/5HZ+J0Ef/vxoakcp W21w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ckvyv3yqh0N5rfJPUu9QrDEEoEtFc8TDq8wz9JW2Q68=; b=D6awx0MPeK9i2zU2s2+M/BPAp/MDs/NfYWieCfd1G+C07MkeL56MWh48Q+yF/QOAeA Du5QOg1YoIApT6ooUb0jfKbdqrAK1Mp89Hu7aXau60zX/NsdbqCSHCCP7Y2q2nMacPMO YqEp90DfVMLkQEYo0vgSDwBUz1+alQmSzWe0nATnvxxJl2HdkurJj6GZ922b/BlqIMfq vLmxQSjU/NI0roeBsOL+BYiv4oj/v4feiQgR/ckcX1U3UxeO91rGPbS6Yfc0gLvsFOy+ 4hSdsXJLhmcCFqWW92vhS2DkIO/VKkUwelB+NOJM5gS6mMd+8vNyzxGw9CiSJBLQTZZe avLg== X-Gm-Message-State: ANhLgQ0Yl+TeBWbv0NLMuftv1p1TqlsuWroNEr2nfnROfa19FNlFHLZA Guubu7tnp/CMylwwEfza+qrN/MCL X-Google-Smtp-Source: ADFU+vstn+Hz4WvmsWzKvalBXwr7+RekvI2mg8+5GdJXBZhTbfEqX09WmIu9urgfXPVw9utb28MbCw== X-Received: by 2002:adf:a512:: with SMTP id i18mr8521818wrb.61.1583567802608; Fri, 06 Mar 2020 23:56:42 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:42 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:26 +0100 Message-Id: <20200307075633.7514-6-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 05/12] support/scripts/pkg-stats: add package status X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Unify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message. The following checks are performed: - url: status of the URL check - license: status of the license presence check - license-files: status of the license file check - hash: status of the hash file presence check - patches: status of the patches count check - pkg-check: status of the check-package script result - developers: status if a package has developers in the DEVELOPERS file - version: status of the version check With that status information the following variables are replaced: has_license, has_license_files, has_hash, url_status Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 32 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 9529ad5c6a..0d4bb13196 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -73,10 +73,10 @@ class Package: self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -85,17 +85,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("error", "missing") fp.close() @property @@ -121,30 +121,42 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field @@ -155,10 +167,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": @@ -169,6 +182,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line.decode()) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def is_cve_ignored(self, cve): @@ -179,7 +196,7 @@ class Package: def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = [ dev.name @@ -187,6 +204,14 @@ class Package: if dev.hasfile(self.path) ] + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + else: + self.status['developers'] = ("warning", "no developers") + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path @@ -195,7 +220,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) class CVE: @@ -409,23 +434,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) del pkg.url_worker pool.terminate() @@ -499,6 +524,18 @@ def check_package_latest_version(packages): results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', "Release Monitoring API error") + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', "Package not found on Release Monitoring") + + if pkg.latest_version['version'] is None: + pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring") + elif pkg.latest_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version'])) + else: + pkg.status['version'] = ('ok', 'up-to-date') worker_pool.terminate() del http_pool @@ -525,15 +562,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1 @@ -676,30 +713,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20: @@ -750,12 +787,12 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] in ("error", "warning"): td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): + if pkg.status['url'][0] == "error": td_class.append("invalid_url") - url_str = "%s" % (pkg.url, pkg.url_status) + url_str = "%s" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "Link" % pkg.url From patchwork Sat Mar 7 07:56:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250844 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=jbdXnVHi; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzF5gPkz9sPK for ; Sat, 7 Mar 2020 18:56:57 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id D773E1FE2F; Sat, 7 Mar 2020 07:56:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WSNw2Ri24cKU; Sat, 7 Mar 2020 07:56:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 0A57A20356; Sat, 7 Mar 2020 07:56:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id BDCF71BF42C for ; Sat, 7 Mar 2020 07:56:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B8F428769A for ; Sat, 7 Mar 2020 07:56:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qaf8n8cJmbGk for ; Sat, 7 Mar 2020 07:56:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id 932DF87666 for ; Sat, 7 Mar 2020 07:56:45 +0000 (UTC) Received: by mail-wm1-f47.google.com with SMTP id u9so4688918wml.3 for ; Fri, 06 Mar 2020 23:56:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wDUwh8agNR86n4MkO1BufE+rk3XvJLtKtPXyHzPGAEM=; b=jbdXnVHilTU7UPTP24G+Isl07ExfD/JZ8NoWVd62x37OOAqX+farT9ZIVfoN3QY0BZ 2ikoyDR8CZ/DuFMyrkZU6nReXRF4QMh9ekevo1SSrWiATnCKX8Czie9VpO35WsCDKjD1 zx50+gqNG1dUsQ+VCWFZ3EtVObhu5KlA7QgCg2rdiXgTnP0orXPle1aVytmFdOtpruhU 6jg1cmI3pTs3jXuJVEFSJG00agRvuM0JxSybNn099sGC2zLHqOFvRX6RnJikekZYXpii I5e0fW5OE0xjWmrLkydJTSf7KKsNVLVYS24O/2Dc2IHutsrNv1g5Qi+fxcjbLUHHJK5H yyrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wDUwh8agNR86n4MkO1BufE+rk3XvJLtKtPXyHzPGAEM=; b=gUpxDe6bOZBoBtBk4eINS+GUQTYkds1N2YTAqYNbZkC6v7rgfWbF6dLivR+xcruA++ Q9D7QAd/9B7OXOdZMmE5hLj+mSQLsnKdbllSOI/2hspG2nwiBwRv28ydachQoOwL06/d pbljddBmNHvGD0uT1EtKQGLuLPFYi49LwYp8eAJ4slG1451dODwtaEFLFdYmCaNmUaEw wUUoXP2AY7jvsbgfjtTNqM5E7YglFjr/wSiI5rDny5V7kOoBRlRx5gSXchOnmgj223nd IBwLqK7C2h7e2nym/HssqmHFbj6wvziHjCHgsJsZvzdsXoVSaPCstjzYcHEMCdwO8qZa CJaw== X-Gm-Message-State: ANhLgQ0pxBrtRCZOl4dV++ZWK5Rcj+5Fs9LVZ0G4xQE/5Rb8qB2MC/TJ DVtU9JUSpjzrN5D0CAI5p5o1r26l X-Google-Smtp-Source: ADFU+vustPvHM2/w0c+ajWoO2B1zFJK6EbWsZ/kH2WEOGlKTBqZI1t/TViQm/y0w37syRsrBq5RXng== X-Received: by 2002:a1c:7dc8:: with SMTP id y191mr3103383wmc.167.1583567803707; Fri, 06 Mar 2020 23:56:43 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:43 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:27 +0100 Message-Id: <20200307075633.7514-7-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 06/12] support/scripts/pkg-stats: add package count to stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 0d4bb13196..0e7d60bdbb 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -552,6 +552,7 @@ def check_package_cves(nvd_path, packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the From patchwork Sat Mar 7 07:56:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250846 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=AKsT9auR; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzJ2KGgz9sPK for ; Sat, 7 Mar 2020 18:57:00 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6A4D5876A6; Sat, 7 Mar 2020 07:56:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XvcouRWY+SgH; Sat, 7 Mar 2020 07:56:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 6AB698766A; Sat, 7 Mar 2020 07:56:56 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 9F7B91BF42C for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 9C61988216 for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qurix97fF9ZE for ; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by hemlock.osuosl.org (Postfix) with ESMTPS id 8D20188222 for ; Sat, 7 Mar 2020 07:56:46 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id v2so4920809wrp.12 for ; Fri, 06 Mar 2020 23:56:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jfexy0ORLO+UEQLnTB4sKdqtlHl6BuOSiQTX6QLSRPw=; b=AKsT9auRu9wwzAZlKjffPKl4v2Njd5WyQ0TAHS2Y1g+gYVcoIPZjt+g0uxMYJ4MVau SHdWprvP3WIfplHJYVN5v+NwIbJG6IkzvT57+efFAIbcceDo6fbfqQNNxCy90XUMgv29 0BOX9qYs8Bhtcx+5dOjQOElpf8YeXqe/XqJXhy2VbFSLI/LIAPy8mBr+UAikr9mgEWqz /9RlOeiKvktKEWe5EcYSbFdS1VqJ0RKzkY0grORfi4rGYD63XZuhrLjUCcgZKTB5sw4O h2nE9WyZuVQHySR0I0dfDSsc0b5JDFiclRGpdAXK16ie5CjjvVguKxb8ovJm46hDT0UD m9eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jfexy0ORLO+UEQLnTB4sKdqtlHl6BuOSiQTX6QLSRPw=; b=dXv26/6P47C447dl9kJFbhnzZv2Pe8f+nURrDF5faYM/kX0Y5q/KRJJ8kJrxd4WwHI CgBo+XLwG6lFbkjYVVer1NWn/ZhQReyTQLbPUTSzujdZkEV22EUDhw9fs/Acv6R6uMD5 l10Pnvrf5sFVSNgpNPeoeC5BrR4SzPeOOiaZn7fI1EfsxL42QAGKZLOPngHkSw18P7J8 amZzFxYCQRcgrp8NuLgz2zh+cJf77+a1CpmPKH4tFoKZtBDqaXlAGvKmfOBbMpPW5try fJnf+Cuieo3O5o/RDFJzu2bvmR0rWFzhtMeyEfT1jrXPIlrs96/3y//m5cPkPJJr0OcC b8Gw== X-Gm-Message-State: ANhLgQ05KXAEyErO/0pE+bmC+HG4HBj/WU6ezW7YgGW6Ll+6KpF++wM7 Qk/zRip7Je5Hp4wi0tkmLv3/fgQj X-Google-Smtp-Source: ADFU+vtl+N7EWwgCpA/qU9Upytds/2E1KeGWqRurL2nnRqEFKcPaY26mDBC985HeMgjSSrl/67MU8A== X-Received: by 2002:adf:fcc3:: with SMTP id f3mr8383331wrs.348.1583567804714; Fri, 06 Mar 2020 23:56:44 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:44 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:28 +0100 Message-Id: <20200307075633.7514-8-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 07/12] support/scripts/pkg-stats: store pkg dir path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This value can be used for later processing. In the buildroot-stats application this is used to create links pointing to the git repo of buildroot. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 0e7d60bdbb..016c2322e6 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -64,6 +64,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.has_license = False From patchwork Sat Mar 7 07:56:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250847 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=IR55yyFJ; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzJ50W7z9sPR for ; Sat, 7 Mar 2020 18:57:00 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id EE149881CC; Sat, 7 Mar 2020 07:56:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cYGko4OtKpx3; Sat, 7 Mar 2020 07:56:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 42D9E8823F; Sat, 7 Mar 2020 07:56:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id D1C081BF867 for ; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CC3B1203C7 for ; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8dflvCaxng2J for ; Sat, 7 Mar 2020 07:56:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by silver.osuosl.org (Postfix) with ESMTPS id 9983D1FE2F for ; Sat, 7 Mar 2020 07:56:47 +0000 (UTC) Received: by mail-wm1-f43.google.com with SMTP id f7so297592wml.4 for ; Fri, 06 Mar 2020 23:56:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AoQu5sWT7f4A8DdyOJu8qNMDVBJuhJTU1b9lv9nRRX8=; b=IR55yyFJYX4ViYXg/ZN04zxQHCiyHWzw96Fh+nvCKsm3YEe2i7agRKfZlTx5FrW05J Cm6UkNLiSSbc6u+83AS1eHLJfaprH91g+JipWz7n0i4quzqE2CgBtpB0ezodDO5VMLwA 4YmZbFfKb/LIBab2TxZ1E+7vcPpu7s9B/UrpqU5iUGgx7o2wTX80eSDE8RoLJxOOApIj +N9G695GSYKDFAPKmsXUxw59yV4eLi59UFF183b2VzZQuFILCIfHL1amLoCJjlhenNk4 ft6O+vbmVwEGjaAT2LuUOwvPtEEGWHtJD+iSXCMka02DA+SgLMGQHlTch3cnb4PXl2aJ gDGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AoQu5sWT7f4A8DdyOJu8qNMDVBJuhJTU1b9lv9nRRX8=; b=mfkUOZ9VfT3rGv4X+Mj+7NRbQsqvCKZWwkFVcoAlgvu2jZnkscX8jU7+MpRrWR5j/Y 5Uc9yijbSjXwslwbatgNTh/s7JNUrmV+5ZJ7L4HwBGulNnh/nnb+rc3ncIXOnnNQN3ZN 5DYrmS20xjUvy6viliDXnkeA4OT386P+ngjBPNT54/OVjKSwl/epZU1kZfNqcCgtfj2Y NIScPZu4RCEY/fqXCNuHyRO9tredTE0JB9Th0rx5ul+J/0cYxbQjKDl+aH9gMtxVuTEM PGqaUZqpmU4bi6Y+hO/m5QC2moAWsdoGKCtlXgckQ3AUTt7jpqoi6lFPn0cNIczT6UDX 5Sqg== X-Gm-Message-State: ANhLgQ0YaqJQ4n19PJHSSiwoZ/FeXsW8+/Tbqp06iS42OMiOv/KcR2Vo TnpQzEYwo/rz8YAmv2/EeH4EeX5B X-Google-Smtp-Source: ADFU+vtqPsPepUJQhFm1NP3kquSoBkUCg/3c/LA0TMXVBTxBpz2g+LrtOqYuwA6KmaZRFpRag0VcDA== X-Received: by 2002:a1c:2d4f:: with SMTP id t76mr8418387wmt.60.1583567805743; Fri, 06 Mar 2020 23:56:45 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:45 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:29 +0100 Message-Id: <20200307075633.7514-9-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 08/12] support/scripts/pkg-stats: add defconfig support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Scan configs directory and create Defconfig objects. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 016c2322e6..a15d0ec24b 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + developer.name + for developer in developers + if developer.hasfile(self.path) + ] + + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + return [ + Defconfig(name[:-len('_defconfig')], os.path.join('configs', name)) + for name in os.listdir('configs') + if name.endswith('_defconfig') + ] + class Package: all_licenses = dict() @@ -889,7 +917,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name'] @@ -900,6 +928,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v @@ -910,6 +944,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)} @@ -951,6 +986,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -977,7 +1016,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__() From patchwork Sat Mar 7 07:56:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250849 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Wl1cHvGe; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzM1yTtz9sPK for ; Sat, 7 Mar 2020 18:57:03 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2951C86FB7; Sat, 7 Mar 2020 07:56:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ECWY28cILf1m; Sat, 7 Mar 2020 07:56:55 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5AEAF86F58; Sat, 7 Mar 2020 07:56:55 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 583731BF42C for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4F98F1FE2F for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WYTWuw6LNTUS for ; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by silver.osuosl.org (Postfix) with ESMTPS id 5EEDF20356 for ; Sat, 7 Mar 2020 07:56:48 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id p9so4829404wmc.2 for ; Fri, 06 Mar 2020 23:56:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=F/7kEDsmJS9hhS2CnUu+S0cmS1hL0U33l9oiAd27Ppc=; b=Wl1cHvGe4nptx6+QXvZvtwYmSJqMMLGREYy+zeccn7Pc3quSbcz3vfWrv9WWWbL8DQ +xjOLqQMaWks4iBS6JpJhr0PmttlORCQBu6EQSZCQj16fKeRx0TYbZ3Z8SKkIeXoEv2T o2hxywIeX1c9/xSS1D3djTGJy0++FZ4AeKR70l+ZdzaU5TA5JdTeNX27QOgtGOXxJNlk zp8Gs+qCtwTnGg8iYFve+tcGoT9mGeGbtxx8t8AEZoVJ7s81E5unMdavu85y88CZOezk mXAKE5aOi/e8fRN07RgU+kMuXbs7vXTwFDRluEt9BROMyN6xidiOgXxAnWD+HFVKSf1b jVDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=F/7kEDsmJS9hhS2CnUu+S0cmS1hL0U33l9oiAd27Ppc=; b=YmSA3Qg/6xYZBEU1McDmU7/DzNp1E8n4j3PwOOQS7W2Sb3jwx2CuHnDPiEYhwDrtz6 wNRvl3pcoyiVBQclLvvoF8NKDznIy727EIBywMsorzmb3PmFTpXC9Mz1kWlk6zEnA3PN 8U8FGZOz+8JjzvQjjtbMLEmRKQY+znOqZ+BuU0rxQcsud3QluHSB2zhRPgkxtorVu+8F kOJOe06aR3RQAjgyJwisV9FXWEW0ZksmaNS8Oz26aZmiSzlYZ0x24qrNausSh8JF64ZW ka0gUKDOsopA09+OGOgfjv8z4AieZSh0FrTes8oITaCHOiufM9BSc6WnrcnuQvbzqpSw O7/w== X-Gm-Message-State: ANhLgQ0+hkLsez1n2jwVC3LttQO3GiRmiVfrSPGBJ7lLw3U2J1qFUP1Q 8zUQO8TtO0Bhzb7qwpXIxONtJFHZ X-Google-Smtp-Source: ADFU+vtsm4BreomZjIturjMfKB5TC86AVouy5c9DY49sLN5AdjghGkfdkPBVwZLytIe9LxIC+7rcPQ== X-Received: by 2002:a05:600c:301:: with SMTP id q1mr8784587wmd.182.1583567806638; Fri, 06 Mar 2020 23:56:46 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:46 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:30 +0100 Message-Id: <20200307075633.7514-10-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 09/12] support/scripts/pkg-stats: add support for license hash check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Store the names of license files and check if they are in the hash file. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index a15d0ec24b..8849c7a1c3 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -85,7 +85,7 @@ def get_defconfig_list(): class Package: all_licenses = dict() - all_license_files = list() + all_license_files = dict() all_versions = dict() all_ignored_cves = dict() @@ -98,6 +98,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False + self.license_files = None self.patch_files = [] self.warnings = 0 self.current_version = None @@ -159,6 +160,7 @@ class Package: self.license = self.all_licenses[var] self.status['license'] = ("ok", "found") if var in self.all_license_files: + self.license_files = self.all_license_files[var].split(' ') self.status['license-files'] = ("ok", "found") def set_hash_info(self): @@ -166,8 +168,18 @@ class Package: Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") + self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): self.status['hash'] = ("ok", "found") + self.status['hash-license'] = ("error", "no license in hash file") + # check if license files are in hash file + if self.license_files is not None: + self.status['hash-license'] = ("ok", "found") + with open(hashpath) as f: + content = f.read() + for license in self.license_files: + if content.find(license) == -1: + self.status['hash-license'] = ("error", "license missing in hash file") else: self.status['hash'] = ("error", "missing") @@ -449,7 +461,7 @@ def package_init_make_info(): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): continue pkgvar = pkgvar[:-14] - Package.all_license_files.append(pkgvar) + Package.all_license_files[pkgvar] = value elif pkgvar.endswith("_VERSION"): if pkgvar.endswith("_DL_VERSION"): From patchwork Sat Mar 7 07:56:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250850 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=tQ3WrqQw; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzP3Fckz9sPK for ; Sat, 7 Mar 2020 18:57:05 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 881EE8772C; Sat, 7 Mar 2020 07:57:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YwIGQXD97rd8; Sat, 7 Mar 2020 07:56:58 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 591B2876BE; Sat, 7 Mar 2020 07:56:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 71E0D1BF42C for ; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 69D4220356 for ; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yQcLK-j2TbZj for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by silver.osuosl.org (Postfix) with ESMTPS id 71B131FE2F for ; Sat, 7 Mar 2020 07:56:49 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id p9so4829431wmc.2 for ; Fri, 06 Mar 2020 23:56:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sAlE2bjgUnL1TQWhgVacG1jE9im3iPhF2ek85Z5uU5k=; b=tQ3WrqQwPCkwU6ifJxrWJthsaJwkNJu2mSASNXX/IedZMS862BLBVSV9JfUA4zqYyK l/CWJkKqJZkcj70kkWNsMArnH7N8BbXVyWtwPjUwR3yp2OIO6RZxS6sxlaD9ikWXTs3o BN7lCLW9dbzYLdgwub+wYVVWo1GCF/v5zPBtcwa4vf7bpcrDuznoStiEp5vJZEgdqDMH jLVvdH3KvBd3d0S0tnpL6BnrQ1oxeoDlvwmK8QqL+ZK3JBVJIOINpgxkLTuG2zn/zQxP Alg89y7VLlO84F9AumIu9rnRRaWt8LxG4q/rLwOIvUbK2REjZR8p8Iho43qooPYutO5e loEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sAlE2bjgUnL1TQWhgVacG1jE9im3iPhF2ek85Z5uU5k=; b=nvrsQUUOi6k7oeSioDlSfLFsnNrrvg2A0q8HSDVURiKPp4zVOrbytjlGS4WDh3WHq2 vICq83ZJ79YIhtlsq5xe+z7CsKsBvCUH6S7VCpNSDbwqIM6GgNXk2ECViIxUmKEpriv5 wRECNikVWq979HyAuj2vlGYxFrNTZp5BmZYuNZH5lekLumo3lKp0Ac7KmAC7jWk0Cs1y BDrBAu1GpiVpQsUkkKSDeRONDlQpP+0vnIrs21KAGABXw6qWwJqoTqThgboZ9uOOlpkG lzR2ZZJVSXd34+1a8Pq1IxmF/3DB/TDx1HrZfOeIesQRc1GhLCn4J3um/1WFZ4DBm7D0 tX8A== X-Gm-Message-State: ANhLgQ0EQa4d4Lagdxr2KTUcR6gjPzyJudjcbK4V3kd0zhIgoT1FmPs6 /+mDHJ0AYUAlQr5KmxKKVoYfNVaE X-Google-Smtp-Source: ADFU+vuo9iUdRa3BPzsIi7Pab3Xl5akv/56KlD9T8GHrBFqIWbNY2CizutCCB5e8aEvz2a1t9etSFQ== X-Received: by 2002:a05:600c:d4:: with SMTP id u20mr71295wmm.169.1583567807657; Fri, 06 Mar 2020 23:56:47 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:47 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:31 +0100 Message-Id: <20200307075633.7514-11-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" If there is no infra set or infra is virtual the status is set to 'na'. This is done for the follwing checks: - license - license-files - hash - hash-license - patches - version Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8849c7a1c3..8e75abe1d3 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -132,6 +132,15 @@ class Package: def patch_count(self): return len(self.patch_files) + @property + def has_valid_infra(self): + try: + if self.infras[0][1] == 'virtual': + return False + except IndexError: + return False + return True + def set_infra(self): """ Fills in the .infras field @@ -153,6 +162,11 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ + if not self.has_valid_infra: + self.status['license'] = ("na", "no valid package infra") + self.status['license-files'] = ("na", "no valid package infra") + return + var = self.pkgvar() self.status['license'] = ("error", "missing") self.status['license-files'] = ("error", "missing") @@ -167,6 +181,11 @@ class Package: """ Fills in the .status['hash'] field """ + if not self.has_valid_infra: + self.status['hash'] = ("na", "no valid package infra") + self.status['hash-license'] = ("na", "no valid package infra") + return + hashpath = self.path.replace(".mk", ".hash") self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): @@ -187,6 +206,10 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ + if not self.has_valid_infra: + self.status['patches'] = ("na", "no valid package infra") + return + pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') @@ -566,6 +589,10 @@ def check_package_latest_version(packages): for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + if not pkg.has_valid_infra: + pkg.status['version'] = ("na", "no valid package infra") + continue + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: pkg.status['version'] = ('warning', "Release Monitoring API error") elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: From patchwork Sat Mar 7 07:56:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250851 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=hSRcndsK; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzQ39M5z9sPR for ; Sat, 7 Mar 2020 18:57:06 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C6EE486F0C; Sat, 7 Mar 2020 07:56:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yCrYU3yfJlGQ; Sat, 7 Mar 2020 07:56:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4A4C686F3A; Sat, 7 Mar 2020 07:56:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 2C62C1BF42C for ; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 297AA87666 for ; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fwsrXYcInlJ1 for ; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by whitealder.osuosl.org (Postfix) with ESMTPS id 680B187660 for ; Sat, 7 Mar 2020 07:56:50 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id e26so4672524wme.5 for ; Fri, 06 Mar 2020 23:56:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cH58+2NLvBefppZftFKR3xVRZAnduFfdxnB49znk4FI=; b=hSRcndsKdc+ZWK3Kb03O9whonbhCPcMM8XdrsAXZQIQCFLiCU+//KJqJ1dbINpBYNL lw86WzJnnriC7CLBixiW80DB/ODz4USmPVHDRwQ48hBMSr1SwNKL76EUeiHiV0LOY4UO 8g7GlnOLf2lU8y9gMLaGmOIOeC2DxkaMwuhy1vQd3mWIzgJK0CKy0CWvrg24RaSL156g AUkYUZDJmWckpf1HLkg9a1zkw13cgQojYMmnHTpNWN80EXhtkFOQ8/yNXqmIT3o82YF/ 9RNjRXxhfBUI5j19ys3zH1ecrS1gC96DbkwSQ4lJd0vC9zoWq1JrvbS2In3Q4tR1icSn 4sXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cH58+2NLvBefppZftFKR3xVRZAnduFfdxnB49znk4FI=; b=XuRQ7jWcLIqFze0+SNhyzGZwxCzcID5xbqwZ/tXgfXVSjBaV3vhwhJpsSUARKIIj4j 3uTPniSB6Nzdfoq52dZ1r3HpMFeayI/BVsnkuJiOXDWHK7PK4rm/1nVmiAD0nC3XvIZA RBSK/7ftCG2zz3JRValwiZLC0ukCcG4TEz8ExLF3YjP14MtujuM5pR9aWogs28ZvlYy/ h/rX8a3tEU1/xoj4pwP8E9jK7Vx8b5OzdoYxVKQkZEPUAk2aYozKg+0NLoUxSprQLGsM P8D/w1DUVdGvS3GU/AysBYvKRKK2+k9WWX0havK0xYpzQWFe+LYGHjY5N8STB5a36tVc m2qg== X-Gm-Message-State: ANhLgQ1O+IVk+bKbds0R6i2GNnlnkTUtWuUzjbkaw7btsal/kR/Tdny7 tV/ACpEJJrTSpGI5f6ma68VjoNy9 X-Google-Smtp-Source: ADFU+vtYc9gTWsmNbCnWuZoQ4q7O+Log+frRIqA3E11O8WUr0fB6wdZZi8AfzneTBRQHjS4+AnOMkA== X-Received: by 2002:a7b:c756:: with SMTP id w22mr8518856wmk.90.1583567808596; Fri, 06 Mar 2020 23:56:48 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:48 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:32 +0100 Message-Id: <20200307075633.7514-12-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 11/12] support/scripts/pkg-stats: add list of status checks to the json output X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8e75abe1d3..15aa48180e 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -88,6 +88,11 @@ class Package: all_license_files = dict() all_versions = dict() all_ignored_cves = dict() + # This is the list of all possible checks. Add new checks to this list so + # a tool that post-processeds the json output knows the checks before + # iterating over the packages. + status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license', + 'license-files', 'patches', 'pkg-check', 'url', 'version'] def __init__(self, name, path): self.name = name @@ -984,6 +989,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output): final = {'packages': pkgs, 'stats': statistics, 'defconfigs': defconfigs, + 'package_status_checks': Package.status_checks, 'commit': commit, 'date': str(date)} From patchwork Sat Mar 7 07:56:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1250848 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=E77us+J6; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48ZGzM2TZnz9sPR for ; Sat, 7 Mar 2020 18:57:03 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9240B2281E; Sat, 7 Mar 2020 07:57:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 92VpOCC79fzM; Sat, 7 Mar 2020 07:57:00 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 5D75120419; Sat, 7 Mar 2020 07:57:00 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 2286A1BF42C for ; Sat, 7 Mar 2020 07:56:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 1FD8187666 for ; Sat, 7 Mar 2020 07:56:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FakkcWJdeZP6 for ; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id 684F187660 for ; Sat, 7 Mar 2020 07:56:51 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id f7so297669wml.4 for ; Fri, 06 Mar 2020 23:56:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rI6WA2u5yo1PFRLqCbKI42nurvCD22FOHGd+PDzhnTE=; b=E77us+J6K+5qz/okl6ZEoCMwWLFY7UAZ2dBATLzZtZmd2zSux5623p9tu+VWUjW2kt sZ7ecdEDdxxI8RbPaujOFIgLTk70YOVVMndeGkmdnFGuKRfY8tZyvJAD9gfwpWGskLlr MLWvbeTWF8OTt9bciM/PWrFZGQCG5Hl7pGuj+wBHDMjFNY4olMGIvbK2Eu143k/g0qSE M0evM0yCBo/VzuBTOQ0Upfej5v0J83avCPdlsq7v7mMDyxTgS8xNhOjgS3YhdKcWPpHm ArlIi80JK3868/jJUgrgbLBKjzCexOSITX/JFe3PYBHccU38FjFb26MEX4CV0Za3ChQ0 cOZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rI6WA2u5yo1PFRLqCbKI42nurvCD22FOHGd+PDzhnTE=; b=CHBvd4Ltm9HlgIr3p/51X+G6cgL6h3mCnw5PVAs2sEPASDyB0Ak/hUAAROxXKfL3AW 8/YU5k6VFXMdSvLG7lq6B0rEqO5IqLicuHhdgNdxhv/WNPCacKy6Kgr1295iCVeQqC0u obPU8MCHYFbfG+xTcowHyBHO1zDdvGXysSrv4pm4QiwAJYUiYdCKIUSAr+QGOaQej+u+ def7qVJY6kEsV21iK9sIhX8epyTknhaK11bmuJtedcOBHYtwrdMR2hd7mKAb+t/D778Y TjJ3nTIH4Hd5cg6jMLppLidnyBo6kTgh2yNx+BBwhvlTyNF+4D6alcOxVfl0e1T1PQ4c nq7w== X-Gm-Message-State: ANhLgQ2kj64NW6hqy526pjgexPxlsFbzqWC6ZNoVBaQEW4LcOe0owzD8 PgxgbUH6PxdYpmlirVeqhvdDqlhN X-Google-Smtp-Source: ADFU+vu+tvI3ytMiBkQ3aCImUmJYgKAaPQj4ChBGCWJL7m7FES5tSoAR7k/3IVWrSKqxwjqjFgRYIQ== X-Received: by 2002:a7b:cc95:: with SMTP id p21mr8018783wma.184.1583567809669; Fri, 06 Mar 2020 23:56:49 -0800 (PST) Received: from t450s.fritz.box (ip5f58981a.dynamic.kabel-deutschland.de. [95.88.152.26]) by smtp.gmail.com with ESMTPSA id g129sm19189071wmg.12.2020.03.06.23.56.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 23:56:49 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Sat, 7 Mar 2020 08:56:33 +0100 Message-Id: <20200307075633.7514-13-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200307075633.7514-1-heiko.thiery@gmail.com> References: <20200307075633.7514-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 12/12] support/scripts/pkg-stats: add status for cve check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 15aa48180e..985a0fd079 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -622,6 +622,12 @@ def check_package_cves(nvd_path, packages): if pkg_name in packages and cve.affects(packages[pkg_name]): packages[pkg_name].cves.append(cve.identifier) + for pkg_name, pkg in packages.items(): + if len(pkg.cves) > 0: + pkg.status['cve'] = ('error', 'affected by cve') + else: + pkg.status['cve'] = ('ok', 'no cve found') + def calculate_stats(packages): stats = defaultdict(int)