From patchwork Tue May 18 18:21:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Weber X-Patchwork-Id: 1480383 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Fl4944Gmnz9sWp for ; Wed, 19 May 2021 04:22:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 0AE24406A5; Tue, 18 May 2021 18:22:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XzgR2bf__53W; Tue, 18 May 2021 18:22:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 01BBC4069E; Tue, 18 May 2021 18:22:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 05CE71BF9AF for ; Tue, 18 May 2021 18:22:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E997C60B73 for ; Tue, 18 May 2021 18:22:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E99uqNzrVyzQ for ; Tue, 18 May 2021 18:22:01 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from da1vs04.rockwellcollins.com (da1vs04.rockwellcollins.com [205.175.227.52]) by smtp3.osuosl.org (Postfix) with ESMTPS id E15E260B69 for ; Tue, 18 May 2021 18:22:00 +0000 (UTC) IronPort-SDR: zlsrt5c/PHG+Z3VDmUZ+aZWYh4JK5Ftzmaiiu6QeozL+NF8v0HFcjJ13/lWZOPcdbK4one0EvN BwSosS+ErrPW84CUf7Es45u8srRB8s0S7O+5OVHUEMaOvuH7M6kvurRhqRp8VkjSo/XnufMLUZ +QPlov8sg92eRHO8siypmd7c9V/CTShUrH+prrsO1YgAaVCqiVYnG1q7Uw9CWJlVHsQvzoUGZo Xg57yB3grWhrrTOFF4A/ObJyCsZIXnXdpb8h7KEwC1VfnbX3YH6yUvT5wOo5+TRASUVKC0budR OFY= Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs04.rockwellcollins.com with ESMTP; 18 May 2021 13:22:00 -0500 X-Received: from biscuits.rockwellcollins.com (biscuits.rockwellcollins.lab [10.148.119.137]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 9842B60249; Tue, 18 May 2021 13:21:59 -0500 (CDT) To: buildroot@buildroot.org Date: Tue, 18 May 2021 13:21:53 -0500 Message-Id: <20210518182155.47055-1-matthew.weber@collins.com> X-Mailer: git-send-email 2.17.1 Subject: [Buildroot] [PATCH 1/3] support/scripts/pkg-stats: fix flake8 E741 ambiguous variable name X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Matthew Weber via buildroot Reply-To: Matthew Weber Cc: Matthew Weber MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Matthew Weber --- support/scripts/pkg-stats | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index c7e30dfd2b..0cd3674c52 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -375,9 +375,9 @@ def package_init_make_info(): variable_list = [x[5:] for x in variable_list if x.startswith("HOST_")] + \ [x for x in variable_list if not x.startswith("HOST_")] - for l in variable_list: + for item in variable_list: # Get variable name and value - pkgvar, value = l.split("=") + pkgvar, value = item.split("=") # Strip the suffix according to the variable if pkgvar.endswith("_LICENSE"): From patchwork Tue May 18 18:21:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Weber X-Patchwork-Id: 1480384 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Fl49J3NFFz9sXb for ; Wed, 19 May 2021 04:22:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4E82B406AD; Tue, 18 May 2021 18:22:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vzpIP4PtdT6d; Tue, 18 May 2021 18:22:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 5B0CA40E52; Tue, 18 May 2021 18:22:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7D53C1BF393 for ; Tue, 18 May 2021 18:22:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 6C9F260B76 for ; Tue, 18 May 2021 18:22:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lD0cdx9Gx-zm for ; Tue, 18 May 2021 18:22:04 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from secvs04.rockwellcollins.com (secvs04.rockwellcollins.com [205.175.225.130]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2E0B660B69 for ; Tue, 18 May 2021 18:22:04 +0000 (UTC) IronPort-SDR: 3KIQN5JrnYzu5H15VtDgncQAUfFlARmjNUT7QTgT2PrS5eTWYxzmHIm1aK4cLX1ZTCk6UpwBJr 2+GyARdvk8hY1A1q9rqWoHN/EyGoOKeSupEgm3d8T78Zvrgt15r+n9hkCsonHLUZVvbjmVwWlV 4HKtDu5ruqVxpceVqg+x/IOtyQqGgUwTp12JnwMjp4vuK3TejcTBongCdqOW79Mcv6WkGC0bOV uS3iCruasZH6m5EHMhNXE/tLVTzo13Zk+PSY4yrqjN1jVHCUAWtEj1/VwO3Qa1sIsTKM4nlWD8 tU0= Received: from ofwgwc03.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.225.12]) by secvs04.rockwellcollins.com with ESMTP; 18 May 2021 13:22:03 -0500 X-Received: from biscuits.rockwellcollins.com (biscuits.rockwellcollins.lab [10.148.119.137]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 17F2E60215; Tue, 18 May 2021 13:22:03 -0500 (CDT) To: buildroot@buildroot.org Date: Tue, 18 May 2021 13:21:54 -0500 Message-Id: <20210518182155.47055-2-matthew.weber@collins.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210518182155.47055-1-matthew.weber@collins.com> References: <20210518182155.47055-1-matthew.weber@collins.com> Subject: [Buildroot] [PATCH 2/3] support/scripts/pkg-stats: add is_actual_package() and rework has_valid_infra() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Matthew Weber via buildroot Reply-To: Matthew Weber Cc: Matthew Weber , "Yann E . MORIN" MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" has_valid_infra() is incorrectly named; it probably should be named is_actual_package(), and has_valid_infra() would be changed to actually represent having an actual infra. This resolves packages reporting as having no valid package infra and cleans up reporting cases of CPE and CVEs where there isn't a valid version or package definition outside Buildroot Cc: Yann E. MORIN Signed-off-by: Matthew Weber --- Yann, hopefully I got this right :-) --- support/scripts/pkg-stats | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 0cd3674c52..ca55a301de 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -131,7 +131,15 @@ class Package: @property def has_valid_infra(self): + if self.infras is None: + return False + return len(self.infras) > 0 + + @property + def is_actual_package(self): try: + if not self.has_valid_infra: + return False if self.infras[0][1] == 'virtual': return False except IndexError: @@ -159,7 +167,7 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ - if not self.has_valid_infra: + if not self.is_actual_package: self.status['license'] = ("na", "no valid package infra") self.status['license-files'] = ("na", "no valid package infra") return @@ -177,7 +185,7 @@ class Package: """ Fills in the .status['hash'] field """ - if not self.has_valid_infra: + if not self.is_actual_package: self.status['hash'] = ("na", "no valid package infra") self.status['hash-license'] = ("na", "no valid package infra") return @@ -192,7 +200,7 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ - if not self.has_valid_infra: + if not self.is_actual_package: self.status['patches'] = ("na", "no valid package infra") return @@ -220,7 +228,7 @@ class Package: Fills in the .cpeid field """ var = self.pkgvar() - if not self.has_valid_infra: + if not self.is_actual_package: self.status['cpe'] = ("na", "no valid package infra") return @@ -551,13 +559,13 @@ async def check_package_latest_version(packages): package, as known by release-monitoring.org """ - for pkg in [p for p in packages if not p.has_valid_infra]: + for pkg in [p for p in packages if not p.is_actual_package]: pkg.status['version'] = ("na", "no valid package infra") tasks = [] connector = aiohttp.TCPConnector(limit_per_host=5) async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess: - packages = [p for p in packages if p.has_valid_infra] + packages = [p for p in packages if p.is_actual_package] for pkg in packages: tasks.append(asyncio.ensure_future(check_package_latest_version_get(sess, pkg, len(packages)))) await asyncio.wait(tasks) @@ -578,7 +586,7 @@ def check_package_cves(nvd_path, packages): cpe_product_pkgs = defaultdict(list) for pkg in packages: - if not pkg.has_valid_infra: + if not pkg.is_actual_package: pkg.status['cve'] = ("na", "no valid package infra") continue if not pkg.current_version: From patchwork Tue May 18 18:21:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Weber X-Patchwork-Id: 1480385 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Fl49Z2BdZz9sVt for ; Wed, 19 May 2021 04:22:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1BF6140E6D; Tue, 18 May 2021 18:22:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d4xisqkVsKGF; Tue, 18 May 2021 18:22:32 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id D4FB640E68; Tue, 18 May 2021 18:22:30 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 7AD611BF393 for ; Tue, 18 May 2021 18:22:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6B064403F0 for ; Tue, 18 May 2021 18:22:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id koQ3FhV4-Loo for ; Tue, 18 May 2021 18:22:05 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from ch3vs05.rockwellcollins.com (ch3vs05.rockwellcollins.com [205.175.226.130]) by smtp2.osuosl.org (Postfix) with ESMTPS id 3087C403E5 for ; Tue, 18 May 2021 18:22:04 +0000 (UTC) IronPort-SDR: nyPeKtHrNXZXTHSW2P0/09fS/bYLdCZ8DQLJQ5ocI/bP5Bt6Zmiu75m9gzmCpeSzStnayKRzK2 qdcAJQ5Or79rvuezJ4bx4Pl8jHejmDLU78jWjFtyW4WKqUxxAoH8CM5r2UwImpxbBmIiemNj9k pHIU4FrS0E04gEEAH63qfZvlf2ZFX5PIpw0c1/6dH4gfuhQtpTEahSTjaoElmPYqp1YPRUNKbk fZd2JLFahn8PF0cVnJXHdROaqEXHvYnxXU1vzaXWjngFhwU2/nXwLonch+qo0tYxAnsMZRynaI cLw= Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs05.rockwellcollins.com with ESMTP; 18 May 2021 13:22:04 -0500 X-Received: from biscuits.rockwellcollins.com (biscuits.rockwellcollins.lab [10.148.119.137]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 06A4D6029B; Tue, 18 May 2021 13:22:04 -0500 (CDT) To: buildroot@buildroot.org Date: Tue, 18 May 2021 13:21:55 -0500 Message-Id: <20210518182155.47055-3-matthew.weber@collins.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210518182155.47055-1-matthew.weber@collins.com> References: <20210518182155.47055-1-matthew.weber@collins.com> Subject: [Buildroot] [PATCH 3/3] support/scripts/pkg-stats: clarify when a CVE/CPE should report as N/A X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Matthew Weber via buildroot Reply-To: Matthew Weber Cc: Matthew Weber MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" - If a package doesn't have any versioning, ignore and state that - If a package is virtual, CVE=ignore and CPE state virtual - For any of these NA cases, don't provide search link Signed-off-by: Matthew Weber --- support/scripts/pkg-stats | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index ca55a301de..3aaf1169cb 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -229,7 +229,10 @@ class Package: """ var = self.pkgvar() if not self.is_actual_package: - self.status['cpe'] = ("na", "no valid package infra") + self.status['cpe'] = ("na", "N/A - virtual pkg") + return + if not self.current_version: + self.status['cpe'] = ("na", "no version information available") return if var in self.all_cpeids: @@ -587,7 +590,7 @@ def check_package_cves(nvd_path, packages): cpe_product_pkgs = defaultdict(list) for pkg in packages: if not pkg.is_actual_package: - pkg.status['cve'] = ("na", "no valid package infra") + pkg.status['cve'] = ("na", "N/A") continue if not pkg.current_version: pkg.status['cve'] = ("na", "no version information available") @@ -942,12 +945,15 @@ def dump_html_pkg(f, pkg): if pkg.cpeid: f.write(" %s\n" % pkg.cpeid) if not pkg.is_status_ok("cpe"): - if pkg.cpeid: - f.write("
%s (Search)\n" % # noqa: E501 - (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5]))) + if pkg.is_actual_package and pkg.current_version: + if pkg.cpeid: + f.write("
%s (Search)\n" % # noqa: E501 + (pkg.status['cpe'][1], ":".join(pkg.cpeid.split(":")[0:5]))) + else: + f.write(" %s (Search)\n" % # noqa: E501 + (pkg.status['cpe'][1], pkg.name)) else: - f.write(" %s (Search)\n" % - (pkg.status['cpe'][1], pkg.name)) + f.write(" %s\n" % pkg.status['cpe'][1]) f.write(" \n")