From patchwork Fri Jul 10 11:22:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326688 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39ds4y7Nz9sRK for ; Fri, 10 Jul 2020 21:23:29 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id DE3E889694; Fri, 10 Jul 2020 11:23:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2-29fIgbs6pq; Fri, 10 Jul 2020 11:23:25 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id C459D8965D; Fri, 10 Jul 2020 11:23:25 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id D6FE51BF319 for ; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1AC02879E0 for ; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XpqSvypVKUII for ; Fri, 10 Jul 2020 11:23:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 8D7AE878CC for ; Fri, 10 Jul 2020 11:23:04 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id E6FCFE0009; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:37 +0200 Message-Id: <20200710112245.1044073-12-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/9] support/script/pkg-stat: Handle exception when version comparison fails X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" With python 3, when a package has a version number x-y-z instead of x.y.z, then the version returned by LooseVersion can't be compared which raises an exception. This patch handles this exception by adding a new return value when the comparison can't be done. As a third value has been introduce, the booelan are no more used. Signed-off-by: Gregory CLEMENT --- support/scripts/pkg-stats | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index c1f41fc9e8..a75cb68581 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -50,6 +50,10 @@ RM_API_STATUS_FOUND_BY_DISTRO = 2 RM_API_STATUS_FOUND_BY_PATTERN = 3 RM_API_STATUS_NOT_FOUND = 4 +CVE_AFFECTS = 1 +CVE_DOESNT_AFFECT = 2 +CVE_UNKNOWN = 3 + # Used to make multiple requests to the same host. It is global # because it's used by sub-processes. http_pool = None @@ -364,7 +368,7 @@ class CVE: by this CVE. """ if br_pkg.is_cve_ignored(self.identifier): - return False + return CVE_DOESNT_AFFECT for product in self.each_product(): if product['product_name'] != br_pkg.name: @@ -373,7 +377,7 @@ class CVE: for v in product['version']['version_data']: if v["version_affected"] == "=": if br_pkg.current_version == v["version_value"]: - return True + return CVE_AFFECTS elif v["version_affected"] == "<=": pkg_version = distutils.version.LooseVersion(br_pkg.current_version) if not hasattr(pkg_version, "version"): @@ -383,10 +387,18 @@ class CVE: if not hasattr(cve_affected_version, "version"): print("Cannot parse CVE affected version '%s'" % v["version_value"]) continue - return pkg_version <= cve_affected_version + try: + affected = pkg_version <= cve_affected_version + break + except: + return CVE_UNKNOWN + if affected: + return CVE_AFFECTS + else: + return CVE_DOESNT_AFFECT else: print("version_affected: %s" % v['version_affected']) - return False + return CVE_DOESNT_AFFECT def get_pkglist(npackages, package_list): @@ -610,7 +622,7 @@ def check_package_cves(nvd_path, packages): for cve in CVE.read_nvd_dir(nvd_path): for pkg_name in cve.pkg_names: - if pkg_name in packages and cve.affects(packages[pkg_name]): + if pkg_name in packages and cve.affects(packages[pkg_name]) == CVE_AFFECTS: packages[pkg_name].cves.append(cve.identifier) From patchwork Fri Jul 10 11:22:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326697 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39fH70H2z9sRK for ; Fri, 10 Jul 2020 21:23:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5F0882038D; Fri, 10 Jul 2020 11:23:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0urniUuvbAwD; Fri, 10 Jul 2020 11:23:38 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 1AE952DEBA; Fri, 10 Jul 2020 11:23:34 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 15F9D1BF319 for ; Fri, 10 Jul 2020 11:23:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 6D77720406 for ; Fri, 10 Jul 2020 11:22:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zsTNmnXAso+H for ; Fri, 10 Jul 2020 11:22:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by silver.osuosl.org (Postfix) with ESMTPS id 10101203DF for ; Fri, 10 Jul 2020 11:22:55 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 438031C000E; Fri, 10 Jul 2020 11:22:52 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:28 +0200 Message-Id: <20200710112245.1044073-3-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 2/9] support/scripts/cve.py: Switch to JSON 1.1 X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In 2019, the JSON vulnerability feeds switched from version 1.0 to 1.1. The main difference is the removal of the affects element that was used to check if a package was affected by a CVE. This information is duplicated in the configuration element which contains in the end the cpeid as well as properties about the versions affected. Instead of having a list of the versions affected, with these properties, it is possible to have a range of versions. Signed-off-by: Gregory CLEMENT --- support/scripts/cve.py | 119 +++++++++++++++++++++++++++++++++-------- 1 file changed, 96 insertions(+), 23 deletions(-) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index 874ab4482d..e911fe0c65 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -31,9 +31,19 @@ import sys sys.path.append('utils/') NVD_START_YEAR = 2002 -NVD_JSON_VERSION = "1.0" +NVD_JSON_VERSION = "1.1" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION +import operator + +ops = { + '>=' : operator.ge, + '>' : operator.gt, + '<=' : operator.le, + '<' : operator.lt, + '=' : operator.eq +} + class CVE: """An accessor class for CVE Items in NVD files""" def __init__(self, nvd_cve): @@ -92,23 +102,83 @@ class CVE: print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename) raise for cve in content: - yield cls(cve['cve']) + yield cls(cve) def each_product(self): """Iterate over each product section of this cve""" - for vendor in self.nvd_cve['affects']['vendor']['vendor_data']: + for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']: for product in vendor['product']['product_data']: yield product + def parse_node(self, node): + """ + Parse the node inside the configurations section to extract the + cpe information usefull to know if a product is affected by + the CVE. Actually only the product name and the version + descriptor are needed, but we also provide the vendor name. + """ + + # The node containing the cpe entries matching the CVE can also + # contain sub-nodes, so we need to manage it. + for child in node.get('children', ()): + self.parse_node(child) + + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if version != '*' and version != '-': + # Version is defined, this is a '=' match + op_start = '=' + v_start = version + elif version == '-': + # no version information is available + op_start = '=' + v_start = version + else: + # Parse start version, end version and operators + if 'versionStartIncluding' in cpe: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + + if 'versionStartExcluding' in cpe: + op_start = '>' + v_start = cpe['versionStartExcluding'] + + if 'versionEndIncluding' in cpe: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + + if 'versionEndExcluding' in cpe: + op_end = '<' + v_end = cpe['versionEndExcluding'] + + key =['vendor', 'product', 'v_start', 'op_start', 'v_end', 'op_end'] + val = [vendor, product, v_start, op_start, v_end, op_end] + yield dict(zip(key, val)) + + def each_cpe(self): + for node in self.nvd_cve['configurations']['nodes']: + for cpe in self.parse_node(node): + yield cpe + @property def identifier(self): """The CVE unique identifier""" - return self.nvd_cve['CVE_data_meta']['ID'] + return self.nvd_cve['cve']['CVE_data_meta']['ID'] @property def pkg_names(self): """The set of package names referred by this CVE definition""" - return set(p['product_name'] for p in self.each_product()) + return set(p['product'] for p in self.each_cpe()) def affects(self, br_pkg): """ @@ -118,24 +188,27 @@ class CVE: if br_pkg.is_cve_ignored(self.identifier): return False - for product in self.each_product(): - if product['product_name'] != br_pkg.name: + for cpe in self.each_cpe(): + affected = True + if cpe['product'] != br_pkg.name: + continue + if cpe['v_start'] == '-': + return True + if not (cpe['v_start'] or cpe['v_end']): + print("No CVE affected version") continue + pkg_version = distutils.version.LooseVersion(br_pkg.current_version) + if not hasattr(pkg_version, "version"): + print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version)) + continue + + if cpe['v_start']: + cve_affected_version = distutils.version.LooseVersion(cpe['v_start']) + affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version) - for v in product['version']['version_data']: - if v["version_affected"] == "=": - if br_pkg.current_version == v["version_value"]: - return True - elif v["version_affected"] == "<=": - pkg_version = distutils.version.LooseVersion(br_pkg.current_version) - if not hasattr(pkg_version, "version"): - print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version)) - continue - cve_affected_version = distutils.version.LooseVersion(v["version_value"]) - if not hasattr(cve_affected_version, "version"): - print("Cannot parse CVE affected version '%s'" % v["version_value"]) - continue - return pkg_version <= cve_affected_version - else: - print("version_affected: %s" % v['version_affected']) + if (affected and cpe['v_end']): + cve_affected_version = distutils.version.LooseVersion(cpe['v_end']) + affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version) + if (affected): + return True return False From patchwork Fri Jul 10 11:22:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326681 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dT2T3Dz9sRK for ; Fri, 10 Jul 2020 21:23:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5EFF388C15; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GdLNy10Eg2GW; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 588E088C9C; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id ABA6E1BF319 for ; Fri, 10 Jul 2020 11:22:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A8A2E8793A for ; Fri, 10 Jul 2020 11:22:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E6VCd0wGD7CF for ; Fri, 10 Jul 2020 11:22:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E1AE2878CC for ; Fri, 10 Jul 2020 11:22:56 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 053F3E000B; Fri, 10 Jul 2020 11:22:52 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:29 +0200 Message-Id: <20200710112245.1044073-4-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 3/9] package/pkg-utils: show-info: report the list of the CVEs ignored X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Add the list of the CVEs to ignore for each package because they already have a fix for it. This information will be useful for a cve-checker. Signed-off-by: Gregory CLEMENT --- package/pkg-utils.mk | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index d88a14ab0f..49ce6dc6f1 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -117,7 +117,10 @@ define _json-info-pkg $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES))) ], "reverse_dependencies": [ - $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))) + $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))), + ], + "ignored_cves": [ + $(call make-comma-list,$(sort $($(1)_IGNORE_CVES))) ] endef From patchwork Fri Jul 10 11:22:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326683 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dZ6zxhz9sSn for ; Fri, 10 Jul 2020 21:23:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 28BD887C27; Fri, 10 Jul 2020 11:23:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lti6rQcICoKW; Fri, 10 Jul 2020 11:23:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E1FA6879E0; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 021EC1BF319 for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id F1FB58963E for ; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwRC4QepEzuM for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by whitealder.osuosl.org (Postfix) with ESMTPS id B94658962B for ; Fri, 10 Jul 2020 11:23:00 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id CBA16FF806; Fri, 10 Jul 2020 11:22:58 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:30 +0200 Message-Id: <20200710112245.1044073-5-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 4/9] package/pkg-utils: Make CVE class independent of the Pacakage class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The affects method of the CVE use the Package class defined in pkg-stats. The purpose of migrating the CVE class outside of pkg-stats was to be able to reuse it from other scripts. So let's remove the Package dependency and only use the needed information. Signed-off-by: Gregory CLEMENT --- support/scripts/cve.py | 10 +++++----- support/scripts/pkg-stats | 14 ++++++++------ 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index e911fe0c65..b754a17991 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -180,26 +180,26 @@ class CVE: """The set of package names referred by this CVE definition""" return set(p['product'] for p in self.each_cpe()) - def affects(self, br_pkg): + def affects(self, name, version, cve_ignore_list): """ True if the Buildroot Package object passed as argument is affected by this CVE. """ - if br_pkg.is_cve_ignored(self.identifier): + if (self.identifier in cve_ignore_list): return False for cpe in self.each_cpe(): affected = True - if cpe['product'] != br_pkg.name: + if cpe['product'] != name: continue if cpe['v_start'] == '-': return True if not (cpe['v_start'] or cpe['v_end']): print("No CVE affected version") continue - pkg_version = distutils.version.LooseVersion(br_pkg.current_version) + pkg_version = distutils.version.LooseVersion(version) if not hasattr(pkg_version, "version"): - print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version)) + print("Cannot parse package '%s' version '%s'" % (name, version)) continue if cpe['v_start']: diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 1c941104fe..883a5bd2be 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -245,11 +245,12 @@ class Package: self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return - def is_cve_ignored(self, cve): + def cve_ignored_list(self): """ - Tells if the CVE is ignored by the package + Give the list of CVEs ignored by the package """ - return cve in self.all_ignored_cves.get(self.pkgvar(), []) + print(self.all_ignored_cves.get(self.pkgvar(), [])) + return list(self.all_ignored_cves.get(self.pkgvar(), [])) def set_developers(self, developers): """ @@ -501,9 +502,10 @@ def check_package_cves(nvd_path, packages): for cve in cvecheck.CVE.read_nvd_dir(nvd_path): for pkg_name in cve.pkg_names: - if pkg_name in packages and cve.affects(packages[pkg_name]): - packages[pkg_name].cves.append(cve.identifier) - + if pkg_name in packages: + pkg = packages[pkg_name] + if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()): + pkg.cves.append(cve.identifier) def calculate_stats(packages): stats = defaultdict(int) From patchwork Fri Jul 10 11:22:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326682 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dZ12MTz9sRK for ; Fri, 10 Jul 2020 21:23:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B393689643; Fri, 10 Jul 2020 11:23:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yXNJ-OyhEZEC; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 1497F8963E; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 750641BF319 for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6F1D98962B for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PtCsOVz0Yrk6 for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by whitealder.osuosl.org (Postfix) with ESMTPS id 562BA89637 for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 69EDD1BF203; Fri, 10 Jul 2020 11:22:59 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:31 +0200 Message-Id: <20200710112245.1044073-6-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This scripts takes as entry on stdin a JSON description of the package used for a given configuration. This description is the one generated by "make show-info". The script generates the list of all the package used and if they are affected by a CVE. The output is either a JSON or an HTML file similar to the one generated by pkg-stats. Signed-off-by: Gregory CLEMENT --- support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100755 support/scripts/cve-checker diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker new file mode 100755 index 0000000000..db8497d7aa --- /dev/null +++ b/support/scripts/cve-checker @@ -0,0 +1,291 @@ +#!/usr/bin/env python + +# Copyright (C) 2009 by Thomas Petazzoni +# Copyright (C) 2020 by Gregory CLEMENT +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import argparse +import datetime +import fnmatch +import os +from collections import defaultdict +import re +import subprocess +import requests # URL checking +import json +import ijson +import certifi +import distutils.version +import time +import gzip +import sys +from urllib3 import HTTPSConnectionPool +from urllib3.exceptions import HTTPError +from multiprocessing import Pool + +sys.path.append('utils/') + +import cve as cvecheck + + +INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") +URL_RE = re.compile(r"\s*https?://\S*\s*$") + +RM_API_STATUS_ERROR = 1 +RM_API_STATUS_FOUND_BY_DISTRO = 2 +RM_API_STATUS_FOUND_BY_PATTERN = 3 +RM_API_STATUS_NOT_FOUND = 4 + +# Used to make multiple requests to the same host. It is global +# because it's used by sub-processes. +http_pool = None + + +class Package: + def __init__(self, name, version, ignored_cves): + self.name = name + self.version = version + self.cves = list() + self.ignored_cves = ignored_cves + +def check_package_cves(nvd_path, packages): + if not os.path.isdir(nvd_path): + os.makedirs(nvd_path) + + for cve in cvecheck.CVE.read_nvd_dir(nvd_path): + for pkg_name in cve.pkg_names: + pkg = packages.get(pkg_name, '') + if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves): + pkg.cves.append(cve.identifier) + +html_header = """ + + + +CVE status for Buildroot packages + + +CVE Status
+ +

+""" + + +html_footer = """ + + + +""" + + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + + +def dump_html_pkg(f, pkg): + f.write(" \n") + f.write(" %s\n" % pkg.name) + + # Current version + if len(pkg.version) > 20: + version = pkg.version[:20] + "..." + else: + version = pkg.version + f.write(" %s\n" % version) + + # CVEs + td_class = ["centered"] + if len(pkg.cves) == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" \n" % " ".join(td_class)) + for cve in pkg.cves: + f.write(" %s
\n" % (cve, cve)) + f.write(" \n") + + f.write(" \n") + + +def dump_html_all_pkgs(f, packages): + f.write(""" + + + + + + +""") + for pkg in packages: + dump_html_pkg(f, pkg) + f.write("
PackageVersionCVEs
") + + +def dump_html_gen_info(f, date, commit): + # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032 + f.write("

Updated on %s, git commit %s

\n" % (str(date), commit)) + + +def dump_html(packages, date, commit, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_gen_info(f, date, commit) + f.write(html_footer) + + +def dump_json(packages, date, commit, output): + # Format packages as a dictionnary instead of a list + # Exclude local field that does not contains real date + excluded_fields = ['url_worker', 'name'] + pkgs = { + pkg.name: { + k: v + for k, v in pkg.__dict__.items() + if k not in excluded_fields + } for pkg in packages + } + # The actual structure to dump, add commit and date to it + final = {'packages': pkgs, + 'commit': commit, + 'date': str(date)} + with open(output, 'w') as f: + json.dump(final, f, indent=2, separators=(',', ': ')) + f.write('\n') + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path)) + + +def parse_args(): + parser = argparse.ArgumentParser() + output = parser.add_argument_group('output', 'Output file(s)') + output.add_argument('--html', dest='html', type=resolvepath, + help='HTML output file') + output.add_argument('--json', dest='json', type=resolvepath, + help='JSON output file') + packages = parser.add_mutually_exclusive_group() + packages.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + packages.add_argument('-p', dest='packages', action='store', + help='List of packages (comma separated)') + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath) + args = parser.parse_args() + if not args.html and not args.json: + parser.error('at least one of --html or --json (or both) is required') + return args + + +def __main__(): + packages = list() + exclude_pacakges = ["linux", "gcc"] + content = json.load(sys.stdin) + for item in content: + if item in exclude_pacakges: + continue + pkg = content[item] + p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', '')) + packages.append(p) + + args = parse_args() + date = datetime.datetime.utcnow() + commit = subprocess.check_output(['git', 'rev-parse', + 'HEAD']).splitlines()[0].decode() + + if args.nvd_path: + print("Checking packages CVEs") + check_package_cves(args.nvd_path, {p.name: p for p in packages}) + if args.html: + print("Write HTML") + dump_html(packages, date, commit, args.html) + if args.json: + print("Write JSON") + dump_json(packages, date, commit, args.json) + +__main__() From patchwork Fri Jul 10 11:22:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326680 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dT3M2Kz9sSn for ; Fri, 10 Jul 2020 21:23:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 11C6389BF0; Fri, 10 Jul 2020 11:23:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gmUiJSehUegW; Fri, 10 Jul 2020 11:23:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 13D5C895FA; Fri, 10 Jul 2020 11:23:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 356701BF319 for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2EEE688C15 for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QWtvOzFyma94 for ; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by hemlock.osuosl.org (Postfix) with ESMTPS id CE2B588D0D for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id E40F01C0008; Fri, 10 Jul 2020 11:22:59 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:32 +0200 Message-Id: <20200710112245.1044073-7-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 6/9] package/pkg-utils: cve.py: Handle exception when version comparison fails X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" With python 3, when a package has a version number x-y-z instead of x.y.z, then the version returned by LooseVersion can't be compared which raises an exception. This patch handles this exception by adding a new return value when the comparison can't be done. Signed-off-by: Gregory CLEMENT --- support/scripts/cve.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index b754a17991..03afdeb54a 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -203,12 +203,21 @@ class CVE: continue if cpe['v_start']: + try: cve_affected_version = distutils.version.LooseVersion(cpe['v_start']) affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version) + break + except: + return 'Unknown' if (affected and cpe['v_end']): + try: cve_affected_version = distutils.version.LooseVersion(cpe['v_end']) affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version) + break + except: + return 'Unknown' + if (affected): return True return False From patchwork Fri Jul 10 11:22:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326686 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dm3Mmmz9sRK for ; Fri, 10 Jul 2020 21:23:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id EAAA029478; Fri, 10 Jul 2020 11:23:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0JZzrOtY38eZ; Fri, 10 Jul 2020 11:23:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 2D2A6203DF; Fri, 10 Jul 2020 11:23:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 85E411BF319 for ; Fri, 10 Jul 2020 11:23:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 57EE289A08 for ; Fri, 10 Jul 2020 11:23:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3yRtBRedSTzn for ; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id 6A6368972A for ; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 6EE1EE0005; Fri, 10 Jul 2020 11:23:00 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:33 +0200 Message-Id: <20200710112245.1044073-8-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 7/9] support/script/pkg-stats: Manage the CVEs that need to be check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version. This patch exposes this new information in json and html format. Signed-off-by: Gregory CLEMENT --- support/scripts/pkg-stats | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 883a5bd2be..e033e15e07 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -106,9 +106,11 @@ class Package: self.patch_files = [] self.warnings = 0 self.current_version = None + self.unknown_cve = False self.url = None self.url_worker = None self.cves = list() + self.cves_to_check = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} self.status = {} @@ -504,7 +506,12 @@ def check_package_cves(nvd_path, packages): for pkg_name in cve.pkg_names: if pkg_name in packages: pkg = packages[pkg_name] - if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()): + affected = cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()) + print(affected) + if (affected == 'Unknown'): + pkg.cves_to_check.append(cve.identifier) + elif affected == True: + print(cve.identifier) pkg.cves.append(cve.identifier) def calculate_stats(packages): @@ -544,8 +551,11 @@ def calculate_stats(packages): stats["version-not-uptodate"] += 1 stats["patches"] += pkg.patch_count stats["total-cves"] += len(pkg.cves) + stats["total-cves-to-check"] += len(pkg.cves_to_check) if len(pkg.cves) != 0: stats["pkg-cves"] += 1 + if len(pkg.cves_to_check) != 0: + stats["pkg-cves_to_check"] += 1 return stats @@ -763,11 +773,22 @@ def dump_html_pkg(f, pkg): td_class.append("correct") else: td_class.append("wrong") - f.write(" \n" % " ".join(td_class)) + f.write(" \n" % " ".join(td_class)) for cve in pkg.cves: f.write(" %s
\n" % (cve, cve)) f.write(" \n") + # CVEs to check + td_class = ["centered"] + if len(pkg.cves_to_check) == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" \n" % " ".join(td_class)) + for cve in pkg.cves_to_check: + f.write(" %s
\n" % (cve, cve)) + f.write(" \n") + f.write(" \n") @@ -786,6 +807,7 @@ def dump_html_all_pkgs(f, packages): Warnings Upstream URL CVEs +CVEs to check """) for pkg in sorted(packages): @@ -824,10 +846,14 @@ def dump_html_stats(f, stats): stats["version-not-uptodate"]) f.write("Packages with no known upstream version%s\n" % stats["version-unknown"]) - f.write("Packages affected by CVEs%s\n" % + f.write("Packages might affected by CVEs, where version needed to be checked%s\n" % stats["pkg-cves"]) - f.write("Total number of CVEs affecting all packages%s\n" % + f.write("Total number of CVEs that might affect all packages, where version needed to be checked%s\n" % stats["total-cves"]) + f.write("Packages affected by CVEs%s\n" % + stats["pkg-cves_to_check"]) + f.write("Total number of CVEs affecting all packages%s\n" % + stats["total-cves_to_check"]) f.write("\n") From patchwork Fri Jul 10 11:22:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326685 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dj5sJ4z9sRK for ; Fri, 10 Jul 2020 21:23:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C8E5026922; Fri, 10 Jul 2020 11:23:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1mli6TX5f7HN; Fri, 10 Jul 2020 11:23:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 07CEF2915B; Fri, 10 Jul 2020 11:23:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 027A71BF319 for ; Fri, 10 Jul 2020 11:23:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E1DDA88C9C for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v2peCMBbvQgd for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by hemlock.osuosl.org (Postfix) with ESMTPS id CF95D89B82 for ; Fri, 10 Jul 2020 11:23:02 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 00D13FF80E; Fri, 10 Jul 2020 11:23:00 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:34 +0200 Message-Id: <20200710112245.1044073-9-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 8/9] support/script/cve-checker: Manage the CVEs that need to be check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version. This patch exposes this new information in json and html format. Signed-off-by: Gregory CLEMENT --- support/scripts/cve-checker | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker index db8497d7aa..2a8a2b4d18 100755 --- a/support/scripts/cve-checker +++ b/support/scripts/cve-checker @@ -59,6 +59,7 @@ class Package: self.name = name self.version = version self.cves = list() + self.cves_to_check = list() self.ignored_cves = ignored_cves def check_package_cves(nvd_path, packages): @@ -68,8 +69,12 @@ def check_package_cves(nvd_path, packages): for cve in cvecheck.CVE.read_nvd_dir(nvd_path): for pkg_name in cve.pkg_names: pkg = packages.get(pkg_name, '') - if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves): - pkg.cves.append(cve.identifier) + if pkg: + affected = cve.affects(pkg.name, pkg.version, pkg.ignored_cves) + if (affected == 'Unknown'): + pkg.cves_to_check.append(cve.identifier) + elif affected == True: + pkg.cves.append(cve.identifier) html_header = """ @@ -188,6 +193,17 @@ def dump_html_pkg(f, pkg): f.write(" %s
\n" % (cve, cve)) f.write(" \n") + # CVEs to check + td_class = ["centered"] + if len(pkg.cves_to_check) == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" \n" % " ".join(td_class)) + for cve in pkg.cves_to_check: + f.write(" %s
\n" % (cve, cve)) + f.write(" \n") + f.write(" \n") @@ -198,6 +214,7 @@ def dump_html_all_pkgs(f, packages): Package Version CVEs +CVEs to check """) for pkg in packages: From patchwork Fri Jul 10 11:22:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326692 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39f658hXz9sRK for ; Fri, 10 Jul 2020 21:23:42 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2E9D888C15; Fri, 10 Jul 2020 11:23:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y9U6mXDIbaXg; Fri, 10 Jul 2020 11:23:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 9004888C9C; Fri, 10 Jul 2020 11:23:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id F0A361BF319 for ; Fri, 10 Jul 2020 11:23:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 02C7A2722E for ; Fri, 10 Jul 2020 11:23:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0areB+d534Id for ; Fri, 10 Jul 2020 11:23:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by silver.osuosl.org (Postfix) with ESMTPS id BB79C2038D for ; Fri, 10 Jul 2020 11:23:05 +0000 (UTC) Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay11.mail.gandi.net (Postfix) with ESMTPSA id C230A100004; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:35 +0200 Message-Id: <20200710112245.1044073-10-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 9/9] package/pkg-utils/cve.py: Manage case when package version doesn't exist X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Until now, when a package didn't report a version, then the CVE comparison was just skipped. It leads most of the time to declare the package not affected by the CVE. Instead of it, report the 'Unknown' status in order to be aware that the CVE related to this package has to be checked. Signed-off-by: Gregory CLEMENT --- support/scripts/cve.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index 03afdeb54a..d3480d68dd 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -188,6 +188,7 @@ class CVE: if (self.identifier in cve_ignore_list): return False + unknown_pkg_version = False for cpe in self.each_cpe(): affected = True if cpe['product'] != name: @@ -200,6 +201,7 @@ class CVE: pkg_version = distutils.version.LooseVersion(version) if not hasattr(pkg_version, "version"): print("Cannot parse package '%s' version '%s'" % (name, version)) + unknown_pkg_version = True continue if cpe['v_start']: @@ -220,4 +222,8 @@ class CVE: if (affected): return True - return False + + if unknown_pkg_version: + return 'Unknown' + else: + return False