From patchwork Wed Oct 17 21:41:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985550 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HZ4zQ3z9sBn for ; Thu, 18 Oct 2018 08:42:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0E86987697; Wed, 17 Oct 2018 21:42:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id exji0wf48bJu; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 5FB4E8795E; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id DFC3B1BF3E3 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id DDDA5882B2 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r6RBUc0lXS6Q for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs01.rockwellcollins.com (ch3vs01.rockwellcollins.com [205.175.226.27]) by hemlock.osuosl.org (Postfix) with ESMTPS id C8947882B8 for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs01.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:04 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id B8C6560191; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:58 -0500 Message-Id: <1539812522-7171-6-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 06/10] support/scripts/pkg-stats: add CPE reporting X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pkg status now includes CPE as an item reported in the html output (stat summary and for each pkg) Signed-off-by: Matthew Weber --- Changes v6 -> v7 - Rebased to work after url checking code was added upstream v5 -> v6 - Rebased to capture formatting changes v4 -> v5 [Ricardo - Renamed patch to correctly match file name - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup [Arnout - Collectly with Ricardo, decided to move cpe report analysis to a seperate script and breakout a module that's imported for the cpedb class - Rename cpe_dict to instead be cpedb v3 -> v4 - Collapsed patch 5 and 6 together into this single patch [Eric - added except handling around file io - fixed condition where buildroot isn't generating a CPE string as part of the infra and output that is the case. (eventually these probably could be fixed but there aren't many at this point) [Ricardo - fixed patch naming and resolved flake8 issues - took the opportunity to also fix other flake8 syntax update suggestions - added except handling to have proper exits - cleaned up csv file header skippin - condensed partial cve string split - updated help txt as suggested - reworked output file requirement. Removed -o as required but added check if provided when -c isn't used v3 - New patch --- support/scripts/pkg-stats | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index d0b06b1..21d1767 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -26,6 +26,7 @@ import subprocess import sys import requests # URL checking from multiprocessing import Pool +from cpedb import CPEDB INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile("\s*https?://\S*\s*$") @@ -35,6 +36,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_cpe_id = dict() def __init__(self, name, path): self.name = name @@ -49,6 +51,8 @@ class Package: self.url = None self.url_status = None self.url_worker = None + self.cpe_id = None + self.has_cpe = False def pkgvar(self): return self.name.upper().replace("-", "_") @@ -139,6 +143,26 @@ class Package: self.warnings = int(m.group(1)) return + def set_cpe_info(self, cpe_dict): + """ + Fills in the .has_cpe field + """ + var = self.pkgvar() + if var in self.all_cpe_id: + self.cpe_id = self.all_cpe_id[var] + if self.cpe_id is None: + # BR infra did not build a CPE ID for this pkg + # as it's most likely a host pkg + return + result = cpe_dict.find(self.cpe_id) + if not result: + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id)) + if result: + self.has_cpe = "Update" + # Unset case for has_cpe is assumed missing/does not exist + else: + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id) + def __eq__(self, other): return self.path == other.path @@ -277,6 +301,20 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value + # CPE ID + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_CPE_ID"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # Strip _CPE_ID + pkgvar = pkgvar[:-7] + if pkgvar in ("LINUX", "LINUX_HEADERS"): + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*" + else: + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*" + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in": @@ -322,6 +360,12 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.has_cpe == "Update": + stats["update-cpe"] += 1 + elif pkg.has_cpe: + stats["cpe"] += 1 + else: + stats["no-cpe"] += 1 stats["patches"] += pkg.patch_count return stats @@ -488,6 +532,20 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % (" ".join(td_class), url_str)) + # CPE Valid + td_class = ["centered"] + if not pkg.has_cpe: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_cpe))) + elif pkg.has_cpe == "Update": + td_class.append("wrong") + f.write(" Update\n" % + (" ".join(td_class))) + else: + td_class.append("correct") + f.write(" %s\n" % + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe))) f.write(" \n") @@ -504,6 +562,7 @@ def dump_html_all_pkgs(f, packages): Current version Warnings Upstream URL +CPE Valid """) for pkg in sorted(packages): @@ -530,6 +589,12 @@ def dump_html_stats(f, stats): stats["hash"]) f.write(" Packages not having a hash file%s\n" % stats["no-hash"]) + f.write(" Packages having a registered CPE%s\n" % + stats["cpe"]) + f.write(" Packages needing CPE update%s\n" % + stats["update-cpe"]) + f.write(" Packages missing a registered CPE%s\n" % + stats["no-cpe"]) f.write(" Total number of patches%s\n" % stats["patches"]) f.write("\n") @@ -572,6 +637,8 @@ def __main__(): package_list = args.packages.split(",") else: package_list = None + cpedb = CPEDB() + cpedb.get_xml_dict() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting package make info ...") @@ -585,6 +652,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_cpe_info(cpedb) print("Checking URL status") check_package_urls(packages) print("Calculate stats")