From patchwork Fri Jul 10 11:22:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory CLEMENT X-Patchwork-Id: 1326682 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B39dZ12MTz9sRK for ; Fri, 10 Jul 2020 21:23:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B393689643; Fri, 10 Jul 2020 11:23:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yXNJ-OyhEZEC; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 1497F8963E; Fri, 10 Jul 2020 11:23:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 750641BF319 for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6F1D98962B for ; Fri, 10 Jul 2020 11:23:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PtCsOVz0Yrk6 for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by whitealder.osuosl.org (Postfix) with ESMTPS id 562BA89637 for ; Fri, 10 Jul 2020 11:23:01 +0000 (UTC) X-Originating-IP: 91.175.115.186 Received: from localhost (91-175-115-186.subs.proxad.net [91.175.115.186]) (Authenticated sender: gregory.clement@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 69EDD1BF203; Fri, 10 Jul 2020 11:22:59 +0000 (UTC) From: Gregory CLEMENT To: buildroot@buildroot.org Date: Fri, 10 Jul 2020 13:22:31 +0200 Message-Id: <20200710112245.1044073-6-gregory.clement@bootlin.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200710112245.1044073-1-gregory.clement@bootlin.com> References: <20200710112245.1044073-1-gregory.clement@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 5/9] support/scripts: Add a per configuration CVE checker X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This scripts takes as entry on stdin a JSON description of the package used for a given configuration. This description is the one generated by "make show-info". The script generates the list of all the package used and if they are affected by a CVE. The output is either a JSON or an HTML file similar to the one generated by pkg-stats. Signed-off-by: Gregory CLEMENT --- support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100755 support/scripts/cve-checker diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker new file mode 100755 index 0000000000..db8497d7aa --- /dev/null +++ b/support/scripts/cve-checker @@ -0,0 +1,291 @@ +#!/usr/bin/env python + +# Copyright (C) 2009 by Thomas Petazzoni +# Copyright (C) 2020 by Gregory CLEMENT +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import argparse +import datetime +import fnmatch +import os +from collections import defaultdict +import re +import subprocess +import requests # URL checking +import json +import ijson +import certifi +import distutils.version +import time +import gzip +import sys +from urllib3 import HTTPSConnectionPool +from urllib3.exceptions import HTTPError +from multiprocessing import Pool + +sys.path.append('utils/') + +import cve as cvecheck + + +INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") +URL_RE = re.compile(r"\s*https?://\S*\s*$") + +RM_API_STATUS_ERROR = 1 +RM_API_STATUS_FOUND_BY_DISTRO = 2 +RM_API_STATUS_FOUND_BY_PATTERN = 3 +RM_API_STATUS_NOT_FOUND = 4 + +# Used to make multiple requests to the same host. It is global +# because it's used by sub-processes. +http_pool = None + + +class Package: + def __init__(self, name, version, ignored_cves): + self.name = name + self.version = version + self.cves = list() + self.ignored_cves = ignored_cves + +def check_package_cves(nvd_path, packages): + if not os.path.isdir(nvd_path): + os.makedirs(nvd_path) + + for cve in cvecheck.CVE.read_nvd_dir(nvd_path): + for pkg_name in cve.pkg_names: + pkg = packages.get(pkg_name, '') + if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves): + pkg.cves.append(cve.identifier) + +html_header = """ + + + +CVE status for Buildroot packages + + +CVE Status
+ +

+""" + + +html_footer = """ + + + +""" + + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + + +def dump_html_pkg(f, pkg): + f.write(" \n") + f.write(" %s\n" % pkg.name) + + # Current version + if len(pkg.version) > 20: + version = pkg.version[:20] + "..." + else: + version = pkg.version + f.write(" %s\n" % version) + + # CVEs + td_class = ["centered"] + if len(pkg.cves) == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" \n" % " ".join(td_class)) + for cve in pkg.cves: + f.write(" %s
\n" % (cve, cve)) + f.write(" \n") + + f.write(" \n") + + +def dump_html_all_pkgs(f, packages): + f.write(""" + + + + + + +""") + for pkg in packages: + dump_html_pkg(f, pkg) + f.write("
PackageVersionCVEs
") + + +def dump_html_gen_info(f, date, commit): + # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032 + f.write("

Updated on %s, git commit %s

\n" % (str(date), commit)) + + +def dump_html(packages, date, commit, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_gen_info(f, date, commit) + f.write(html_footer) + + +def dump_json(packages, date, commit, output): + # Format packages as a dictionnary instead of a list + # Exclude local field that does not contains real date + excluded_fields = ['url_worker', 'name'] + pkgs = { + pkg.name: { + k: v + for k, v in pkg.__dict__.items() + if k not in excluded_fields + } for pkg in packages + } + # The actual structure to dump, add commit and date to it + final = {'packages': pkgs, + 'commit': commit, + 'date': str(date)} + with open(output, 'w') as f: + json.dump(final, f, indent=2, separators=(',', ': ')) + f.write('\n') + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path)) + + +def parse_args(): + parser = argparse.ArgumentParser() + output = parser.add_argument_group('output', 'Output file(s)') + output.add_argument('--html', dest='html', type=resolvepath, + help='HTML output file') + output.add_argument('--json', dest='json', type=resolvepath, + help='JSON output file') + packages = parser.add_mutually_exclusive_group() + packages.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + packages.add_argument('-p', dest='packages', action='store', + help='List of packages (comma separated)') + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath) + args = parser.parse_args() + if not args.html and not args.json: + parser.error('at least one of --html or --json (or both) is required') + return args + + +def __main__(): + packages = list() + exclude_pacakges = ["linux", "gcc"] + content = json.load(sys.stdin) + for item in content: + if item in exclude_pacakges: + continue + pkg = content[item] + p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', '')) + packages.append(p) + + args = parse_args() + date = datetime.datetime.utcnow() + commit = subprocess.check_output(['git', 'rev-parse', + 'HEAD']).splitlines()[0].decode() + + if args.nvd_path: + print("Checking packages CVEs") + check_package_cves(args.nvd_path, {p.name: p for p in packages}) + if args.html: + print("Write HTML") + dump_html(packages, date, commit, args.html) + if args.json: + print("Write JSON") + dump_json(packages, date, commit, args.json) + +__main__()