diff mbox series

[v3,5/8] support/scripts: Add a per configuration CVE checker

Message ID 20200724154356.2607639-6-gregory.clement@bootlin.com
State Accepted
Headers show
Series Improving CVE reporting | expand

Commit Message

Gregory CLEMENT July 24, 2020, 3:43 p.m. UTC
This scripts takes as entry on stdin a JSON description of the package
used for a given configuration. This description is the one generated
by "make show-info".

The script generates the list of all the package used and if they are
affected by a CVE. The output is either a JSON or an HTML file similar
to the one generated by pkg-stats.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
---
 support/scripts/cve-checker | 258 ++++++++++++++++++++++++++++++++++++
 1 file changed, 258 insertions(+)
 create mode 100755 support/scripts/cve-checker

Comments

Matthew Weber July 29, 2020, 6:03 p.m. UTC | #1
Gregory,

On Fri, Jul 24, 2020 at 10:44 AM Gregory CLEMENT
<gregory.clement@bootlin.com> wrote:
>
> This scripts takes as entry on stdin a JSON description of the package
> used for a given configuration. This description is the one generated
> by "make show-info".
>
> The script generates the list of all the package used and if they are
> affected by a CVE. The output is either a JSON or an HTML file similar
> to the one generated by pkg-stats.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>

Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>=
Thomas Petazzoni Aug. 28, 2020, 9:45 a.m. UTC | #2
Hello,

On Fri, 24 Jul 2020 17:43:53 +0200
Gregory CLEMENT <gregory.clement@bootlin.com> wrote:

> This scripts takes as entry on stdin a JSON description of the package
> used for a given configuration. This description is the one generated
> by "make show-info".
> 
> The script generates the list of all the package used and if they are
> affected by a CVE. The output is either a JSON or an HTML file similar
> to the one generated by pkg-stats.
> 
> Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>

Thanks, I have applied to next, but after doing a number of changes,
see below.

> +import argparse
> +import datetime
> +import os
> +import json
> +import sys
> +
> +sys.path.append('utils/')

This was not needed.

> +
> +import cve as cvecheck
> +
> +class Package:
> +    def __init__(self, name, version, ignored_cves):
> +        self.name = name
> +        self.version = version
> +        self.cves = list()
> +        self.ignored_cves = ignored_cves
> +
> +def check_package_cves(nvd_path, packages):
> +    if not os.path.isdir(nvd_path):
> +        os.makedirs(nvd_path)
> +
> +    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
> +        for pkg_name in cve.pkg_names:
> +            pkg = packages.get(pkg_name, '')
> +            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):

This was not correct as cve.affects() no longer returns a boolean. Due
to this, all existing CVEs were reported in the generated HTML/JSON as
affecting the package.

> +                pkg.cves.append(cve.identifier)
> +
> +html_header = """
> +<head>
> +<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
> +<style type=\"text/css\">
> +table {
> +  width: 100%;
> +}
> +td {
> +  border: 1px solid black;
> +}
> +td.centered {
> +  text-align: center;
> +}
> +td.wrong {
> +  background: #ff9a69;
> +}
> +td.correct {
> +  background: #d2ffc4;
> +}
> +td.nopatches {
> +  background: #d2ffc4;
> +}
> +td.somepatches {
> +  background: #ffd870;
> +}
> +td.lotsofpatches {
> +  background: #ff9a69;
> +}
> +
> +td.good_url {
> +  background: #d2ffc4;
> +}
> +td.missing_url {
> +  background: #ffd870;
> +}
> +td.invalid_url {
> +  background: #ff9a69;
> +}
> +
> +td.version-good {
> +  background: #d2ffc4;
> +}
> +td.version-needs-update {
> +  background: #ff9a69;
> +}
> +td.version-unknown {
> + background: #ffd870;
> +}
> +td.version-error {
> + background: #ccc;
> +}

Lots of these CSS classes were not useful, so I dropped them.

> +
> +</style>
> +<title>CVE status for Buildroot packages</title>

Changed "Buildroot packages" for "Buildroot configuration". Indeed,
compared to pkg-stats which operates on all packages (it's a tool for
Buildroot maintenance), cve-checker is really about a given Buildroot
configuration.

> +def infra_str(infra_list):

This function was not used anywhere, so I dropped it.

> +def boolean_str(b):

This function was not used anywhere, so I dropped it.


> +def dump_json(packages, date, output):
> +    # Format packages as a dictionnary instead of a list
> +    # Exclude local field that does not contains real date
> +    excluded_fields = ['url_worker', 'name']
> +    pkgs = {
> +        pkg.name: {
> +            k: v
> +            for k, v in pkg.__dict__.items()
> +            if k not in excluded_fields
> +        } for pkg in packages

I simplified that a bit, as we don't want all fields in the JSON I
believe, just the version and list of CVEs. For example, the list of
ignored CVEs is not really relevant.

> +
> +def parse_args():
> +    parser = argparse.ArgumentParser()
> +    output = parser.add_argument_group('output', 'Output file(s)')
> +    output.add_argument('--html', dest='html', type=resolvepath,
> +                        help='HTML output file')
> +    output.add_argument('--json', dest='json', type=resolvepath,
> +                        help='JSON output file')
> +    packages = parser.add_mutually_exclusive_group()

This line was not used.

> +    parser.add_argument('--nvd-path', dest='nvd_path',
> +                        help='Path to the local NVD database',type=resolvepath,
> +                        default='./nvd_dl')

The default value doesn't exist for pkg-stats, I'm not sure it makes
sense to have a default value. I've however added a required=True
because this script doesn't do anything useful if we don't have access
to the NVD data.

> +def __main__():
> +    packages = list()
> +    exclude_pacakges = ["linux", "gcc"]

I'm not sure why those two packages were excluded, so I've dropped
that, at least for now. We can of course improve things later on.

> +    content = json.load(sys.stdin)
> +    for item in content:
> +        if item in exclude_pacakges:
> +            continue
> +        pkg = content[item]
> +        p = Package(item, pkg.get('version', ''), pkg.get('ignore_cves', ''))
> +        packages.append(p)
> +
> +    args = parse_args()
> +    date = datetime.datetime.utcnow()
> +
> +    if args.nvd_path:

I've dropped this "if", since args.nvd_path is a required option.

As said above: applied to next with all those changes. Thanks!

Thomas
diff mbox series

Patch

diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
new file mode 100755
index 0000000000..19fd104b56
--- /dev/null
+++ b/support/scripts/cve-checker
@@ -0,0 +1,258 @@ 
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import os
+import json
+import sys
+
+sys.path.append('utils/')
+
+import cve as cvecheck
+
+class Package:
+    def __init__(self, name, version, ignored_cves):
+        self.name = name
+        self.version = version
+        self.cves = list()
+        self.ignored_cves = ignored_cves
+
+def check_package_cves(nvd_path, packages):
+    if not os.path.isdir(nvd_path):
+        os.makedirs(nvd_path)
+
+    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+        for pkg_name in cve.pkg_names:
+            pkg = packages.get(pkg_name, '')
+            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
+                pkg.cves.append(cve.identifier)
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+  width: 100%;
+}
+td {
+  border: 1px solid black;
+}
+td.centered {
+  text-align: center;
+}
+td.wrong {
+  background: #ff9a69;
+}
+td.correct {
+  background: #d2ffc4;
+}
+td.nopatches {
+  background: #d2ffc4;
+}
+td.somepatches {
+  background: #ffd870;
+}
+td.lotsofpatches {
+  background: #ff9a69;
+}
+
+td.good_url {
+  background: #d2ffc4;
+}
+td.missing_url {
+  background: #ffd870;
+}
+td.invalid_url {
+  background: #ff9a69;
+}
+
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
+</style>
+<title>CVE status for Buildroot packages</title>
+</head>
+
+<a href=\"#results\">CVE Status</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+  document.getElementById(\"sortable_hint\").innerHTML =
+  \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+    if not infra_list:
+        return "Unknown"
+    elif len(infra_list) == 1:
+        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+    elif infra_list[0][1] == infra_list[1][1]:
+        return "<b>%s</b><br/>%s + %s" % \
+            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+    else:
+        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+            (infra_list[0][1], infra_list[0][0],
+             infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+    if b:
+        return "Yes"
+    else:
+        return "No"
+
+
+def dump_html_pkg(f, pkg):
+    f.write(" <tr>\n")
+    f.write("  <td>%s</td>\n" % pkg.name)
+
+    # Current version
+    if len(pkg.version) > 20:
+        version = pkg.version[:20] + "..."
+    else:
+        version = pkg.version
+    f.write("  <td class=\"centered\">%s</td>\n" % version)
+
+    # CVEs
+    td_class = ["centered"]
+    if len(pkg.cves) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
+    f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+    f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Version</td>
+<td class=\"centered\">CVEs</td>
+</tr>
+""")
+    for pkg in packages:
+        dump_html_pkg(f, pkg)
+    f.write("</table>")
+
+
+def dump_html_gen_info(f, date):
+    # Updated on Mon Feb 19 08:12:08 CET 2018
+    f.write("<p><i>Updated on %s</i></p>\n" % (str(date)))
+
+
+def dump_html(packages, date, output):
+    with open(output, 'w') as f:
+        f.write(html_header)
+        dump_html_all_pkgs(f, packages)
+        dump_html_gen_info(f, date)
+        f.write(html_footer)
+
+
+def dump_json(packages, date, output):
+    # Format packages as a dictionnary instead of a list
+    # Exclude local field that does not contains real date
+    excluded_fields = ['url_worker', 'name']
+    pkgs = {
+        pkg.name: {
+            k: v
+            for k, v in pkg.__dict__.items()
+            if k not in excluded_fields
+        } for pkg in packages
+    }
+     # The actual structure to dump, add date to it
+    final = {'packages': pkgs,
+             'date': str(date)}
+    with open(output, 'w') as f:
+        json.dump(final, f, indent=2, separators=(',', ': '))
+        f.write('\n')
+
+
+def resolvepath(path):
+        return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    output = parser.add_argument_group('output', 'Output file(s)')
+    output.add_argument('--html', dest='html', type=resolvepath,
+                        help='HTML output file')
+    output.add_argument('--json', dest='json', type=resolvepath,
+                        help='JSON output file')
+    packages = parser.add_mutually_exclusive_group()
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database',type=resolvepath,
+                        default='./nvd_dl')
+    args = parser.parse_args()
+    if not args.html and not args.json:
+        parser.error('at least one of --html or --json (or both) is required')
+    return args
+
+
+def __main__():
+    packages = list()
+    exclude_pacakges = ["linux", "gcc"]
+    content = json.load(sys.stdin)
+    for item in content:
+        if item in exclude_pacakges:
+            continue
+        pkg = content[item]
+        p = Package(item, pkg.get('version', ''), pkg.get('ignore_cves', ''))
+        packages.append(p)
+
+    args = parse_args()
+    date = datetime.datetime.utcnow()
+
+    if args.nvd_path:
+        print("Checking packages CVEs")
+        check_package_cves(args.nvd_path, {p.name: p for p in packages})
+    if args.html:
+        print("Write HTML")
+        dump_html(packages, date, args.html)
+    if args.json:
+        print("Write JSON")
+        dump_json(packages, date, args.json)
+
+__main__()