diff mbox series

[RFC,v9,08/10] support/scripts/cpe-report: new script

Message ID 20200616170341.45098-8-matthew.weber@rockwellcollins.com
State New
Headers show
Series [RFC,v9,01/10] cpe-info: new make target | expand

Commit Message

Matthew Weber June 16, 2020, 5:03 p.m. UTC
The script supports looking up all the CPEs provided in a
make cpe-info csv file export from a target Buildroot build.
It checks the current version and suggests a CPE needs update
or possibly an initial submission is required to NIST.

Adds option to allow alternate locations for the dictionary
URL and caching of a processed dictionary to speed up execution.

Outputs a cpe/ folder with propsed xml generated from the
dictionary contents to propose updated versions to NIST.

For missing CPE matches, a cpe-report-missing.txt is created
by the script that can be used later to manually create proposed
new NIST dictionary entries.

Ref: NIST has a group email (cpe_dictionary@nist.gov) used to
recieve these version update and new entry xml files.  They do
process the XML and provide feedback. In some cases they will
propose back something different where the vendor or version is
slightly different.

Limitations
 - Currently any use of non-number version identifiers isn't
   supported by NIST as they use ranges to determine impact
   of a CVE
 - Any Linux version from a non-upstream is also not supported
   without manually adjusting the information as the custom
   kernel will more then likely not match the upstream version
   used in the dictionary

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
Changes

v8
 - Updated to just output missing and needs version update
 - Optional processed dictionary caching support
 - Optional dictionary URL
 - Creation of a missing status file (cpe-report-missing.txt)
 - Adjusted index used in CSV for removal of CVE patched item

v5 -> v7
 - No change

v5
[Ricardo
 - Updated v4 comments about general flake formatting cleanup
 - Incorporated parts of patch 1/2 suggestions for optimizations

[Ricardo/Arnout
 - Collectly, decided to move cpe report analysis to this
   script and use a seperate module cpedb class

[Arnout
 - Rename cpe_dict to instead be cpedb

v1 -> v4
 - Patch did not exist and was part of pkg-stats file
---
 support/scripts/cpe-report | 70 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100755 support/scripts/cpe-report

Comments

Thomas Petazzoni June 25, 2020, 11:18 a.m. UTC | #1
On Tue, 16 Jun 2020 12:03:39 -0500
Matt Weber <matthew.weber@rockwellcollins.com> wrote:

> The script supports looking up all the CPEs provided in a
> make cpe-info csv file export from a target Buildroot build.
> It checks the current version and suggests a CPE needs update
> or possibly an initial submission is required to NIST.
> 
> Adds option to allow alternate locations for the dictionary
> URL and caching of a processed dictionary to speed up execution.
> 
> Outputs a cpe/ folder with propsed xml generated from the
> dictionary contents to propose updated versions to NIST.
> 
> For missing CPE matches, a cpe-report-missing.txt is created
> by the script that can be used later to manually create proposed
> new NIST dictionary entries.
> 
> Ref: NIST has a group email (cpe_dictionary@nist.gov) used to
> recieve these version update and new entry xml files.  They do
> process the XML and provide feedback. In some cases they will
> propose back something different where the vendor or version is
> slightly different.
> 
> Limitations
>  - Currently any use of non-number version identifiers isn't
>    supported by NIST as they use ranges to determine impact
>    of a CVE
>  - Any Linux version from a non-upstream is also not supported
>    without manually adjusting the information as the custom
>    kernel will more then likely not match the upstream version
>    used in the dictionary
> 
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

At this point, I am not really clear what this script does. Indeed,
what I would have initially expected is a script that based on the
"show-info" output, tells the user what are the known unfixed CVEs
affecting his configuration. But this is not what this cpe-report
script is doing.

I am not sure to understand what are the CPE updates that this script
generates ? Does the NVD database needs to know about all versions of
all software components ? I though the database was indexed by CVE, and
then provided for each CVE the range of versions of the software
component affected by that CVE.

Could you clarify a bit the whole process, and what are those "CPE
updates" sent to NIST useful for ?

> +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"

Or perhaps this "dictionary" is not about CVEs, but about listing all
versions of all software components ?

Thomas
diff mbox series

Patch

diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report
new file mode 100755
index 0000000000..7242a372b2
--- /dev/null
+++ b/support/scripts/cpe-report
@@ -0,0 +1,70 @@ 
+#!/usr/bin/env python
+
+import argparse
+import sys
+import csv
+from cpedb import CPEDB
+
+CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+
+def get_target_cpe_report(cpe_report_file, cpedb):
+    report_cpe_exact_match = ""
+    report_cpe_needing_update = ""
+    report_cpe_needing_update_list = ""
+    report_cpe_missing = ""
+
+    print("CPE: Checking for matches...")
+    try:
+        with open(cpe_report_file) as cpe_file:
+            cpe_list = csv.reader(cpe_file)
+            next(cpe_list)  # make cpe-info has a one line header
+            for cpe in cpe_list:
+                result = cpedb.find(cpe[0])
+                if not result:
+                    result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0]))
+                    if not result:
+                        report_cpe_missing += cpe[0] + "," + cpe[1] + "," + cpe[3] + "\n"
+                    else:
+                        latest_version = cpedb.find_partial_latest_version(cpedb.get_cpe_no_version(cpe[0]))
+                        report_cpe_needing_update += cpe[0] + ", Latest Version Guess from Dict[" + latest_version + "]\n"
+                        report_cpe_needing_update_list += cpe[0] + "\n"
+                else:
+                    report_cpe_exact_match += cpe[0] + "\n"
+    except (OSError, IOError) as e:
+        print("CPE: report csv file (%s): %s" % (e.errno, e.strerror))
+        sys.exit(1)
+
+    print("CPE: Found but may REQUIRE an UPDATE:\n" + report_cpe_needing_update)
+    print("CPE: Not found:\n" + report_cpe_missing)
+
+    fp = open('cpe-report-missing.txt', 'w+')
+    fp.write(report_cpe_missing)
+    fp.close()
+
+    for cpe in report_cpe_needing_update_list.splitlines():
+        cpedb.update(cpe)
+    print("XML Generation Complete of NIST update files, see ./cpe/*")
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-c', dest='cpe_report', action='store', required=True,
+                        help='CPE Report generated by make cpe-info (csv format)')
+    parser.add_argument('-u', dest='url', action='store', required=False,
+                        help='(optional)URL to the NIST dict (official-cpe-dictionary_v2.3.xml.gz)')
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    cpedb = CPEDB()
+    url = CPE_XML_URL
+    if args.url:
+        url = args.url
+    cpedb.get_xml_dict(url)
+    print("Performing Target CPE Report Analysis...")
+    get_target_cpe_report(args.cpe_report, cpedb)
+
+
+__main__()