Message ID | 20201006134250.22738-9-gregory.clement@bootlin.com |
---|---|
State | Superseded |
Headers | show |
Series | Adding CPE ID support for CVEs | expand |
Hi Gregory, Hi Matt, Am Di., 6. Okt. 2020 um 15:44 Uhr schrieb Gregory CLEMENT <gregory.clement@bootlin.com>: > > From: Matt Weber <matthew.weber@rockwellcollins.com> > > The script supports looking up all the CPEs provided in a > make cpe-info csv file export from a target Buildroot build. > It checks the current version and suggests a CPE needs update > or possibly an initial submission is required to NIST. Is there a way to create this kind of list/output also for all packages in buildroot and not only the one that is generated by a configuration? > Adds option to allow alternate locations for the dictionary > URL and caching of a processed dictionary to speed up execution. > > Outputs a cpe/ folder with propsed xml generated from the > dictionary contents to propose updated versions to NIST. > > For missing CPE matches, a cpe-report-missing.txt is created > by the script that can be used later to manually create proposed > new NIST dictionary entries. > > Ref: NIST has a group email (cpe_dictionary@nist.gov) used to > recieve these version update and new entry xml files. They do > process the XML and provide feedback. In some cases they will > propose back something different where the vendor or version is > slightly different. > > Limitations > - Currently any use of non-number version identifiers isn't > supported by NIST as they use ranges to determine impact > of a CVE > - Any Linux version from a non-upstream is also not supported > without manually adjusting the information as the custom > kernel will more then likely not match the upstream version > used in the dictionary > > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com> > --- > support/scripts/cpe-report | 70 ++++++++++++++++++++++++++++++++++++++ > 1 file changed, 70 insertions(+) > create mode 100755 support/scripts/cpe-report > > diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report > new file mode 100755 > index 0000000000..7242a372b2 > --- /dev/null > +++ b/support/scripts/cpe-report > @@ -0,0 +1,70 @@ > +#!/usr/bin/env python Shouldn't we use python3 here? > + > +import argparse > +import sys > +import csv > +from cpedb import CPEDB > + > +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" > + > + > +def get_target_cpe_report(cpe_report_file, cpedb): > + report_cpe_exact_match = "" > + report_cpe_needing_update = "" > + report_cpe_needing_update_list = "" > + report_cpe_missing = "" > + > + print("CPE: Checking for matches...") > + try: > + with open(cpe_report_file) as cpe_file: > + cpe_list = csv.reader(cpe_file) > + next(cpe_list) # make cpe-info has a one line header > + for cpe in cpe_list: > + result = cpedb.find(cpe[0]) > + if not result: > + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) > + if not result: > + report_cpe_missing += cpe[0] + "," + cpe[1] + "," + cpe[3] + "\n" > + else: > + latest_version = cpedb.find_partial_latest_version(cpedb.get_cpe_no_version(cpe[0])) > + report_cpe_needing_update += cpe[0] + ", Latest Version Guess from Dict[" + latest_version + "]\n" > + report_cpe_needing_update_list += cpe[0] + "\n" > + else: > + report_cpe_exact_match += cpe[0] + "\n" > + except (OSError, IOError) as e: > + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) > + sys.exit(1) > + > + print("CPE: Found but may REQUIRE an UPDATE:\n" + report_cpe_needing_update) > + print("CPE: Not found:\n" + report_cpe_missing) > + > + fp = open('cpe-report-missing.txt', 'w+') > + fp.write(report_cpe_missing) > + fp.close() > + > + for cpe in report_cpe_needing_update_list.splitlines(): > + cpedb.update(cpe) > + print("XML Generation Complete of NIST update files, see ./cpe/*") > + > + > +def parse_args(): > + parser = argparse.ArgumentParser() > + parser.add_argument('-c', dest='cpe_report', action='store', required=True, > + help='CPE Report generated by make cpe-info (csv format)') > + parser.add_argument('-u', dest='url', action='store', required=False, > + help='(optional)URL to the NIST dict (official-cpe-dictionary_v2.3.xml.gz)') > + return parser.parse_args() > + > + > +def __main__(): > + args = parse_args() > + cpedb = CPEDB() > + url = CPE_XML_URL > + if args.url: > + url = args.url > + cpedb.get_xml_dict(url) > + print("Performing Target CPE Report Analysis...") > + get_target_cpe_report(args.cpe_report, cpedb) > + > + > +__main__()
Heiko, On Wed, Oct 7, 2020 at 3:13 AM Heiko Thiery <heiko.thiery@gmail.com> wrote: > > Hi Gregory, Hi Matt, > > Am Di., 6. Okt. 2020 um 15:44 Uhr schrieb Gregory CLEMENT > <gregory.clement@bootlin.com>: > > > > From: Matt Weber <matthew.weber@rockwellcollins.com> > > > > The script supports looking up all the CPEs provided in a > > make cpe-info csv file export from a target Buildroot build. > > It checks the current version and suggests a CPE needs update > > or possibly an initial submission is required to NIST. > > Is there a way to create this kind of list/output also for all > packages in buildroot and not only the one that is generated by a > configuration? Yeah. The CPE maintaining tooling can look at whatever list of CPE you feed it, so we'd just need to add an option to dump the complete CPE listing similar to how pkg-stats does it. There was some debate on just using "show vars" directly by the tooling and not generating this in between csv file. I'm still of the argument that having this hand-off file in some format is still valuable for 3rd party analysis of the pkg listing (similar to what we have with the legal info csv's). However if we allow full package list dumps of CPE and the current defconfig, it may make sense to convert the CPE tools to just directly use the "show vars" and have a command line option to select the type of analysis. Then the tool could be improved to also output an optional report similar to the CSV. This would allow us to remove the cpe-info make target and simplify the buildsystem changeset. Regards, Matt
diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report new file mode 100755 index 0000000000..7242a372b2 --- /dev/null +++ b/support/scripts/cpe-report @@ -0,0 +1,70 @@ +#!/usr/bin/env python + +import argparse +import sys +import csv +from cpedb import CPEDB + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +def get_target_cpe_report(cpe_report_file, cpedb): + report_cpe_exact_match = "" + report_cpe_needing_update = "" + report_cpe_needing_update_list = "" + report_cpe_missing = "" + + print("CPE: Checking for matches...") + try: + with open(cpe_report_file) as cpe_file: + cpe_list = csv.reader(cpe_file) + next(cpe_list) # make cpe-info has a one line header + for cpe in cpe_list: + result = cpedb.find(cpe[0]) + if not result: + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) + if not result: + report_cpe_missing += cpe[0] + "," + cpe[1] + "," + cpe[3] + "\n" + else: + latest_version = cpedb.find_partial_latest_version(cpedb.get_cpe_no_version(cpe[0])) + report_cpe_needing_update += cpe[0] + ", Latest Version Guess from Dict[" + latest_version + "]\n" + report_cpe_needing_update_list += cpe[0] + "\n" + else: + report_cpe_exact_match += cpe[0] + "\n" + except (OSError, IOError) as e: + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) + sys.exit(1) + + print("CPE: Found but may REQUIRE an UPDATE:\n" + report_cpe_needing_update) + print("CPE: Not found:\n" + report_cpe_missing) + + fp = open('cpe-report-missing.txt', 'w+') + fp.write(report_cpe_missing) + fp.close() + + for cpe in report_cpe_needing_update_list.splitlines(): + cpedb.update(cpe) + print("XML Generation Complete of NIST update files, see ./cpe/*") + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-c', dest='cpe_report', action='store', required=True, + help='CPE Report generated by make cpe-info (csv format)') + parser.add_argument('-u', dest='url', action='store', required=False, + help='(optional)URL to the NIST dict (official-cpe-dictionary_v2.3.xml.gz)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + cpedb = CPEDB() + url = CPE_XML_URL + if args.url: + url = args.url + cpedb.get_xml_dict(url) + print("Performing Target CPE Report Analysis...") + get_target_cpe_report(args.cpe_report, cpedb) + + +__main__()