| Message ID | 20210107133948.2997849-9-thomas.petazzoni@bootlin.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | CPE validation | expand |
Gregory / Thomas, On Thu, Jan 7, 2021 at 7:40 AM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > From: Matt Weber <matthew.weber@rockwellcollins.com> > > This script queries the list of CPE IDs for the packages of the > current configuration (based on the "make show-info" output), and: > > - for CPE IDs that do not have any matching entry in the CPE > database, it emits a warning > > - for CPE IDs that do have a matching entry, but not with the same > version, it generates a snippet of XML that can be used to propose > an updated version to NIST. > > Ref: NIST has a group email (cpe_dictionary@nist.gov) used to > recieve these version update and new entry xml files. They do > process the XML and provide feedback. In some cases they will > propose back something different where the vendor or version is > slightly different. > > Limitations > - Currently any use of non-number version identifiers isn't > supported by NIST as they use ranges to determine impact > of a CVE > - Any Linux version from a non-upstream is also not supported > without manually adjusting the information as the custom > kernel will more then likely not match the upstream version > used in the dictionary > > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) > create mode 100755 support/scripts/gen-missing-cpe > > diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe > new file mode 100755 > index 0000000000..22801ca488 > --- /dev/null > +++ b/support/scripts/gen-missing-cpe > @@ -0,0 +1,65 @@ > +#!/usr/bin/env python3 > + > +import argparse > +import sys > +import json > +import subprocess > +import os > +from cpedb import CPEDB, CPE > + > + > +def gen_update_xml_reports(cpes, cpedb, output): > + cpe_need_update = [] > + > + for cpe in cpes: > + result = cpedb.find(cpe) > + if not result: > + result = cpedb.find_partial(CPE.no_version(cpe)) > + if result: > + cpe_need_update.append(cpe) > + else: > + print("WARNING: no match found for '%s'" % cpe) I've tested with the following defconfig and I get the following output. I know for sure that some of these are missing/need update/match. i.e. busybox has a exact match and kmod is pending update and at 25. BR2_aarch64=y BR2_TOOLCHAIN_EXTERNAL=y BR2_LINUX_KERNEL=y BR2_LINUX_KERNEL_CUSTOM_VERSION=y BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.16.7" BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config" BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y BR2_TARGET_ROOTFS_EXT2=y # BR2_TARGET_ROOTFS_TAR is not set make missing-cpe mkdir -p /accts/mlweber1/tmp.G9papvL0Eg-buildroot/output/cpe-updates CPE: Setting up NIST dictionary CPE: Loading CACHED dictionary Getting list of CPE for enabled packages WARNING: no match found for 'cpe:2.3:a:kernel:util-linux:2.36.1:*:*:*:*:*:*:*' WARNING: no match found for 'cpe:2.3:a:busybox:busybox:1.32.0:*:*:*:*:*:*:*' WARNING: no match found for 'cpe:2.3:a:kernel:kmod:27:*:*:*:*:*:*:*' WARNING: no match found for 'cpe:2.3:a:gnu:zlib:1.2.11:*:*:*:*:*:*:*' WARNING: no match found for 'cpe:2.3:a:linux:linux_kernel:4.16.7:*:*:*:*:*:*:*' WARNING: no match found for 'cpe:2.3:a:openssl:openssl:1.1.1i:*:*:*:*:*:*:*' Generated 0 update files out of 6 CPEs Also I noticed in this build that I get a list of 6 CPEs but pkgstats doesn't list the same ones. it looks like the kernel dependencies of kmod / openssl / zlib don't get reflected in the per defconfig pkgstats. Just checking if that was a known bug. > + > + for cpe in cpe_need_update: > + xml = cpedb.gen_update_xml(cpe) > + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml' > + print("Generating %s" % fname) > + fp = open(os.path.join(output, fname), 'w+') > + fp.write(xml) > + fp.close() > + > + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes))) > + > + > +def get_cpe_ids(): > + print("Getting list of CPE for enabled packages") > + cmd = ["make", "--no-print-directory", "show-info"] > + js = json.loads(subprocess.check_output(cmd)) > + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v]) > + > + > +def resolvepath(path): > + return os.path.abspath(os.path.expanduser(path)) > + > + > +def parse_args(): > + parser = argparse.ArgumentParser() > + parser.add_argument('--output', dest='output', > + help='Path to the output CPE update files', type=resolvepath, required=True) > + parser.add_argument('--nvd-path', dest='nvd_path', > + help='Path to the local NVD database', type=resolvepath, required=True) > + return parser.parse_args() > + > + > +def __main__(): > + args = parse_args() > + if not os.path.isdir(args.output): > + print("ERROR: output directory %s does not exist" % args.output) > + sys.exit(1) > + cpedb = CPEDB(args.nvd_path) > + cpedb.get_xml_dict() > + cpes = get_cpe_ids() > + gen_update_xml_reports(cpes, cpedb, args.output) > + > + > +__main__() > -- > 2.29.2 >
Hello, On Thu, 7 Jan 2021 13:33:18 -0600 Matthew Weber via buildroot <buildroot@busybox.net> wrote: > I've tested with the following defconfig and I get the following > output. I know for sure that some of these are missing/need > update/match. i.e. busybox has a exact match and kmod is pending > update and at 25. > > BR2_aarch64=y > BR2_TOOLCHAIN_EXTERNAL=y > BR2_LINUX_KERNEL=y > BR2_LINUX_KERNEL_CUSTOM_VERSION=y > BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.16.7" > BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y > BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config" > BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y > BR2_TARGET_ROOTFS_EXT2=y > # BR2_TARGET_ROOTFS_TAR is not set > > > make missing-cpe > mkdir -p /accts/mlweber1/tmp.G9papvL0Eg-buildroot/output/cpe-updates > CPE: Setting up NIST dictionary > CPE: Loading CACHED dictionary > Getting list of CPE for enabled packages > WARNING: no match found for 'cpe:2.3:a:kernel:util-linux:2.36.1:*:*:*:*:*:*:*' > WARNING: no match found for 'cpe:2.3:a:busybox:busybox:1.32.0:*:*:*:*:*:*:*' > WARNING: no match found for 'cpe:2.3:a:kernel:kmod:27:*:*:*:*:*:*:*' > WARNING: no match found for 'cpe:2.3:a:gnu:zlib:1.2.11:*:*:*:*:*:*:*' > WARNING: no match found for 'cpe:2.3:a:linux:linux_kernel:4.16.7:*:*:*:*:*:*:*' > WARNING: no match found for 'cpe:2.3:a:openssl:openssl:1.1.1i:*:*:*:*:*:*:*' > Generated 0 update files out of 6 CPEs So here as well, the issue was that your .pkl cache was empty, due to parsing issue. This should be resolved in v2. > Also I noticed in this build that I get a list of 6 CPEs but pkgstats > doesn't list the same ones. it looks like the kernel dependencies of > kmod / openssl / zlib don't get reflected in the per defconfig > pkgstats. Just checking if that was a known bug. gen-missing-cpe simply gets the CPE IDs visible in the "make show-info" output, without caring whether things are a target or a host package, so it captures all packages that have a CPE ID. On the other hand, "make pkg-stats" looks at the package names in the "show-info" output and these contain "host-" prefixes for host packages which prevents those names from matching actual package names. I proposed a patch at http://lists.busybox.net/pipermail/buildroot/2021-January/300286.html, but Yann/Arnout made some suggestions on how to better handle this. It'll be part of my v2 of this series. Thanks! Thomas
diff --git a/support/scripts/gen-missing-cpe b/support/scripts/gen-missing-cpe new file mode 100755 index 0000000000..22801ca488 --- /dev/null +++ b/support/scripts/gen-missing-cpe @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 + +import argparse +import sys +import json +import subprocess +import os +from cpedb import CPEDB, CPE + + +def gen_update_xml_reports(cpes, cpedb, output): + cpe_need_update = [] + + for cpe in cpes: + result = cpedb.find(cpe) + if not result: + result = cpedb.find_partial(CPE.no_version(cpe)) + if result: + cpe_need_update.append(cpe) + else: + print("WARNING: no match found for '%s'" % cpe) + + for cpe in cpe_need_update: + xml = cpedb.gen_update_xml(cpe) + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml' + print("Generating %s" % fname) + fp = open(os.path.join(output, fname), 'w+') + fp.write(xml) + fp.close() + + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes))) + + +def get_cpe_ids(): + print("Getting list of CPE for enabled packages") + cmd = ["make", "--no-print-directory", "show-info"] + js = json.loads(subprocess.check_output(cmd)) + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v]) + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path)) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('--output', dest='output', + help='Path to the output CPE update files', type=resolvepath, required=True) + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath, required=True) + return parser.parse_args() + + +def __main__(): + args = parse_args() + if not os.path.isdir(args.output): + print("ERROR: output directory %s does not exist" % args.output) + sys.exit(1) + cpedb = CPEDB(args.nvd_path) + cpedb.get_xml_dict() + cpes = get_cpe_ids() + gen_update_xml_reports(cpes, cpedb, args.output) + + +__main__()