From patchwork Fri May 18 03:13:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 916000 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40nCtc6P1mz9s33 for ; Fri, 18 May 2018 13:13:44 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9440B874D3; Fri, 18 May 2018 03:13:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3y272RS3YdEJ; Fri, 18 May 2018 03:13:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D3AD98750F; Fri, 18 May 2018 03:13:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 721421CF1CB for ; Fri, 18 May 2018 03:13:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6F6278747C for ; Fri, 18 May 2018 03:13:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VAghowTxPYZx for ; Fri, 18 May 2018 03:13:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (da1vs02.rockwellcollins.com [205.175.227.29]) by fraxinus.osuosl.org (Postfix) with ESMTPS id B25D3874D3 for ; Fri, 18 May 2018 03:13:31 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO dtulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 17 May 2018 22:13:31 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by dtulimr02.rockwellcollins.com (Postfix) with ESMTP id E853820081; Thu, 17 May 2018 22:13:30 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 17 May 2018 22:13:18 -0500 Message-Id: <1526613200-48452-6-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1526613200-48452-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1526613200-48452-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v5 5/7] support/scripts/cpedb.py: new CPE XML helper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data. Signed-off-by: Matthew Weber --- v5 [Ricardo - Fixed typo in join/split of cpe str without version - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Arnout - added pre-processing of cpe values into two sets, one with and one without version - Collectly with Ricardo, decided to move cpe class to this seperate script v1 -> v4 - No version --- support/scripts/cpedb.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 support/scripts/cpedb.py diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py new file mode 100644 index 0000000..77d1d17 --- /dev/null +++ b/support/scripts/cpedb.py @@ -0,0 +1,52 @@ +import sys +import urllib2 +import xmltodict +import gzip +from StringIO import StringIO + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +class CPEDB: + all_cpedb = dict() + all_cpes = set() + all_cpes_no_version = set() + + def get_xml_dict(self): + print("CPE: Fetching xml manifest...") + try: + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL) + print("CPE: Unzipping xml manifest...") + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read() + print("CPE: Converting xml manifest to dict...") + self.all_cpedb = xmltodict.parse(cpe_file) + + for cpe in self.all_cpedb['cpe-list']['cpe-item']: + cpe_str = cpe['cpe-23:cpe23-item']['@name'] + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + self.all_cpes.add(cpe_str) + self.all_cpes_no_version.add(cpe_str_no_version) + + except urllib2.HTTPError: + print("CPE: HTTP Error: %s" % CPE_XML_URL) + sys.exit(1) + except urllib2.URLError: + print("CPE: URL Error: %s" % CPE_XML_URL) + sys.exit(1) + + def find_partial(self, cpe_str): + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find(self, cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def get_cpe_no_version(self, cpe): + return ":".join(cpe.split(":")[:5]) + + def get_nvd_url(self, cpe_str): + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \ + urllib2.quote(cpe_str) + \ + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"