From patchwork Thu Jun 21 17:15:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932834 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyV1zlMz9s2L for ; Fri, 22 Jun 2018 03:15:45 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id EE61B8902D; Thu, 21 Jun 2018 17:15:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KXMvk1IGqzMB; Thu, 21 Jun 2018 17:15:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 4A76889007; Thu, 21 Jun 2018 17:15:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id DD90A1BFFD0 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DA6B188017 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UOeKBs77FgN3 for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs02.rockwellcollins.com (ch3vs02.rockwellcollins.com [205.175.226.29]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 50099881E3 for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.226.14]) by ch3vs02.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:37 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 1EF4F603EC; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:29 -0500 Message-Id: <1529601335-38937-2-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 1/7] cpe-info: new make target X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Similar to make legal-info, produce a csv delimited file containing all selected packages CPE identification. Have the pkg infra define CPE_ID_* defaults using the package name for the vendor and name as most CPE IDs seem to align with that assumption. Also use the pkg version as the CPE ID's version field. Signed-off-by: Matt Weber --- Changes v2 [Thomas P - Moved comment on conditionals back to this patchset where the conditional is created vs later v3 [Thomas P - Merged infra define CPE_ID_* into this patch - Report all packages vs restricting to just allowing based on if the VENDOR was set (v2). This now represents Thomas P's original idea to report everything. At first I felt I should restrict the reporting to those CPE IDs we had made sure were correct. Turns out we should have actually let the script handle fixing the CPEs and just make a complete design of this up front. [Matt - Moved to using the _project on all vendors instead of just name --- Makefile | 17 ++++++++++++++++- package/pkg-generic.mk | 13 +++++++++++++ package/pkg-utils.mk | 8 ++++++++ 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 8d25c8a..4cc95b7 100644 --- a/Makefile +++ b/Makefile @@ -146,7 +146,7 @@ nobuild_targets := source %-source \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \ graph-build graph-size list-defconfigs \ - savedefconfig printvars + savedefconfig printvars cpe-info %-cpe-info ifeq ($(MAKECMDGOALS),) BR_BUILDING = y else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),) @@ -233,6 +233,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings LEGAL_REPORT = $(LEGAL_INFO_DIR)/README +CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv BR2_CONFIG = $(CONFIG_DIR)/.config @@ -802,6 +803,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p mv .legal-info.sha256 legal-info.sha256) @echo "Legal info produced in $(LEGAL_INFO_DIR)" +.PHONY: cpe-info-clean +cpe-info-clean: + @rm -f $(CPE_MANIFEST_CSV) + +.PHONY: cpe-info-prepare +cpe-info-prepare: + @$(call MESSAGE,"Gathering CPE info") + @$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE) + +.PHONY: cpe-info +cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info) + @echo "CPE info produced in $(CPE_MANIFEST_CSV)" + .PHONY: show-targets show-targets: @echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS)) @@ -1070,6 +1084,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' cpe-info - generate info about security CPE identification' @echo ' printvars - dump all the internal variables' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build' diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 501dd3d..1bd3f47 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -870,6 +870,18 @@ else $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2) endif +$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project +$(2)_CPE_ID_NAME ?= $$($(2)_NAME) +$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) +$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) + +$(1)-cpe-info: PKG=$(2) +$(1)-cpe-info: +ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + @$$(call MESSAGE,"Collecting cpe info") + $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) +endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES) @@ -1011,6 +1023,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-clean-for-reconfigure \ $(1)-clean-for-reinstall \ $(1)-configure \ + $(1)-cpe-info \ $(1)-depends \ $(1)-dirclean \ $(1)-external-deps \ diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index c3acc22..11a2457 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -95,3 +95,11 @@ define legal-license-file # pkgname, pkgname-pkgver, pkgdir, filename, file-full } && \ cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4) endef + +# +# cpe-info helper functions +# + +define cpe-manifest # cpe, cve patched, pkg name, version, url + echo '"$(1)","$(2)","$(3)","$(4)","$(5)"' >>$(CPE_MANIFEST_CSV) +endef From patchwork Thu Jun 21 17:15:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932840 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyn3Mtrz9s2L for ; Fri, 22 Jun 2018 03:16:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 25DF289731; Thu, 21 Jun 2018 17:15:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EcAYHmPUmoHf; Thu, 21 Jun 2018 17:15:49 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A55AB8972D; Thu, 21 Jun 2018 17:15:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id DC9A81CF047 for ; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C18D788FF9 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3J82NxywrNBv for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs01.rockwellcollins.com (secvs01.rockwellcollins.com [205.175.225.240]) by whitealder.osuosl.org (Postfix) with ESMTPS id C35F288FFD for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) Received: from ofwgwc03.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.225.12]) by secvs01.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:37 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 23C376040B; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:30 -0500 Message-Id: <1529601335-38937-3-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 2/7] cpe-info: id prefix/suffix X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" There are two types of software cpe prefixes, one for applications and one for operating systems. Note: There is a third type for hardware. This patchset determines which should be used and stores that information with the package for later use when assembling the CPE report. There is also a suffix which we just default to wildcards at this point. Refs: https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf https://cpe.mitre.org/specification/ Signed-off-by: Matthew Weber --- Changes v1 -> v2 [Thomas P - Change to using a filter on pkg name value vs ifelse v3 [Arnout - Moved CPE prefix and suffix defines to package/Makefile.in --- package/Makefile.in | 4 ++++ package/pkg-generic.mk | 8 +++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/package/Makefile.in b/package/Makefile.in index a268016..8c7f37e 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -395,6 +395,10 @@ TARGET_CONFIGURE_ARGS = \ ################################################################################ +CPE_PREFIX_OS = cpe:2.3:o +CPE_PREFIX_APP = cpe:2.3:a +CPE_SUFFIX = *:*:*:*:*:*:* + ifeq ($(BR2_SYSTEM_ENABLE_NLS),y) NLS_OPTS = --enable-nls TARGET_NLS_DEPENDENCIES = host-gettext diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 1bd3f47..9badc54 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -875,11 +875,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME) $(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) $(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) +ifneq ($(filter linux linux-headers,$(1)),) +$(2)_CPE_PREFIX = $(CPE_PREFIX_OS) +else +$(2)_CPE_PREFIX = $(CPE_PREFIX_APP) +endif + $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") - $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) + $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) # legal-info: declare dependencies and set values used later for the manifest From patchwork Thu Jun 21 17:15:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932836 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyd4rPZz9s2R for ; Fri, 22 Jun 2018 03:15:53 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0EDD78471F; Thu, 21 Jun 2018 17:15:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FHEvBSeeQcOk; Thu, 21 Jun 2018 17:15:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7DE00846C9; Thu, 21 Jun 2018 17:15:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 7CF8C1BFFD0 for ; Thu, 21 Jun 2018 17:15:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5E8B388FFA for ; Thu, 21 Jun 2018 17:15:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JFq5DnQ3c9RB for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs01.rockwellcollins.com (secvs01.rockwellcollins.com [205.175.225.240]) by whitealder.osuosl.org (Postfix) with ESMTPS id 3085589003 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) Received: from ofwgwc03.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.225.12]) by secvs01.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:37 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 319656040C; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:31 -0500 Message-Id: <1529601335-38937-4-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 3/7] cpe-info: only report target pkgs X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The reporting of host packages causes some duplication and complicates what is really in the targets configuration. For the purpose of the first version of this patchset, its assumed that host packages aren't relevant for the configuration and we only report the target's contents. Signed-off-by: Matthew Weber --- Changes v1 -> v2 [Thomas P - select if target vs selecting not host v3 - Fixed host build error because cpe-info wasn't defined --- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 9badc54..5df3f07 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -883,10 +883,12 @@ endif $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: +ifeq ($$($(2)_TYPE),target) ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) +endif # ifeq ($$($(2)_TYPE),target) # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) From patchwork Thu Jun 21 17:15:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932839 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyk5Jycz9s2L for ; Fri, 22 Jun 2018 03:15:58 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 420018959C; Thu, 21 Jun 2018 17:15:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vifN62sTW1wB; Thu, 21 Jun 2018 17:15:49 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 9621E895A6; Thu, 21 Jun 2018 17:15:49 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6188B1BFFD0 for ; Thu, 21 Jun 2018 17:15:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0CEF088FFD for ; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BsoZtZhieP0d for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (da1vs02.rockwellcollins.com [205.175.227.29]) by whitealder.osuosl.org (Postfix) with ESMTPS id B41FE88FFA for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:38 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 369EA60487; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:32 -0500 Message-Id: <1529601335-38937-5-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 4/7] cpe-info: update manual for new pkg vars X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Provide guidance on setting up the *_CPE_* and *_CVE_* variables. Signed-off-by: Matt Weber --- Changes v2 [Thomas P - Reworded LIBFOO_CVE_PATCHED description [Matt W - Added definition for new preset variables to auto-gen the CPE ID - Added example LIBFOO_CPE_ID_VENDOR to LIBFOO v3 - Updated to make *_CPE_VENDOR optional - Changed wording around _CPE_ID as there is only one defined now --- docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------ 1 file changed, 74 insertions(+), 43 deletions(-) diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt index cc91e89..8620ee0 100644 --- a/docs/manual/adding-packages-generic.txt +++ b/docs/manual/adding-packages-generic.txt @@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts. 09: LIBFOO_SITE = http://www.foosoftware.org/download 10: LIBFOO_LICENSE = GPL-3.0+ 11: LIBFOO_LICENSE_FILES = COPYING -12: LIBFOO_INSTALL_STAGING = YES -13: LIBFOO_CONFIG_SCRIPTS = libfoo-config -14: LIBFOO_DEPENDENCIES = host-libaaa libbbb -15: -16: define LIBFOO_BUILD_CMDS -17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all -18: endef -19: -20: define LIBFOO_INSTALL_STAGING_CMDS -21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a -22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h -23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib -24: endef -25: -26: define LIBFOO_INSTALL_TARGET_CMDS -27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib -28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d -29: endef -30: -31: define LIBFOO_USERS -32: foo -1 libfoo -1 * - - - LibFoo daemon -33: endef -34: -35: define LIBFOO_DEVICES -36: /dev/foo c 666 0 0 42 0 - - - -37: endef -38: -39: define LIBFOO_PERMISSIONS -40: /bin/foo f 4755 foo libfoo - - - - - -41: endef -42: -43: $(eval $(generic-package)) +12: LIBFOO_CPE_ID_VENDOR = foosoftware +13: LIBFOO_INSTALL_STAGING = YES +14: LIBFOO_CONFIG_SCRIPTS = libfoo-config +15: LIBFOO_DEPENDENCIES = host-libaaa libbbb +16: +17: define LIBFOO_BUILD_CMDS +18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all +19: endef +20: +21: define LIBFOO_INSTALL_STAGING_CMDS +22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a +23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h +24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib +25: endef +26: +27: define LIBFOO_INSTALL_TARGET_CMDS +28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib +29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d +30: endef +31: +32: define LIBFOO_USERS +33: foo -1 libfoo -1 * - - - LibFoo daemon +34: endef +35: +36: define LIBFOO_DEVICES +37: /dev/foo c 666 0 0 42 0 - - - +38: endef +39: +40: define LIBFOO_PERMISSIONS +41: /bin/foo f 4755 foo libfoo - - - - - +42: endef +43: +44: $(eval $(generic-package)) -------------------------------- -The Makefile begins on line 7 to 11 with metadata information: the +The Makefile begins on line 7 to 12 with metadata information: the version of the package (+LIBFOO_VERSION+), the name of the tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended) the Internet location at which the tarball can be downloaded from -(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the -license text (+LIBFOO_LICENSE_FILES+). All variables must start with +(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the +license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability +analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with the same prefix, +LIBFOO_+ in this case. This prefix is always the uppercased version of the package name (see below to understand where the package name is defined). -On line 12, we specify that this package wants to install something to +On line 13, we specify that this package wants to install something to the staging space. This is often needed for libraries, since they must install header files and other development files in the staging space. This will ensure that the commands listed in the +LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed. -On line 13, we specify that there is some fixing to be done to some +On line 14, we specify that there is some fixing to be done to some of the 'libfoo-config' files that were installed during +LIBFOO_INSTALL_STAGING_CMDS+ phase. These *-config files are executable shell script files that are @@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \ -------------------------------- ================================ -On line 14, we specify the list of dependencies this package relies +On line 15, we specify the list of dependencies this package relies on. These dependencies are listed in terms of lower-case package names, which can be packages for the target (without the +host-+ prefix) or packages for the host (with the +host-+) prefix). Buildroot will ensure that all these packages are built and installed 'before' the current package starts its configuration. -The rest of the Makefile, lines 16..29, defines what should be done +The rest of the Makefile, lines 17..29, defines what should be done at the different steps of the package configuration, compilation and installation. +LIBFOO_BUILD_CMDS+ tells what steps should be performed to @@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which contains the directory where the source code of the package has been extracted. -On lines 31..43, we define a user that is used by this package (e.g. +On lines 32..44, we define a user that is used by this package (e.g. to run a daemon as non-root) (+LIBFOO_USERS+). -On line 35..37, we define a device-node file used by this package +On line 36..38, we define a device-node file used by this package (+LIBFOO_DEVICES+). -On line 39..41, we define the permissions to set to specific files +On line 40..42, we define the permissions to set to specific files installed by this package (+LIBFOO_PERMISSIONS+). -Finally, on line 43, we call the +generic-package+ function, which +Finally, on line 44, we call the +generic-package+ function, which generates, according to the variables defined previously, all the Makefile code necessary to make your package working. @@ -469,6 +471,35 @@ information is (assuming the package name is +libfoo+) : locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`, `/usr/lib/modules`, and `/usr/share`, which are automatically excluded. +* +LIBFOO_CPE_ID_VENDOR+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the vendor. By default it's set + to _project. + +* +LIBFOO_CPE_ID_NAME+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the name. By default it's set + to . + +* +LIBFOO_CPE_ID_VERSION+ + This variable is optional. By default it's set to . + +* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the + default case of a single package's Common Product Enumeration (CPE) + identification string. +make cpe-info+ copies all of these into a + +cpe-manifest.csv+ file. To identify a package's possible CPE, + the National Vunerability Database can be searched at + https://nvd.nist.gov/products/cpe/search. + +* +LIBFOO_CVE_PATCHED+ is a space-separated list of the package's Common + Vunerability Enumeration (CVE) identification strings. This list + enumerates CVEs which are fixed by patches added in Buildroot. This + allows the CPE reporting to provide additional detail on CVEs which + have been fixed, even if Buildroot is not yet using an updated upstream + release including the fix. This variable is optional. If it is not + defined, the +CVE PATCHED+ field will appear empty in the manifest + file for this package. + The recommended way to define these variables is to use the following syntax: From patchwork Thu Jun 21 17:15:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932833 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyR6qLWz9s2L for ; Fri, 22 Jun 2018 03:15:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 08B398821E; Thu, 21 Jun 2018 17:15:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BRWLaz6dOXSM; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7DF9E881E3; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 46A3A1BFFD0 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4284F881E7 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id a_zblzPSV_GE for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs04.rockwellcollins.com (ch3vs04.rockwellcollins.com [205.175.226.52]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 4F26388017 for ; Thu, 21 Jun 2018 17:15:38 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.226.14]) by ch3vs04.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:37 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 44F6060497; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:33 -0500 Message-Id: <1529601335-38937-6-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 5/7] support/scripts/cpedb.py: new CPE XML helper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data. Signed-off-by: Matthew Weber --- v5 [Ricardo - Fixed typo in join/split of cpe str without version - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Arnout - added pre-processing of cpe values into two sets, one with and one without version - Collectly with Ricardo, decided to move cpe class to this seperate script v1 -> v4 - No version --- support/scripts/cpedb.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 support/scripts/cpedb.py diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py new file mode 100644 index 0000000..77d1d17 --- /dev/null +++ b/support/scripts/cpedb.py @@ -0,0 +1,52 @@ +import sys +import urllib2 +import xmltodict +import gzip +from StringIO import StringIO + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +class CPEDB: + all_cpedb = dict() + all_cpes = set() + all_cpes_no_version = set() + + def get_xml_dict(self): + print("CPE: Fetching xml manifest...") + try: + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL) + print("CPE: Unzipping xml manifest...") + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read() + print("CPE: Converting xml manifest to dict...") + self.all_cpedb = xmltodict.parse(cpe_file) + + for cpe in self.all_cpedb['cpe-list']['cpe-item']: + cpe_str = cpe['cpe-23:cpe23-item']['@name'] + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + self.all_cpes.add(cpe_str) + self.all_cpes_no_version.add(cpe_str_no_version) + + except urllib2.HTTPError: + print("CPE: HTTP Error: %s" % CPE_XML_URL) + sys.exit(1) + except urllib2.URLError: + print("CPE: URL Error: %s" % CPE_XML_URL) + sys.exit(1) + + def find_partial(self, cpe_str): + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find(self, cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def get_cpe_no_version(self, cpe): + return ":".join(cpe.split(":")[:5]) + + def get_nvd_url(self, cpe_str): + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \ + urllib2.quote(cpe_str) + \ + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3" From patchwork Thu Jun 21 17:15:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932838 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyk1j0Jz9s2R for ; Fri, 22 Jun 2018 03:15:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A06F1228EE; Thu, 21 Jun 2018 17:15:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z-JWJSIz54I3; Thu, 21 Jun 2018 17:15:50 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id C34FB22216; Thu, 21 Jun 2018 17:15:50 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 379C71BFFD0 for ; Thu, 21 Jun 2018 17:15:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D952F88FF3 for ; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EdxLdD1yijcP for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (da1vs02.rockwellcollins.com [205.175.227.29]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1894E89000 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:38 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 49D106049F; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:34 -0500 Message-Id: <1529601335-38937-7-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 6/7] support/scripts/pkg-stats: add CPE reporting X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pkg status now includes CPE as an item reported in the html output (stat summary and for each pkg) Signed-off-by: Matthew Weber --- Changes v5 -> v6 - Rebased to capture formatting changes v4 -> v5 [Ricardo - Renamed patch to correctly match file name - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup [Arnout - Collectly with Ricardo, decided to move cpe report analysis to a seperate script and breakout a module that's imported for the cpedb class - Rename cpe_dict to instead be cpedb v3 -> v4 - Collapsed patch 5 and 6 together into this single patch [Eric - added except handling around file io - fixed condition where buildroot isn't generating a CPE string as part of the infra and output that is the case. (eventually these probably could be fixed but there aren't many at this point) [Ricardo - fixed patch naming and resolved flake8 issues - took the opportunity to also fix other flake8 syntax update suggestions - added except handling to have proper exits - cleaned up csv file header skippin - condensed partial cve string split - updated help txt as suggested - reworked output file requirement. Removed -o as required but added check if provided when -c isn't used v3 - New patch --- support/scripts/pkg-stats | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index b7b00e8..db58a75 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -24,6 +24,7 @@ from collections import defaultdict import re import subprocess import sys +from cpedb import CPEDB INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") @@ -32,6 +33,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_cpe_id = dict() def __init__(self, name, path): self.name = name @@ -43,6 +45,8 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.cpe_id = None + self.has_cpe = False def pkgvar(self): return self.name.upper().replace("-", "_") @@ -116,6 +120,26 @@ class Package: self.warnings = int(m.group(1)) return + def set_cpe_info(self, cpe_dict): + """ + Fills in the .has_cpe field + """ + var = self.pkgvar() + if var in self.all_cpe_id: + self.cpe_id = self.all_cpe_id[var] + if self.cpe_id is None: + # BR infra did not build a CPE ID for this pkg + # as it's most likely a host pkg + return + result = cpe_dict.find(self.cpe_id) + if not result: + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id)) + if result: + self.has_cpe = "Update" + # Unset case for has_cpe is assumed missing/does not exist + else: + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id) + def __eq__(self, other): return self.path == other.path @@ -254,6 +278,20 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value + # CPE ID + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_CPE_ID"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # Strip _CPE_ID + pkgvar = pkgvar[:-7] + if pkgvar in ("LINUX", "LINUX_HEADERS"): + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*" + else: + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*" + def calculate_stats(packages): stats = defaultdict(int) @@ -279,6 +317,12 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.has_cpe == "Update": + stats["update-cpe"] += 1 + elif pkg.has_cpe: + stats["cpe"] += 1 + else: + stats["no-cpe"] += 1 stats["patches"] += pkg.patch_count return stats @@ -422,6 +466,20 @@ def dump_html_pkg(f, pkg): f.write(" %d\n" % (" ".join(td_class), pkg.warnings)) + # CPE Valid + td_class = ["centered"] + if not pkg.has_cpe: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_cpe))) + elif pkg.has_cpe == "Update": + td_class.append("wrong") + f.write(" Update\n" % + (" ".join(td_class))) + else: + td_class.append("correct") + f.write(" %s\n" % + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe))) f.write(" \n") @@ -437,6 +495,7 @@ def dump_html_all_pkgs(f, packages): Hash file Current version Warnings +CPE Valid """) for pkg in sorted(packages): @@ -463,6 +522,12 @@ def dump_html_stats(f, stats): stats["hash"]) f.write(" Packages not having a hash file%s\n" % stats["no-hash"]) + f.write(" Packages having a registered CPE%s\n" % + stats["cpe"]) + f.write(" Packages needing CPE update%s\n" % + stats["update-cpe"]) + f.write(" Packages missing a registered CPE%s\n" % + stats["no-cpe"]) f.write(" Total number of patches%s\n" % stats["patches"]) f.write("\n") @@ -505,6 +570,8 @@ def __main__(): package_list = args.packages.split(",") else: package_list = None + cpedb = CPEDB() + cpedb.get_xml_dict() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting package make info ...") @@ -517,6 +584,7 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() + pkg.set_cpe_info(cpedb) print("Calculate stats") stats = calculate_stats(packages) print("Write HTML") From patchwork Thu Jun 21 17:15:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 932837 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BSyf36dcz9s2L for ; Fri, 22 Jun 2018 03:15:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 8C997897CB; Thu, 21 Jun 2018 17:15:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2COD1bUdFcD1; Thu, 21 Jun 2018 17:15:47 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id D31CF8958B; Thu, 21 Jun 2018 17:15:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C6C341BFFD0 for ; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C367022210 for ; Thu, 21 Jun 2018 17:15:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dTeq1jclhfRU for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs01.rockwellcollins.com (secvs01.rockwellcollins.com [205.175.225.240]) by silver.osuosl.org (Postfix) with ESMTPS id 88742221F9 for ; Thu, 21 Jun 2018 17:15:39 +0000 (UTC) Received: from ofwgwc03.rockwellcollins.com (HELO crulimr01.rockwellcollins.com) ([205.175.225.12]) by secvs01.rockwellcollins.com with ESMTP; 21 Jun 2018 12:15:37 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr01.rockwellcollins.com (Postfix) with ESMTP id 4ED4F604A0; Thu, 21 Jun 2018 12:15:37 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Thu, 21 Jun 2018 12:15:35 -0500 Message-Id: <1529601335-38937-8-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1529601335-38937-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v6 7/7] support/scripts/cpe-report: new script X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The script supports looking up all the CPEs provided in a make cpe-info csv file export from a target Buildroot build. It checks the current version and suggests a CPE needs update or possibly initial submission to NIST. Signed-off-by: Matthew Weber --- Changes v5 [Ricardo - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Ricardo/Arnout - Collectly, decided to move cpe report analysis to this script and use a seperate module cpedb class [Arnout - Rename cpe_dict to instead be cpedb v1 -> v4 - Patch did not exist and was part of pkg-stats file --- support/scripts/cpe-report | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 support/scripts/cpe-report diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report new file mode 100755 index 0000000..036eab2 --- /dev/null +++ b/support/scripts/cpe-report @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +import argparse +import sys +import csv +from cpedb import CPEDB + + +def get_target_cpe_report(cpe_report_file, cpedb): + report_cpe_exact_match = "" + report_cpe_needing_update = "" + report_cpe_missing = "" + + print("CPE: Checking for matches...") + try: + with open(cpe_report_file) as cpe_file: + cpe_list = csv.reader(cpe_file) + next(cpe_list) # make cpe-info has a one line header + for cpe in cpe_list: + result = cpedb.find(cpe[0]) + if not result: + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) + if not result: + report_cpe_missing += cpe[0] + "\n" + else: + report_cpe_needing_update += cpe[0] + "\n" + else: + report_cpe_exact_match += cpe[0] + "\n" + except (OSError, IOError) as e: + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) + sys.exit(1) + + print("CPE: Found EXACT match:\n" + report_cpe_exact_match) + print("CPE: Found but REQUIRES UPDATE:\n" + report_cpe_needing_update) + print("CPE: Not found (proposing the following to be added):\n" + report_cpe_missing) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-c', dest='cpe_report', action='store', required=True, + help='CPE Report generated by make cpe-info (csv format)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + cpedb = CPEDB() + cpedb.get_xml_dict() + print("Performing Target CPE Report Analysis...") + get_target_cpe_report(args.cpe_report, cpedb) + + +__main__()