From patchwork Wed Oct 17 21:41:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985552 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5Hd5YYSz9sCT for ; Thu, 18 Oct 2018 08:42:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 43D7F226B8; Wed, 17 Oct 2018 21:42:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I1Pa4C+psbSs; Wed, 17 Oct 2018 21:42:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 1DC142273B; Wed, 17 Oct 2018 21:42:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 898101BF59B for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 87483857BD for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9rWw2pe251F5 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs04.rockwellcollins.com (da1vs04.rockwellcollins.com [205.175.227.52]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E2B4F858CD for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs04.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id A36E660110; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:53 -0500 Message-Id: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 Subject: [Buildroot] [PATCH v7 01/10] cpe-info: new make target X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Similar to make legal-info, produce a csv delimited file containing all selected packages CPE identification. Have the pkg infra define CPE_ID_* defaults using the package name for the vendor and name as most CPE IDs seem to align with that assumption. Also use the pkg version as the CPE ID's version field. Signed-off-by: Matt Weber --- Changes v4 -> v7 - No change v3 [Thomas P - Merged infra define CPE_ID_* into this patch - Report all packages vs restricting to just allowing based on if the VENDOR was set (v2). This now represents Thomas P's original idea to report everything. At first I felt I should restrict the reporting to those CPE IDs we had made sure were correct. Turns out we should have actually let the script handle fixing the CPEs and just make a complete design of this up front. [Matt - Moved to using the _project on all vendors instead of just name v2 [Thomas P - Moved comment on conditionals back to this patchset where the conditional is created vs later --- Makefile | 17 ++++++++++++++++- package/pkg-generic.mk | 13 +++++++++++++ package/pkg-utils.mk | 8 ++++++++ 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 82dd76e..24f4e97 100644 --- a/Makefile +++ b/Makefile @@ -146,7 +146,7 @@ nobuild_targets := source %-source \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \ graph-build graph-size list-defconfigs \ - savedefconfig printvars + savedefconfig printvars cpe-info %-cpe-info ifeq ($(MAKECMDGOALS),) BR_BUILDING = y else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),) @@ -233,6 +233,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings LEGAL_REPORT = $(LEGAL_INFO_DIR)/README +CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv BR2_CONFIG = $(CONFIG_DIR)/.config @@ -814,6 +815,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p mv .legal-info.sha256 legal-info.sha256) @echo "Legal info produced in $(LEGAL_INFO_DIR)" +.PHONY: cpe-info-clean +cpe-info-clean: + @rm -f $(CPE_MANIFEST_CSV) + +.PHONY: cpe-info-prepare +cpe-info-prepare: + @$(call MESSAGE,"Gathering CPE info") + @$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE) + +.PHONY: cpe-info +cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info) + @echo "CPE info produced in $(CPE_MANIFEST_CSV)" + .PHONY: show-targets show-targets: @echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS)) @@ -1083,6 +1097,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' cpe-info - generate info about security CPE identification' @echo ' printvars - dump all the internal variables' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build' diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index daf2459..503861b 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -870,6 +870,18 @@ else $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2) endif +$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project +$(2)_CPE_ID_NAME ?= $$($(2)_NAME) +$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) +$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) + +$(1)-cpe-info: PKG=$(2) +$(1)-cpe-info: +ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + @$$(call MESSAGE,"Collecting cpe info") + $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) +endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES) @@ -1011,6 +1023,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-clean-for-reconfigure \ $(1)-clean-for-reinstall \ $(1)-configure \ + $(1)-cpe-info \ $(1)-depends \ $(1)-dirclean \ $(1)-external-deps \ diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index c3acc22..11a2457 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -95,3 +95,11 @@ define legal-license-file # pkgname, pkgname-pkgver, pkgdir, filename, file-full } && \ cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4) endef + +# +# cpe-info helper functions +# + +define cpe-manifest # cpe, cve patched, pkg name, version, url + echo '"$(1)","$(2)","$(3)","$(4)","$(5)"' >>$(CPE_MANIFEST_CSV) +endef From patchwork Wed Oct 17 21:41:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985553 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5Hg0kXYz9sBN for ; Thu, 18 Oct 2018 08:42:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A9ED286AEB; Wed, 17 Oct 2018 21:42:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GRjj2YWalTby; Wed, 17 Oct 2018 21:42:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id CC1CC86B0E; Wed, 17 Oct 2018 21:42:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id B26891BF3E3 for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B041187680 for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c+zejoA1v7Sw for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs03.rockwellcollins.com (da1vs03.rockwellcollins.com [205.175.227.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id DDD72875E2 for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs03.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id A784260111; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:54 -0500 Message-Id: <1539812522-7171-2-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 02/10] cpe-info: id prefix/suffix X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" There are two types of software cpe prefixes, one for applications and one for operating systems. Note: There is a third type for hardware. This patchset determines which should be used and stores that information with the package for later use when assembling the CPE report. There is also a suffix which we just default to wildcards at this point. Refs: https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf https://cpe.mitre.org/specification/ Signed-off-by: Matthew Weber --- Changes v4 -> v7 - None v3 [Arnout - Moved CPE prefix and suffix defines to package/Makefile.in v1 -> v2 [Thomas P - Change to using a filter on pkg name value vs ifelse --- package/Makefile.in | 4 ++++ package/pkg-generic.mk | 8 +++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/package/Makefile.in b/package/Makefile.in index abfdb81..a476c0b 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -393,6 +393,10 @@ TARGET_CONFIGURE_ARGS = \ ################################################################################ +CPE_PREFIX_OS = cpe:2.3:o +CPE_PREFIX_APP = cpe:2.3:a +CPE_SUFFIX = *:*:*:*:*:*:* + ifeq ($(BR2_SYSTEM_ENABLE_NLS),y) NLS_OPTS = --enable-nls TARGET_NLS_DEPENDENCIES = host-gettext diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 503861b..e970a66 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -875,11 +875,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME) $(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) $(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) +ifneq ($(filter linux linux-headers,$(1)),) +$(2)_CPE_PREFIX = $(CPE_PREFIX_OS) +else +$(2)_CPE_PREFIX = $(CPE_PREFIX_APP) +endif + $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") - $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) + $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) # legal-info: declare dependencies and set values used later for the manifest From patchwork Wed Oct 17 21:41:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985546 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HS06YHz9sBN for ; Thu, 18 Oct 2018 08:42:10 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9254C8618F; Wed, 17 Oct 2018 21:42:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OE_3s_c0qVGx; Wed, 17 Oct 2018 21:42:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0FFAA858CD; Wed, 17 Oct 2018 21:42:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id BA7181BF59B for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B795187680 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m3e0lJ2sTRsn for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs03.rockwellcollins.com (da1vs03.rockwellcollins.com [205.175.227.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id 298828763C for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs03.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id AB9286016A; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:55 -0500 Message-Id: <1539812522-7171-3-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 03/10] cpe-info: only report target pkgs X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The reporting of host packages causes some duplication and complicates what is really in the targets configuration. For the purpose of the first version of this patchset, its assumed that host packages aren't relevant for the configuration and we only report the target's contents. Signed-off-by: Matthew Weber --- Changes v4 -> v7 - None v3 - Fixed host build error because cpe-info wasn't defined v1 -> v2 [Thomas P - select if target vs selecting not host --- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index e970a66..aba41dc 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -883,10 +883,12 @@ endif $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: +ifeq ($$($(2)_TYPE),target) ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) +endif # ifeq ($$($(2)_TYPE),target) # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) From patchwork Wed Oct 17 21:41:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985555 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5Hl0vMGz9sBn for ; Thu, 18 Oct 2018 08:42:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 99B0A2E8FE; Wed, 17 Oct 2018 21:42:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ujJ5auK7sSDd; Wed, 17 Oct 2018 21:42:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 84F5D2E9B3; Wed, 17 Oct 2018 21:42:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 9EBD31BF3E3 for ; Wed, 17 Oct 2018 21:42:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 99E5E22767 for ; Wed, 17 Oct 2018 21:42:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7wgDEcoteQ6M for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (da1vs02.rockwellcollins.com [205.175.227.29]) by silver.osuosl.org (Postfix) with ESMTPS id C6CEE226B8 for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:04 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id AFE9860173; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:56 -0500 Message-Id: <1539812522-7171-4-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 04/10] cpe-info: update manual for new pkg vars X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Provide guidance on setting up the *_CPE_* and *_CVE_* variables. Signed-off-by: Matt Weber --- Changes v4 -> v7 - None v3 - Updated to make *_CPE_VENDOR optional - Changed wording around _CPE_ID as there is only one defined now v2 [Thomas P - Reworded LIBFOO_CVE_PATCHED description [Matt W - Added definition for new preset variables to auto-gen the CPE ID - Added example LIBFOO_CPE_ID_VENDOR to LIBFOO --- docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------ 1 file changed, 74 insertions(+), 43 deletions(-) diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt index 7be1754..3c2ad66 100644 --- a/docs/manual/adding-packages-generic.txt +++ b/docs/manual/adding-packages-generic.txt @@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts. 09: LIBFOO_SITE = http://www.foosoftware.org/download 10: LIBFOO_LICENSE = GPL-3.0+ 11: LIBFOO_LICENSE_FILES = COPYING -12: LIBFOO_INSTALL_STAGING = YES -13: LIBFOO_CONFIG_SCRIPTS = libfoo-config -14: LIBFOO_DEPENDENCIES = host-libaaa libbbb -15: -16: define LIBFOO_BUILD_CMDS -17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all -18: endef -19: -20: define LIBFOO_INSTALL_STAGING_CMDS -21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a -22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h -23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib -24: endef -25: -26: define LIBFOO_INSTALL_TARGET_CMDS -27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib -28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d -29: endef -30: -31: define LIBFOO_USERS -32: foo -1 libfoo -1 * - - - LibFoo daemon -33: endef -34: -35: define LIBFOO_DEVICES -36: /dev/foo c 666 0 0 42 0 - - - -37: endef -38: -39: define LIBFOO_PERMISSIONS -40: /bin/foo f 4755 foo libfoo - - - - - -41: endef -42: -43: $(eval $(generic-package)) +12: LIBFOO_CPE_ID_VENDOR = foosoftware +13: LIBFOO_INSTALL_STAGING = YES +14: LIBFOO_CONFIG_SCRIPTS = libfoo-config +15: LIBFOO_DEPENDENCIES = host-libaaa libbbb +16: +17: define LIBFOO_BUILD_CMDS +18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all +19: endef +20: +21: define LIBFOO_INSTALL_STAGING_CMDS +22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a +23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h +24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib +25: endef +26: +27: define LIBFOO_INSTALL_TARGET_CMDS +28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib +29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d +30: endef +31: +32: define LIBFOO_USERS +33: foo -1 libfoo -1 * - - - LibFoo daemon +34: endef +35: +36: define LIBFOO_DEVICES +37: /dev/foo c 666 0 0 42 0 - - - +38: endef +39: +40: define LIBFOO_PERMISSIONS +41: /bin/foo f 4755 foo libfoo - - - - - +42: endef +43: +44: $(eval $(generic-package)) -------------------------------- -The Makefile begins on line 7 to 11 with metadata information: the +The Makefile begins on line 7 to 12 with metadata information: the version of the package (+LIBFOO_VERSION+), the name of the tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended) the Internet location at which the tarball can be downloaded from -(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the -license text (+LIBFOO_LICENSE_FILES+). All variables must start with +(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the +license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability +analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with the same prefix, +LIBFOO_+ in this case. This prefix is always the uppercased version of the package name (see below to understand where the package name is defined). -On line 12, we specify that this package wants to install something to +On line 13, we specify that this package wants to install something to the staging space. This is often needed for libraries, since they must install header files and other development files in the staging space. This will ensure that the commands listed in the +LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed. -On line 13, we specify that there is some fixing to be done to some +On line 14, we specify that there is some fixing to be done to some of the 'libfoo-config' files that were installed during +LIBFOO_INSTALL_STAGING_CMDS+ phase. These *-config files are executable shell script files that are @@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \ -------------------------------- ================================ -On line 14, we specify the list of dependencies this package relies +On line 15, we specify the list of dependencies this package relies on. These dependencies are listed in terms of lower-case package names, which can be packages for the target (without the +host-+ prefix) or packages for the host (with the +host-+) prefix). Buildroot will ensure that all these packages are built and installed 'before' the current package starts its configuration. -The rest of the Makefile, lines 16..29, defines what should be done +The rest of the Makefile, lines 17..29, defines what should be done at the different steps of the package configuration, compilation and installation. +LIBFOO_BUILD_CMDS+ tells what steps should be performed to @@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which contains the directory where the source code of the package has been extracted. -On lines 31..43, we define a user that is used by this package (e.g. +On lines 32..44, we define a user that is used by this package (e.g. to run a daemon as non-root) (+LIBFOO_USERS+). -On line 35..37, we define a device-node file used by this package +On line 36..38, we define a device-node file used by this package (+LIBFOO_DEVICES+). -On line 39..41, we define the permissions to set to specific files +On line 40..42, we define the permissions to set to specific files installed by this package (+LIBFOO_PERMISSIONS+). -Finally, on line 43, we call the +generic-package+ function, which +Finally, on line 44, we call the +generic-package+ function, which generates, according to the variables defined previously, all the Makefile code necessary to make your package working. @@ -482,6 +484,35 @@ not and can not work as people would expect it should: locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`, `/usr/lib/modules`, and `/usr/share`, which are automatically excluded. +* +LIBFOO_CPE_ID_VENDOR+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the vendor. By default it's set + to _project. + +* +LIBFOO_CPE_ID_NAME+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the name. By default it's set + to . + +* +LIBFOO_CPE_ID_VERSION+ + This variable is optional. By default it's set to . + +* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the + default case of a single package's Common Product Enumeration (CPE) + identification string. +make cpe-info+ copies all of these into a + +cpe-manifest.csv+ file. To identify a package's possible CPE, + the National Vunerability Database can be searched at + https://nvd.nist.gov/products/cpe/search. + +* +LIBFOO_CVE_PATCHED+ is a space-separated list of the package's Common + Vunerability Enumeration (CVE) identification strings. This list + enumerates CVEs which are fixed by patches added in Buildroot. This + allows the CPE reporting to provide additional detail on CVEs which + have been fixed, even if Buildroot is not yet using an updated upstream + release including the fix. This variable is optional. If it is not + defined, the +CVE PATCHED+ field will appear empty in the manifest + file for this package. + The recommended way to define these variables is to use the following syntax: From patchwork Wed Oct 17 21:41:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985548 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HT1B3Mz9sBq for ; Thu, 18 Oct 2018 08:42:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4BE222EA91; Wed, 17 Oct 2018 21:42:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YENRkcFVClbn; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 541BB22767; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id AC19E1BF3E3 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id A9C31882BF for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u6M4BAIxrHCJ for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs04.rockwellcollins.com (ch3vs04.rockwellcollins.com [205.175.226.52]) by hemlock.osuosl.org (Postfix) with ESMTPS id BB34E882B2 for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs04.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id B488A6018D; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:57 -0500 Message-Id: <1539812522-7171-5-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 05/10] support/scripts/cpedb.py: new CPE XML helper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data. Signed-off-by: Matthew Weber --- v5 -> v7 - No change v5 [Ricardo - Fixed typo in join/split of cpe str without version - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Arnout - added pre-processing of cpe values into two sets, one with and one without version - Collectly with Ricardo, decided to move cpe class to this seperate script v1 -> v4 - No version --- support/scripts/cpedb.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 support/scripts/cpedb.py diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py new file mode 100644 index 0000000..77d1d17 --- /dev/null +++ b/support/scripts/cpedb.py @@ -0,0 +1,52 @@ +import sys +import urllib2 +import xmltodict +import gzip +from StringIO import StringIO + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +class CPEDB: + all_cpedb = dict() + all_cpes = set() + all_cpes_no_version = set() + + def get_xml_dict(self): + print("CPE: Fetching xml manifest...") + try: + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL) + print("CPE: Unzipping xml manifest...") + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read() + print("CPE: Converting xml manifest to dict...") + self.all_cpedb = xmltodict.parse(cpe_file) + + for cpe in self.all_cpedb['cpe-list']['cpe-item']: + cpe_str = cpe['cpe-23:cpe23-item']['@name'] + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + self.all_cpes.add(cpe_str) + self.all_cpes_no_version.add(cpe_str_no_version) + + except urllib2.HTTPError: + print("CPE: HTTP Error: %s" % CPE_XML_URL) + sys.exit(1) + except urllib2.URLError: + print("CPE: URL Error: %s" % CPE_XML_URL) + sys.exit(1) + + def find_partial(self, cpe_str): + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find(self, cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def get_cpe_no_version(self, cpe): + return ":".join(cpe.split(":")[:5]) + + def get_nvd_url(self, cpe_str): + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \ + urllib2.quote(cpe_str) + \ + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3" From patchwork Wed Oct 17 21:41:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985550 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HZ4zQ3z9sBn for ; Thu, 18 Oct 2018 08:42:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0E86987697; Wed, 17 Oct 2018 21:42:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id exji0wf48bJu; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 5FB4E8795E; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id DFC3B1BF3E3 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id DDDA5882B2 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r6RBUc0lXS6Q for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs01.rockwellcollins.com (ch3vs01.rockwellcollins.com [205.175.226.27]) by hemlock.osuosl.org (Postfix) with ESMTPS id C8947882B8 for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs01.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:04 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id B8C6560191; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:58 -0500 Message-Id: <1539812522-7171-6-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 06/10] support/scripts/pkg-stats: add CPE reporting X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pkg status now includes CPE as an item reported in the html output (stat summary and for each pkg) Signed-off-by: Matthew Weber --- Changes v6 -> v7 - Rebased to work after url checking code was added upstream v5 -> v6 - Rebased to capture formatting changes v4 -> v5 [Ricardo - Renamed patch to correctly match file name - Removed extra prints as they aren't needed when we have the output reports/stdout - Updated v4 comments about general flake formatting cleanup [Arnout - Collectly with Ricardo, decided to move cpe report analysis to a seperate script and breakout a module that's imported for the cpedb class - Rename cpe_dict to instead be cpedb v3 -> v4 - Collapsed patch 5 and 6 together into this single patch [Eric - added except handling around file io - fixed condition where buildroot isn't generating a CPE string as part of the infra and output that is the case. (eventually these probably could be fixed but there aren't many at this point) [Ricardo - fixed patch naming and resolved flake8 issues - took the opportunity to also fix other flake8 syntax update suggestions - added except handling to have proper exits - cleaned up csv file header skippin - condensed partial cve string split - updated help txt as suggested - reworked output file requirement. Removed -o as required but added check if provided when -c isn't used v3 - New patch --- support/scripts/pkg-stats | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index d0b06b1..21d1767 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -26,6 +26,7 @@ import subprocess import sys import requests # URL checking from multiprocessing import Pool +from cpedb import CPEDB INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile("\s*https?://\S*\s*$") @@ -35,6 +36,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_cpe_id = dict() def __init__(self, name, path): self.name = name @@ -49,6 +51,8 @@ class Package: self.url = None self.url_status = None self.url_worker = None + self.cpe_id = None + self.has_cpe = False def pkgvar(self): return self.name.upper().replace("-", "_") @@ -139,6 +143,26 @@ class Package: self.warnings = int(m.group(1)) return + def set_cpe_info(self, cpe_dict): + """ + Fills in the .has_cpe field + """ + var = self.pkgvar() + if var in self.all_cpe_id: + self.cpe_id = self.all_cpe_id[var] + if self.cpe_id is None: + # BR infra did not build a CPE ID for this pkg + # as it's most likely a host pkg + return + result = cpe_dict.find(self.cpe_id) + if not result: + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id)) + if result: + self.has_cpe = "Update" + # Unset case for has_cpe is assumed missing/does not exist + else: + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id) + def __eq__(self, other): return self.path == other.path @@ -277,6 +301,20 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value + # CPE ID + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_CPE_ID"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # Strip _CPE_ID + pkgvar = pkgvar[:-7] + if pkgvar in ("LINUX", "LINUX_HEADERS"): + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*" + else: + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*" + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in": @@ -322,6 +360,12 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.has_cpe == "Update": + stats["update-cpe"] += 1 + elif pkg.has_cpe: + stats["cpe"] += 1 + else: + stats["no-cpe"] += 1 stats["patches"] += pkg.patch_count return stats @@ -488,6 +532,20 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % (" ".join(td_class), url_str)) + # CPE Valid + td_class = ["centered"] + if not pkg.has_cpe: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_cpe))) + elif pkg.has_cpe == "Update": + td_class.append("wrong") + f.write(" Update\n" % + (" ".join(td_class))) + else: + td_class.append("correct") + f.write(" %s\n" % + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe))) f.write(" \n") @@ -504,6 +562,7 @@ def dump_html_all_pkgs(f, packages): Current version Warnings Upstream URL +CPE Valid """) for pkg in sorted(packages): @@ -530,6 +589,12 @@ def dump_html_stats(f, stats): stats["hash"]) f.write(" Packages not having a hash file%s\n" % stats["no-hash"]) + f.write(" Packages having a registered CPE%s\n" % + stats["cpe"]) + f.write(" Packages needing CPE update%s\n" % + stats["update-cpe"]) + f.write(" Packages missing a registered CPE%s\n" % + stats["no-cpe"]) f.write(" Total number of patches%s\n" % stats["patches"]) f.write("\n") @@ -572,6 +637,8 @@ def __main__(): package_list = args.packages.split(",") else: package_list = None + cpedb = CPEDB() + cpedb.get_xml_dict() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting package make info ...") @@ -585,6 +652,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_cpe_info(cpedb) print("Checking URL status") check_package_urls(packages) print("Calculate stats") From patchwork Wed Oct 17 21:41:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985554 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5Hh0K99z9sBn for ; Thu, 18 Oct 2018 08:42:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 25F9888318; Wed, 17 Oct 2018 21:42:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WPGJmSrnDRK5; Wed, 17 Oct 2018 21:42:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 6CE59882FE; Wed, 17 Oct 2018 21:42:18 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 7EA551BF3E3 for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7C83C87680 for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A4HdmyuhpkoI for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs03.rockwellcollins.com (da1vs03.rockwellcollins.com [205.175.227.47]) by whitealder.osuosl.org (Postfix) with ESMTPS id 690C98770C for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs03.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id BD222601E5; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:41:59 -0500 Message-Id: <1539812522-7171-7-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 07/10] support/scripts/cpe-report: new script X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The script supports looking up all the CPEs provided in a make cpe-info csv file export from a target Buildroot build. It checks the current version and suggests a CPE needs update or possibly initial submission to NIST. Limitations - Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE - Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary Signed-off-by: Matthew Weber --- Changes v5 -> v7 - No change v5 [Ricardo - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Ricardo/Arnout - Collectly, decided to move cpe report analysis to this script and use a seperate module cpedb class [Arnout - Rename cpe_dict to instead be cpedb v1 -> v4 - Patch did not exist and was part of pkg-stats file --- support/scripts/cpe-report | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 support/scripts/cpe-report diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report new file mode 100755 index 0000000..036eab2 --- /dev/null +++ b/support/scripts/cpe-report @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +import argparse +import sys +import csv +from cpedb import CPEDB + + +def get_target_cpe_report(cpe_report_file, cpedb): + report_cpe_exact_match = "" + report_cpe_needing_update = "" + report_cpe_missing = "" + + print("CPE: Checking for matches...") + try: + with open(cpe_report_file) as cpe_file: + cpe_list = csv.reader(cpe_file) + next(cpe_list) # make cpe-info has a one line header + for cpe in cpe_list: + result = cpedb.find(cpe[0]) + if not result: + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) + if not result: + report_cpe_missing += cpe[0] + "\n" + else: + report_cpe_needing_update += cpe[0] + "\n" + else: + report_cpe_exact_match += cpe[0] + "\n" + except (OSError, IOError) as e: + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) + sys.exit(1) + + print("CPE: Found EXACT match:\n" + report_cpe_exact_match) + print("CPE: Found but REQUIRES UPDATE:\n" + report_cpe_needing_update) + print("CPE: Not found (proposing the following to be added):\n" + report_cpe_missing) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-c', dest='cpe_report', action='store', required=True, + help='CPE Report generated by make cpe-info (csv format)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + cpedb = CPEDB() + cpedb.get_xml_dict() + print("Performing Target CPE Report Analysis...") + get_target_cpe_report(args.cpe_report, cpedb) + + +__main__() From patchwork Wed Oct 17 21:42:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985551 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5Hc3vLKz9sBn for ; Thu, 18 Oct 2018 08:42:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id AF7A5858CD; Wed, 17 Oct 2018 21:42:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xaj5-N8ESoun; Wed, 17 Oct 2018 21:42:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id BB7C185D4B; Wed, 17 Oct 2018 21:42:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 255011BF3E3 for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 22600857BD for ; Wed, 17 Oct 2018 21:42:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tQE_-pSMLDdz for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs04.rockwellcollins.com (da1vs04.rockwellcollins.com [205.175.227.52]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2F2B085B37 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs04.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id CC24C601F8; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:42:00 -0500 Message-Id: <1539812522-7171-8-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 08/10] support/scripts/cpe-report: add NIST xml generation X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bryce Ferguson , Paresh Chaudhary MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Bryce Ferguson Hooks to generate the NIST xml files required to propose an update to the database. The generation uses existing Buildroot information to create the submission information. This design uses the generated cpe-info csv and a path to the relevant buildroot clone for access to the packages/dir structure. Signed-off-by: Paresh Chaudhary Signed-off-by: Bryce Ferguson Signed-off-by: Matt Weber --- Changes v7 - New --- support/scripts/cpe-report | 17 ++++-- support/scripts/cpedb.py | 128 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 132 insertions(+), 13 deletions(-) diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report index 036eab2..da4ad96 100755 --- a/support/scripts/cpe-report +++ b/support/scripts/cpe-report @@ -6,7 +6,7 @@ import csv from cpedb import CPEDB -def get_target_cpe_report(cpe_report_file, cpedb): +def get_target_cpe_report(cpe_report_file, pkg_dir, cpedb): report_cpe_exact_match = "" report_cpe_needing_update = "" report_cpe_missing = "" @@ -21,9 +21,9 @@ def get_target_cpe_report(cpe_report_file, cpedb): if not result: result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) if not result: - report_cpe_missing += cpe[0] + "\n" + report_cpe_missing += cpe[0] + "," + cpe[2] + "," + cpe[4] + "\n" else: - report_cpe_needing_update += cpe[0] + "\n" + report_cpe_needing_update += cpe[0] + "," + cpe[2] + "," + cpe[4] + "\n" else: report_cpe_exact_match += cpe[0] + "\n" except (OSError, IOError) as e: @@ -34,11 +34,20 @@ def get_target_cpe_report(cpe_report_file, cpedb): print("CPE: Found but REQUIRES UPDATE:\n" + report_cpe_needing_update) print("CPE: Not found (proposing the following to be added):\n" + report_cpe_missing) + for cpe in report_cpe_needing_update.splitlines(): + cpedb.update(cpe, pkg_dir, 1) + for cpe in report_cpe_missing.splitlines(): + cpedb.update(cpe, pkg_dir, 1) + print("XML Generation Complete of NIST update files, see ./cpe/*") + def parse_args(): parser = argparse.ArgumentParser() parser.add_argument('-c', dest='cpe_report', action='store', required=True, help='CPE Report generated by make cpe-info (csv format)') + parser.add_argument('-d', dest='pkg_dir', action='store', required=True, + help='Path to dir(s) of Buildroot and BR2_EXTERNAL(s) seperated with ":". ' + + 'By default the script looks for a ./package/ and ./ in each path)') return parser.parse_args() @@ -47,7 +56,7 @@ def __main__(): cpedb = CPEDB() cpedb.get_xml_dict() print("Performing Target CPE Report Analysis...") - get_target_cpe_report(args.cpe_report, cpedb) + get_target_cpe_report(args.cpe_report, args.pkg_dir, cpedb) __main__() diff --git a/support/scripts/cpedb.py b/support/scripts/cpedb.py index 77d1d17..9e2ea91 100644 --- a/support/scripts/cpedb.py +++ b/support/scripts/cpedb.py @@ -1,16 +1,78 @@ import sys import urllib2 +from collections import OrderedDict import xmltodict import gzip from StringIO import StringIO +import re +import os CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" +class CPE: + cpe_str = None + cpe_str_short = None + cpe_desc = None + references = {} + + def __init__(self, cpe_str, product_ref=None, version_ref=None): + self.cpe_str = cpe_str + self.cpe_str_short = ":".join(self.cpe_str.split(":")[2:6]) + pkg_name = "".join(self.cpe_str.split(":")[4:5]).replace("_", " ") + pkg_ver = "".join(self.cpe_str.split(":")[5:6]) + self.cpe_desc = pkg_name.title() + " " + pkg_ver + if product_ref and version_ref: + self.references = { + 'references': { + 'reference': [{ + '@href': product_ref, + '#text': 'PRODUCT' + }, { + '@href': version_ref, + '#text': 'VERSION' + }] + } + } + + def update_ref(self, new_ref): + self.references = new_ref + + def to_dict(self): + cpe_dict = OrderedDict([ + ('cpe-item', OrderedDict([ + ('@name', 'cpe:/' + self.cpe_str_short), + ('title', OrderedDict([ + ('@xml:lang', 'en-US'), + ('#text', self.cpe_desc) + ])), + ('references', self.references), + ('cpe-23:cpe23-item', OrderedDict([ + ('@name', self.cpe_str) + ])) + ])) + ]) + return cpe_dict + + +def find_url_in_config_in(pkg_config_in): + product_url = "Not Found" + in_help_section = False + if os.path.exists(pkg_config_in): + fp = open(pkg_config_in, "r") + for config_line in fp: + if config_line.strip() == "help": + in_help_section = True + if in_help_section and re.match("(.*)https?://", config_line): + product_url = ''.join(config_line.split()) + break + fp.close() + return product_url + + class CPEDB: all_cpedb = dict() - all_cpes = set() - all_cpes_no_version = set() + all_cpes = dict() def get_xml_dict(self): print("CPE: Fetching xml manifest...") @@ -23,9 +85,9 @@ class CPEDB: for cpe in self.all_cpedb['cpe-list']['cpe-item']: cpe_str = cpe['cpe-23:cpe23-item']['@name'] + item = CPE(cpe_str) cpe_str_no_version = self.get_cpe_no_version(cpe_str) - self.all_cpes.add(cpe_str) - self.all_cpes_no_version.add(cpe_str_no_version) + self.all_cpes.update({cpe_str_no_version: {cpe_str: item}}) except urllib2.HTTPError: print("CPE: HTTP Error: %s" % CPE_XML_URL) @@ -36,17 +98,65 @@ class CPEDB: def find_partial(self, cpe_str): cpe_str_no_version = self.get_cpe_no_version(cpe_str) - if cpe_str_no_version in self.all_cpes_no_version: + if cpe_str_no_version in self.all_cpes: return cpe_str_no_version def find(self, cpe_str): - if cpe_str in self.all_cpes: - return cpe_str + if self.find_partial(cpe_str): + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + if cpe_str in self.all_cpes[cpe_str_no_version]: + return cpe_str - def get_cpe_no_version(self, cpe): - return ":".join(cpe.split(":")[:5]) + def check_package(self, cpe_str, pkg_dir): + product_site = "Not Found" + # For each dir, check the boot//, package// and / folders for Config.in + # This should cover regular packages, linux and boot related packages. + for pkg_dir_path in pkg_dir.split(":"): + file_to_open = os.path.join(os.path.abspath(pkg_dir_path), 'package', cpe_str, 'Config.in') + product_site = find_url_in_config_in(file_to_open) + if product_site != "Not Found": + break + file_to_open = os.path.join(os.path.abspath(pkg_dir_path), 'boot', cpe_str, 'Config.in') + product_site = find_url_in_config_in(file_to_open) + if product_site != "Not Found": + break + file_to_open = os.path.join(os.path.abspath(pkg_dir_path), cpe_str, 'Config.in') + product_site = find_url_in_config_in(file_to_open) + if product_site != "Not Found": + break + return product_site + + def update(self, cpe_str, pkg_dir=None, cpe_full_str=None): + if cpe_full_str: + cpe_str = cpe_str.split(",") + to_update = CPE(cpe_str[0], self.check_package(cpe_str[1], pkg_dir), cpe_str[2]) + xml = self.__gen_xml__(to_update.to_dict()) + if not os.path.exists("cpe"): + os.makedirs("cpe") + fp = open(os.path.join('cpe', cpe_str[1] + ".xml"), "w+") + fp.write(xmltodict.unparse(xml, pretty=True)) + fp.close() def get_nvd_url(self, cpe_str): return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \ urllib2.quote(cpe_str) + \ "&status=FINAL&orderBy=CPEURI&namingFormat=2.3" + + def get_cpe_no_version(self, cpe): + return ":".join(cpe.split(":")[:5]) + + def __gen_xml__(self, cpe_list): + list_header = { + "cpe-list": { + "@xmlns:config": "http://scap.nist.gov/schema/configuration/0.1", + "@xmlns": "http://cpe.mitre.org/dictionary/2.0", + "@xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance", + "@xmlnsscap-core": "http://scap.nist.gov/schema/scap-core/0.3", + "@xmlns:cpe-23": "http://scap.nist.gov/schema/cpe-extension/2.3", + "@xmlns:ns6": "http://scap.nist.gov/schema/scap-core/0.1", + "@xmlns:meta": "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2", + "@xsi:schemaLocation": "http://scap.nist.gov/schema/cpe-extension/2.3 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd http://cpe.mitre.org/dictionary/2.0 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2 https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd http://scap.nist.gov/schema/scap-core/0.3 https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd http://scap.nist.gov/schema/configuration/0.1 https://scap.nist.gov/schema/nvd/configuration_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd" + } + } + list_header['cpe-list'].update(cpe_list) + return list_header From patchwork Wed Oct 17 21:42:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985547 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HS4yxDz9sBn for ; Thu, 18 Oct 2018 08:42:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1FFF3882B2; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qZpvsFJkiunn; Wed, 17 Oct 2018 21:42:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 909D1882B8; Wed, 17 Oct 2018 21:42:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id C28CC1BF59D for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C010785D3D for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8GoKPh9AYxhq for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs02.rockwellcollins.com (ch3vs02.rockwellcollins.com [205.175.226.29]) by fraxinus.osuosl.org (Postfix) with ESMTPS id D5829857BD for ; Wed, 17 Oct 2018 21:42:05 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs02.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id D144760248; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:42:01 -0500 Message-Id: <1539812522-7171-9-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 09/10] toolchain/toolchain-ext: glibc cpe-info support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shruthi Singh MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Shruthi Singh This commit adds the correct CPE string for glibc, describing CPE ID, VERSION, PACKAGE NAME and URL. Signed-off-by: Shruthi Singh Signed-off-by: Matt Weber --- Changes v7 - New --- package/pkg-generic.mk | 14 ++++++++++++++ toolchain/toolchain-external/pkg-toolchain-external.mk | 7 +++++++ 2 files changed, 21 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index aba41dc..03db988 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -882,8 +882,22 @@ $(2)_CPE_PREFIX = $(CPE_PREFIX_APP) endif $(1)-cpe-info: PKG=$(2) +ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y) +$(1)-cpe-info: toolchain +endif $(1)-cpe-info: ifeq ($$($(2)_TYPE),target) +ifneq ($$($(2)_NAME),toolchain-external) +ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),) +ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC),y) + $$(eval $(2)_VERSION = $$(shell $$(call TOOLCHAIN_CPE_INFO))) + $$(eval $(2)_CPE_ID_VENDOR = gnu) + $$(eval $(2)_CPE_ID_NAME = glibc) + $$(eval $(2)_ACTUAL_SOURCE_SITE = https://github.com/bminor/glibc/releases) + $$(eval $(2)_RAWNAME = glibc) +endif # ifeq ($(BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC),y) +endif # ifneq ($(findstring TOOLCHAIN_EXTERNAL, $(2)),) +endif # ifneq ($$($(2)_NAME),toolchain-external) ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk index db3570d..aed06c5 100644 --- a/toolchain/toolchain-external/pkg-toolchain-external.mk +++ b/toolchain/toolchain-external/pkg-toolchain-external.mk @@ -440,6 +440,13 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS $(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR}) endef +define TOOLCHAIN_CPE_INFO + ARCH_SYSROOT_DIR="$(call toolchain_find_sysroot,$(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS))" ; \ + MAJ=`awk '{ if ($$1 = /#define/ && ($$2= /__GLIBC__/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \ + MIN=`awk '{ if ($$1 = /#define/ && ($$2 = /_GLIBC_MINOR/)){printf $$3};}' $${ARCH_SYSROOT_DIR}/usr/include/features.h` ; \ + echo $${MAJ}.$${MIN} +endef + # Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib. # Note: the skeleton package additionally creates lib32->lib or lib64->lib # (as appropriate) From patchwork Wed Oct 17 21:42:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 985549 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42b5HT68JXz9sC2 for ; Thu, 18 Oct 2018 08:42:13 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 227AB875E2; Wed, 17 Oct 2018 21:42:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pXeQLeGOaX4W; Wed, 17 Oct 2018 21:42:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id D91F287680; Wed, 17 Oct 2018 21:42:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id BB7221BF59C for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B8C4422767 for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0AjSVzemXltV for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (da1vs02.rockwellcollins.com [205.175.227.29]) by silver.osuosl.org (Postfix) with ESMTPS id 120192273B for ; Wed, 17 Oct 2018 21:42:06 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 17 Oct 2018 16:42:05 -0500 X-Received: from largo.rockwellcollins.com (unknown [192.168.140.76]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id D5D4660257; Wed, 17 Oct 2018 16:42:04 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Wed, 17 Oct 2018 16:42:02 -0500 Message-Id: <1539812522-7171-10-git-send-email-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> References: <1539812522-7171-1-git-send-email-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v7 10/10] docs/manual: security management section X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" - Start a section on security vulnerability management - Capture notes on cpe reporting support and limitations Signed-off-by: Matthew Weber --- Changes v7 - New --- docs/manual/cpe-reporting.txt | 79 +++++++++++++++++++++++++++++++++++++++++++ docs/manual/manual.txt | 2 ++ 2 files changed, 81 insertions(+) create mode 100644 docs/manual/cpe-reporting.txt diff --git a/docs/manual/cpe-reporting.txt b/docs/manual/cpe-reporting.txt new file mode 100644 index 0000000..dde7fb8 --- /dev/null +++ b/docs/manual/cpe-reporting.txt @@ -0,0 +1,79 @@ +// -*- mode:doc; -*- +// vim: set syntax=asciidoc: + +[[cpe-info]] + +== Security Vulnerability Management + +There are many different vulnerability databases (open/paid). This +section documents the use of the National Vulnerability Database(NVD) +provided by the National Institute of Standards and Technology (NIST). + +Within Buildroot, the intent is to provide good reporting of the build +configuration's inventory of software. The vulnerability analysis is +assumed to occur outside of the Buildroot environment. + +=== Common Platform Enumeration (CPE) Reporting + +Buildroot consists of a series of upstream packages. Each of those +packages may have a CPE definition used to map vulnerabilities to Common +Vulnerabilities and Exposures (CVE). A single package CPE has many versions +and each version may have a suite of CVEs associated. + +To make the gathering of the software inventory of CPE easier, Buildroot can +collect for you all the CPE related to the configured defconfig. To produce +this material, after you have configured Buildroot with +make menuconfig+, ++make xconfig+ or +make gconfig+, run: + +-------------------- +make cpe-info +-------------------- + +Buildroot then collects and writes the +$(TOPDIR)/cpe-manifest.csv+. This file +can be used for manual inspection against a CVE database or provided to +external tools which perform CVE inventory/analysis. + +*CPE Maintenance* + +To maintain these CPE strings for version changes against the NIST dictionary, +the manifest can be further processed. First, navigate to your Buildroot +directory and execute the script below. The +-d+ flag requires the paths to +your Buildroot and other BR2_EXTERNAL folder(s) to gather the required +information to generate NIST update XML. If more then one path is provided, +they should be seperated by colons. + +-------------------- +support/scripts/cpe-report -c cpe-manifest.csv -d ":" +-------------------- + +This script retrieves the current NIST dictionary and classifies each CPE +as either matched, requires version update or missing. Based on this +analysis, the script automatically uses Buildroot information to produce a +draft of XML which can be submitted to NIST to update the dictionary. +It is important to review the feedback from this script for cases of +"missing", as there maybe valid CPE strings for the package, however Buildroot +needs to be updated to match. This change is made by adjusting the default +CPE variables in the specific package's +.mk+. See xref:_infrastructure_for_packages_with_specific_build_systems[] +discussion on the use of +LIBFOO_CPE_*+. The NIST search engine is a good +tool for identifing existing strings (https://nvd.nist.gov/products/cpe/search). + +*Limitations* + +Buildroot does not produce or accurately present some of the material. Items +such as any versions which are non-number/hash are not compliant with the CPE +string specification and would require a manual analysis to update the CPE list +before any external CVE analysis should occur. This is a similar situation for +packages like the Linux kernel or U-Boot which may not have a version which +directly maps to a CPE. + +There is an assumed default CPE string for each package which is auto-generated +using existing package information. The output of +make cpe-info+ is based on +this default information and the packages which have been individually tailored +to match existing CPE strings. The Buildroot developers try to do their best to +keep those declarative statements as accurate as possible, to the best of their +knowledge. However, it is very well possible that those declarative statements +are not all fully accurate nor exhaustive. Similar to legal-info, it is your +responsibility to verify this information. A current health of CPE strings for +the latest upstream commit vs the NIST dictionary, can be checked in the +pkg-stats report (http://autobuild.buildroot.net/stats/). + diff --git a/docs/manual/manual.txt b/docs/manual/manual.txt index 9d50760..f15cf50 100644 --- a/docs/manual/manual.txt +++ b/docs/manual/manual.txt @@ -46,6 +46,8 @@ include::legal-notice.txt[] include::beyond-buildroot.txt[] +include::cpe-reporting.txt[] + = Developer guide include::how-buildroot-works.txt[]