From patchwork Mon Apr 15 19:47:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1085891 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="E/WdUqo8"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44jfDj6MMdz9s0W for ; Tue, 16 Apr 2019 05:48:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 54B87857CB; Mon, 15 Apr 2019 19:48:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hakteq1W07kH; Mon, 15 Apr 2019 19:47:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 99153869CA; Mon, 15 Apr 2019 19:47:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 2CFDD1BF37A for ; Mon, 15 Apr 2019 19:47:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 290B282115 for ; Mon, 15 Apr 2019 19:47:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i51fi4b+iEJN for ; Mon, 15 Apr 2019 19:47:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by hemlock.osuosl.org (Postfix) with ESMTPS id 5AEDE81FB4 for ; Mon, 15 Apr 2019 19:47:48 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id o25so22139750wmf.5 for ; Mon, 15 Apr 2019 12:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=GoCi5E/PX8vflIOt9fsR9Qfm6xZH8xkzXZD+fwrZrfE=; b=E/WdUqo8F7Qgmme2Akjhx/TUcCAFu9D8MUD8taVujTCqsIHWn4RUDBAnYjEw91ptn4 6rocj+/x8RJRnKrZoNis0yTFvrZCxh7POLrrGR02LWqporMexoxt7CEDsHDef8m/jlfY IzUt6zbFn1L1snR/cKdgFSDt2Q4AJQ5V/BtMf8b9kCj1j3E4QdrVqOBJnE/JTccw4liI ShBGumxJQsdkTgIPHpY5agrCBj224FZAcrq2R47tFHgWXxWcrdCqHq9VN0WlABpf1E53 pk7QLqUH8LbWC5kewlBbIKknDm2FdE4Mmso+5CW8gxJ84OQ6XEARuBEz5MHfjL4LKrRF wI/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=GoCi5E/PX8vflIOt9fsR9Qfm6xZH8xkzXZD+fwrZrfE=; b=b3+JNKZzZvg/sX50QMi9xwdmEjzaf6dJkqZn9vpTzAJMd9jI9p0h8KHcWnqB9DlmlE xaQ6Igth7dFdfV6wAFN7XUNv1DQqGj5KrUXDXewQ8nKrS0iFTirdp1bf2qWzYehrSTh4 wxiKKo1cfmDeyAqS0Q3Ua0urFmqG52mFs6rMkbtG+knBHGGJ2uAQkbURokpzPAy7Yoq/ fi8PBccXh580Nf0Zkr3z8EP66QLUrE9yMWE63cbKf37kSdmHGwJ1DwKVgfAIqFMbuNrY D1++/G9GfvKNeCVCDOKmDr452fqqY/c63h6xuQuZa6F64C9Dx33RUJW7Go6fAAUnMVx1 RKdg== X-Gm-Message-State: APjAAAXv/m1UHeImJR55TBE5EG7wiDG2XEQ7j86l+ItM/pS4vhcsBua4 GBWui8U2cl3EwF3RTn3ojrvvzpNx X-Google-Smtp-Source: APXvYqwHX7m8gfzQ61Q5C19ZVQMTlv7X7Ow4xNPY9BuYQwtB1FM+xbjsEF3M7zzkfhiJTSDv4enLbQ== X-Received: by 2002:a7b:c353:: with SMTP id l19mr24147721wmj.12.1555357666719; Mon, 15 Apr 2019 12:47:46 -0700 (PDT) Received: from scaer.home ([2a01:cb19:829a:2800:68e8:7a61:9bb9:12a]) by smtp.gmail.com with ESMTPSA id z74sm1479222wmc.2.2019.04.15.12.47.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Apr 2019 12:47:46 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 15 Apr 2019 21:47:31 +0200 Message-Id: <30455cde143814806008e9654974452726602e55.1555357644.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 09/12 v3] core: introduce new global show-info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , "Yann E. MORIN" , Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Users are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package. Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is. Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies... We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq). It also closely matches Python data structure, which makes it easy to use with our own internal tools as well. Finally, JSON being a key-value store, allows for easy expanding the output without requiring existing consumers to be updated; new, unknown keys are simply ignored by those (as long as they are true JSON parsers). The complex part of this change was the conditional output of parts of the data: virtual packages have no source, version, license or downloads, unlike non-virtual packages. Same goes for filesystems. We use a wrapper macro, show-info, that de-multiplexes unto either the package-related- or filesystem-related macros, and for packages, we also use a detailed macro for non-virtual packages. It is non-trivial to properly output correct JSON blurbs, especially when trying to output an array of objects, like so, where the last item shall not be followed by a comma: [ { ... }, { ... } ] So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of ",}" or ", }" or ",]" or ", ]" with only the respective closing symbol, "}" or "]". The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb that fits on a single line with all spaces squashed (but still with spaces, as it is not possible to differentiate spaces between JSON elements from spaces inside JSON strings). Reported-by: Thomas De Schampheleire Signed-off-by: "Yann E. MORIN" Cc: Thomas De Schampheleire Cc: Thomas Petazzoni Cc: Arnout Vandecappelle --- Changes v2 -> v3: - set the virtual field in a single place (Arnout) - rename the macros (Arnout) - don't add opening and closing {} in clean-json (Arnout) - add to 'make help' Changes v1 -> v2: - make it a macro to be called (Arnout) - make it a top-level rule (Arnout) --- Makefile | 16 +++++++++++++ package/pkg-utils.mk | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+) diff --git a/Makefile b/Makefile index 60bf7d7d08..0f5331ec7e 100644 --- a/Makefile +++ b/Makefile @@ -903,6 +903,21 @@ check-dependencies: @cd "$(CONFIG_DIR)"; \ $(TOPDIR)/support/scripts/graph-depends -C +.PHONY: show-info +show-info: + @: + $(info $(call clean-json, \ + { $(foreach p, \ + $(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \ + $(i) \ + $($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \ + ) \ + ), \ + $(call json-info,$(call UPPERCASE,$(p)))$(comma) \ + ) } \ + ) \ + ) + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) # Some subdirectories are also package names. To avoid that "make linux" @@ -1128,6 +1143,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' show-info - generate info about packages, as a JSON blurb' @echo ' printvars - dump internal variables selected with VARS=...' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build' diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index bffd79dfb0..b7280e930f 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -62,6 +62,70 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) endif endef +# json-info -- return package or filesystem metadata formatted as an entry +# of a JSON dictionnary +# $(1): upper-case package or filesystem name +define json-info + "$($(1)_NAME)": { + "type": "$($(1)_TYPE)", + $(if $(filter rootfs,$($(1)_TYPE)), \ + $(call _json-info-fs,$(1)), \ + $(call _json-info-pkg,$(1)), \ + ) + } +endef + +# _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers +# for json-info, above +define _json-info-pkg + $(if $($(1)_IS_VIRTUAL), \ + "virtual": true$(comma), + "virtual": false$(comma) + $(call _json-info-pkg-details,$(1)) \ + ) + "dependencies": [ + $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES))) + ], + "reverse_dependencies": [ + $(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))) + ] +endef + +define _json-info-pkg-details + "version": "$($(1)_DL_VERSION)", + "licenses": "$($(1)_LICENSE)", + "downloads": [ + $(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)), + { + "source": "$(notdir $(dl))", + "uris": [ + $(call make-comma-list, + $(subst \|,|, + $(call DOWNLOAD_URIS,$(dl),$(1)) + ) + ) + ] + }, + ) + ], +endef + +define _json-info-fs + "dependencies": [ + $(call make-comma-list,$(sort $($(1)_DEPENDENCIES))) + ] +endef + +# clean-json -- cleanup pseudo-json into clean json: +# - remove commas before closing ] and } +# - minify with $(strip) +clean-json = $(strip \ + $(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \ + $(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \ + $(strip $(1)) \ + )))) \ +) + # # legal-info helper functions #