From patchwork Mon Nov 4 17:48:06 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Clayton Shotwell X-Patchwork-Id: 288224 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from fraxinus.osuosl.org (fraxinus.osuosl.org [140.211.166.137]) by ozlabs.org (Postfix) with ESMTP id 1C57C2C00AC for ; Tue, 5 Nov 2013 04:48:26 +1100 (EST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 444228C0AC; Mon, 4 Nov 2013 17:48:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5XVMohEzYkSu; Mon, 4 Nov 2013 17:48:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7FD788C0A3; Mon, 4 Nov 2013 17:48:24 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id C4AC71BFA5B for ; Mon, 4 Nov 2013 17:48:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id BE0E38D30C for ; Mon, 4 Nov 2013 17:48:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GQ+8NJzWGEQl for ; Mon, 4 Nov 2013 17:48:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs02.rockwellcollins.com (secvs02.rockwellcollins.com [205.175.225.241]) by whitealder.osuosl.org (Postfix) with ESMTPS id 3A5CC8D2F5 for ; Mon, 4 Nov 2013 17:48:21 +0000 (UTC) Received: from nosuchhost.198.131.in-addr.arpa (HELO collinscrsmtp02.rockwellcollins.com) ([131.198.63.133]) by mail-virt.rockwellcollins.com with ESMTP; 04 Nov 2013 11:48:19 -0600 Received: from nunkun.rockwellcollins.com ([131.198.63.11]) by collinscrsmtp02.rockwellcollins.com (Lotus Domino Release 8.5.2FP2 HF162) with ESMTP id 2013110411481841-3629377 ; Mon, 4 Nov 2013 11:48:18 -0600 From: Clayton Shotwell To: buildroot@busybox.net Date: Mon, 4 Nov 2013 09:48:06 -0800 Message-Id: <1383587286-18094-1-git-send-email-clshotwe@rockwellcollins.com> X-Mailer: git-send-email 1.7.9.5 X-MIMETrack: Itemize by SMTP Server on CollinsCRSMTP02/CedarRapids/RockwellCollins(Release 8.5.2FP2 HF162|May 16, 2011) at 11/04/2013 11:48:18 AM, Serialize by Router on CollinsCRSMTP02/CedarRapids/RockwellCollins(Release 8.5.2FP2 HF162|May 16, 2011) at 11/04/2013 11:48:18 AM, Serialize complete at 11/04/2013 11:48:18 AM X-TNEFEvaluated: 1 Cc: Clayton Shotwell Subject: [Buildroot] [RFC] core: Download all package sources X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: buildroot-bounces@busybox.net Everyone, This is an RFC for a new make option to download all package sources to aid developers who cannot freely download packages. This includes limited internet access or restrictive firewalls. This is a best effort download. Any errors in downloading are ignored rather than causing a failure. The main changes in this patch is the addition of an "allsource" make option that pulls down all of the package sources if a version string is found. The version check is required because packages that do not have a default version cause a invalid file download attempt. I had to abstract the download steps from pkg-generic.mk so they are callable by both the main Makefile and the pkg-generic.mk download step. Implementation details: 1) The allsource command filters the make .VARIABLES for all "*_TARGET_SOURCE" and then strips off the "_TARGET_SOURCE" to get the package name. This results in a list of all of the packages that can be downloaded. 2) The newly created PKG_DOWNLOAD define is called on all of the package names to download the source. This call does not cause the allsource command to fail if there is an error. This is only a best effort call. 3) All of the sources are downloaded to the BR2_DL_DIR folder. Signed-off-by: Clayton Shotwell --- Makefile | 8 +++++++- package/pkg-download.mk | 17 +++++++++++++++++ package/pkg-generic.mk | 9 +-------- 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/Makefile b/Makefile index f266e2d..ef2cd74 100644 --- a/Makefile +++ b/Makefile @@ -388,7 +388,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf world: $(BASE_TARGETS) $(TARGETS_ALL) -.PHONY: all world toolchain dirs clean distclean source outputmakefile \ +.PHONY: all world toolchain dirs clean distclean source allsource outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \ $(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \ $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \ @@ -564,6 +564,11 @@ toolchain-eclipse-register: source: dirs $(TARGETS_SOURCE) $(HOST_SOURCE) +allsource: + $(foreach V, \ + $(subst _TARGET_SOURCE,,$(filter %_TARGET_SOURCE,$(sort $(.VARIABLES)))), \ + -$(if $($(V)_VERSION),$(call PKG_DOWNLOAD,$(V)))) + external-deps: @$(MAKE) -Bs DL_MODE=SHOW_EXTERNAL_DEPS $(EXTRAMAKEARGS) source | sort -u @@ -802,6 +807,7 @@ endif @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' allsource - download sources for all packages, not just selected ones' @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 8e4a1ec..b75f039 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -226,6 +226,23 @@ define SHOW_EXTERNAL_DEPS_LOCALFILES echo $(2) endef +############################################################################### +# PKG_DOWNLOAD -- Package Download helper. +# +# Argument 1 is the package name in all caps +# +############################################################################### +define PKG_DOWNLOAD + $(if $($(1)_SOURCE),$(call DOWNLOAD,$($(1)_SITE:/=)/$($(1)_SOURCE))) + $(foreach p,$($(1)_EXTRA_DOWNLOADS),$(call DOWNLOAD,$($(1)_SITE:/=)/$(p))$(sep)) + $(foreach p,$($(1)_PATCH),\ + $(if $(findstring ://,$(p)),\ + $(call DOWNLOAD,$(p)),\ + $(call DOWNLOAD,$($(1)_SITE:/=)/$(p))\ + )\ + $(sep)) +endef + ################################################################################ # DOWNLOAD -- Download helper. Will try to download source from: # 1) BR2_PRIMARY_SITE if enabled diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 4bba4b5..c993f47 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -39,14 +39,7 @@ ifeq ($(DL_MODE),DOWNLOAD) done ; \ fi endif - $(if $($(PKG)_SOURCE),$(call DOWNLOAD,$($(PKG)_SITE:/=)/$($(PKG)_SOURCE))) - $(foreach p,$($(PKG)_EXTRA_DOWNLOADS),$(call DOWNLOAD,$($(PKG)_SITE:/=)/$(p))$(sep)) - $(foreach p,$($(PKG)_PATCH),\ - $(if $(findstring ://,$(p)),\ - $(call DOWNLOAD,$(p)),\ - $(call DOWNLOAD,$($(PKG)_SITE:/=)/$(p))\ - )\ - $(sep)) + $(call PKG_DOWNLOAD,$(PKG)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) ifeq ($(DL_MODE),DOWNLOAD) $(Q)mkdir -p $(@D)