From patchwork Thu Nov 19 21:23:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1403280 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=njseEkhc; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcXkb5VBNz9sTR for ; Fri, 20 Nov 2020 08:24:39 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 960F727A5C; Thu, 19 Nov 2020 21:24:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dlpkpTC-CGiI; Thu, 19 Nov 2020 21:24:11 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id E2731275B0; Thu, 19 Nov 2020 21:23:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 9FA761BF32B for ; Thu, 19 Nov 2020 21:23:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 96F9A8324B for ; Thu, 19 Nov 2020 21:23:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UjL+JfAp2GSv for ; Thu, 19 Nov 2020 21:23:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1DFA684D06 for ; Thu, 19 Nov 2020 21:23:56 +0000 (UTC) Received: by mail-wm1-f44.google.com with SMTP id x13so1628118wmj.1 for ; Thu, 19 Nov 2020 13:23:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2Z50m1wBwhkBRy7GEjiBOV5cyNMRnfR+E6czeeAdgBw=; b=njseEkhcwcgcUd9ZVHpoxhOJA+BAbccm0g4l/u0zHlxUeQDWB3T678PjUGgvhG/Xvm yZzGJMIWPc4Lw4krpILCoO8/t16hakFANrH1BLy1+qNguDU5IFYj1SJzhtxl0LAKrQg/ A4/HlWTC4Ny0zjVtiC/lQMpezSvY35KjKTZNHHfyoWBpdFdsh3WHKPy6ko1d0sKQIBby 49a/NO8SA+LKID7QYKJRXEnlTbnpoe6lwveJkQ2cMbz6ZNBiLu5u+0c1rdC1c4hlq9mR vkpfv2RzbkySMxfPbxpTMfs55R0f3cGoPWn2ryL7ScqjNTvlxbCFdUfsUvk/kZ4PanbG MLSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=2Z50m1wBwhkBRy7GEjiBOV5cyNMRnfR+E6czeeAdgBw=; b=AdFPWjNfrQV6WLmsb23CbvwlpA7VUv0eIxI5KBnhmIMGIHQ9802078Es7fQbARMZ8Q f5MJwt5j2YfFXkHG2tRXruFqRzxiFtO8h0QAyc4PkXVtKiDMQecDXQb3B48+ZKlSXqG/ 2ZtD1sftxnAezqecDqqRYdjG80yM9OBvAWMw01ku17TIAI7Nnu0jEP33wR1ofy3dRqQQ IMJG1TSe80t1vJabV03menZyhZWPh3I0TKRrujWl4YH004Uc1P2d71Q1Jkd6nOIJhQxM 9jt4oVp0BmvpNjPMFk5bctan9tRTIU7Xyp6PPJv2DVAosH3kD/2cNd/5ajj1VXVT7qW1 A0bA== X-Gm-Message-State: AOAM531W8HZVh/TQSUW9tYr7wFwjt9azzwowVVuMuAnCXRzCbBgN/7FL VAI2dRl49mcmfMU39BMehunUbpaZFFg= X-Google-Smtp-Source: ABdhPJw5sT4mUxck5OUdg+N7DIlBEzZQUIU5+1AEXgrqzzL40hlJGwR6J3hUYkWkD7lTEGymUX98FA== X-Received: by 2002:a1c:2ecc:: with SMTP id u195mr6451222wmu.27.1605821034052; Thu, 19 Nov 2020 13:23:54 -0800 (PST) Received: from scaer.home ([2a01:cb19:8b51:cb00:b94c:b327:ef3f:e064]) by smtp.gmail.com with ESMTPSA id l16sm1935392wrx.5.2020.11.19.13.23.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 13:23:53 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Thu, 19 Nov 2020 22:23:46 +0100 Message-Id: <5c59ed6901ff916883b240c7811d8ba723b11a3d.1605821010.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Buildroot] [PATCH 2/7] WIP: support/download: change format of archives generated from git X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas Petazzoni , Vincent Fazio Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" ** WIP: needs an update to all the hashes. Currently, our git archives are reproducible because we ensure that we use one of the few tar versions that generate identical gnu-formatted archives. However, than means that any tar version greater or equal to 1.30 is not compatible. I.e. we're stuck in the past, forever. However, thanks to some grunt work by Vincent, we now have a set of options that we can pass tar, to generate reproducible archives back from tar-1.27 and up through tar-1.32, the latest released version. However, those archives are not identical to the previous ones generated in the (now-broken) gnu format. To avoid any clashing between old and new archives, and new and old Buildroot versions, we need to name the new generated archives differently from the existing ones. The only latitude we have is to change the extension. The .tar.gz extension is a historical accident, dating back to when we introduced downloads from VCS, at which time we decided to use the same compression as was used for the tarballs directly downloaded via wget. We could switch over to use .tgz, which is just a shorter name for a .tar.gz. But while at it, lets also switch the compression, from the venerable gzip, to the not-so-new-nowadays xz. But since xz is quite slower than xz, we add traces that something is going on, so users do not wonder why there does not seem to be any progress. Setting the _SOURCE_EXT needs _SITE_METHOD to be known, which only happens later in the file, so we move it down. This is OK, as long as the variable are set before they are used to generate dependency rules (or other conditional code). The --pax-option, to set specific PAX headers, does not accept RFC2822 timestamps which value diverge too much: tar: Time stamp is out of allowed range However, the same timestamps passed as strict compliant ISO 8601 is accepted, so that's what we switch to as the date representation (%ci has been supported by git back to 1.6.0, released August 2008). Signed-off-by: Yann E. MORIN Cc: Vincent Fazio Cc: Thomas Petazzoni --- Here is a Makefile used to test all the versions of tar, along with a set of options: # Versions prior to 1.27 do not build on recent machines, because 'gets' # got removed (rightfully so), so don't count them as candidates. VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32 DATE = Thu 21 May 2020 06:44:11 PM CEST TARS = \ $(patsubst %,test_gnu_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS)) all: $(TARS) sha1sum $(^) .INTERMEDIATE: test_%.tar test_gnu_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=gnu \ -T list \ >$(@) test_posix_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ -T list \ >$(@) test_posix_paxoption_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ --pax-option='delete=atime,delete=ctime,delete=mtime' \ --pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \ -T list \ >$(@) list: .FORCE list: test (cd test && find . -not -type d ) |LC_ALL=C sort >$(@) LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng test: .FORCE test: rm -rf test mkdir -p test/bar echo foo >test/Foo echo bar >test/bar/Bar ln -s bar/Bar test/buz echo long >test/Very-$(LONG)-filename ln test/Very-$(LONG)-filename \ test/short .PRECIOUS: tar.% tar.%: tar-% cd $(<) && ./configure $(MAKE) -C $(<) install -m 0755 $(<)/src/tar $(@) .PRECIOUS: tar-% tar-%: tar-%.tar.gz tar xzf $(<) .PRECIOUS: tar-%.tar.gz tar-%.tar.gz: wget "https://ftp.gnu.org/gnu/tar/$(@)" .FORCE: clean: rm -rf tar-* tar.* test_* test list --- package/pkg-generic.mk | 8 +++++++- support/download/git | 20 ++++++++++++++------ 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 5fe1bfe0e2..115115a345 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -514,11 +514,11 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),) $(2)_VERSION = custom endif -$(2)_SOURCE_EXT = .tar.gz ifndef $(2)_SOURCE ifdef $(3)_SOURCE $(2)_SOURCE = $$($(3)_SOURCE) else ifdef $(2)_VERSION + # _SOURCE_EXT is defined below, after we compute the _SITE_METHOD $(2)_SOURCE ?= $$($(2)_BASENAME_RAW)$$($(2)_SOURCE_EXT) endif endif @@ -564,6 +564,12 @@ ifndef $(2)_DL_OPTS endif endif +ifneq ($$(filter git,$$($(2)_SITE_METHOD)),) +$(2)_SOURCE_EXT = .tar.xz +else +$(2)_SOURCE_EXT = .tar.gz +endif + ifneq ($$(filter bzr cvs hg,$$($(2)_SITE_METHOD)),) BR_NO_CHECK_HASH_FOR += $$($(2)_SOURCE) endif diff --git a/support/download/git b/support/download/git index 15d8c66e05..7bdc807ca7 100755 --- a/support/download/git +++ b/support/download/git @@ -170,8 +170,8 @@ _git checkout -f -q "'${cset}'" _git clean -ffdx # Get date of commit to generate a reproducible archive. -# %cD is RFC2822, so it's fully qualified, with TZ and all. -date="$( _git log -1 --pretty=format:%cD )" +# %ci is ISO 8601, so it's fully qualified, with TZ and all. +date="$( _git log -1 --pretty=format:%ci )" # There might be submodules, so fetch them. if [ ${recurse} -eq 1 ]; then @@ -201,12 +201,20 @@ find . -not -type d \ -and -not -path "./.git/*" >"${output}.list" LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" -# Create GNU-format tarballs, since that's the format of the tarballs on -# sources.buildroot.org and used in the *.hash files +# Explicit options to ensure reproducibility of the archive +pax_options="delete=atime,delete=ctime,delete=mtime" +pax_options+=",exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={${date}}" + +# Create tarballs in the posix format, since that's the most +# reproducible format +printf 'Creating tarball (%d files)...\n' "$( cat "${output}.list.sorted" |wc -l )" tar cf - --transform="s#^\./#${basename}/#" \ - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" \ + --format=posix \ + --pax-option="${pax_options}" \ -T "${output}.list.sorted" >"${output}.tar" -gzip -6 -n <"${output}.tar" >"${output}" +printf 'Compressing tarball (%d bytes)...\n' "$( stat -c %s "${output}.tar" )" +xz -9 <"${output}.tar" >"${output}" rm -f "${output}.list" rm -f "${output}.list.sorted"