From patchwork Tue Dec 29 11:01:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1421181 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=EuFljmch; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4D4s1F2FTlz9sVq for ; Tue, 29 Dec 2020 22:01:37 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D265981DD7; Tue, 29 Dec 2020 11:01:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nN250L1iXTXJ; Tue, 29 Dec 2020 11:01:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 5717F871B4; Tue, 29 Dec 2020 11:01:30 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 649861BF3DA for ; Tue, 29 Dec 2020 11:01:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5DFDE871AA for ; Tue, 29 Dec 2020 11:01:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CbnTBUmS+h4a for ; Tue, 29 Dec 2020 11:01:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by hemlock.osuosl.org (Postfix) with ESMTPS id CE10B871A4 for ; Tue, 29 Dec 2020 11:01:23 +0000 (UTC) Received: by mail-wm1-f54.google.com with SMTP id a6so2315360wmc.2 for ; Tue, 29 Dec 2020 03:01:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+5GJkZ3idj1iCSVcdP2fwLYztnfPo16XTsqJT7USuBA=; b=EuFljmchTEK0f8WiaREJjOjmmh4f3yaUl0WawqXFbQrcoR2o7NZuCPUZRWyPGVmv3E hkjYS57wDwbNw4VegR/5uoo7hWRBfRy8E1gZSI9ws9LMRIKBwDzK0wnJqTPYtHvaGKsY 15R3fqmOWYSjDpKtf6YJZeDen81NCCbD4hdaLr55yfCvxpwO23tD+1Qu4Jd2QbpsCuEV tf+YCqImm+WfT3hY6jE2/vGtGiXLa9GQeJcQROvWOme7mmXP/QNKJOfRc76H7FmalLcG HEfi+BKyMt/F9xwa3mkCWZPwEfXg/q7qVvtEY3G9CXyAOCYJS6qkTBtfZe8E6wKX6iLE sEKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=+5GJkZ3idj1iCSVcdP2fwLYztnfPo16XTsqJT7USuBA=; b=fDtbiox7kfKksYHP72eLbH40J0kz3CnLRmG5qr3cvdAb8TjkXlOw0+XE4a0/KoI7rN ZaBOcFGrRH3pYSKIGw7DwtWqm+Vtid11QuHOtXpraJ23BNlJD3Zni6+cxTeeln5zJY/6 n91VGQWlGI1sBmi9h/c4UAVnWMIjAsy3kcd6QizuAbQBhWnSXaTERNnYMjPaZkCK3PEg rWFugzAoSgB6wz8BpiZ54V1we3FRDUWZtYVTAAEwIgQt8Gag2BpKcEZj5P2i58lvwYIf mU2gJTR6ajnL4B55a0NFF6sQkroIFOrdWOk7fBdOB9rvfzQJN6SPB2SbxVg/M6Tz1QvZ sccw== X-Gm-Message-State: AOAM531g9NiNIq05DTQJovC7Ht1csMj1Zw2fguwnUhRCa0pKhIa/OKqn Jsva9cnirWPsPBURjK/fSQ8a2p162yQ= X-Google-Smtp-Source: ABdhPJy6KOsUl44xLS5mqjXa39RUQf9SYXzk80wkv3UO1gQJgP6Hk6gqLUCMhesKtEaTaGCEboY8wA== X-Received: by 2002:a1c:220a:: with SMTP id i10mr2986198wmi.93.1609239682339; Tue, 29 Dec 2020 03:01:22 -0800 (PST) Received: from scaer.home ([2a01:cb19:8b51:cb00:5459:5fc6:57c1:a482]) by smtp.gmail.com with ESMTPSA id u9sm2986787wmb.32.2020.12.29.03.01.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Dec 2020 03:01:21 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 29 Dec 2020 12:01:11 +0100 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Buildroot] [PATCH 03/10 v3] support/download: add helper to generate a reproducible archive X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas Petazzoni , Vincent Fazio Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We currently need to generate reproducible archives in at least two locations: the git and svn download backends. We also know of some future potential use (e.g. the other download backends, like cvs, or in the upcoming download post-processors for vendoring, like cargo and go). However, we are currently limited to a narrow range of tar versions that we support, to create reproducible archives, because the gnu format we use has changed with tar 1.30. As a consequence, and as time advances, more and more distros are, or will eventually start, shipping with tar 1.30 or later, and thus we need to always build our on host-tar. Now, thanks to some grunt work by Vincent, we have a set of options that we can pass tar, to generate reproducible archives back from tar-1.27 and up through tar-1.32, the latest released version. However, those options are non-trivial, so we do not want to have to repeat those (and maintain them) in multiple locations. Introduce a helper that can generate a reproducible archive from an input directory. The --pax-option, to set specific PAX headers, does not accept RFC2822 timestamps which value are too away from some fixed point (set atcompile-time?): tar: Time stamp is out of allowed range However, the same timestamps passed as strict compliant ISO 8601 are accepted, so that's what we expect as a date format. Signed-off-by: Yann E. MORIN Cc: Thomas Petazzoni Cc: Vincent Fazio PS. Here is a Makefile used to test all the versions of tar, with different output formats and different sets of options: ---8<------8<------8<------8<--- # Versions prior to 1.27 do not build on recent machines, because 'gets' # got removed (rightfully so), so don't count them as candidates. VERSIONS = 1.27 1.27.1 1.28 1.29 1.30 1.31 1.32 DATE = Thu 21 May 2020 06:44:11 PM CEST TARS = \ $(patsubst %,test_gnu_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_%.tar,$(VERSIONS)) \ $(patsubst %,test_posix_paxoption_%.tar,$(VERSIONS)) all: $(TARS) sha1sum $(^) .INTERMEDIATE: test_%.tar test_gnu_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=gnu \ -T list \ >$(@) test_posix_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ -T list \ >$(@) test_posix_paxoption_%.tar: tar.% list ./$(<) cf - -C test \ --transform="s#^\./#test-version/#" \ --numeric-owner --owner=0 --group=0 \ --mtime="$(DATE)" \ --format=posix \ --pax-option='delete=atime,delete=ctime,delete=mtime' \ --pax-option='exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={$(DATE)}' \ -T list \ >$(@) list: .FORCE list: test (cd test && find . -not -type d ) |LC_ALL=C sort >$(@) LONG = L$$(for i in $$(seq 1 200); do printf 'o'; done)ng test: .FORCE test: rm -rf test mkdir -p test/bar echo foo >test/Foo echo bar >test/bar/Bar ln -s bar/Bar test/buz echo long >test/Very-$(LONG)-filename ln test/Very-$(LONG)-filename \ test/short .PRECIOUS: tar.% tar.%: tar-% cd $(<) && ./configure $(MAKE) -C $(<) install -m 0755 $(<)/src/tar $(@) .PRECIOUS: tar-% tar-%: tar-%.tar.gz tar xzf $(<) .PRECIOUS: tar-%.tar.gz tar-%.tar.gz: wget "https://ftp.gnu.org/gnu/tar/$(@)" .FORCE: clean: rm -rf tar-* tar.* test_* test list ---8<------8<------8<------8<--- Acked-by: Arnout Vandecappelle (Essensium/Mind) Reviewed-by: Vincent Fazio --- support/download/helpers | 70 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100755 support/download/helpers diff --git a/support/download/helpers b/support/download/helpers new file mode 100755 index 0000000000..0e0432c884 --- /dev/null +++ b/support/download/helpers @@ -0,0 +1,70 @@ +# Generate a reproducible archive from the content of a directory +# +# $1 : input directory +# $2 : leading component in archive +# $3 : ISO8601 date: YYYY-MM-DDThh:mm:ssZZ +# $4 : output file +# $5... : globs of filenames to exclude from the archive, suitable for +# find's -path option, and relative to the input directory $1 +# +# Notes : +# - must not be called with CWD as, or below, the input directory +# - some temporary files are created in CWD, and removed at the end +# +# Example: +# $ find /path/to/temp/dir +# /path/to/temp/dir/ +# /path/to/temp/dir/some-file +# /path/to/temp/dir/some-dir/ +# /path/to/temp/dir/some-dir/some-other-file +# +# $ mk_tar_gz /path/to/some/dir \ +# foo_bar-1.2.3 \ +# 1970-01-01T00:00:00Z \ +# /path/to/foo.tar.gz \ +# '.git/*' '.svn/*' +# +# $ tar tzf /path/to/foo.tar.gz +# foo_bar-1.2.3/some-file +# foo_bar-1.2.3/some-dir/some-other-file +# +mk_tar_gz() { + local in_dir="${1}" + local base_dir="${2}" + local date="${3}" + local out="${4}" + shift 4 + local glob tmp pax_options + local -a find_opts + + for glob; do + find_opts+=( -or -path "./${glob#./}" ) + done + + pax_options="delete=atime,delete=ctime,delete=mtime" + pax_options+=",exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={${date}}" + + tmp="$(mktemp --tmpdir="$(pwd)")" + pushd "${in_dir}" >/dev/null + + # Establish list + find . -not -type d -and -not \( -false "${find_opts[@]}" \) >"${tmp}.list" + # Sort list for reproducibility + LC_ALL=C sort <"${tmp}.list" >"${tmp}.sorted" + + # Create POSIX tarballs, since that's the format the most reproducible + tar cf - --transform="s#^\./#${base_dir}/#" \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" \ + --format=posix --pax-option="${pax_options}" \ + -T "${tmp}.sorted" >"${tmp}.tar" + + # Compress the archive + gzip -6 -n <"${tmp}.tar" >"${out}" + + rm -f "${tmp}"{.list,.sorted,.tar} + + popd >/dev/null +} + +# Keep this line and the following as last lines in this file. +# vim: ft=bash