| Message ID | 20181116152719.24591-1-yann.morin.1998@free.fr |
|---|---|
| State | Superseded |
| Headers | show |
| Series | support/dependencies: add a check for a suitable gzip | expand |
Hello, On Fri, 16 Nov 2018 16:27:19 +0100, Yann E. MORIN wrote: > Recently, some hash mismatch have been reported, both by users as well > as autobuilder failures, about tarballs generated from git repositories. > > This turned out to be caused by users having the 'gzip' command somehow > aliased to 'pigz' (which stand for: parallel implementation of gzip, > which takes advantage of multi-processor system to parallelise the > compression). > > Unfortunately, the output of pigz-compressed archives differ from that > of gzip (even though they *are* valid gzip-compressed streams). > > Add a dependency check that ensures that gzip is not pigz. If that is > the case, bail out and refuse to build. > > This is a stop-gap measure in preparation of the release. A complete > solution would accept pigz as a decompressor (because that is totally > OK), and ensure that we do build a host-gzip package should that be > needed. This is a much bigger endeavour, so this simple solution is > deemed enough for the release (after all, use of pigz is just atypical > enough that it should not pose such a problem for users to reverti to > using plain gzip). Is it really that more complicated to add and use host-gzip ? (This is a real question, not one asked with some irony.) Thomas
Thomas, All, On 2018-11-16 16:32 +0100, Thomas Petazzoni spake thusly: > On Fri, 16 Nov 2018 16:27:19 +0100, Yann E. MORIN wrote: > > Recently, some hash mismatch have been reported, both by users as well > > as autobuilder failures, about tarballs generated from git repositories. > > > > This turned out to be caused by users having the 'gzip' command somehow > > aliased to 'pigz' (which stand for: parallel implementation of gzip, > > which takes advantage of multi-processor system to parallelise the > > compression). > > > > Unfortunately, the output of pigz-compressed archives differ from that > > of gzip (even though they *are* valid gzip-compressed streams). > > > > Add a dependency check that ensures that gzip is not pigz. If that is > > the case, bail out and refuse to build. > > > > This is a stop-gap measure in preparation of the release. A complete > > solution would accept pigz as a decompressor (because that is totally > > OK), and ensure that we do build a host-gzip package should that be > > needed. This is a much bigger endeavour, so this simple solution is > > deemed enough for the release (after all, use of pigz is just atypical > > enough that it should not pose such a problem for users to reverti to > > using plain gzip). > > Is it really that more complicated to add and use host-gzip ? We need to add an actual host-gzip package, then ensure it is used when needed (e.g. in _DOWNLOAD_DEPENDENCIES of git/svn/cvs based packages). Definitiely not overly complex as I stated, no, but I was mostly looking through the release-incoming prism, with a goal to do the strict minimum to fail in a sane way, and was seeing the host-gzip as too far reaching for the release. But I can do that if you prefer. > (This is a real question, not one asked with some irony.) ;-) Even then, I can stomach a bit of irony. Even if I am very often incapable of noticing it. :-] Regards, Yann E. MORIN.
diff --git a/support/dependencies/check-host-gzip.mk b/support/dependencies/check-host-gzip.mk new file mode 100644 index 0000000000..867bbb04f8 --- /dev/null +++ b/support/dependencies/check-host-gzip.mk @@ -0,0 +1,3 @@ +ifeq (,$(call suitable-host-package,gzip)) +$(error No suitable gzip found) +endif diff --git a/support/dependencies/check-host-gzip.sh b/support/dependencies/check-host-gzip.sh new file mode 100755 index 0000000000..350fe552d5 --- /dev/null +++ b/support/dependencies/check-host-gzip.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +candidate="$1" # ignored + +gzip="$(which gzip)" +if [ ! -x "${gzip}" ]; then + exit 1 +fi + +# gzip displays its version string on stdout +# pigz displays its version string on stderr +version="$("${gzip}" --version 2>&1)" +case "${version}" in + (*pigz*) + printf "%s (%s) is not acceptable as a gzip implementation\n" \ + "${gzip}" "${version}" >&2 + exit 1 + ;; +esac + +printf "%s" "${gzip}"
Recently, some hash mismatch have been reported, both by users as well as autobuilder failures, about tarballs generated from git repositories. This turned out to be caused by users having the 'gzip' command somehow aliased to 'pigz' (which stand for: parallel implementation of gzip, which takes advantage of multi-processor system to parallelise the compression). Unfortunately, the output of pigz-compressed archives differ from that of gzip (even though they *are* valid gzip-compressed streams). Add a dependency check that ensures that gzip is not pigz. If that is the case, bail out and refuse to build. This is a stop-gap measure in preparation of the release. A complete solution would accept pigz as a decompressor (because that is totally OK), and ensure that we do build a host-gzip package should that be needed. This is a much bigger endeavour, so this simple solution is deemed enough for the release (after all, use of pigz is just atypical enough that it should not pose such a problem for users to reverti to using plain gzip). Fixes: http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/ Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Marcin Niestrój <m.niestroj@grinn-global.com> Cc: Erico Nunes <nunes.erico@gmail.com> --- support/dependencies/check-host-gzip.mk | 3 +++ support/dependencies/check-host-gzip.sh | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 support/dependencies/check-host-gzip.mk create mode 100755 support/dependencies/check-host-gzip.sh