| Message ID | 1479460224-6119-5-git-send-email-jezz@sysmic.org |
|---|---|
| State | Superseded |
| Headers | show |
Hello, On Fri, 18 Nov 2016 10:10:13 +0100, Jérôme Pouiller wrote: > -# Minimal version = 1.17 (previous versions do not correctly unpack archives > -# containing hard-links if the --strip-components option is used). > +# Minimal version = 1.28 (previous versions do not does not support --sort=name) > major_min=1 > -minor_min=17 > +minor_min=28 Unfortunately, this version dependency is not really acceptable I'm afraid. Ubuntu 14.04, which is really not that old, only has tar 1.27. And the old Debian 6.0 that I have for the autobuilders to test "old systems" uses tar 1.23. Can we find a better way? Thomas
On 2016-11-18 12:40, Thomas Petazzoni wrote: > Hello, > > On Fri, 18 Nov 2016 10:10:13 +0100, Jérôme Pouiller wrote: > >> -# Minimal version = 1.17 (previous versions do not correctly unpack >> archives >> -# containing hard-links if the --strip-components option is used). >> +# Minimal version = 1.28 (previous versions do not does not support >> --sort=name) >> major_min=1 >> -minor_min=17 >> +minor_min=28 > > Unfortunately, this version dependency is not really acceptable I'm > afraid. Ubuntu 14.04, which is really not that old, only has tar 1.27. > > And the old Debian 6.0 that I have for the autobuilders to test "old > systems" uses tar 1.23. > > Can we find a better way? My workstation only have tar 1.27 installed. When tar version is wrong, compilation does not fail. Instead, host-tar is compiled before any other packages (see support/dependencies/check-host-tar.mk) -- Jérôme Pouiller
Hello, On Fri, 18 Nov 2016 14:02:58 +0100, Jérôme Pouiller wrote: > My workstation only have tar 1.27 installed. When tar version is wrong, > compilation does not fail. Instead, host-tar is compiled before any > other packages (see support/dependencies/check-host-tar.mk) Yes, I know, but it's a pain to force everyone using an old distro to build host-tar. Maybe we should make this conditional on BR2_REPRODUCIBLE? Thanks, Thomas
On 2016-11-18 14:29, Thomas Petazzoni wrote: > Hello, > > On Fri, 18 Nov 2016 14:02:58 +0100, Jérôme Pouiller wrote: > >> My workstation only have tar 1.27 installed. When tar version is >> wrong, >> compilation does not fail. Instead, host-tar is compiled before any >> other packages (see support/dependencies/check-host-tar.mk) > > Yes, I know, but it's a pain to force everyone using an old distro to > build host-tar. Maybe we should make this conditional on > BR2_REPRODUCIBLE? Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months (maybe simultaneous with next Buildroot version). So, IMHO it not justified to add a dirty condition in check-host-tar.sh. -- Jérôme Pouiller
Hello, On Fri, 18 Nov 2016 14:44:50 +0100, Jérôme Pouiller wrote: > > Yes, I know, but it's a pain to force everyone using an old distro to > > build host-tar. Maybe we should make this conditional on > > BR2_REPRODUCIBLE? > > Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months > (maybe simultaneous with next Buildroot version). > > So, IMHO it not justified to add a dirty condition in check-host-tar.sh. I think you don't really realize how big companies work. They will be using such version in 3 or 5 years maybe. We have people still using RHEL5, even though it's almost 10 years old, and its support going to stop next year. Supporting old distributions is important for Buildroot, so you can't just sweep away the problem by pretending that it doesn't exist with modern distributions. Best regards, Thomas
On 18-11-16 22:28, Thomas Petazzoni wrote: > Hello, > > On Fri, 18 Nov 2016 14:44:50 +0100, Jérôme Pouiller wrote: > >>> Yes, I know, but it's a pain to force everyone using an old distro to >>> build host-tar. Maybe we should make this conditional on >>> BR2_REPRODUCIBLE? >> >> Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months >> (maybe simultaneous with next Buildroot version). >> >> So, IMHO it not justified to add a dirty condition in check-host-tar.sh. > > I think you don't really realize how big companies work. They will be > using such version in 3 or 5 years maybe. > > We have people still using RHEL5, even though it's almost 10 years old, > and its support going to stop next year. Supporting old distributions > is important for Buildroot, so you can't just sweep away the problem by > pretending that it doesn't exist with modern distributions. I'm absolutely with Thomas on this one. Also, it doesn't have to be very dirty. In check-host-tar.sh the major_min and minor_min can be passed as arguments, and in check-host-tar.mk we can do ifeq ($(BR2_REPRODUCIBLE),y) TAR_VERSION_MIN = 1 28 else TAR_VERSION_MIN = 1 17 endif ifeq (,$(call suitable-host-package,tar,$(TAR) $(TAR_VERSION_MIN)) ... Regards, Arnout
On 18-11-16 10:10, Jérôme Pouiller wrote: > In order to make tar images reproducible, we use --sort flag. However, > this flags is available only from tar 1.28. So we also bump necessary > host-tar version. > > This work was sponsored by `BA Robotic Systems'. > > Signed-off-by: Jérôme Pouiller <jezz@sysmic.org> > --- > fs/tar/tar.mk | 2 +- > support/dependencies/check-host-tar.sh | 5 ++--- > 2 files changed, 3 insertions(+), 4 deletions(-) > > diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk > index 11c69c5..5a1b263 100644 > --- a/fs/tar/tar.mk > +++ b/fs/tar/tar.mk > @@ -7,7 +7,7 @@ > TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) > > define ROOTFS_TAR_CMD > - tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . > + tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) . Instead of requiring a specific version of tar, I would prefer to use the same solution we already have in the git wrapper, and which is also what is proposed by reproducible-builds.org: find src -print0 | LC_ALL=C sort -z | tar --null -T - --no-recursion -cf $@ --numeric-owner -C $(TARGET_DIR) The find | sort part we probably want to factor into the rootfs infra, because the same thing will have to be done for all other rootfs types as well. But that refactoring can be done later (when the other rootfs types are tackled). Regards, Arnout > endef > > $(eval $(call ROOTFS_TARGET,tar)) > diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh > index 932d3c4..cef6d82 100755 > --- a/support/dependencies/check-host-tar.sh > +++ b/support/dependencies/check-host-tar.sh > @@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then > minor=0 > fi > > -# Minimal version = 1.17 (previous versions do not correctly unpack archives > -# containing hard-links if the --strip-components option is used). > +# Minimal version = 1.28 (previous versions do not does not support --sort=name) > major_min=1 > -minor_min=17 > +minor_min=28 > if [ $major -gt $major_min ]; then > echo $tar > else >
On Friday 18 November 2016 22:28:23 Thomas Petazzoni wrote: > On Fri, 18 Nov 2016 14:44:50 +0100, Jérôme Pouiller wrote: > > > > Yes, I know, but it's a pain to force everyone using an old distro to > > > build host-tar. Maybe we should make this conditional on > > > BR2_REPRODUCIBLE? > > > > Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months > > (maybe simultaneous with next Buildroot version). > > > > So, IMHO it not justified to add a dirty condition in check-host-tar.sh. > > I think you don't really realize how big companies work. They will be > using such version in 3 or 5 years maybe. > > We have people still using RHEL5, even though it's almost 10 years old, > and its support going to stop next year. Supporting old distributions > is important for Buildroot, so you can't just sweep away the problem by > pretending that it doesn't exist with modern distributions. This patch did not break support for old distributions, but OK.
On Saturday 19 November 2016 10:12:24 Arnout Vandecappelle wrote: > On 18-11-16 10:10, Jérôme Pouiller wrote: > > In order to make tar images reproducible, we use --sort flag. However, > > this flags is available only from tar 1.28. So we also bump necessary > > host-tar version. > > > > This work was sponsored by `BA Robotic Systems'. > > > > Signed-off-by: Jérôme Pouiller <jezz@sysmic.org> > > --- > > fs/tar/tar.mk | 2 +- > > support/dependencies/check-host-tar.sh | 5 ++--- > > 2 files changed, 3 insertions(+), 4 deletions(-) > > > > diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk > > index 11c69c5..5a1b263 100644 > > --- a/fs/tar/tar.mk > > +++ b/fs/tar/tar.mk > > @@ -7,7 +7,7 @@ > > TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) > > > > define ROOTFS_TAR_CMD > > - tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . > > + tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) . > > Instead of requiring a specific version of tar, I would prefer to use the same > solution we already have in the git wrapper, and which is also what is proposed > by reproducible-builds.org: > > find src -print0 | LC_ALL=C sort -z | > tar --null -T - --no-recursion -cf $@ --numeric-owner -C $(TARGET_DIR) > > The find | sort part we probably want to factor into the rootfs infra, because > the same thing will have to be done for all other rootfs types as well. But that > refactoring can be done later (when the other rootfs types are tackled). It makes sense. I will do that.
diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk index 11c69c5..5a1b263 100644 --- a/fs/tar/tar.mk +++ b/fs/tar/tar.mk @@ -7,7 +7,7 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) define ROOTFS_TAR_CMD - tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . + tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) . endef $(eval $(call ROOTFS_TARGET,tar)) diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh index 932d3c4..cef6d82 100755 --- a/support/dependencies/check-host-tar.sh +++ b/support/dependencies/check-host-tar.sh @@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then minor=0 fi -# Minimal version = 1.17 (previous versions do not correctly unpack archives -# containing hard-links if the --strip-components option is used). +# Minimal version = 1.28 (previous versions do not does not support --sort=name) major_min=1 -minor_min=17 +minor_min=28 if [ $major -gt $major_min ]; then echo $tar else
In order to make tar images reproducible, we use --sort flag. However, this flags is available only from tar 1.28. So we also bump necessary host-tar version. This work was sponsored by `BA Robotic Systems'. Signed-off-by: Jérôme Pouiller <jezz@sysmic.org> --- fs/tar/tar.mk | 2 +- support/dependencies/check-host-tar.sh | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-)