diff mbox series

[1/1] support/dependencies: bump minimal tar version to 1.35

Message ID 20231018141155.533944-1-vfazio@gmail.com
State Changes Requested
Headers show
Series [1/1] support/dependencies: bump minimal tar version to 1.35 | expand

Commit Message

Vincent Fazio Oct. 18, 2023, 2:11 p.m. UTC 
GNU tar 1.35 includes a breaking change [0] that changes the tar header
created for each regular file even for "stable" formats like pax (which
we use in support/download/helpers::mk_tar_gz).

Previously, normal files had the devmajor/devminor fields of the header
filled with ASCII zero (0x30). These fields are now null which also
affects the checksum value in the header as well.

  $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output)
  10c10
  < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000  075.015167. 0...
  ---
  > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000  075.013727. 0...
  21,22c21,22
  < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030  .........0000000
  < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000  .0000000........
  ---
  > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
  > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................

This has the consequence of causing hash mismatches for any tarball
created with a version prior to 1.35.

Instead of reverting back to a host specific version (see 37a909cacf)
and adding a cap to the maximal tar version (see ec50e407be), just
establish that 1.35+ should be used going forward.

This change requires an updated BR_FMT_VERSION for both git and svn.

[0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
---
This patch should be coordinated with other patches [1] that update the
BR_FMT_VERSION revision for git or svn as all of the tarball hashes will
need to be regenerated tree-wide.

[1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742
---
 package/pkg-download.mk                |  4 ++--
 support/dependencies/check-host-tar.sh | 10 ++++++----
 2 files changed, 8 insertions(+), 6 deletions(-)

Comments

Arnout Vandecappelle Oct. 18, 2023, 5:43 p.m. UTC | #1 
On 18/10/2023 16:11, Vincent Fazio wrote:
> GNU tar 1.35 includes a breaking change [0] that changes the tar header
> created for each regular file even for "stable" formats like pax (which
> we use in support/download/helpers::mk_tar_gz).

  The funny thing is that the tar docs have a section about making the archive 
reproducible [1]. They call it "more reproducible", though, so perhaps they 
realize that they fail...


> Previously, normal files had the devmajor/devminor fields of the header
> filled with ASCII zero (0x30). These fields are now null which also
> affects the checksum value in the header as well.
> 
>    $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output)
>    10c10
>    < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000  075.015167. 0...
>    ---
>    > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000  075.013727. 0...
>    21,22c21,22
>    < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030  .........0000000
>    < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000  .0000000........
>    ---
>    > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
>    > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 
> This has the consequence of causing hash mismatches for any tarball
> created with a version prior to 1.35.
> 
> Instead of reverting back to a host specific version (see 37a909cacf)
> and adding a cap to the maximal tar version (see ec50e407be), just
> establish that 1.35+ should be used going forward.
> 
> This change requires an updated BR_FMT_VERSION for both git and svn.
> 
> [0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0
> 
> Signed-off-by: Vincent Fazio <vfazio@gmail.com>
> ---
> This patch should be coordinated with other patches [1] that update the
> BR_FMT_VERSION revision for git or svn as all of the tarball hashes will
> need to be regenerated tree-wide.
> 
> [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742
> ---
>   package/pkg-download.mk                |  4 ++--
>   support/dependencies/check-host-tar.sh | 10 ++++++----
>   2 files changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index e5cd83d859..d33eb2d811 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>   
>   # Version of the format of the archives we generate in the corresponding
>   # download backend:
> -BR_FMT_VERSION_git = -br1
> -BR_FMT_VERSION_svn = -br3
> +BR_FMT_VERSION_git = -br2
> +BR_FMT_VERSION_svn = -br4

  Vendored cargo and go packages will have the same issue, no? Those will need 
to be updated as well...

  Perhaps we should look for an alternative archive format that is actually 
reproducible. If we anyway have to do a mass update, then it's better if now is 
the last time... Though I don't know if an actually reproducible format exists. 
GNU cpio has the --reproducible option, but I don't know if it guarantees 
reproducibility across cpio versions. Also, we still have gzip to contend with.

  Regards,
  Arnout

[1] 
https://www.gnu.org/software/tar/manual/html_chapter/Formats.html#Reproducibility


>   
>   DL_WRAPPER = support/download/dl-wrapper
>   
> diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
> index b7d607a47a..d56d0242a9 100755
> --- a/support/dependencies/check-host-tar.sh
> +++ b/support/dependencies/check-host-tar.sh
> @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then
>       exit 1
>   fi
>   
> -# Minimal version = 1.27 (previous versions do not correctly unpack archives
> -# containing hard-links if the --strip-components option is used or create
> -# different gnu long link headers for path elements > 100 characters).
> +# Minimal version = 1.35
> +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
> +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
> +# generated with 1.35 will not hash to the same value as those generated
> +# by previous versions of tar.
>   major_min=1
> -minor_min=27
> +minor_min=35
>   
>   if [ $major -lt $major_min ]; then
>   	# echo nothing: no suitable tar found
Yann E. MORIN Oct. 21, 2023, 8:25 p.m. UTC | #2 
Vincent, Arnout, All,

On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly:
> On 18/10/2023 16:11, Vincent Fazio wrote:
> >GNU tar 1.35 includes a breaking change [0] that changes the tar header
> >created for each regular file even for "stable" formats like pax (which
> >we use in support/download/helpers::mk_tar_gz).
>  The funny thing is that the tar docs have a section about making the
> archive reproducible [1]. They call it "more reproducible", though, so
> perhaps they realize that they fail...

Yes, I think this is what served as a base for Vincent to come up with
the options we have in the tar helper.

[--SNIP--]
> >diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> >index e5cd83d859..d33eb2d811 100644
> >--- a/package/pkg-download.mk
> >+++ b/package/pkg-download.mk
> >@@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
> >  # Version of the format of the archives we generate in the corresponding
> >  # download backend:
> >-BR_FMT_VERSION_git = -br1
> >-BR_FMT_VERSION_svn = -br3
> >+BR_FMT_VERSION_git = -br2
> >+BR_FMT_VERSION_svn = -br4
>  Vendored cargo and go packages will have the same issue, no? Those will
> need to be updated as well...

Good catch, indeed. I'll try and resurect my patch to add a -br suffix
to those...

>  Perhaps we should look for an alternative archive format that is actually
> reproducible. If we anyway have to do a mass update, then it's better if now
> is the last time... Though I don't know if an actually reproducible format
> exists.

There is a lost there:
    https://en.wikipedia.org/wiki/List_of_archive_formats

but it does not provide any information about stability...

We could of course come up with our own, but I would very much like that
we do avoid that at all cost. The odds of us failing if we go that route
are higher than those of us succeeding, I'd say...

> GNU cpio has the --reproducible option, but I don't know if it
> guarantees reproducibility across cpio versions.

[0] states that --reproducible "is equivalent to ‘--ignore-devno
--ignore-dirnlink --renumber-inodes’", which to me means that some
metadata is not recorded (or is faked), not that there is any
guarantee on the stability of the stream.

[0] https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

> Also, we still have gzip to
> contend with.

I did test that a while ago, and gzip *has* been ultra-super stable over
the years, to the point that all the versions that I could build (we was
almost back to the initial release 30 yerrs ago) did produce the same
output as the latest version. So, if I had to worry about
reproducibility, I would not look at gzip.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> [1] https://www.gnu.org/software/tar/manual/html_chapter/Formats.html#Reproducibility
> 
> 
> >  DL_WRAPPER = support/download/dl-wrapper
> >diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
> >index b7d607a47a..d56d0242a9 100755
> >--- a/support/dependencies/check-host-tar.sh
> >+++ b/support/dependencies/check-host-tar.sh
> >@@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then
> >      exit 1
> >  fi
> >-# Minimal version = 1.27 (previous versions do not correctly unpack archives
> >-# containing hard-links if the --strip-components option is used or create
> >-# different gnu long link headers for path elements > 100 characters).
> >+# Minimal version = 1.35
> >+# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
> >+# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
> >+# generated with 1.35 will not hash to the same value as those generated
> >+# by previous versions of tar.
> >  major_min=1
> >-minor_min=27
> >+minor_min=35
> >  if [ $major -lt $major_min ]; then
> >  	# echo nothing: no suitable tar found
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN Oct. 21, 2023, 8:36 p.m. UTC | #3 
Vincent, Arnout, All,

On 2023-10-21 22:25 +0200, Yann E. MORIN spake thusly:
> On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly:
> > On 18/10/2023 16:11, Vincent Fazio wrote:
> > >GNU tar 1.35 includes a breaking change [0] that changes the tar header
> > >created for each regular file even for "stable" formats like pax (which
> > >we use in support/download/helpers::mk_tar_gz).
[--SNIP--]
> > >diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > >index e5cd83d859..d33eb2d811 100644
> > >--- a/package/pkg-download.mk
> > >+++ b/package/pkg-download.mk
> > >@@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
> > >  # Version of the format of the archives we generate in the corresponding
> > >  # download backend:
> > >-BR_FMT_VERSION_git = -br1
> > >-BR_FMT_VERSION_svn = -br3
> > >+BR_FMT_VERSION_git = -br2
> > >+BR_FMT_VERSION_svn = -br4
> >  Vendored cargo and go packages will have the same issue, no? Those will
> > need to be updated as well...
> Good catch, indeed. I'll try and resurect my patch to add a -br suffix
> to those...

I think it was basically just that:

     b/package/pkg-download.mk |    1     1     0     0 +
     b/package/pkg-utils.mk    |    2     1     1     0 +-
     2 files changed, 2 insertions(+), 1 deletion(-)

    diff --git a/package/pkg-download.mk b/package/pkg-download.mk
    index e5cd83d859..5ad3f2baaf 100644
    --- a/package/pkg-download.mk
    +++ b/package/pkg-download.mk
    @@ -22,6 +22,7 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
     # download backend:
     BR_FMT_VERSION_git = -br1
     BR_FMT_VERSION_svn = -br3
    +BR_FMT_VERSION_cargo = -cargo1

     DL_WRAPPER = support/download/dl-wrapper

    diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
    index 7c670ee615..95a8452db8 100644
    --- a/package/pkg-utils.mk
    +++ b/package/pkg-utils.mk
    @@ -48,7 +48,7 @@ pkgname = $(lastword $(subst /, ,$(pkgdir)))
     # Helper to build the extension for a package archive, based on various
     # conditions.
     # $(1): upper-case package name
    -pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz
    +pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD))$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)).tar.gz

     # Define extractors for different archive suffixes
     INFLATE.bz2  = $(BZCAT)

Toally untested.

Regards,
Yann E. MORIN.
Arnout Vandecappelle Oct. 23, 2023, 7:45 a.m. UTC | #4 
On 21/10/2023 22:36, Yann E. MORIN wrote:
> Vincent, Arnout, All,
> 
> On 2023-10-21 22:25 +0200, Yann E. MORIN spake thusly:
>> On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly:
>>> On 18/10/2023 16:11, Vincent Fazio wrote:
>>>> GNU tar 1.35 includes a breaking change [0] that changes the tar header
>>>> created for each regular file even for "stable" formats like pax (which
>>>> we use in support/download/helpers::mk_tar_gz).
> [--SNIP--]
>>>> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
>>>> index e5cd83d859..d33eb2d811 100644
>>>> --- a/package/pkg-download.mk
>>>> +++ b/package/pkg-download.mk
>>>> @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>>>>   # Version of the format of the archives we generate in the corresponding
>>>>   # download backend:
>>>> -BR_FMT_VERSION_git = -br1
>>>> -BR_FMT_VERSION_svn = -br3
>>>> +BR_FMT_VERSION_git = -br2
>>>> +BR_FMT_VERSION_svn = -br4
>>>   Vendored cargo and go packages will have the same issue, no? Those will
>>> need to be updated as well...
>> Good catch, indeed. I'll try and resurect my patch to add a -br suffix
>> to those...
> 
> I think it was basically just that:
> 
>       b/package/pkg-download.mk |    1     1     0     0 +
>       b/package/pkg-utils.mk    |    2     1     1     0 +-
>       2 files changed, 2 insertions(+), 1 deletion(-)
> 
>      diff --git a/package/pkg-download.mk b/package/pkg-download.mk
>      index e5cd83d859..5ad3f2baaf 100644
>      --- a/package/pkg-download.mk
>      +++ b/package/pkg-download.mk
>      @@ -22,6 +22,7 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>       # download backend:
>       BR_FMT_VERSION_git = -br1
>       BR_FMT_VERSION_svn = -br3
>      +BR_FMT_VERSION_cargo = -cargo1
> 
>       DL_WRAPPER = support/download/dl-wrapper
> 
>      diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>      index 7c670ee615..95a8452db8 100644
>      --- a/package/pkg-utils.mk
>      +++ b/package/pkg-utils.mk
>      @@ -48,7 +48,7 @@ pkgname = $(lastword $(subst /, ,$(pkgdir)))
>       # Helper to build the extension for a package archive, based on various
>       # conditions.
>       # $(1): upper-case package name
>      -pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz
>      +pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD))$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)).tar.gz

  When POST_PROCESS is used, the SITE_METHOD is not actually relevant (we don't 
actually make a tarball of the git, or at least we don't hash it). So maybe:

pkg_source_ext = $(or 
$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)),$(BR_FMT_VERSION_$($(1)_SITE_METHOD))).tar.gz


  Then we an also make BR_FMT_VERSION_cargo = -br4 (not -br1 or -br3 to avoid 
conflict with already existing git or svn based tarballs).

  Regards,
  Arnout

> 
>       # Define extractors for different archive suffixes
>       INFLATE.bz2  = $(BZCAT)
> 
> Toally untested.
> 
> Regards,
> Yann E. MORIN.
>
Arnout Vandecappelle Nov. 8, 2023, 9:01 p.m. UTC | #5 
Hi Vincent,

On 18/10/2023 16:11, Vincent Fazio wrote:
> GNU tar 1.35 includes a breaking change [0] that changes the tar header
> created for each regular file even for "stable" formats like pax (which
> we use in support/download/helpers::mk_tar_gz).
> 
> Previously, normal files had the devmajor/devminor fields of the header
> filled with ASCII zero (0x30). These fields are now null which also
> affects the checksum value in the header as well.
> 
>    $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output)
>    10c10
>    < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000  075.015167. 0...
>    ---
>    > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000  075.013727. 0...
>    21,22c21,22
>    < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030  .........0000000
>    < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000  .0000000........
>    ---
>    > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
>    > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 
> This has the consequence of causing hash mismatches for any tarball
> created with a version prior to 1.35.
> 
> Instead of reverting back to a host specific version (see 37a909cacf)
> and adding a cap to the maximal tar version (see ec50e407be), just
> establish that 1.35+ should be used going forward.
> 
> This change requires an updated BR_FMT_VERSION for both git and svn.
> 
> [0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0
> 
> Signed-off-by: Vincent Fazio <vfazio@gmail.com>

  We discussed with 3 more maintainers (Peter, Romain and myself) and came to 
the conclusion that it is really too annoying that almost everybody would have 
to build host-tar (unconditionally!). At the moment, of the main distros, only 
Fedora 39 (just released) and Arch Linux already have 1.35.

  The idea is to apply this "bump minimal tar" only when more major distros have 
switched to 1.35 or later. Or, perhaps with the additional time we find an 
alternative that is hopefully more stable. I was thinking of switching to 
sharutils, for example - last release is from 2015 so very stable! :-)


  So, for the time being, we can do the following:

- in check-host-tar.sh, check that system tar < 1.35;
- in package/tar, add a patch that reverts 738de9ecde (probably need to revert 
c3663838 as well).


  Regarding ec50e407be: we're indeed in a similar situation as back then. It's 
not ideal to have a maximal version, but for now it's better than putting the 
minimum at 1.35.


  Regards,
  Arnout


> ---
> This patch should be coordinated with other patches [1] that update the
> BR_FMT_VERSION revision for git or svn as all of the tarball hashes will
> need to be regenerated tree-wide.
> 
> [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742
> ---
>   package/pkg-download.mk                |  4 ++--
>   support/dependencies/check-host-tar.sh | 10 ++++++----
>   2 files changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index e5cd83d859..d33eb2d811 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>   
>   # Version of the format of the archives we generate in the corresponding
>   # download backend:
> -BR_FMT_VERSION_git = -br1
> -BR_FMT_VERSION_svn = -br3
> +BR_FMT_VERSION_git = -br2
> +BR_FMT_VERSION_svn = -br4
>   
>   DL_WRAPPER = support/download/dl-wrapper
>   
> diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
> index b7d607a47a..d56d0242a9 100755
> --- a/support/dependencies/check-host-tar.sh
> +++ b/support/dependencies/check-host-tar.sh
> @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then
>       exit 1
>   fi
>   
> -# Minimal version = 1.27 (previous versions do not correctly unpack archives
> -# containing hard-links if the --strip-components option is used or create
> -# different gnu long link headers for path elements > 100 characters).
> +# Minimal version = 1.35
> +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
> +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
> +# generated with 1.35 will not hash to the same value as those generated
> +# by previous versions of tar.
>   major_min=1
> -minor_min=27
> +minor_min=35
>   
>   if [ $major -lt $major_min ]; then
>   	# echo nothing: no suitable tar found
Vincent Fazio Nov. 9, 2023, 3:18 p.m. UTC | #6 
All

On Wed, Nov 8, 2023 at 3:01 PM Arnout Vandecappelle <arnout@mind.be> wrote:

>   Hi Vincent,
>
> On 18/10/2023 16:11, Vincent Fazio wrote:
> > GNU tar 1.35 includes a breaking change [0] that changes the tar header
> > created for each regular file even for "stable" formats like pax (which
> > we use in support/download/helpers::mk_tar_gz).
> >
> > Previously, normal files had the devmajor/devminor fields of the header
> > filled with ASCII zero (0x30). These fields are now null which also
> > affects the checksum value in the header as well.
> >
> >    $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output)
> >    10c10
> >    < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000  075.015167. 0...
> >    ---
> >    > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000  075.013727. 0...
> >    21,22c21,22
> >    < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030  .........0000000
> >    < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000  .0000000........
> >    ---
> >    > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> >    > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> >
> > This has the consequence of causing hash mismatches for any tarball
> > created with a version prior to 1.35.
> >
> > Instead of reverting back to a host specific version (see 37a909cacf)
> > and adding a cap to the maximal tar version (see ec50e407be), just
> > establish that 1.35+ should be used going forward.
> >
> > This change requires an updated BR_FMT_VERSION for both git and svn.
> >
> > [0]
> https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0
> >
> > Signed-off-by: Vincent Fazio <vfazio@gmail.com>
>
>   We discussed with 3 more maintainers (Peter, Romain and myself) and came
> to
> the conclusion that it is really too annoying that almost everybody would
> have
> to build host-tar (unconditionally!). At the moment, of the main distros,
> only
>


The host-tar package takes between 20-45 seconds to build. I don't see that
as overly burdensome on a build that can potentially take upwards of half
an hour.



> Fedora 39 (just released) and Arch Linux already have 1.35.
>
>

We're already in a situation where people are using the latest Arch and
encountering this (a colleague of Thomas in fact).
It's what drove the investigation in October and this patch. They were
using an older buildroot, so there's not much that can be done there.
It's not ideal, but we can document the limitation with distributions using
tar 1.35 and recommend utilizing a build container that uses a previous
version of tar.



>   The idea is to apply this "bump minimal tar" only when more major
> distros have
> switched to 1.35 or later. Or, perhaps with the additional time we find an
> alternative that is hopefully more stable. I was thinking of switching to
> sharutils, for example - last release is from 2015 so very stable! :-)
>
>

Based on testing Yann did back in the day
(cbe95b1a455bbefcaa90a08cf3dd1a590630921e) the --format=posix format has
been stable since at least GNU tar 1.27 released 10 years ago.
I did open an issue in GNU tar for this change and asked about the
guarantee of stability: https://savannah.gnu.org/bugs/index.php?64875
Pending the outcome of that issue, we can maybe not do anything for now
except recommend against using distros with tar 1.35

I personally think it's better to do the bump and account for the behavior
change now. The build of host-tar 1.35 will eventually not be required as
the new version is adopted by distros vs sticking with 1.34 and then slowly
forcing more builds of older tar on newer distros.



>
  So, for the time being, we can do the following:
>
> - in check-host-tar.sh, check that system tar < 1.35;
> - in package/tar, add a patch that reverts 738de9ecde (probably need to
> revert
> c3663838 as well).
>


I don't agree with reverting these commits in 1.35 as it causes
functionally different behavior between BR built tar and distribution built
tar on the same version.
The only way to guarantee previous/stable behavior would be to _always_ use
BR built tar with reverts, which you didn't want to do above.

It would also make the recipe a bit funky, where we're specifically
applying a patch to 1.35 for the host because we rely on the behavior or
earlier versions
but probably don't want to do that for the target.



>
>
>   Regarding ec50e407be: we're indeed in a similar situation as back then.
> It's
> not ideal to have a maximal version, but for now it's better than putting
> the
> minimum at 1.35.
>
>

We could set the host tar version to 1.34, but it sounded like there were
concerns about security? The only CVE I see at a glance is CVE-2022-48303.
Upstream
https://git.savannah.gnu.org/cgit/tar.git/commit/?id=3da78400eafcccb97e2f2fd4b227ea40d794ede8
Oddly, Debian hasn't ported the patch but Ubuntu has.


I obviously defer to the decision of the maintainers, but wanted to voice
my thoughts.


>
>   Regards,
>   Arnout
>
>
> > ---
> > This patch should be coordinated with other patches [1] that update the
> > BR_FMT_VERSION revision for git or svn as all of the tarball hashes will
> > need to be regenerated tree-wide.
> >
> > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742
> > ---
> >   package/pkg-download.mk                |  4 ++--
> >   support/dependencies/check-host-tar.sh | 10 ++++++----
> >   2 files changed, 8 insertions(+), 6 deletions(-)
> >
> > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > index e5cd83d859..d33eb2d811 100644
> > --- a/package/pkg-download.mk
> > +++ b/package/pkg-download.mk
> > @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
> >
> >   # Version of the format of the archives we generate in the
> corresponding
> >   # download backend:
> > -BR_FMT_VERSION_git = -br1
> > -BR_FMT_VERSION_svn = -br3
> > +BR_FMT_VERSION_git = -br2
> > +BR_FMT_VERSION_svn = -br4
> >
> >   DL_WRAPPER = support/download/dl-wrapper
> >
> > diff --git a/support/dependencies/check-host-tar.sh
> b/support/dependencies/check-host-tar.sh
> > index b7d607a47a..d56d0242a9 100755
> > --- a/support/dependencies/check-host-tar.sh
> > +++ b/support/dependencies/check-host-tar.sh
> > @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then
> >       exit 1
> >   fi
> >
> > -# Minimal version = 1.27 (previous versions do not correctly unpack
> archives
> > -# containing hard-links if the --strip-components option is used or
> create
> > -# different gnu long link headers for path elements > 100 characters).
> > +# Minimal version = 1.35
> > +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
> > +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
> > +# generated with 1.35 will not hash to the same value as those generated
> > +# by previous versions of tar.
> >   major_min=1
> > -minor_min=27
> > +minor_min=35
> >
> >   if [ $major -lt $major_min ]; then
> >       # echo nothing: no suitable tar found
>
diff mbox series

Patch

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e5cd83d859..d33eb2d811 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -20,8 +20,8 @@  export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 # Version of the format of the archives we generate in the corresponding
 # download backend:
-BR_FMT_VERSION_git = -br1
-BR_FMT_VERSION_svn = -br3
+BR_FMT_VERSION_git = -br2
+BR_FMT_VERSION_svn = -br4
 
 DL_WRAPPER = support/download/dl-wrapper
 
diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
index b7d607a47a..d56d0242a9 100755
--- a/support/dependencies/check-host-tar.sh
+++ b/support/dependencies/check-host-tar.sh
@@ -27,11 +27,13 @@  if [ -n "${version_bsd}" ] ; then
     exit 1
 fi
 
-# Minimal version = 1.27 (previous versions do not correctly unpack archives
-# containing hard-links if the --strip-components option is used or create
-# different gnu long link headers for path elements > 100 characters).
+# Minimal version = 1.35
+# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
+# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
+# generated with 1.35 will not hash to the same value as those generated
+# by previous versions of tar.
 major_min=1
-minor_min=27
+minor_min=35
 
 if [ $major -lt $major_min ]; then
 	# echo nothing: no suitable tar found