<PACKAGE>_SOURCE with <PACKAGE>_SITE_METHOD = git can result to tar.gz with mismatched file extension

Hello,

I noticed odd behavior when <PACKAGE>_SOURCE is set whereas <PACKAGE>
is downloaded via git (<PACKAGE>_SITE_METHOD = git).

For example, I'm trying to bump Git package to commit
c000d916380bb59db69c78546928eadd076b9c7d (v2.39.0-rc0). On the makefile
(package/git/git.mk), I bumped by:

---- >8 ----

(note: I fetch from my local Git project repository, hence file:// URI).

When I fetch the sources (make source), the generated tarball is actually
.tar.gz with $(GIT_VERSION) as the filename. This causes extracting the
tarball to fail. In this case, the tarball is decompressed first with
xzcat, and it complains due to unrecognized compressed format (gzip
instead of xz).

Thanks.

Hello,

+Yann Morin in Cc.

On Fri, 2 Dec 2022 21:02:06 +0700
Bagas Sanjaya <bagasdotme@gmail.com> wrote:

> Hello,
> 
> I noticed odd behavior when <PACKAGE>_SOURCE is set whereas <PACKAGE>
> is downloaded via git (<PACKAGE>_SITE_METHOD = git).
> 
> For example, I'm trying to bump Git package to commit
> c000d916380bb59db69c78546928eadd076b9c7d (v2.39.0-rc0). On the makefile
> (package/git/git.mk), I bumped by:
> 
> ---- >8 ----  
> diff --git a/package/git/git.mk b/package/git/git.mk
> index dc587170e8..1990bf8e67 100644
> --- a/package/git/git.mk
> +++ b/package/git/git.mk
> @@ -4,9 +4,10 @@
>  #
>  ################################################################################
>  
> -GIT_VERSION = 2.31.4
> +GIT_VERSION = c000d916380bb59db69c78546928eadd076b9c7d
>  GIT_SOURCE = git-$(GIT_VERSION).tar.xz
> -GIT_SITE = $(BR2_KERNEL_MIRROR)/software/scm/git
> +GIT_SITE = file:///home/bagas/repo/git-scm
> +GIT_SITE_METHOD = git
>  GIT_LICENSE = GPL-2.0, LGPL-2.1+
>  GIT_LICENSE_FILES = COPYING LGPL-2.1
>  GIT_CPE_ID_VENDOR = git-scm
> 
> (note: I fetch from my local Git project repository, hence file:// URI).
> 
> When I fetch the sources (make source), the generated tarball is actually
> .tar.gz with $(GIT_VERSION) as the filename. This causes extracting the
> tarball to fail. In this case, the tarball is decompressed first with
> xzcat, and it complains due to unrecognized compressed format (gzip
> instead of xz).

I think you're right. We assume that _SOURCE is not defined when a
_SITE_METHOD using a VCS (git, cvs, svn, hg) is used.

I guess we have two options here:

 (1) Detect this situation, and error out, to prevent this situation
 from happening

 (2) Actually support overriding _SOURCE, but in this case, we should
 comply with the specified compression algorithm

Yann, thoughts? (1) seems easier to me, I don't know if the benefits of
(2) are really relevant.

Thomas

Thomas, Bagas, All,

On 2022-12-03 15:52 +0100, Thomas Petazzoni spake thusly:
> On Fri, 2 Dec 2022 21:02:06 +0700
> Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > Hello,
> > 
> > I noticed odd behavior when <PACKAGE>_SOURCE is set whereas <PACKAGE>
> > is downloaded via git (<PACKAGE>_SITE_METHOD = git).
> > 
> > For example, I'm trying to bump Git package to commit
> > c000d916380bb59db69c78546928eadd076b9c7d (v2.39.0-rc0). On the makefile
> > (package/git/git.mk), I bumped by:
> > 
> > ---- >8 ----  
> > diff --git a/package/git/git.mk b/package/git/git.mk
> > index dc587170e8..1990bf8e67 100644
> > --- a/package/git/git.mk
> > +++ b/package/git/git.mk
> > @@ -4,9 +4,10 @@
> >  #
> >  ################################################################################
> >  
> > -GIT_VERSION = 2.31.4
> > +GIT_VERSION = c000d916380bb59db69c78546928eadd076b9c7d
> >  GIT_SOURCE = git-$(GIT_VERSION).tar.xz
> > -GIT_SITE = $(BR2_KERNEL_MIRROR)/software/scm/git
> > +GIT_SITE = file:///home/bagas/repo/git-scm
> > +GIT_SITE_METHOD = git
> >  GIT_LICENSE = GPL-2.0, LGPL-2.1+
> >  GIT_LICENSE_FILES = COPYING LGPL-2.1
> >  GIT_CPE_ID_VENDOR = git-scm
> > 
> > (note: I fetch from my local Git project repository, hence file:// URI).
> > 
> > When I fetch the sources (make source), the generated tarball is actually
> > .tar.gz with $(GIT_VERSION) as the filename. This causes extracting the
> > tarball to fail. In this case, the tarball is decompressed first with
> > xzcat, and it complains due to unrecognized compressed format (gzip
> > instead of xz).
> 
> I think you're right. We assume that _SOURCE is not defined when a
> _SITE_METHOD using a VCS (git, cvs, svn, hg) is used.
> 
> I guess we have two options here:
> 
>  (1) Detect this situation, and error out, to prevent this situation
>  from happening
> 
>  (2) Actually support overriding _SOURCE, but in this case, we should
>  comply with the specified compression algorithm
> 
> Yann, thoughts? (1) seems easier to me, I don't know if the benefits of
> (2) are really relevant.

I think the only meaningful solution is to go for (1): detect and abort.
I too do not think letting packages each request a different compression
would be interesting...

If we believe that gzip is too slow/big, then we can think about
switching to another scheme globaly, but that's orthogonal...

So, yes: detect and abort.

Regards,
Yann E. MORIN.

Hello,

On Sat, 3 Dec 2022 17:05:06 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> > Yann, thoughts? (1) seems easier to me, I don't know if the benefits of
> > (2) are really relevant.  
> 
> I think the only meaningful solution is to go for (1): detect and abort.
> I too do not think letting packages each request a different compression
> would be interesting...
> 
> If we believe that gzip is too slow/big, then we can think about
> switching to another scheme globaly, but that's orthogonal...

Actually, (2) would have an advantage: it would allow to migrate from
one compression to another package per package, instead of requiring a
flag day where we switch all VCS-fetched packages to the new
compression.

Best regards,

Thomas

Thomas, All,

On 2022-12-03 18:23 +0100, Thomas Petazzoni spake thusly:
> On Sat, 3 Dec 2022 17:05:06 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > > Yann, thoughts? (1) seems easier to me, I don't know if the benefits of
> > > (2) are really relevant.  
> > I think the only meaningful solution is to go for (1): detect and abort.
> > I too do not think letting packages each request a different compression
> > would be interesting...
> > If we believe that gzip is too slow/big, then we can think about
> > switching to another scheme globaly, but that's orthogonal...
> Actually, (2) would have an advantage: it would allow to migrate from
> one compression to another package per package, instead of requiring a
> flag day where we switch all VCS-fetched packages to the new
> compression.

Sure, but we already need a flag-day when we actually change the way we
generate the archives from VCS checkouts:
    5b95a5dc27c0 support/download: change format of archives generated from git
    c043ecb20ce6 support/download: change format of archives generated from svn
    c92be85e3a29 support/download: make the svn backend more reproducible

Having a flag-day is not too bad, as we can quite easily script the hash
updates...

So, I still believe we should not allow _SOURCE when _SITE_METHOD is one
of our VCS backends.

Regards,
Yann E. MORIN.

On 12/3/22 23:05, Yann E. MORIN wrote:
>> I think you're right. We assume that _SOURCE is not defined when a
>> _SITE_METHOD using a VCS (git, cvs, svn, hg) is used.
>>
>> I guess we have two options here:
>>
>>  (1) Detect this situation, and error out, to prevent this situation
>>  from happening
>>
>>  (2) Actually support overriding _SOURCE, but in this case, we should
>>  comply with the specified compression algorithm
>>
>> Yann, thoughts? (1) seems easier to me, I don't know if the benefits of
>> (2) are really relevant.
> 
> I think the only meaningful solution is to go for (1): detect and abort.
> I too do not think letting packages each request a different compression
> would be interesting...
> 
> If we believe that gzip is too slow/big, then we can think about
> switching to another scheme globaly, but that's orthogonal...
> 
> So, yes: detect and abort.
> 

Actually I think we can just do ``tar xvf`` since tar will automatically
figure out the decompressor to use.

Bagas, All,

On 2022-12-04 19:31 +0700, Bagas Sanjaya spake thusly:
> On 12/3/22 23:05, Yann E. MORIN wrote:
> >> I guess we have two options here:
> >>  (1) Detect this situation, and error out, to prevent this situation
> >>  from happening
> >>  (2) Actually support overriding _SOURCE, but in this case, we should
> >>  comply with the specified compression algorithm
[--SNIP--]
> > So, yes: detect and abort.
> Actually I think we can just do ``tar xvf`` since tar will automatically
> figure out the decompressor to use.

The issue is not so much about decompressing, than about compressing.

The download backends for VCS all expect to generate gzip-compressed
tarballs; see: support/download/git@231, support/download/svn@67,
support/download/cvs@70. For mercurial, it's even going deeper, as this
is ingrained in hg itself, and we rely on hg to generate the archive,
see support/download/hg@48..50 (it knows other compression formats, but
that may not align with the one we'd choose for the others).

The reason to request for a compression other than gzip is to achieve
one or more of:
  - decreasing the size of the generated archives;
  - decreasing the time needed to compress the archives;
  - decreasing the time needed to decompress the archives.

Aiming for those goals I believe only really makes sense globally, not
on a per-package basis.

If we were to go for a per-package support, we would need to convert all
our download backends anyway, and as I said previously, I don't think it
would be too cumbersome to migrate all the packages in one go at the
same time:
  - no cvs-hosted package;
  - a single svn-hosted package for which we have a hash: libxmlrpc (the
    other four have no hash);
  - two hg-hosted packages: dvb-apps, python-pygmes (the other three
    have no hash);
  - 112 git-hosted packages, so a bit more than 100 with a hash, and
    updating them can be easily scripted (that's what I did when we
    introduced the -br1 suffix for git).

Sure, we would also need to add that support to {go,cargo}-post-process,
but we do not have that many impacted packages either. However, for
those, I could well see a reason to have a per-package support, that we
shoved aside when introducing the download post-process: if we have to
download an unvendored archive that is not a tar.gz already, then we'd
have no way to represent that. But we assumed that all the packages that
would require vendoring would come from github or gitlab, they would all
be .tar.gz already, and that if we were to download a released archive,
it would already be vendored; this is turning to be actually the case
in practice: we only use archives from github/gitlab, so the point is
mostly academic.

So I still think we do not need support for different per-package
compression support, and that we shouldddetect the case that _SOURCE is
not set for a VCS-based download.

Regards,
Yann E. MORIN.