[v2] linux-firmware: bump version to latest 1baa348

Message ID 20181108153322.17362-1-m.niestroj@grinn-global.com
State Changes Requested
Headers show
Series
  • [v2] linux-firmware: bump version to latest 1baa348
Related show

Commit Message

Marcin Niestrój Nov. 8, 2018, 3:33 p.m.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
---
 package/linux-firmware/linux-firmware.hash | 6 +++---
 package/linux-firmware/linux-firmware.mk   | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

Comments

Thomas Petazzoni Nov. 9, 2018, 8:57 p.m. | #1
Hello,

On Thu,  8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote:

> -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
> +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz

I am sorry, but this hash is still not good for me, I get:

  5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

I.e:

ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
ERROR: got     : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

And this morning, Yann E. Morin reported having the same hash as me:

08:07 < y_morin> kos_tom: I also have a different hash here.
08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

My system tar is 1.29, which is considered as a "good" version by
support/dependencies/check-host-tar.sh. I have nonetheless forced
building host-tar, and I still get the same hash.

I have uploaded the tarball at
https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
Could you upload the tarball that was generated on your side so that we
can compare them, and see where the problem lies ?

Best regards,

Thomas
Yann E. MORIN Nov. 9, 2018, 9:06 p.m. | #2
Marcin, Thomas, All,

On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly:
> On Thu,  8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote:
> 
> > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
> > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz
> 
> I am sorry, but this hash is still not good for me, I get:
> 
>   5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
> 
> I.e:
> 
> ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
> ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
> ERROR: got     : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
> 
> And this morning, Yann E. Morin reported having the same hash as me:
> 
> 08:07 < y_morin> kos_tom: I also have a different hash here.
> 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1

Right.

> My system tar is 1.29, which is considered as a "good" version by
> support/dependencies/check-host-tar.sh. I have nonetheless forced
> building host-tar, and I still get the same hash.

Recently, another user reported hash issues as well, and it turned out
that they had changed gzip to be really pigz (a parallel gzip). Can you
check if that is not your case too?

Regards,
Yann E. MORIN.

> I have uploaded the tarball at
> https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
> Could you upload the tarball that was generated on your side so that we
> can compare them, and see where the problem lies ?
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
Marcin Niestrój Nov. 13, 2018, 5:20 p.m. | #3
Hi All,

Yann E. MORIN <yann.morin.1998@free.fr> writes:

> Marcin, Thomas, All,
>
> On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly:
>> On Thu,  8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote:
>> 
>> > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
>> > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz
>> 
>> I am sorry, but this hash is still not good for me, I get:
>> 
>>   5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>> 
>> I.e:
>> 
>> ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
>> ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
>> ERROR: got     : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>> 
>> And this morning, Yann E. Morin reported having the same hash as me:
>> 
>> 08:07 < y_morin> kos_tom: I also have a different hash here.
>> 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>
> Right.

Thank you both for testing.

>
>> My system tar is 1.29, which is considered as a "good" version by
>> support/dependencies/check-host-tar.sh. I have nonetheless forced
>> building host-tar, and I still get the same hash.
>
> Recently, another user reported hash issues as well, and it turned out
> that they had changed gzip to be really pigz (a parallel gzip). Can you
> check if that is not your case too?

I have investigated the issue on my side. It turns out that gzip is
really the issue here.

I have two PCs with Arch Linux. PC_1 is the one I have prepared
linux-firmware. Here gzip package info on that machine:

  [mniestroj@gm ~]$ LANG=C yaourt -Si gzip
  Repository      : core
  Name            : gzip
  Version         : 1.9-1
  Description     : GNU compression utility
  Architecture    : x86_64
  URL             : https://www.gnu.org/software/gzip/
  Licenses        : GPL3
  Groups          : base  base-devel
  Provides        : None
  Depends On      : glibc  bash  less
  Optional Deps   : None
  Conflicts With  : None
  Replaces        : None
  Download Size   : 77.78 KiB
  Installed Size  : 150.00 KiB
  Packager        : S
  Build Date      : Mon Jan 22 00:52:54 2018
  Validated By    : MD5 Sum  SHA-256 Sum  Signature

PC_2 is also Arch Linux, but with slightly more up-to-date
packages. gzip package info looks like this:

  [macius@zm ~]$ LANG=C yaourt -Si gzip
  Repository      : core
  Name            : gzip
  Version         : 1.9-2
  Description     : GNU compression utility
  Architecture    : x86_64
  URL             : https://www.gnu.org/software/gzip/
  Licenses        : GPL3
  Groups          : base  base-devel
  Provides        : None
  Depends On      : glibc  bash  less
  Optional Deps   : None
  Conflicts With  : None
  Replaces        : None
  Download Size   : 78.14 KiB
  Installed Size  : 185.00 KiB
  Packager        : Allan McRae <allan@archlinux.org>
  Build Date      : Sat Nov 3 23:10:39 2018
  Validated By    : MD5 Sum  SHA-256 Sum  Signature

You can find differences in package here:
https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip

I have also checked output of gzip command on another PC with pigz
configured as gzip drop-in replacement. It outputs even different file,
with different sha256 hash.

I think the overall conclusion is that a host-gzip package is needed,
just like host-tar. In the meantime I will send v3 of this patch with
proper hash (the same as you calculated above).

>
> Regards,
> Yann E. MORIN.
>
>> I have uploaded the tarball at
>> https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
>> Could you upload the tarball that was generated on your side so that we
>> can compare them, and see where the problem lies ?

Thank for sharing. After gunzipping your version and mine I ended with
the same *.tar files. So the difference was clearly because of gzip.

Regards,
Marcin

>> 
>> Best regards,
>> 
>> Thomas
>> -- 
>> Thomas Petazzoni, CTO, Bootlin
>> Embedded Linux and Kernel engineering
>> https://bootlin.com
Marcin Niestrój Nov. 13, 2018, 5:32 p.m. | #4
Marcin Niestrój <m.niestroj@grinn-global.com> writes:

> Hi All,
>
> Yann E. MORIN <yann.morin.1998@free.fr> writes:
>
>> Marcin, Thomas, All,
>>
>> On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly:
>>> On Thu,  8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote:
>>> 
>>> > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
>>> > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz
>>> 
>>> I am sorry, but this hash is still not good for me, I get:
>>> 
>>>   5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>>> 
>>> I.e:
>>> 
>>> ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash:
>>> ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755
>>> ERROR: got     : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>>> 
>>> And this morning, Yann E. Morin reported having the same hash as me:
>>> 
>>> 08:07 < y_morin> kos_tom: I also have a different hash here.
>>> 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1
>>
>> Right.
>
> Thank you both for testing.
>
>>
>>> My system tar is 1.29, which is considered as a "good" version by
>>> support/dependencies/check-host-tar.sh. I have nonetheless forced
>>> building host-tar, and I still get the same hash.
>>
>> Recently, another user reported hash issues as well, and it turned out
>> that they had changed gzip to be really pigz (a parallel gzip). Can you
>> check if that is not your case too?
>
> I have investigated the issue on my side. It turns out that gzip is
> really the issue here.
>
> I have two PCs with Arch Linux. PC_1 is the one I have prepared
> linux-firmware. Here gzip package info on that machine:
>
>   [mniestroj@gm ~]$ LANG=C yaourt -Si gzip
>   Repository      : core
>   Name            : gzip
>   Version         : 1.9-1
>   Description     : GNU compression utility
>   Architecture    : x86_64
>   URL             : https://www.gnu.org/software/gzip/
>   Licenses        : GPL3
>   Groups          : base  base-devel
>   Provides        : None
>   Depends On      : glibc  bash  less
>   Optional Deps   : None
>   Conflicts With  : None
>   Replaces        : None
>   Download Size   : 77.78 KiB
>   Installed Size  : 150.00 KiB
>   Packager        : S
>   Build Date      : Mon Jan 22 00:52:54 2018
>   Validated By    : MD5 Sum  SHA-256 Sum  Signature
>
> PC_2 is also Arch Linux, but with slightly more up-to-date
> packages. gzip package info looks like this:

Forgot to mention - on PC_2 I get the right hash (the same as you).

>
>   [macius@zm ~]$ LANG=C yaourt -Si gzip
>   Repository      : core
>   Name            : gzip
>   Version         : 1.9-2
>   Description     : GNU compression utility
>   Architecture    : x86_64
>   URL             : https://www.gnu.org/software/gzip/
>   Licenses        : GPL3
>   Groups          : base  base-devel
>   Provides        : None
>   Depends On      : glibc  bash  less
>   Optional Deps   : None
>   Conflicts With  : None
>   Replaces        : None
>   Download Size   : 78.14 KiB
>   Installed Size  : 185.00 KiB
>   Packager        : Allan McRae <allan@archlinux.org>
>   Build Date      : Sat Nov 3 23:10:39 2018
>   Validated By    : MD5 Sum  SHA-256 Sum  Signature
>
> You can find differences in package here:
> https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip
>
> I have also checked output of gzip command on another PC with pigz
> configured as gzip drop-in replacement. It outputs even different file,
> with different sha256 hash.
>
> I think the overall conclusion is that a host-gzip package is needed,
> just like host-tar. In the meantime I will send v3 of this patch with
> proper hash (the same as you calculated above).
>
>>
>> Regards,
>> Yann E. MORIN.
>>
>>> I have uploaded the tarball at
>>> https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz.
>>> Could you upload the tarball that was generated on your side so that we
>>> can compare them, and see where the problem lies ?
>
> Thank for sharing. After gunzipping your version and mine I ended with
> the same *.tar files. So the difference was clearly because of gzip.
>
> Regards,
> Marcin
>
>>> 
>>> Best regards,
>>> 
>>> Thomas
>>> -- 
>>> Thomas Petazzoni, CTO, Bootlin
>>> Embedded Linux and Kernel engineering
>>> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Thomas Petazzoni Nov. 13, 2018, 8:02 p.m. | #5
Hello,

On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote:

> I have investigated the issue on my side. It turns out that gzip is
> really the issue here.

Thanks for the additional investigation!

> You can find differences in package here:
> https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip
> 
> I have also checked output of gzip command on another PC with pigz
> configured as gzip drop-in replacement. It outputs even different file,
> with different sha256 hash.
> 
> I think the overall conclusion is that a host-gzip package is needed,
> just like host-tar. In the meantime I will send v3 of this patch with
> proper hash (the same as you calculated above).

This is getting really horrible.

On my side, I have updated to Fedora 29, which ships tar 1.30, that we
consider as broken, and therefore now for everything single small build
that I do, I'm paying the price of building tar, which is super
annoying. This is especially annoying as tar 1.30 is only problematic
when *creating* tarballs of git-fetched packages, which clearly does
not happen at every build.

It would be truly horrible to also have to build our own gzip. Maybe we
need to somehow revisit our requirement that we need to have
reproducible tarballs for git-fetched packages ?

Best regards,

Thomas
Yann E. MORIN Nov. 13, 2018, 8:29 p.m. | #6
Thomas, All,

On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly:
> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote:
[--SNIP--]
> > I think the overall conclusion is that a host-gzip package is needed,
> > just like host-tar. In the meantime I will send v3 of this patch with
> > proper hash (the same as you calculated above).
> 
> This is getting really horrible.
> 
> On my side, I have updated to Fedora 29, which ships tar 1.30, that we
> consider as broken, and therefore now for everything single small build
> that I do, I'm paying the price of building tar, which is super
> annoying. This is especially annoying as tar 1.30 is only problematic
> when *creating* tarballs of git-fetched packages, which clearly does
> not happen at every build.

Well, we only need host-tar and host-gzip when creating tarballs, not
extracting them, correct? So, we could very well envision the fact that
host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when
FOO_SITE_METHOD == git (and they are needed, of course), no?

(note: tar is ugly, because its configure step takes ~26s, while the
build is only ~4s).

> It would be truly horrible to also have to build our own gzip. Maybe we
> need to somehow revisit our requirement that we need to have
> reproducible tarballs for git-fetched packages ?

I am ready to pay the price for reproducibility any time.

Regards,
Yann E. MORIN.
Thomas Petazzoni Nov. 13, 2018, 8:57 p.m. | #7
Hello,

On Tue, 13 Nov 2018 21:29:46 +0100, Yann E. MORIN wrote:

> > On my side, I have updated to Fedora 29, which ships tar 1.30, that we
> > consider as broken, and therefore now for everything single small build
> > that I do, I'm paying the price of building tar, which is super
> > annoying. This is especially annoying as tar 1.30 is only problematic
> > when *creating* tarballs of git-fetched packages, which clearly does
> > not happen at every build.  
> 
> Well, we only need host-tar and host-gzip when creating tarballs, not
> extracting them, correct? So, we could very well envision the fact that
> host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when
> FOO_SITE_METHOD == git (and they are needed, of course), no?

For host-tar, it's a bit complicated, because we have two conditions:

 - tar must be >= 1.27 for *extraction* to work properly.

 - tar must be < 1.30 for *creation* to work properly

So, if we add $(BR2_TAR_HOST_DEPENDENCY) to the package dependencies
only when FOO_SITE_METHOD = git, we would miss the case where it is
needed for extracting plain regular tarballs.

> > It would be truly horrible to also have to build our own gzip. Maybe we
> > need to somehow revisit our requirement that we need to have
> > reproducible tarballs for git-fetched packages ?  
> 
> I am ready to pay the price for reproducibility any time.

Well, for a "normal" build, I agree. When doing active BR development,
where one regularly builds from scratch small configurations, every
additional amount of time/stuff to build is annoying.

Thomas
Marcin Niestrój Nov. 13, 2018, 9:34 p.m. | #8
Hi,

Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

> Hello,
>
> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote:
>
>> I have investigated the issue on my side. It turns out that gzip is
>> really the issue here.
>
> Thanks for the additional investigation!

Some new findings now. It turned out that I actually used pigz instead
of gzip. It was not directly clear, because I had PATH variable
prepended with a directory containing "gzip -> /usr/bin/pigz". I have
set this up ~3 years ago and totally forgot about that. Sorry for
misleading that gzip version / patch could be the reason of wrong hash
in my case.

I need to really go through all the patches that got applied to
master...

>
>> You can find differences in package here:
>> https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip
>> 
>> I have also checked output of gzip command on another PC with pigz
>> configured as gzip drop-in replacement. It outputs even different file,
>> with different sha256 hash.
>> 
>> I think the overall conclusion is that a host-gzip package is needed,
>> just like host-tar. In the meantime I will send v3 of this patch with
>> proper hash (the same as you calculated above).
Arnout Vandecappelle Nov. 13, 2018, 11:54 p.m. | #9
On 13/11/2018 21:29, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly:
>> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote:
> [--SNIP--]
>>> I think the overall conclusion is that a host-gzip package is needed,
>>> just like host-tar. In the meantime I will send v3 of this patch with
>>> proper hash (the same as you calculated above).
>>
>> This is getting really horrible.
>>
>> On my side, I have updated to Fedora 29, which ships tar 1.30, that we
>> consider as broken, and therefore now for everything single small build
>> that I do, I'm paying the price of building tar, which is super
>> annoying. This is especially annoying as tar 1.30 is only problematic
>> when *creating* tarballs of git-fetched packages, which clearly does
>> not happen at every build.
> 
> Well, we only need host-tar and host-gzip when creating tarballs, not
> extracting them, correct? So, we could very well envision the fact that
> host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when
> FOO_SITE_METHOD == git (and they are needed, of course), no?

 Since we have the same problems sometimes with github tarballs, I think we need
a more fundamental solution. I would propose a new tarsha256 hash type, which
extracts the tarball to calculate the hash. In a simple version it's not so
complicated, something like

tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -

where tarsha256 contains:

{ echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
	sha256sum - | cut -f 1 -d ' '

As usually, entirely untested.

 Regards,
 Arnout

> 
> (note: tar is ugly, because its configure step takes ~26s, while the
> build is only ~4s).
> 
>> It would be truly horrible to also have to build our own gzip. Maybe we
>> need to somehow revisit our requirement that we need to have
>> reproducible tarballs for git-fetched packages ?
> 
> I am ready to pay the price for reproducibility any time.
> 
> Regards,
> Yann E. MORIN.
>
Yann E. MORIN Nov. 15, 2018, 7:05 p.m. | #10
Arnout, All,

On 2018-11-14 00:54 +0100, Arnout Vandecappelle spake thusly:
> On 13/11/2018 21:29, Yann E. MORIN wrote:
> > On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly:
> >> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote:
> >>> I think the overall conclusion is that a host-gzip package is needed,
> >>> just like host-tar. In the meantime I will send v3 of this patch with
> >>> proper hash (the same as you calculated above).
> >> This is getting really horrible.
> > Well, we only need host-tar and host-gzip when creating tarballs, not
> > extracting them, correct? So, we could very well envision the fact that
> > host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when
> > FOO_SITE_METHOD == git (and they are needed, of course), no?
> 
>  Since we have the same problems sometimes with github tarballs, I think we need
> a more fundamental solution. I would propose a new tarsha256 hash type, which
> extracts the tarball to calculate the hash. In a simple version it's not so
> complicated, something like
> 
> tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -
> 
> where tarsha256 contains:
> 
> { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
> 	sha256sum - | cut -f 1 -d ' '
> 
> As usually, entirely untested.

I don't like it, because this is totally non-standard. People expect to
be able to check hashes by running the *usual* XXXsum commands, directly
on the shipped/received files.

Introducing our own hash mechanism, how reliable or simple as it would
be, breaks this assumption, and the tool to actually check them is not
available at all except internally to Buildroot, so it is not possible
to reproduce the checks outside of Buildroot.

This goes counter one of the initial goal of hashes, which is to be able
to track archives and their validity across a supply chain, inbound (as
sent by a provider to Buildroot, to do the build) or outbound (as received
by a recepient, from Buildroot, for compliance) alike.

Regards,
Yann E. MORIN.
Thomas Petazzoni Nov. 16, 2018, 8:35 a.m. | #11
Hello,

On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote:

> >  Since we have the same problems sometimes with github tarballs, I think we need
> > a more fundamental solution. I would propose a new tarsha256 hash type, which
> > extracts the tarball to calculate the hash. In a simple version it's not so
> > complicated, something like
> > 
> > tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -
> > 
> > where tarsha256 contains:
> > 
> > { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
> > 	sha256sum - | cut -f 1 -d ' '
> > 
> > As usually, entirely untested.  
> 
> I don't like it, because this is totally non-standard. People expect to
> be able to check hashes by running the *usual* XXXsum commands, directly
> on the shipped/received files.
> 
> Introducing our own hash mechanism, how reliable or simple as it would
> be, breaks this assumption, and the tool to actually check them is not
> available at all except internally to Buildroot, so it is not possible
> to reproduce the checks outside of Buildroot.
> 
> This goes counter one of the initial goal of hashes, which is to be able
> to track archives and their validity across a supply chain, inbound (as
> sent by a provider to Buildroot, to do the build) or outbound (as received
> by a recepient, from Buildroot, for compliance) alike.

I understand this argument, but do you have some alternative solution ?

Even building our own host-tar and host-gzip doesn't solve entirely the
problem, because it doesn't solve the case of tarballs fetched from
github, that tend to change in subtle ways once in a while.

Best regards,

Thomas
Yann E. MORIN Nov. 20, 2018, 6:50 p.m. | #12
Thomas, All,

On 2018-11-16 09:35 +0100, Thomas Petazzoni spake thusly:
> On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote:
> > >  Since we have the same problems sometimes with github tarballs, I think we need
> > > a more fundamental solution. I would propose a new tarsha256 hash type, which
> > > extracts the tarball to calculate the hash. In a simple version it's not so
> > > complicated, something like
> > > 
> > > tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -
> > > 
> > > where tarsha256 contains:
> > > 
> > > { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
> > > 	sha256sum - | cut -f 1 -d ' '
> > > 
> > > As usually, entirely untested.  
> > 
> > I don't like it, because this is totally non-standard. People expect to
> > be able to check hashes by running the *usual* XXXsum commands, directly
> > on the shipped/received files.
> > 
> > Introducing our own hash mechanism, how reliable or simple as it would
> > be, breaks this assumption, and the tool to actually check them is not
> > available at all except internally to Buildroot, so it is not possible
> > to reproduce the checks outside of Buildroot.
> > 
> > This goes counter one of the initial goal of hashes, which is to be able
> > to track archives and their validity across a supply chain, inbound (as
> > sent by a provider to Buildroot, to do the build) or outbound (as received
> > by a recepient, from Buildroot, for compliance) alike.
> 
> I understand this argument, but do you have some alternative solution ?

No.

Also, the tarshasum thingy would not work either, because it is not
enough to hash the content of files. There are cases where the metadat
also matters, for example if a tarball is regenerated with new dates in
it. This could trigger a rebuild of parts of the package, for example if
configure.ac is now younger than configure. Obviously, we could add that
info to be hashed...

Until we find anotehr type of info we can't get via the --to-command,
like extended attributes for example...

So, I am still opposed to the tarshasum idea.

> Even building our own host-tar and host-gzip doesn't solve entirely the
> problem, because it doesn't solve the case of tarballs fetched from
> github, that tend to change in subtle ways once in a while.

For that, we can simply decide to ditch the github macro, and revert
to using Github for what it is: a git repository, and switch to the git
method.

Yes, yes, I know, this is a shame, especially since there are big
repositories out there. But OTOH, we now have the git cache, so it
should not be such a hassle in the end.

Regards,
Yann E. MORIN.
Arnout Vandecappelle Nov. 20, 2018, 11:47 p.m. | #13
On 20/11/2018 19:50, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2018-11-16 09:35 +0100, Thomas Petazzoni spake thusly:
>> On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote:
>>>>  Since we have the same problems sometimes with github tarballs, I think we need
>>>> a more fundamental solution. I would propose a new tarsha256 hash type, which
>>>> extracts the tarball to calculate the hash. In a simple version it's not so
>>>> complicated, something like
>>>>
>>>> tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum -
>>>>
>>>> where tarsha256 contains:
>>>>
>>>> { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \
>>>> 	sha256sum - | cut -f 1 -d ' '
>>>>
>>>> As usually, entirely untested.  
>>>
>>> I don't like it, because this is totally non-standard. People expect to
>>> be able to check hashes by running the *usual* XXXsum commands, directly
>>> on the shipped/received files.
>>>
>>> Introducing our own hash mechanism, how reliable or simple as it would
>>> be, breaks this assumption, and the tool to actually check them is not
>>> available at all except internally to Buildroot, so it is not possible
>>> to reproduce the checks outside of Buildroot.
>>>
>>> This goes counter one of the initial goal of hashes, which is to be able
>>> to track archives and their validity across a supply chain, inbound (as
>>> sent by a provider to Buildroot, to do the build) or outbound (as received
>>> by a recepient, from Buildroot, for compliance) alike.
>>
>> I understand this argument, but do you have some alternative solution ?
> 
> No.
> 
> Also, the tarshasum thingy would not work either, because it is not
> enough to hash the content of files. There are cases where the metadat
> also matters, for example if a tarball is regenerated with new dates in
> it. This could trigger a rebuild of parts of the package, for example if
> configure.ac is now younger than configure. Obviously, we could add that
> info to be hashed...

 Yes, I forgot to include the comment that we need to investigate which of
mtime, ctime and atime need to be included in the hash.


> Until we find anotehr type of info we can't get via the --to-command,
> like extended attributes for example...

 Other metadata we explicitly exclude. Hm, we don't but we should :-)


 BTW, turns out that https://github.com/mikemccabe/tarsump is not usable - it
doesn't sort the files, and it doesn't compute an overall hash.

 Regards,
 Arnout

> 
> So, I am still opposed to the tarshasum idea.
> 
>> Even building our own host-tar and host-gzip doesn't solve entirely the
>> problem, because it doesn't solve the case of tarballs fetched from
>> github, that tend to change in subtle ways once in a while.
> 
> For that, we can simply decide to ditch the github macro, and revert
> to using Github for what it is: a git repository, and switch to the git
> method.
> 
> Yes, yes, I know, this is a shame, especially since there are big
> repositories out there. But OTOH, we now have the git cache, so it
> should not be such a hassle in the end.
> 
> Regards,
> Yann E. MORIN.
>
Peter Korsgaard Nov. 21, 2018, 7:13 a.m. | #14
>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >  BTW, turns out that https://github.com/mikemccabe/tarsump is not usable - it
 > doesn't sort the files, and it doesn't compute an overall hash.

Isn't that just a question of piping the output to:

LANG=C sort | sha256sum

Or am I missing something?

Patch

diff --git a/package/linux-firmware/linux-firmware.hash b/package/linux-firmware/linux-firmware.hash
index ca6ad8f59d..6ade218888 100644
--- a/package/linux-firmware/linux-firmware.hash
+++ b/package/linux-firmware/linux-firmware.hash
@@ -1,11 +1,11 @@ 
 # Locally calculated
-sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz
+sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz
 sha256 8116433f4004fc0c24d72b3d9e497808b724aa0e5e1cd63fc1bf66b715b1e2e9 LICENCE.Abilis
 sha256 4b3ea5d5a03c0db81bee0bcb14b30d75b30ef568597bb5be7d4dee57f434265f LICENSE.amdgpu
 sha256 38f2037aa14631b4b29826d7a99379613c41a97064d1defdee30a7a022138b20 LICENCE.Marvell
 sha256 802b7014b26c606cf6248ae8b0ab1ce6d2d1b0db236d38dd269e676cd70710f2 LICENCE.atheros_firmware
 sha256 3b5eb392b2d9d8c46d6aae26d06c187e5ea3029b12d13bc2b8deb8b3ce6bfa53 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-4.txt
-sha256 c565861ff7c42f5df98e15239241f1f42614e5e15f362094a2d3e8da724dc842 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt
+sha256 8ce5c6ea0542bf4aac31fc3ae16a39792ad22d0eae4543063fac56fb3380f021 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt
 sha256 b16056fc91b82a0e3e8de8f86c2dac98201aa9dc3cbd33e8d38f1b087fcec30d LICENCE.broadcom_bcm43xx
 sha256 a5777f9e80aca0603b0648454de996168b1c530322550ccda94d6d78bcf6c061 LICENCE.chelsio_firmware
 sha256 60fbc9cccb455e1a3306c97db942d6f24fa93664be61d54c497637e6d0e2ae83 LICENCE.fw_sst_0f28
@@ -27,5 +27,5 @@  sha256 8542aeabf2761935122d693561e16766ce1bcc2b0d003204f9040b7d6d929f2e LICENSE.
 sha256 be904cd28cb292b80cdb6cf412ab0d9159d431671e987ad433c1f62e0988a9bc LICENSE.qcom
 sha256 fc6223d4bfe9f2f9e2eddc44b9fe5721d0caf49f01cb08d602906add686d8c6f LICENSE.radeon
 sha256 2bdd2e716f05d9737d3f9a20f9a3a3c0caee0e866100ddb0673f1178e42f92b9 LICENSE.sdma_firmware
-sha256 ef38a9a8bb4b0f72b369d337426eea63ef8fc9d48453f127028d935f7dbc5820 WHENCE
+sha256 9b873499a822762177a7a02d2a3ead9fdf0d514c0f9899fb16a2d22ed99f4acc WHENCE
 sha256 fa43e1b9a13b341a07adca9dbe73d0f9072d7966fdfe811c01f0dd2872d7309a qcom/NOTICE.txt
diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index 4eba57434b..60f39804a2 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -4,7 +4,7 @@ 
 #
 ################################################################################
 
-LINUX_FIRMWARE_VERSION = 44d4fca9922a252a0bd81f6307bcc072a78da54a
+LINUX_FIRMWARE_VERSION = 1baa34868b2c0a004dc595b20678145e3fff83e7
 LINUX_FIRMWARE_SITE = http://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
 LINUX_FIRMWARE_SITE_METHOD = git