| Message ID | 20181108153322.17362-1-m.niestroj@grinn-global.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [v2] linux-firmware: bump version to latest 1baa348 | expand |
Hello, On Thu, 8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote: > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz I am sorry, but this hash is still not good for me, I get: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 I.e: ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash: ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 And this morning, Yann E. Morin reported having the same hash as me: 08:07 < y_morin> kos_tom: I also have a different hash here. 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 My system tar is 1.29, which is considered as a "good" version by support/dependencies/check-host-tar.sh. I have nonetheless forced building host-tar, and I still get the same hash. I have uploaded the tarball at https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz. Could you upload the tarball that was generated on your side so that we can compare them, and see where the problem lies ? Best regards, Thomas
Marcin, Thomas, All, On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly: > On Thu, 8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote: > > > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz > > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz > > I am sorry, but this hash is still not good for me, I get: > > 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 > > I.e: > > ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash: > ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 > ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 > > And this morning, Yann E. Morin reported having the same hash as me: > > 08:07 < y_morin> kos_tom: I also have a different hash here. > 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 Right. > My system tar is 1.29, which is considered as a "good" version by > support/dependencies/check-host-tar.sh. I have nonetheless forced > building host-tar, and I still get the same hash. Recently, another user reported hash issues as well, and it turned out that they had changed gzip to be really pigz (a parallel gzip). Can you check if that is not your case too? Regards, Yann E. MORIN. > I have uploaded the tarball at > https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz. > Could you upload the tarball that was generated on your side so that we > can compare them, and see where the problem lies ? > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Hi All, Yann E. MORIN <yann.morin.1998@free.fr> writes: > Marcin, Thomas, All, > > On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly: >> On Thu, 8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote: >> >> > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz >> > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz >> >> I am sorry, but this hash is still not good for me, I get: >> >> 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 >> >> I.e: >> >> ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash: >> ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 >> ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 >> >> And this morning, Yann E. Morin reported having the same hash as me: >> >> 08:07 < y_morin> kos_tom: I also have a different hash here. >> 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 > > Right. Thank you both for testing. > >> My system tar is 1.29, which is considered as a "good" version by >> support/dependencies/check-host-tar.sh. I have nonetheless forced >> building host-tar, and I still get the same hash. > > Recently, another user reported hash issues as well, and it turned out > that they had changed gzip to be really pigz (a parallel gzip). Can you > check if that is not your case too? I have investigated the issue on my side. It turns out that gzip is really the issue here. I have two PCs with Arch Linux. PC_1 is the one I have prepared linux-firmware. Here gzip package info on that machine: [mniestroj@gm ~]$ LANG=C yaourt -Si gzip Repository : core Name : gzip Version : 1.9-1 Description : GNU compression utility Architecture : x86_64 URL : https://www.gnu.org/software/gzip/ Licenses : GPL3 Groups : base base-devel Provides : None Depends On : glibc bash less Optional Deps : None Conflicts With : None Replaces : None Download Size : 77.78 KiB Installed Size : 150.00 KiB Packager : S Build Date : Mon Jan 22 00:52:54 2018 Validated By : MD5 Sum SHA-256 Sum Signature PC_2 is also Arch Linux, but with slightly more up-to-date packages. gzip package info looks like this: [macius@zm ~]$ LANG=C yaourt -Si gzip Repository : core Name : gzip Version : 1.9-2 Description : GNU compression utility Architecture : x86_64 URL : https://www.gnu.org/software/gzip/ Licenses : GPL3 Groups : base base-devel Provides : None Depends On : glibc bash less Optional Deps : None Conflicts With : None Replaces : None Download Size : 78.14 KiB Installed Size : 185.00 KiB Packager : Allan McRae <allan@archlinux.org> Build Date : Sat Nov 3 23:10:39 2018 Validated By : MD5 Sum SHA-256 Sum Signature You can find differences in package here: https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip I have also checked output of gzip command on another PC with pigz configured as gzip drop-in replacement. It outputs even different file, with different sha256 hash. I think the overall conclusion is that a host-gzip package is needed, just like host-tar. In the meantime I will send v3 of this patch with proper hash (the same as you calculated above). > > Regards, > Yann E. MORIN. > >> I have uploaded the tarball at >> https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz. >> Could you upload the tarball that was generated on your side so that we >> can compare them, and see where the problem lies ? Thank for sharing. After gunzipping your version and mine I ended with the same *.tar files. So the difference was clearly because of gzip. Regards, Marcin >> >> Best regards, >> >> Thomas >> -- >> Thomas Petazzoni, CTO, Bootlin >> Embedded Linux and Kernel engineering >> https://bootlin.com
Marcin Niestrój <m.niestroj@grinn-global.com> writes: > Hi All, > > Yann E. MORIN <yann.morin.1998@free.fr> writes: > >> Marcin, Thomas, All, >> >> On 2018-11-09 21:57 +0100, Thomas Petazzoni spake thusly: >>> On Thu, 8 Nov 2018 16:33:22 +0100, Marcin Niestroj wrote: >>> >>> > -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz >>> > +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz >>> >>> I am sorry, but this hash is still not good for me, I get: >>> >>> 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 >>> >>> I.e: >>> >>> ERROR: linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz has wrong sha256 hash: >>> ERROR: expected: 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 >>> ERROR: got : 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 >>> >>> And this morning, Yann E. Morin reported having the same hash as me: >>> >>> 08:07 < y_morin> kos_tom: I also have a different hash here. >>> 08:10 < y_morin> kos_tom: FTR, I got: 5c636765fd1ac638176893feccfd4a4854f59fc3d01b38f3ccdbb89bd5bb6ef1 >> >> Right. > > Thank you both for testing. > >> >>> My system tar is 1.29, which is considered as a "good" version by >>> support/dependencies/check-host-tar.sh. I have nonetheless forced >>> building host-tar, and I still get the same hash. >> >> Recently, another user reported hash issues as well, and it turned out >> that they had changed gzip to be really pigz (a parallel gzip). Can you >> check if that is not your case too? > > I have investigated the issue on my side. It turns out that gzip is > really the issue here. > > I have two PCs with Arch Linux. PC_1 is the one I have prepared > linux-firmware. Here gzip package info on that machine: > > [mniestroj@gm ~]$ LANG=C yaourt -Si gzip > Repository : core > Name : gzip > Version : 1.9-1 > Description : GNU compression utility > Architecture : x86_64 > URL : https://www.gnu.org/software/gzip/ > Licenses : GPL3 > Groups : base base-devel > Provides : None > Depends On : glibc bash less > Optional Deps : None > Conflicts With : None > Replaces : None > Download Size : 77.78 KiB > Installed Size : 150.00 KiB > Packager : S > Build Date : Mon Jan 22 00:52:54 2018 > Validated By : MD5 Sum SHA-256 Sum Signature > > PC_2 is also Arch Linux, but with slightly more up-to-date > packages. gzip package info looks like this: Forgot to mention - on PC_2 I get the right hash (the same as you). > > [macius@zm ~]$ LANG=C yaourt -Si gzip > Repository : core > Name : gzip > Version : 1.9-2 > Description : GNU compression utility > Architecture : x86_64 > URL : https://www.gnu.org/software/gzip/ > Licenses : GPL3 > Groups : base base-devel > Provides : None > Depends On : glibc bash less > Optional Deps : None > Conflicts With : None > Replaces : None > Download Size : 78.14 KiB > Installed Size : 185.00 KiB > Packager : Allan McRae <allan@archlinux.org> > Build Date : Sat Nov 3 23:10:39 2018 > Validated By : MD5 Sum SHA-256 Sum Signature > > You can find differences in package here: > https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip > > I have also checked output of gzip command on another PC with pigz > configured as gzip drop-in replacement. It outputs even different file, > with different sha256 hash. > > I think the overall conclusion is that a host-gzip package is needed, > just like host-tar. In the meantime I will send v3 of this patch with > proper hash (the same as you calculated above). > >> >> Regards, >> Yann E. MORIN. >> >>> I have uploaded the tarball at >>> https://bootlin.com/~thomas/pub/linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz. >>> Could you upload the tarball that was generated on your side so that we >>> can compare them, and see where the problem lies ? > > Thank for sharing. After gunzipping your version and mine I ended with > the same *.tar files. So the difference was clearly because of gzip. > > Regards, > Marcin > >>> >>> Best regards, >>> >>> Thomas >>> -- >>> Thomas Petazzoni, CTO, Bootlin >>> Embedded Linux and Kernel engineering >>> https://bootlin.com > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Hello, On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote: > I have investigated the issue on my side. It turns out that gzip is > really the issue here. Thanks for the additional investigation! > You can find differences in package here: > https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip > > I have also checked output of gzip command on another PC with pigz > configured as gzip drop-in replacement. It outputs even different file, > with different sha256 hash. > > I think the overall conclusion is that a host-gzip package is needed, > just like host-tar. In the meantime I will send v3 of this patch with > proper hash (the same as you calculated above). This is getting really horrible. On my side, I have updated to Fedora 29, which ships tar 1.30, that we consider as broken, and therefore now for everything single small build that I do, I'm paying the price of building tar, which is super annoying. This is especially annoying as tar 1.30 is only problematic when *creating* tarballs of git-fetched packages, which clearly does not happen at every build. It would be truly horrible to also have to build our own gzip. Maybe we need to somehow revisit our requirement that we need to have reproducible tarballs for git-fetched packages ? Best regards, Thomas
Thomas, All, On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly: > On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote: [--SNIP--] > > I think the overall conclusion is that a host-gzip package is needed, > > just like host-tar. In the meantime I will send v3 of this patch with > > proper hash (the same as you calculated above). > > This is getting really horrible. > > On my side, I have updated to Fedora 29, which ships tar 1.30, that we > consider as broken, and therefore now for everything single small build > that I do, I'm paying the price of building tar, which is super > annoying. This is especially annoying as tar 1.30 is only problematic > when *creating* tarballs of git-fetched packages, which clearly does > not happen at every build. Well, we only need host-tar and host-gzip when creating tarballs, not extracting them, correct? So, we could very well envision the fact that host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when FOO_SITE_METHOD == git (and they are needed, of course), no? (note: tar is ugly, because its configure step takes ~26s, while the build is only ~4s). > It would be truly horrible to also have to build our own gzip. Maybe we > need to somehow revisit our requirement that we need to have > reproducible tarballs for git-fetched packages ? I am ready to pay the price for reproducibility any time. Regards, Yann E. MORIN.
Hello, On Tue, 13 Nov 2018 21:29:46 +0100, Yann E. MORIN wrote: > > On my side, I have updated to Fedora 29, which ships tar 1.30, that we > > consider as broken, and therefore now for everything single small build > > that I do, I'm paying the price of building tar, which is super > > annoying. This is especially annoying as tar 1.30 is only problematic > > when *creating* tarballs of git-fetched packages, which clearly does > > not happen at every build. > > Well, we only need host-tar and host-gzip when creating tarballs, not > extracting them, correct? So, we could very well envision the fact that > host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when > FOO_SITE_METHOD == git (and they are needed, of course), no? For host-tar, it's a bit complicated, because we have two conditions: - tar must be >= 1.27 for *extraction* to work properly. - tar must be < 1.30 for *creation* to work properly So, if we add $(BR2_TAR_HOST_DEPENDENCY) to the package dependencies only when FOO_SITE_METHOD = git, we would miss the case where it is needed for extracting plain regular tarballs. > > It would be truly horrible to also have to build our own gzip. Maybe we > > need to somehow revisit our requirement that we need to have > > reproducible tarballs for git-fetched packages ? > > I am ready to pay the price for reproducibility any time. Well, for a "normal" build, I agree. When doing active BR development, where one regularly builds from scratch small configurations, every additional amount of time/stuff to build is annoying. Thomas
Hi, Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > Hello, > > On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote: > >> I have investigated the issue on my side. It turns out that gzip is >> really the issue here. > > Thanks for the additional investigation! Some new findings now. It turned out that I actually used pigz instead of gzip. It was not directly clear, because I had PATH variable prepended with a directory containing "gzip -> /usr/bin/pigz". I have set this up ~3 years ago and totally forgot about that. Sorry for misleading that gzip version / patch could be the reason of wrong hash in my case. I need to really go through all the patches that got applied to master... > >> You can find differences in package here: >> https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/gzip >> >> I have also checked output of gzip command on another PC with pigz >> configured as gzip drop-in replacement. It outputs even different file, >> with different sha256 hash. >> >> I think the overall conclusion is that a host-gzip package is needed, >> just like host-tar. In the meantime I will send v3 of this patch with >> proper hash (the same as you calculated above).
On 13/11/2018 21:29, Yann E. MORIN wrote: > Thomas, All, > > On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly: >> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote: > [--SNIP--] >>> I think the overall conclusion is that a host-gzip package is needed, >>> just like host-tar. In the meantime I will send v3 of this patch with >>> proper hash (the same as you calculated above). >> >> This is getting really horrible. >> >> On my side, I have updated to Fedora 29, which ships tar 1.30, that we >> consider as broken, and therefore now for everything single small build >> that I do, I'm paying the price of building tar, which is super >> annoying. This is especially annoying as tar 1.30 is only problematic >> when *creating* tarballs of git-fetched packages, which clearly does >> not happen at every build. > > Well, we only need host-tar and host-gzip when creating tarballs, not > extracting them, correct? So, we could very well envision the fact that > host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when > FOO_SITE_METHOD == git (and they are needed, of course), no? Since we have the same problems sometimes with github tarballs, I think we need a more fundamental solution. I would propose a new tarsha256 hash type, which extracts the tarball to calculate the hash. In a simple version it's not so complicated, something like tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum - where tarsha256 contains: { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \ sha256sum - | cut -f 1 -d ' ' As usually, entirely untested. Regards, Arnout > > (note: tar is ugly, because its configure step takes ~26s, while the > build is only ~4s). > >> It would be truly horrible to also have to build our own gzip. Maybe we >> need to somehow revisit our requirement that we need to have >> reproducible tarballs for git-fetched packages ? > > I am ready to pay the price for reproducibility any time. > > Regards, > Yann E. MORIN. >
Arnout, All, On 2018-11-14 00:54 +0100, Arnout Vandecappelle spake thusly: > On 13/11/2018 21:29, Yann E. MORIN wrote: > > On 2018-11-13 21:02 +0100, Thomas Petazzoni spake thusly: > >> On Tue, 13 Nov 2018 18:20:57 +0100, Marcin Niestrój wrote: > >>> I think the overall conclusion is that a host-gzip package is needed, > >>> just like host-tar. In the meantime I will send v3 of this patch with > >>> proper hash (the same as you calculated above). > >> This is getting really horrible. > > Well, we only need host-tar and host-gzip when creating tarballs, not > > extracting them, correct? So, we could very well envision the fact that > > host-tar (and host-gzip) are only added to FOO_DOWNLOAD dependecies when > > FOO_SITE_METHOD == git (and they are needed, of course), no? > > Since we have the same problems sometimes with github tarballs, I think we need > a more fundamental solution. I would propose a new tarsha256 hash type, which > extracts the tarball to calculate the hash. In a simple version it's not so > complicated, something like > > tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum - > > where tarsha256 contains: > > { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \ > sha256sum - | cut -f 1 -d ' ' > > As usually, entirely untested. I don't like it, because this is totally non-standard. People expect to be able to check hashes by running the *usual* XXXsum commands, directly on the shipped/received files. Introducing our own hash mechanism, how reliable or simple as it would be, breaks this assumption, and the tool to actually check them is not available at all except internally to Buildroot, so it is not possible to reproduce the checks outside of Buildroot. This goes counter one of the initial goal of hashes, which is to be able to track archives and their validity across a supply chain, inbound (as sent by a provider to Buildroot, to do the build) or outbound (as received by a recepient, from Buildroot, for compliance) alike. Regards, Yann E. MORIN.
Hello, On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote: > > Since we have the same problems sometimes with github tarballs, I think we need > > a more fundamental solution. I would propose a new tarsha256 hash type, which > > extracts the tarball to calculate the hash. In a simple version it's not so > > complicated, something like > > > > tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum - > > > > where tarsha256 contains: > > > > { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \ > > sha256sum - | cut -f 1 -d ' ' > > > > As usually, entirely untested. > > I don't like it, because this is totally non-standard. People expect to > be able to check hashes by running the *usual* XXXsum commands, directly > on the shipped/received files. > > Introducing our own hash mechanism, how reliable or simple as it would > be, breaks this assumption, and the tool to actually check them is not > available at all except internally to Buildroot, so it is not possible > to reproduce the checks outside of Buildroot. > > This goes counter one of the initial goal of hashes, which is to be able > to track archives and their validity across a supply chain, inbound (as > sent by a provider to Buildroot, to do the build) or outbound (as received > by a recepient, from Buildroot, for compliance) alike. I understand this argument, but do you have some alternative solution ? Even building our own host-tar and host-gzip doesn't solve entirely the problem, because it doesn't solve the case of tarballs fetched from github, that tend to change in subtle ways once in a while. Best regards, Thomas
Thomas, All, On 2018-11-16 09:35 +0100, Thomas Petazzoni spake thusly: > On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote: > > > Since we have the same problems sometimes with github tarballs, I think we need > > > a more fundamental solution. I would propose a new tarsha256 hash type, which > > > extracts the tarball to calculate the hash. In a simple version it's not so > > > complicated, something like > > > > > > tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum - > > > > > > where tarsha256 contains: > > > > > > { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \ > > > sha256sum - | cut -f 1 -d ' ' > > > > > > As usually, entirely untested. > > > > I don't like it, because this is totally non-standard. People expect to > > be able to check hashes by running the *usual* XXXsum commands, directly > > on the shipped/received files. > > > > Introducing our own hash mechanism, how reliable or simple as it would > > be, breaks this assumption, and the tool to actually check them is not > > available at all except internally to Buildroot, so it is not possible > > to reproduce the checks outside of Buildroot. > > > > This goes counter one of the initial goal of hashes, which is to be able > > to track archives and their validity across a supply chain, inbound (as > > sent by a provider to Buildroot, to do the build) or outbound (as received > > by a recepient, from Buildroot, for compliance) alike. > > I understand this argument, but do you have some alternative solution ? No. Also, the tarshasum thingy would not work either, because it is not enough to hash the content of files. There are cases where the metadat also matters, for example if a tarball is regenerated with new dates in it. This could trigger a rebuild of parts of the package, for example if configure.ac is now younger than configure. Obviously, we could add that info to be hashed... Until we find anotehr type of info we can't get via the --to-command, like extended attributes for example... So, I am still opposed to the tarshasum idea. > Even building our own host-tar and host-gzip doesn't solve entirely the > problem, because it doesn't solve the case of tarballs fetched from > github, that tend to change in subtle ways once in a while. For that, we can simply decide to ditch the github macro, and revert to using Github for what it is: a git repository, and switch to the git method. Yes, yes, I know, this is a shame, especially since there are big repositories out there. But OTOH, we now have the git cache, so it should not be such a hassle in the end. Regards, Yann E. MORIN.
On 20/11/2018 19:50, Yann E. MORIN wrote: > Thomas, All, > > On 2018-11-16 09:35 +0100, Thomas Petazzoni spake thusly: >> On Thu, 15 Nov 2018 20:05:58 +0100, Yann E. MORIN wrote: >>>> Since we have the same problems sometimes with github tarballs, I think we need >>>> a more fundamental solution. I would propose a new tarsha256 hash type, which >>>> extracts the tarball to calculate the hash. In a simple version it's not so >>>> complicated, something like >>>> >>>> tar -xf - --to-command=$(TOPDIR)/support/scripts/tarsha256 | sort | sha256sum - >>>> >>>> where tarsha256 contains: >>>> >>>> { echo $TAR_FILENAME; echo $TAR_MODE; echo $TAR_FILETYPE; cat - } | \ >>>> sha256sum - | cut -f 1 -d ' ' >>>> >>>> As usually, entirely untested. >>> >>> I don't like it, because this is totally non-standard. People expect to >>> be able to check hashes by running the *usual* XXXsum commands, directly >>> on the shipped/received files. >>> >>> Introducing our own hash mechanism, how reliable or simple as it would >>> be, breaks this assumption, and the tool to actually check them is not >>> available at all except internally to Buildroot, so it is not possible >>> to reproduce the checks outside of Buildroot. >>> >>> This goes counter one of the initial goal of hashes, which is to be able >>> to track archives and their validity across a supply chain, inbound (as >>> sent by a provider to Buildroot, to do the build) or outbound (as received >>> by a recepient, from Buildroot, for compliance) alike. >> >> I understand this argument, but do you have some alternative solution ? > > No. > > Also, the tarshasum thingy would not work either, because it is not > enough to hash the content of files. There are cases where the metadat > also matters, for example if a tarball is regenerated with new dates in > it. This could trigger a rebuild of parts of the package, for example if > configure.ac is now younger than configure. Obviously, we could add that > info to be hashed... Yes, I forgot to include the comment that we need to investigate which of mtime, ctime and atime need to be included in the hash. > Until we find anotehr type of info we can't get via the --to-command, > like extended attributes for example... Other metadata we explicitly exclude. Hm, we don't but we should :-) BTW, turns out that https://github.com/mikemccabe/tarsump is not usable - it doesn't sort the files, and it doesn't compute an overall hash. Regards, Arnout > > So, I am still opposed to the tarshasum idea. > >> Even building our own host-tar and host-gzip doesn't solve entirely the >> problem, because it doesn't solve the case of tarballs fetched from >> github, that tend to change in subtle ways once in a while. > > For that, we can simply decide to ditch the github macro, and revert > to using Github for what it is: a git repository, and switch to the git > method. > > Yes, yes, I know, this is a shame, especially since there are big > repositories out there. But OTOH, we now have the git cache, so it > should not be such a hassle in the end. > > Regards, > Yann E. MORIN. >
>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes: Hi, > BTW, turns out that https://github.com/mikemccabe/tarsump is not usable - it > doesn't sort the files, and it doesn't compute an overall hash. Isn't that just a question of piping the output to: LANG=C sort | sha256sum Or am I missing something?
diff --git a/package/linux-firmware/linux-firmware.hash b/package/linux-firmware/linux-firmware.hash index ca6ad8f59d..6ade218888 100644 --- a/package/linux-firmware/linux-firmware.hash +++ b/package/linux-firmware/linux-firmware.hash @@ -1,11 +1,11 @@ # Locally calculated -sha256 b279ca4d086887c2efab13e28a7ca36e409410d3df38a62d7c7b5799ee3de916 linux-firmware-44d4fca9922a252a0bd81f6307bcc072a78da54a.tar.gz +sha256 3e4fcbac18990a14d52159fefdc70081a56c5244adba28b14242e159a748e755 linux-firmware-1baa34868b2c0a004dc595b20678145e3fff83e7.tar.gz sha256 8116433f4004fc0c24d72b3d9e497808b724aa0e5e1cd63fc1bf66b715b1e2e9 LICENCE.Abilis sha256 4b3ea5d5a03c0db81bee0bcb14b30d75b30ef568597bb5be7d4dee57f434265f LICENSE.amdgpu sha256 38f2037aa14631b4b29826d7a99379613c41a97064d1defdee30a7a022138b20 LICENCE.Marvell sha256 802b7014b26c606cf6248ae8b0ab1ce6d2d1b0db236d38dd269e676cd70710f2 LICENCE.atheros_firmware sha256 3b5eb392b2d9d8c46d6aae26d06c187e5ea3029b12d13bc2b8deb8b3ce6bfa53 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-4.txt -sha256 c565861ff7c42f5df98e15239241f1f42614e5e15f362094a2d3e8da724dc842 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt +sha256 8ce5c6ea0542bf4aac31fc3ae16a39792ad22d0eae4543063fac56fb3380f021 ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt sha256 b16056fc91b82a0e3e8de8f86c2dac98201aa9dc3cbd33e8d38f1b087fcec30d LICENCE.broadcom_bcm43xx sha256 a5777f9e80aca0603b0648454de996168b1c530322550ccda94d6d78bcf6c061 LICENCE.chelsio_firmware sha256 60fbc9cccb455e1a3306c97db942d6f24fa93664be61d54c497637e6d0e2ae83 LICENCE.fw_sst_0f28 @@ -27,5 +27,5 @@ sha256 8542aeabf2761935122d693561e16766ce1bcc2b0d003204f9040b7d6d929f2e LICENSE. sha256 be904cd28cb292b80cdb6cf412ab0d9159d431671e987ad433c1f62e0988a9bc LICENSE.qcom sha256 fc6223d4bfe9f2f9e2eddc44b9fe5721d0caf49f01cb08d602906add686d8c6f LICENSE.radeon sha256 2bdd2e716f05d9737d3f9a20f9a3a3c0caee0e866100ddb0673f1178e42f92b9 LICENSE.sdma_firmware -sha256 ef38a9a8bb4b0f72b369d337426eea63ef8fc9d48453f127028d935f7dbc5820 WHENCE +sha256 9b873499a822762177a7a02d2a3ead9fdf0d514c0f9899fb16a2d22ed99f4acc WHENCE sha256 fa43e1b9a13b341a07adca9dbe73d0f9072d7966fdfe811c01f0dd2872d7309a qcom/NOTICE.txt diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk index 4eba57434b..60f39804a2 100644 --- a/package/linux-firmware/linux-firmware.mk +++ b/package/linux-firmware/linux-firmware.mk @@ -4,7 +4,7 @@ # ################################################################################ -LINUX_FIRMWARE_VERSION = 44d4fca9922a252a0bd81f6307bcc072a78da54a +LINUX_FIRMWARE_VERSION = 1baa34868b2c0a004dc595b20678145e3fff83e7 LINUX_FIRMWARE_SITE = http://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git LINUX_FIRMWARE_SITE_METHOD = git
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com> --- package/linux-firmware/linux-firmware.hash | 6 +++--- package/linux-firmware/linux-firmware.mk | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)