| Message ID | 20220704064254.18187-2-thuth@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | scripts/make-release: Decrease the size of the release tarballs | expand |
On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: > > The edk2 sources are bigger than the sources of QEMU - so they double the > size of our release tarballs if we include them. Fortunately, edk2 has a > permissive license, so there is no need for us to do this as long as we > continue to distribute the edk2 license information in our release tarball. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > scripts/make-release | 27 +++++++++++++++++++++------ > 1 file changed, 21 insertions(+), 6 deletions(-) > > diff --git a/scripts/make-release b/scripts/make-release > index 05b14ecc95..f79ed7a2ae 100755 > --- a/scripts/make-release > +++ b/scripts/make-release > @@ -20,19 +20,34 @@ git checkout "v${version}" > git submodule update --init > (cd roms/seabios && git describe --tags --long --dirty > .version) > (cd roms/skiboot && ./make_version.sh > .version) > -# Fetch edk2 submodule's submodules, since it won't have access to them via > -# the tarball later. > + > +# Fetch edk2 submodule's submodules to get their license information. > +# Edk2 is distributed under permissive licenses, so we don't have to > +# include the full source tree here (which is bigger than QEMU's sources) > +# as long as we provide the license information in the tarball. > # > # A more uniform way to handle this sort of situation would be nice, but we > # don't necessarily have much control over how a submodule handles its > # submodule dependencies, so we continue to handle these on a case-by-case > # basis for now. > -(cd roms/edk2 && \ > - git submodule update --init -- \ > +cd roms/edk2 > +git submodule update --init --depth 1 -- \ > ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ > BaseTools/Source/C/BrotliCompress/brotli \ > - CryptoPkg/Library/OpensslLib/openssl \ > - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) > + CryptoPkg/Library/OpensslLib/openssl > +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ > + License-softfloat.txt > +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt > +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt > +edk2rev=$(git rev-parse --short=12 HEAD) > +rm -r $(ls | grep -v License) .[a-z]* > +cat > sources.txt <<EOF > +The edk2 sources can be downloaded from: > + > +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz Please use QEMU's edk2 mirror: https://gitlab.com/qemu-project/edk2 QEMU mirrors all dependencies so that even if upstream projects go offline we can still rebuild all of QEMU from source. Thanks, Stefan
On 04/07/2022 09.26, Stefan Hajnoczi wrote: > On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: >> >> The edk2 sources are bigger than the sources of QEMU - so they double the >> size of our release tarballs if we include them. Fortunately, edk2 has a >> permissive license, so there is no need for us to do this as long as we >> continue to distribute the edk2 license information in our release tarball. >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> scripts/make-release | 27 +++++++++++++++++++++------ >> 1 file changed, 21 insertions(+), 6 deletions(-) >> >> diff --git a/scripts/make-release b/scripts/make-release >> index 05b14ecc95..f79ed7a2ae 100755 >> --- a/scripts/make-release >> +++ b/scripts/make-release >> @@ -20,19 +20,34 @@ git checkout "v${version}" >> git submodule update --init >> (cd roms/seabios && git describe --tags --long --dirty > .version) >> (cd roms/skiboot && ./make_version.sh > .version) >> -# Fetch edk2 submodule's submodules, since it won't have access to them via >> -# the tarball later. >> + >> +# Fetch edk2 submodule's submodules to get their license information. >> +# Edk2 is distributed under permissive licenses, so we don't have to >> +# include the full source tree here (which is bigger than QEMU's sources) >> +# as long as we provide the license information in the tarball. >> # >> # A more uniform way to handle this sort of situation would be nice, but we >> # don't necessarily have much control over how a submodule handles its >> # submodule dependencies, so we continue to handle these on a case-by-case >> # basis for now. >> -(cd roms/edk2 && \ >> - git submodule update --init -- \ >> +cd roms/edk2 >> +git submodule update --init --depth 1 -- \ >> ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ >> BaseTools/Source/C/BrotliCompress/brotli \ >> - CryptoPkg/Library/OpensslLib/openssl \ >> - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) >> + CryptoPkg/Library/OpensslLib/openssl >> +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ >> + License-softfloat.txt >> +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt >> +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt >> +edk2rev=$(git rev-parse --short=12 HEAD) >> +rm -r $(ls | grep -v License) .[a-z]* >> +cat > sources.txt <<EOF >> +The edk2 sources can be downloaded from: >> + >> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz > > Please use QEMU's edk2 mirror: > https://gitlab.com/qemu-project/edk2 > > QEMU mirrors all dependencies so that even if upstream projects go > offline we can still rebuild all of QEMU from source. Sure, that makes sense, indeed. I'll wait a little bit longer for other comments, then I'll respin the series with the URLs updated (also in the second patch). Thomas
On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote: > On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: > > > > The edk2 sources are bigger than the sources of QEMU - so they double the > > size of our release tarballs if we include them. Fortunately, edk2 has a > > permissive license, so there is no need for us to do this as long as we > > continue to distribute the edk2 license information in our release tarball. > > > > Signed-off-by: Thomas Huth <thuth@redhat.com> > > --- > > scripts/make-release | 27 +++++++++++++++++++++------ > > 1 file changed, 21 insertions(+), 6 deletions(-) > > > > diff --git a/scripts/make-release b/scripts/make-release > > index 05b14ecc95..f79ed7a2ae 100755 > > --- a/scripts/make-release > > +++ b/scripts/make-release > > @@ -20,19 +20,34 @@ git checkout "v${version}" > > git submodule update --init > > (cd roms/seabios && git describe --tags --long --dirty > .version) > > (cd roms/skiboot && ./make_version.sh > .version) > > -# Fetch edk2 submodule's submodules, since it won't have access to them via > > -# the tarball later. > > + > > +# Fetch edk2 submodule's submodules to get their license information. > > +# Edk2 is distributed under permissive licenses, so we don't have to > > +# include the full source tree here (which is bigger than QEMU's sources) > > +# as long as we provide the license information in the tarball. > > # > > # A more uniform way to handle this sort of situation would be nice, but we > > # don't necessarily have much control over how a submodule handles its > > # submodule dependencies, so we continue to handle these on a case-by-case > > # basis for now. > > -(cd roms/edk2 && \ > > - git submodule update --init -- \ > > +cd roms/edk2 > > +git submodule update --init --depth 1 -- \ > > ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ > > BaseTools/Source/C/BrotliCompress/brotli \ > > - CryptoPkg/Library/OpensslLib/openssl \ > > - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) > > + CryptoPkg/Library/OpensslLib/openssl > > +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ > > + License-softfloat.txt > > +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt > > +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt > > +edk2rev=$(git rev-parse --short=12 HEAD) > > +rm -r $(ls | grep -v License) .[a-z]* > > +cat > sources.txt <<EOF > > +The edk2 sources can be downloaded from: > > + > > +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz > > Please use QEMU's edk2 mirror: > https://gitlab.com/qemu-project/edk2 > > QEMU mirrors all dependencies so that even if upstream projects go > offline we can still rebuild all of QEMU from source. Note that the github/lab generated tarballs are not signed, while QEMU's release tarballs are gpg signed, so from that POV this would be a regression no matter which site we point to. Also it would need more guidance on what to actually do with the tarball, as if you merely unpack it into this dir, it won't work as it will be one level of dirs nesting too deep for QEMU's build scripts to work. With regards, Daniel
On 04/07/2022 11.14, Daniel P. Berrangé wrote: > On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote: >> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: ... >>> +cat > sources.txt <<EOF >>> +The edk2 sources can be downloaded from: >>> + >>> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz >> >> Please use QEMU's edk2 mirror: >> https://gitlab.com/qemu-project/edk2 >> >> QEMU mirrors all dependencies so that even if upstream projects go >> offline we can still rebuild all of QEMU from source. > > Note that the github/lab generated tarballs are not signed, while > QEMU's release tarballs are gpg signed, so from that POV this would > be a regression no matter which site we point to. Maybe we should point to the edk2 release page instead? I.e.: https://github.com/tianocore/edk2/releases Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here. > Also it would need more guidance on what to actually do with the > tarball, as if you merely unpack it into this dir, it won't work > as it will be one level of dirs nesting too deep for QEMU's > build scripts to work. I could add some wording how to use the edk2-build.sh script with a downloaded edk2 tarball if that helps ... not sure whether it is really required, though, we also don't provide instruction for building any other firmware yet, as far as I know. Thomas
> > Also it would need more guidance on what to actually do with the > > tarball, as if you merely unpack it into this dir, it won't work > > as it will be one level of dirs nesting too deep for QEMU's > > build scripts to work. > > I could add some wording how to use the edk2-build.sh script with a > downloaded edk2 tarball if that helps ... not sure whether it is really > required, though, we also don't provide instruction for building any other > firmware yet, as far as I know. The instructions are printed by "make -C roms help". We could teach roms/Makefile to either automatically fetch the sources (be that the upstream edk2 tarball or a qemu-firmware-src tarball should we go with the approach suggested by Daniel elsewhere in this thread), or throw an error with instructions when they are not present. take care, Gerd
On Mon, Jul 04, 2022 at 11:37:11AM +0200, Thomas Huth wrote: > On 04/07/2022 11.14, Daniel P. Berrangé wrote: > > On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote: > > > On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: > ... > > > > +cat > sources.txt <<EOF > > > > +The edk2 sources can be downloaded from: > > > > + > > > > +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz > > > > > > Please use QEMU's edk2 mirror: > > > https://gitlab.com/qemu-project/edk2 > > > > > > QEMU mirrors all dependencies so that even if upstream projects go > > > offline we can still rebuild all of QEMU from source. > > > > Note that the github/lab generated tarballs are not signed, while > > QEMU's release tarballs are gpg signed, so from that POV this would > > be a regression no matter which site we point to. > > Maybe we should point to the edk2 release page instead? I.e.: > > https://github.com/tianocore/edk2/releases > > Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here. > > > Also it would need more guidance on what to actually do with the > > tarball, as if you merely unpack it into this dir, it won't work > > as it will be one level of dirs nesting too deep for QEMU's > > build scripts to work. > > I could add some wording how to use the edk2-build.sh script with a > downloaded edk2 tarball if that helps ... not sure whether it is really > required, though, we also don't provide instruction for building any other > firmware yet, as far as I know. Well my thought is that currently you can do 'make -C roms efi' from the unpacked release tarball, and with this change, that no longer works. So we ought to explain what to do to get that working again. This doesn't neccessarily means docs. It could just be 'make' prints out an error telling people what to do, or figure it out itself. If we split off the entire of the 'roms' directory into a self-contained firmware-src tarball, the 'roms/Makefile' would still "just work". With regards, Daniel
On 04/07/2022 12.45, Daniel P. Berrangé wrote: > On Mon, Jul 04, 2022 at 11:37:11AM +0200, Thomas Huth wrote: >> On 04/07/2022 11.14, Daniel P. Berrangé wrote: >>> On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote: >>>> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote: >> ... >>>>> +cat > sources.txt <<EOF >>>>> +The edk2 sources can be downloaded from: >>>>> + >>>>> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz >>>> >>>> Please use QEMU's edk2 mirror: >>>> https://gitlab.com/qemu-project/edk2 >>>> >>>> QEMU mirrors all dependencies so that even if upstream projects go >>>> offline we can still rebuild all of QEMU from source. >>> >>> Note that the github/lab generated tarballs are not signed, while >>> QEMU's release tarballs are gpg signed, so from that POV this would >>> be a regression no matter which site we point to. >> >> Maybe we should point to the edk2 release page instead? I.e.: >> >> https://github.com/tianocore/edk2/releases >> >> Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here. >> >>> Also it would need more guidance on what to actually do with the >>> tarball, as if you merely unpack it into this dir, it won't work >>> as it will be one level of dirs nesting too deep for QEMU's >>> build scripts to work. >> >> I could add some wording how to use the edk2-build.sh script with a >> downloaded edk2 tarball if that helps ... not sure whether it is really >> required, though, we also don't provide instruction for building any other >> firmware yet, as far as I know. > > Well my thought is that currently you can do 'make -C roms efi' from > the unpacked release tarball, and with this change, that no longer > works. So we ought to explain what to do to get that working again. > > This doesn't neccessarily means docs. It could just be 'make' prints > out an error telling people what to do, or figure it out itself. > > If we split off the entire of the 'roms' directory into a self-contained > firmware-src tarball, the 'roms/Makefile' would still "just work". Sorry for the late reply, I've been pondering about this for a while and was busy with other stuff... So I think I tend to agree - if we still want to ship the edk2 (and skiboot) sources, it's likely best if we put all roms/* sources into a separate tarball. Maybe even split the roms folder into a separate git repository. However, there's at least one minor obstacle: The sources of the s390-ccw bios (in pc-bios/s390-ccw) currently partly depend on the roms/SLOF sources to be available for the full build. So if we slice out the roms/SLOF stuff, we should likely move the s390-ccw bios also to same location... I guess it's now too late for 7.1, so I'll postpone my further work on this to the 7.2 release cycle. Thomas
diff --git a/scripts/make-release b/scripts/make-release index 05b14ecc95..f79ed7a2ae 100755 --- a/scripts/make-release +++ b/scripts/make-release @@ -20,19 +20,34 @@ git checkout "v${version}" git submodule update --init (cd roms/seabios && git describe --tags --long --dirty > .version) (cd roms/skiboot && ./make_version.sh > .version) -# Fetch edk2 submodule's submodules, since it won't have access to them via -# the tarball later. + +# Fetch edk2 submodule's submodules to get their license information. +# Edk2 is distributed under permissive licenses, so we don't have to +# include the full source tree here (which is bigger than QEMU's sources) +# as long as we provide the license information in the tarball. # # A more uniform way to handle this sort of situation would be nice, but we # don't necessarily have much control over how a submodule handles its # submodule dependencies, so we continue to handle these on a case-by-case # basis for now. -(cd roms/edk2 && \ - git submodule update --init -- \ +cd roms/edk2 +git submodule update --init --depth 1 -- \ ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \ BaseTools/Source/C/BrotliCompress/brotli \ - CryptoPkg/Library/OpensslLib/openssl \ - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + CryptoPkg/Library/OpensslLib/openssl +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \ + License-softfloat.txt +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt +edk2rev=$(git rev-parse --short=12 HEAD) +rm -r $(ls | grep -v License) .[a-z]* +cat > sources.txt <<EOF +The edk2 sources can be downloaded from: + +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz +EOF +cd ../.. + popd tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
The edk2 sources are bigger than the sources of QEMU - so they double the size of our release tarballs if we include them. Fortunately, edk2 has a permissive license, so there is no need for us to do this as long as we continue to distribute the edk2 license information in our release tarball. Signed-off-by: Thomas Huth <thuth@redhat.com> --- scripts/make-release | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)