| Message ID | 20230527092851.705884-1-pbonzini@redhat.com |
|---|---|
| Headers | show |
| Series | meson: replace submodules with wrap files | expand |
On 27/05/2023 11.28, Paolo Bonzini wrote: > This series replaces git submodules for bundled libraries with .wrap > files that can be used directly by meson for subprojects. ... > The remaining submodules consist of tests/lcitool/libvirt-ci and the > firmware in roms/. We talked about moving the contents of roms/ to a separate repository a couple of times ... maybe it's time now that we really do it? (However, when I tried to tackle the "do we need to ship the firmware sources with the main tarball" problem the last time, there was no consensus how to do it best, see https://lore.kernel.org/qemu-devel/20221128092555.37102-1-thuth@redhat.com/ ... maybe something to discuss at KVM forum...) > The former is only used in very specific cases, > while the latter is mostly used only as a pointer used to create the QEMU > tarball. Unfortunately, git-submodule.sh is still needed for roms/SLOF, > parts of which are used in the QEMU build process for pc-bios/s390-ccw; > more on this later in this cover letter. > > I'm not sure what's the best way to proceed for roms/SLOF. Some > possibilities, in no particular order, include: > > * doing nothing > > * merging --with-git-submodules with --enable-download, and > moving the git-submodule.sh rules from the main Makefile to > pc-bios/s390-ccw/ (my favorite option) > > * copying the relevant SLOF files into pc-bios/ Considering that SLOF is also rather on life support already (there is now VOF for the sPAPR machine instead, Alexey left IBM, ...), I also wouldn't mind the third option, I think. But of course we can also start with option 2 and go for option 3 later. Thomas
On Sat, May 27, 2023 at 11:28:46AM +0200, Paolo Bonzini wrote: > The remaining submodules consist of tests/lcitool/libvirt-ci and the > firmware in roms/. The former is only used in very specific cases, > while the latter is mostly used only as a pointer used to create the QEMU > tarball. Unfortunately, git-submodule.sh is still needed for roms/SLOF, > parts of which are used in the QEMU build process for pc-bios/s390-ccw; > more on this later in this cover letter. > > I'm not sure what's the best way to proceed for roms/SLOF. Some > possibilities, in no particular order, include: > > * doing nothing > > * merging --with-git-submodules with --enable-download, and > moving the git-submodule.sh rules from the main Makefile to > pc-bios/s390-ccw/ (my favorite option) > > * copying the relevant SLOF files into pc-bios/ > > Also, getting into more overengineered territory: > > * same as the second option, but also replace the roms/ submodules > with text files, in a format similar to .wrap files; meson uses the > standard configparser to read them, so it would not be a lot of > code. The files would be parsed by scripts/make-release and > pc-bios/s390-ccw/Makefile. > > * adding support for firmware with a meson build system to > configure; turn SLOF into a wrap and roms/SLOF into a symlink > for ../pc-bios/s390-ccw/subprojects/SLOF. I'm mentioning this for > completeness but this is not something I would like. On the other > hand it could reuse some (or most?) of the code currently used to > generate config-meson.cross, so maybe it isn't that bad... Is there a reason why SLOF/s390-ccw is handled differently from the other ROMs ? ie, why haven't we checked in the pre-built firmware binaries, such that we don't build SLOF by default ? With regards, Daniel
On Tue, May 30, 2023 at 2:31 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > * adding support for firmware with a meson build system to > > configure; turn SLOF into a wrap and roms/SLOF into a symlink > > for ../pc-bios/s390-ccw/subprojects/SLOF. I'm mentioning this for > > completeness but this is not something I would like. On the other > > hand it could reuse some (or most?) of the code currently used to > > generate config-meson.cross, so maybe it isn't that bad... > > Is there a reason why SLOF/s390-ccw is handled differently from > the other ROMs ? ie, why haven't we checked in the pre-built > firmware binaries, such that we don't build SLOF by default ? The SLOF ROM is checked in. s390-ccw is also checked in, but it is a QEMU-specific ROM like, on x86, linuxboot.bin or multiboot.bin. Therefore it's rebuilt by "make" and its build system is part of QEMU's. The relationship between s390-ccw and SLOF is that s390-ccw _reuses_ the network stack of SLOF, and pc-bios/s390-ccw/Makefile does so by simply looking at sources at $(SOURCE_PATH)../../roms/SLOF. Paolo > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, May 30, 2023 at 02:18:30PM +0200, Thomas Huth wrote: > On 27/05/2023 11.28, Paolo Bonzini wrote: > > This series replaces git submodules for bundled libraries with .wrap > > files that can be used directly by meson for subprojects. > ... > > The remaining submodules consist of tests/lcitool/libvirt-ci and the > > firmware in roms/. > > We talked about moving the contents of roms/ to a separate repository a > couple of times ... maybe it's time now that we really do it? > > (However, when I tried to tackle the "do we need to ship the firmware > sources with the main tarball" problem the last time, there was no consensus > how to do it best, see > https://lore.kernel.org/qemu-devel/20221128092555.37102-1-thuth@redhat.com/ > ... maybe something to discuss at KVM forum...) > > > The former is only used in very specific cases, > > while the latter is mostly used only as a pointer used to create the QEMU > > tarball. Unfortunately, git-submodule.sh is still needed for roms/SLOF, > > parts of which are used in the QEMU build process for pc-bios/s390-ccw; > > more on this later in this cover letter. > > > > I'm not sure what's the best way to proceed for roms/SLOF. Some > > possibilities, in no particular order, include: > > > > * doing nothing > > > > * merging --with-git-submodules with --enable-download, and > > moving the git-submodule.sh rules from the main Makefile to > > pc-bios/s390-ccw/ (my favorite option) > > > > * copying the relevant SLOF files into pc-bios/ > > Considering that SLOF is also rather on life support already (there is now > VOF for the sPAPR machine instead, Alexey left IBM, ...), I also wouldn't > mind the third option, I think. > > But of course we can also start with option 2 and go for option 3 later. My inclination would be for option 3 too, just copy the relevant files into s390-ccw dir, so the two distinct ROMs are fully separated from each other. With regards, Daniel
On Tue, May 30, 2023 at 2:57 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > I'm not sure what's the best way to proceed for roms/SLOF. Some > > > possibilities, in no particular order, include: > > > > > > * doing nothing > > > > > > * merging --with-git-submodules with --enable-download, and > > > moving the git-submodule.sh rules from the main Makefile to > > > pc-bios/s390-ccw/ (my favorite option) > > > > > > * copying the relevant SLOF files into pc-bios/ > > > > Considering that SLOF is also rather on life support already (there is now > > VOF for the sPAPR machine instead, Alexey left IBM, ...), I also wouldn't > > mind the third option, I think. > > > > But of course we can also start with option 2 and go for option 3 later. > > My inclination would be for option 3 too, just copy the relevant files > into s390-ccw dir, so the two distinct ROMs are fully separated from > each other. Note that the amount of code is not small (7.000 lines of source for the network stack and 1.000 for libc); but yeah, that can be done. Paolo
This series replaces git submodules for bundled libraries with .wrap files that can be used directly by meson for subprojects. These have several advantages, either immediate or potential: * option parsing and downloading is delegated to meson * the commit is stored in a text file instead of a magic entry in the git tree object, and can be a branch name or a version number as well * now that QEMU's configure script knows how to install missing Python dependencies, we could stop shipping external dependencies that are only used as a fallback, and download them on demand if the build platform lacks them. For example dtc could be downloaded at build time, controlled by --enable-download, even when building from a tarball. This is _not_ done in this patch series, but Marc-André has tried it before[1]. * we could also add .wrap files for other dependencies that are missing on the GCC compile farm machines, or for people who build on Windows and might enjoy getting the mandatory dependencies (pixman, zlib, glib, possibly SDL?) via wraps. In theory meson already supports "meson wrap update-db" to automatically use wraps for anything required but missing, but one would need to test that it actually works; see for example https://github.com/mesonbuild/meson/issues/11821. dtc and keycodemapdb both support meson, and previously reviewed patches already run their build system via subproject(), so the wraps are automatically taken into account. Two other submodules, berkeley-softfloat-3 and berkeley-testfloat-3, are used to test QEMU and can be changed to use wraps; however this requires a few more changes to extract the corresponding parts of tests/fp/meson.build. It's mostly code movement rather than new code. The remaining submodules consist of tests/lcitool/libvirt-ci and the firmware in roms/. The former is only used in very specific cases, while the latter is mostly used only as a pointer used to create the QEMU tarball. Unfortunately, git-submodule.sh is still needed for roms/SLOF, parts of which are used in the QEMU build process for pc-bios/s390-ccw; more on this later in this cover letter. I'm not sure what's the best way to proceed for roms/SLOF. Some possibilities, in no particular order, include: * doing nothing * merging --with-git-submodules with --enable-download, and moving the git-submodule.sh rules from the main Makefile to pc-bios/s390-ccw/ (my favorite option) * copying the relevant SLOF files into pc-bios/ Also, getting into more overengineered territory: * same as the second option, but also replace the roms/ submodules with text files, in a format similar to .wrap files; meson uses the standard configparser to read them, so it would not be a lot of code. The files would be parsed by scripts/make-release and pc-bios/s390-ccw/Makefile. * adding support for firmware with a meson build system to configure; turn SLOF into a wrap and roms/SLOF into a symlink for ../pc-bios/s390-ccw/subprojects/SLOF. I'm mentioning this for completeness but this is not something I would like. On the other hand it could reuse some (or most?) of the code currently used to generate config-meson.cross, so maybe it isn't that bad... Patch 1 removes the --with-git= option for consistency, since git cannot be overridden for "meson subprojects download". Patch 2 renames the recently introduced --enable-pypi command line option to --enable-download, and makes it control meson's -Dwrapmode option as well. Patch 3 replaces submodule update with meson's subproject download for existing subprojects (including libfdt and keycodemapdb). Patch 5 converts berkeley-{soft,test}float-3 to subprojects too. However, those are the submodules that are used in configure to check the validity of the source tree, so patch 4 adds an extra check for the presence of the aforementioned SLOF submodule. Still a bit RFCish, in case people prefer to have a solution for git-submodule.sh and roms/SLOF before the switch to wraps. For this reason, and because I have exhausted my gitlab CI minutes for this month, it's only tested lightly (it passed gitlab CI in a previous version and passes vm-build-netbsd in this one). Paolo Supersedes: <20230519085647.1104775-1-pbonzini@redhat.com> [1] https://patchew.org/QEMU/20230302131848.1527460-1-marcandre.lureau@redhat.com/20230302131848.1527460-5-marcandre.lureau@redhat.com/ Paolo Bonzini (5): configure: remove --with-git= option configure: rename --enable-pypi to --enable-download, control subprojects too meson: subprojects: replace submodules with wrap files configure: check for SLOF submodule before building pc-bios/s390-ccw meson: subprojects: replace berkeley-{soft,test}float-3 with wraps .gitlab-ci.d/buildtest-template.yml | 1 + .gitmodules | 15 - Makefile | 2 +- configure | 126 ++-- meson.build | 16 +- scripts/archive-source.sh | 27 +- scripts/git-submodule.sh | 8 +- scripts/make-release | 5 + subprojects/.gitignore | 8 + subprojects/berkeley-softfloat-3.wrap | 5 + subprojects/berkeley-testfloat-3.wrap | 5 + subprojects/dtc | 1 - subprojects/dtc.wrap | 4 + subprojects/keycodemapdb | 1 - subprojects/keycodemapdb.wrap | 4 + subprojects/libvfio-user | 1 - subprojects/libvfio-user.wrap | 4 + .../berkeley-softfloat-3/meson.build | 339 +++++++++++ .../berkeley-softfloat-3/meson_options.txt | 1 + .../berkeley-testfloat-3/meson.build | 220 +++++++ .../berkeley-testfloat-3/meson_options.txt | 1 + tests/fp/meson.build | 541 +----------------- 22 files changed, 668 insertions(+), 667 deletions(-) create mode 100644 subprojects/.gitignore create mode 100644 subprojects/berkeley-softfloat-3.wrap create mode 100644 subprojects/berkeley-testfloat-3.wrap delete mode 160000 subprojects/dtc create mode 100644 subprojects/dtc.wrap delete mode 160000 subprojects/keycodemapdb create mode 100644 subprojects/keycodemapdb.wrap delete mode 160000 subprojects/libvfio-user create mode 100644 subprojects/libvfio-user.wrap create mode 100644 subprojects/packagefiles/berkeley-softfloat-3/meson.build create mode 100644 subprojects/packagefiles/berkeley-softfloat-3/meson_options.txt create mode 100644 subprojects/packagefiles/berkeley-testfloat-3/meson.build create mode 100644 subprojects/packagefiles/berkeley-testfloat-3/meson_options.txt