| Message ID | 20181114105557.12599-1-thomas.petazzoni@bootlin.com |
|---|---|
| Headers | show |
| Series | Per-package host/target directory support | expand |
Hello, On Wed, 14 Nov 2018 11:55:51 +0100, Thomas Petazzoni wrote: > Here is a fourth iteration of the per-package SDK and target directory > implementation. FYI, I did a test build of some given configuration (randomly chosen), in the following situations, which provided the following results - Per-package folder support disabled real 5m11.261s user 8m58.011s sys 1m54.650s - Per-package folder support enabled, sequential build real 5m15.744s user 9m4.848s sys 1m55.885s - Per-package folder support enabled, parallel build real 4m17.059s user 9m55.024s sys 2m2.344s So, there is a small performance hit in doing all the rsyncs needed for per-package folders: 4.5 seconds on a build of ~5 minutes. However, this performance hit is largely compensated by the parallelism of the build. It is worth mentioning that this specific configuration only lead to a small amount of parallelism, as can be seen in the attached PDF files. Best regards, Thomas
Hi Thomas, Am 14.11.18 um 11:55 schrieb Thomas Petazzoni: > Hello, > > Here is a fourth iteration of the per-package SDK and target directory > implementation. > > If you're interested in testing it, you can find the patch series at: > > http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4 > thank you for the updated version. I tried to run it with my configs. For that I needed a) a generic patch for all kconfig packages, see my answer on the final patch b) linux-specific patch posted two days ago: [PATCH v2 1/1] linux: Make dtc install step more reliable On my local machine things seemed to work, however final rootfs creation failed in our CI environment due to broken host- binaries, e.g. mkfs.ubifs not finding some required lib (liblzo2.so.2 in that particular case). It seems some things with rpath manipulation/checking do not yet work properly. Looking into the binary with ldd I see libs used from the host-machine that probably should come from buildroots host/usr/lib... linux-vdso.so.1 => (0x00007ffc06fcc000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f760428f000) liblzo2.so.2 => not found libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f760408a000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7603d81000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76039b7000) /lib64/ld-linux-x86-64.so.2 (0x00007f76044a9000) I couldnt find out in reasonable time where things go wrong so I'd be thankful for a pointer on how to debug this. regards, Andreas
Hello Andreas, Thanks for your feedback! On Thu, 15 Nov 2018 17:41:27 +0100, Andreas Naumann wrote: > thank you for the updated version. I tried to run it with my configs. Thanks a lot for testing! > For that I needed > a) a generic patch for all kconfig packages, see my answer on the final > patch For this one, we need to continue the discussion with Yann and Thomas De Schampheleire to understand what is the right approach. > b) linux-specific patch posted two days ago: [PATCH v2 1/1] linux: Make > dtc install step more reliable This one is a package-specific fix, which can be merged separately from the core of the PPSH series. I want to keep the PPSH series focused on introducing PPSH support, and not make it grow infinitely with fixup patches for lots of packages. However, perhaps what we could do is to collect those patches in branch, this way we could have two branches: - One with the PPSH core stuff - One with all the PPSH-related fixes, which are being submitted/applied progressively to Buildroot, even before PPSH itself is merged. What do you think ? > On my local machine things seemed to work, however final rootfs creation > failed in our CI environment due to broken host- binaries, e.g. > mkfs.ubifs not finding some required lib (liblzo2.so.2 in that > particular case). > It seems some things with rpath manipulation/checking do not yet work > properly. Looking into the binary with ldd I see libs used from the > host-machine that probably should come from buildroots host/usr/lib... > linux-vdso.so.1 => (0x00007ffc06fcc000) > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f760428f000) > liblzo2.so.2 => not found > libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f760408a000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7603d81000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76039b7000) > /lib64/ld-linux-x86-64.so.2 (0x00007f76044a9000) This is weird, because it works fine here: thomas@windsurf:~/projets/outputs/t$ ldd host/sbin/mkfs.ubifs linux-vdso.so.1 (0x00007ffe8cd3b000) libz.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libz.so.1 (0x00007f1a683ff000) liblzo2.so.2 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/liblzo2.so.2 (0x00007f1a683db000) libuuid.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libuuid.so.1 (0x00007f1a683d2000) libm.so.6 => /lib64/libm.so.6 (0x00007f1a6821e000) libc.so.6 => /lib64/libc.so.6 (0x00007f1a68058000) /lib64/ld-linux-x86-64.so.2 (0x00007f1a6841b000) Could you dump the RPATH of the mkfs.ubifs library, like this: thomas@windsurf:~/projets/outputs/t$ readelf -d host/sbin/mkfs.ubifs Dynamic section at offset 0x11dd0 contains 29 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libz.so.1] 0x0000000000000001 (NEEDED) Shared library: [liblzo2.so.2] 0x0000000000000001 (NEEDED) Shared library: [libuuid.so.1] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000000f (RPATH) Library rpath: [/home/thomas/projets/outputs/t/per-package/host-mtd/host/lib:/home/thomas/projets/outputs/t/per-package/host-util-linux/host/lib] 0x000000000000000c (INIT) 0x402000 0x000000000000000d (FINI) 0x40bb08 0x0000000000000019 (INIT_ARRAY) 0x412dc0 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes) 0x000000000000001a (FINI_ARRAY) 0x412dc8 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x400308 0x0000000000000005 (STRTAB) 0x4009b8 0x0000000000000006 (SYMTAB) 0x400340 0x000000000000000a (STRSZ) 781 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000015 (DEBUG) 0x0 0x0000000000000003 (PLTGOT) 0x413000 0x0000000000000002 (PLTRELSZ) 1464 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x400e58 0x0000000000000007 (RELA) 0x400db0 0x0000000000000008 (RELASZ) 168 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x400d50 0x000000006fffffff (VERNEEDNUM) 2 0x000000006ffffff0 (VERSYM) 0x400cc6 0x0000000000000000 (NULL) 0x0 Thanks a lot! Thomas
Hi Thomas, On 14/11/2018 11:55, Thomas Petazzoni wrote: > Hello, > > Here is a fourth iteration of the per-package SDK and target directory > implementation. It's very late to be coming with this, and I haven't thought it through, but I realized something now... What we are doing is to create a per-package directory in which all of that package's dependencies are collected. However, NixOS for example is doing the reverse: each package gets installed in a separate staging directory, and dependant packages are built with a long list of -I and -L flags to point to all the dependencies. Well, to be exact, in many cases it's just PKG_CONFIG_LIBDIR that needs to be set. The big advantage of such an approach is that no rewriting of absolute paths is needed. It's perfectly fine if there are absolute paths, because we're not going to change the location of any library or include directory. There are a few disadvantages to this approach that I already thought of: - It doesn't fix the issue with qmake that the installation location is hardcoded in the qmake configuration - but I guess a solution for that must exist. - We're not doing exactly the same as NixOS, since NixOS will set --prefix to the real installation location, while we will use --prefix=/usr and DESTDIR=$($(PKG)_STAGING_DIR). I'm not sure what the impact is, but I'm pretty sure it will have an impact. We can't use --prefix because we don't want that absolute path on the target. - We still need to combine everything for the SDK. - Probably others... Still, given all the problems that we already found with PPD, it's something to consider I think. Regards, Arnout > > If you're interested in testing it, you can find the patch series at: > > http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4 > > Changes since v4 > ================ > > - Dropped patches that have been merged upstream, mainly the "extract > dependency" patches, but also a few other preparation patches. > > - Dropped the change of the RPATH handling. As discussed during > previous Buildroot Developers meeting, it is not a big problem if > during the build, the RPATH of a binary points to the library > folder of another package per-package folder. Therefore, instead of > fixing up RPATH at the end of each package installation, we keep > what Buildroot does today: fix them at the very end of the build. > > - Made the support for per-package folders optional. Indeed, we > realized there was no way to make sure it would be perfect from day > 1. Even if the principle works, there are lots of package-specific > details to solve, and solving all of them before merging the base > per-package folder support is not reasonable. So for now, an option > BR2_PER_PACKAGE_FOLDERS is introduced, which defaults to off. When > this option is enabled, the .NOTPARALLEL statement of the main > Makefile goes away, which allows to do top-level parallel build. > > - Addition of a commit that reworks how the top-level directories are > created. > > - Introduction of a prepare-per-package-folder function in > pkg-generic.mk to encapsulate the logic to create the per-package > folders from the dependencies of a package. > > - Rebased on top of the latest master. > > Changes since v3 > ================ > > - Dropped patches that have been merged upstream: > pkgconf: use relative path to STAGING_DIR instead of absolute path > toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN > > - For now, removed the most controversial/complicated patches > (changing the rpath strategy, and switching to per-package > host/target directories). The goal for now is to merge only the > preparation patches. > > Changes since v2 > ================ > > - Solved the problem of all DEPENDENCIES_HOST_PREREQ targets: > host-ccache, host-tar, host-lzip, host-xz, host-fakedate. > > To achieve this, introduced <pkg>_EXTRACT_DEPENDENCIES and used > that to handle the dependencies on host-tar, host-lzip and host-xz. > > Used regular dependencies for host-ccache and host-fakedate. > > See below for more details. > > Changes since v1 > ================ > > - Rebased on the latest Buildroot next branch > > - The pkg-config changes are reworked according to Arnout's comments: > only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate > pkg-config, the rest of the logic is internal to the script. > > - New patch to move the "host skeleton" logic into a proper > host-skeleton package. > > - New patch to have the CMake related files generated as part of the > toolchain package. > > - New patch to add a new "install" step that depends on the different > install steps (target, staging, images, host). This is needed to > later add some logic that is executed once all install steps of a > package have been done. > > - Fix the approach to solve the RPATH logic: instead of using > LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host > binaries right after their installation. > > - Misc other improvements in the per-package SDK implementation. > > What is this all about ? > ======================== > > See the cover letter of v3 for details: > http://lists.busybox.net/pipermail/buildroot/2017-December/208384.html > > What remains to be done? > ======================== > > The main problem that remains is fixing packages that don't work well > with per-package folders. I have already sent a few fixes (such as > adding missing host-pkgconf dependencies, or fixing the freetype > package), but I have setup a separate autobuilder instance to test > this, and it uncovered a number of other problematic packages: > > - apr-util doesn't build, because some hardcoded paths from the "apr" > per-package folders leak into the apr-util build. > > - Meson packages currently do not build, because pkg-meson.mk doesn't > use $$(HOST_DIR) to reference cross-compilation.conf, but more > importantly because cross-compilation.conf uses hardcoded paths. > > - The qt5 stack is entirely broken, because qmake hardcodes the > installation path to be $(STAGING_DIR), so all qt5 packages install > to the per-package $(STAGING_DIR) of qt5base. I started working on > this topic, and will be sending a separate e-mail about this. > > Best regards, > > Thomas > > Thomas Petazzoni (6): > Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use > support/scripts/check-host-rpath: split condition on two statements > Makefile: rework main directory creation logic > Makefile: move .NOTPARALLEL statement after including .config file > Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR > core: implement per-package SDK and target > > Config.in | 18 ++++++ > Makefile | 94 ++++++++++++++++++++------------ > fs/common.mk | 3 +- > package/pkg-generic.mk | 27 ++++++++- > support/scripts/check-host-rpath | 8 ++- > 5 files changed, 110 insertions(+), 40 deletions(-) >
Hi Thomas, Am 16.11.18 um 15:43 schrieb Thomas Petazzoni: > Hello Andreas, > > Thanks for your feedback! > > On Thu, 15 Nov 2018 17:41:27 +0100, Andreas Naumann wrote: > >> thank you for the updated version. I tried to run it with my configs. > > Thanks a lot for testing! > >> For that I needed >> a) a generic patch for all kconfig packages, see my answer on the final >> patch > > For this one, we need to continue the discussion with Yann and Thomas > De Schampheleire to understand what is the right approach. Ok. > >> b) linux-specific patch posted two days ago: [PATCH v2 1/1] linux: Make >> dtc install step more reliable > > This one is a package-specific fix, which can be merged separately from > the core of the PPSH series. I want to keep the PPSH series focused on > introducing PPSH support, and not make it grow infinitely with fixup > patches for lots of packages. > > However, perhaps what we could do is to collect those patches in > branch, this way we could have two branches: > > - One with the PPSH core stuff > > - One with all the PPSH-related fixes, which are being > submitted/applied progressively to Buildroot, even before PPSH > itself is merged. > > What do you think ? Ok, but right now I have only this one additional patch. > >> On my local machine things seemed to work, however final rootfs creation >> failed in our CI environment due to broken host- binaries, e.g. >> mkfs.ubifs not finding some required lib (liblzo2.so.2 in that >> particular case). >> It seems some things with rpath manipulation/checking do not yet work >> properly. Looking into the binary with ldd I see libs used from the >> host-machine that probably should come from buildroots host/usr/lib... >> linux-vdso.so.1 => (0x00007ffc06fcc000) >> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f760428f000) >> liblzo2.so.2 => not found >> libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f760408a000) >> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7603d81000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76039b7000) >> /lib64/ld-linux-x86-64.so.2 (0x00007f76044a9000) > > This is weird, because it works fine here: > > thomas@windsurf:~/projets/outputs/t$ ldd host/sbin/mkfs.ubifs > linux-vdso.so.1 (0x00007ffe8cd3b000) > libz.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libz.so.1 (0x00007f1a683ff000) > liblzo2.so.2 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/liblzo2.so.2 (0x00007f1a683db000) > libuuid.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libuuid.so.1 (0x00007f1a683d2000) > libm.so.6 => /lib64/libm.so.6 (0x00007f1a6821e000) > libc.so.6 => /lib64/libc.so.6 (0x00007f1a68058000) > /lib64/ld-linux-x86-64.so.2 (0x00007f1a6841b000) > > Could you dump the RPATH of the mkfs.ubifs library, like this: > > thomas@windsurf:~/projets/outputs/t$ readelf -d host/sbin/mkfs.ubifs > > Dynamic section at offset 0x11dd0 contains 29 entries: > Tag Type Name/Value > 0x0000000000000001 (NEEDED) Shared library: [libz.so.1] > 0x0000000000000001 (NEEDED) Shared library: [liblzo2.so.2] > 0x0000000000000001 (NEEDED) Shared library: [libuuid.so.1] > 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] > 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] > 0x000000000000000f (RPATH) Library rpath: [/home/thomas/projets/outputs/t/per-package/host-mtd/host/lib:/home/thomas/projets/outputs/t/per-package/host-util-linux/host/lib] > 0x000000000000000c (INIT) 0x402000 > 0x000000000000000d (FINI) 0x40bb08 > 0x0000000000000019 (INIT_ARRAY) 0x412dc0 > 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes) > 0x000000000000001a (FINI_ARRAY) 0x412dc8 > 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) > 0x000000006ffffef5 (GNU_HASH) 0x400308 > 0x0000000000000005 (STRTAB) 0x4009b8 > 0x0000000000000006 (SYMTAB) 0x400340 > 0x000000000000000a (STRSZ) 781 (bytes) > 0x000000000000000b (SYMENT) 24 (bytes) > 0x0000000000000015 (DEBUG) 0x0 > 0x0000000000000003 (PLTGOT) 0x413000 > 0x0000000000000002 (PLTRELSZ) 1464 (bytes) > 0x0000000000000014 (PLTREL) RELA > 0x0000000000000017 (JMPREL) 0x400e58 > 0x0000000000000007 (RELA) 0x400db0 > 0x0000000000000008 (RELASZ) 168 (bytes) > 0x0000000000000009 (RELAENT) 24 (bytes) > 0x000000006ffffffe (VERNEED) 0x400d50 > 0x000000006fffffff (VERNEEDNUM) 2 > 0x000000006ffffff0 (VERSYM) 0x400cc6 > 0x0000000000000000 (NULL) 0x0 > I've done that and found that in the troubling case the RPATH is empty. This might be a problem of a custom wrapper I use which creates multiple images in different output folders. Digging deeper will take some time, thanks for the pointer! regards, Andreas > Thanks a lot! > > Thomas >
Hello, Here is a fourth iteration of the per-package SDK and target directory implementation. If you're interested in testing it, you can find the patch series at: http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4 Changes since v4 ================ - Dropped patches that have been merged upstream, mainly the "extract dependency" patches, but also a few other preparation patches. - Dropped the change of the RPATH handling. As discussed during previous Buildroot Developers meeting, it is not a big problem if during the build, the RPATH of a binary points to the library folder of another package per-package folder. Therefore, instead of fixing up RPATH at the end of each package installation, we keep what Buildroot does today: fix them at the very end of the build. - Made the support for per-package folders optional. Indeed, we realized there was no way to make sure it would be perfect from day 1. Even if the principle works, there are lots of package-specific details to solve, and solving all of them before merging the base per-package folder support is not reasonable. So for now, an option BR2_PER_PACKAGE_FOLDERS is introduced, which defaults to off. When this option is enabled, the .NOTPARALLEL statement of the main Makefile goes away, which allows to do top-level parallel build. - Addition of a commit that reworks how the top-level directories are created. - Introduction of a prepare-per-package-folder function in pkg-generic.mk to encapsulate the logic to create the per-package folders from the dependencies of a package. - Rebased on top of the latest master. Changes since v3 ================ - Dropped patches that have been merged upstream: pkgconf: use relative path to STAGING_DIR instead of absolute path toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN - For now, removed the most controversial/complicated patches (changing the rpath strategy, and switching to per-package host/target directories). The goal for now is to merge only the preparation patches. Changes since v2 ================ - Solved the problem of all DEPENDENCIES_HOST_PREREQ targets: host-ccache, host-tar, host-lzip, host-xz, host-fakedate. To achieve this, introduced <pkg>_EXTRACT_DEPENDENCIES and used that to handle the dependencies on host-tar, host-lzip and host-xz. Used regular dependencies for host-ccache and host-fakedate. See below for more details. Changes since v1 ================ - Rebased on the latest Buildroot next branch - The pkg-config changes are reworked according to Arnout's comments: only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate pkg-config, the rest of the logic is internal to the script. - New patch to move the "host skeleton" logic into a proper host-skeleton package. - New patch to have the CMake related files generated as part of the toolchain package. - New patch to add a new "install" step that depends on the different install steps (target, staging, images, host). This is needed to later add some logic that is executed once all install steps of a package have been done. - Fix the approach to solve the RPATH logic: instead of using LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host binaries right after their installation. - Misc other improvements in the per-package SDK implementation. What is this all about ? ======================== See the cover letter of v3 for details: http://lists.busybox.net/pipermail/buildroot/2017-December/208384.html What remains to be done? ======================== The main problem that remains is fixing packages that don't work well with per-package folders. I have already sent a few fixes (such as adding missing host-pkgconf dependencies, or fixing the freetype package), but I have setup a separate autobuilder instance to test this, and it uncovered a number of other problematic packages: - apr-util doesn't build, because some hardcoded paths from the "apr" per-package folders leak into the apr-util build. - Meson packages currently do not build, because pkg-meson.mk doesn't use $$(HOST_DIR) to reference cross-compilation.conf, but more importantly because cross-compilation.conf uses hardcoded paths. - The qt5 stack is entirely broken, because qmake hardcodes the installation path to be $(STAGING_DIR), so all qt5 packages install to the per-package $(STAGING_DIR) of qt5base. I started working on this topic, and will be sending a separate e-mail about this. Best regards, Thomas Thomas Petazzoni (6): Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use support/scripts/check-host-rpath: split condition on two statements Makefile: rework main directory creation logic Makefile: move .NOTPARALLEL statement after including .config file Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR core: implement per-package SDK and target Config.in | 18 ++++++ Makefile | 94 ++++++++++++++++++++------------ fs/common.mk | 3 +- package/pkg-generic.mk | 27 ++++++++- support/scripts/check-host-rpath | 8 ++- 5 files changed, 110 insertions(+), 40 deletions(-)