| Message ID | b5cbc35d4d8e4457563fdf5b8dfd04360c6616a5.1487811280.git.sam.bobroff@au1.ibm.com |
|---|---|
| State | RFC |
| Headers | show |
Hello, On Thu, 23 Feb 2017 11:54:54 +1100, Sam Bobroff wrote: > Add gobject-introspection, built using host-qemu. Ouch, this is complicated dependency. Have you taken into account that it's only available for some architectures? Also there's the major issue of kernel headers version between host and target. See the comment in qemu.mk: # The principle of qemu-user is that it emulates the instructions of # the target architecture when running the binary, and then when this # binary does a system call, it converts this system call into a # system call on the host machine. This mechanism makes an assumption: # that the target binary will not do system calls that do not exist on # the host. This basically requires that the target binary should be # built with kernel headers that are older or the same as the kernel # version running on the host machine. This makes using host-qemu user mode emulation very unpractical, because you must have target kernel headers older than the host kernel headers, which is frequently not true. > Also updates libglib2 to a more recent version to satasify a dependency. Which dependency? Your SoB line is missing. > diff --git a/package/gobject-introspection/0001-ldd-cross-launcher.patch b/package/gobject-introspection/0001-ldd-cross-launcher.patch > new file mode 100644 > index 000000000..9129431cb > --- /dev/null > +++ b/package/gobject-introspection/0001-ldd-cross-launcher.patch > @@ -0,0 +1,22 @@ > +*** a/giscanner/shlibs.py 2016-06-01 13:42:52.559661299 +1000 > +--- b/giscanner/shlibs.py 2016-06-01 13:44:43.028025807 +1000 > +*************** > +*** 103,109 **** > + if platform_system == 'Darwin': > + args.extend(['otool', '-L', binary.args[0]]) > + else: > +! args.extend(['ldd', binary.args[0]]) > + proc = subprocess.Popen(args, stdout=subprocess.PIPE) > + patterns = {} > + for library in libraries: > +--- 103,112 ---- > + if platform_system == 'Darwin': > + args.extend(['otool', '-L', binary.args[0]]) > + else: > +! ldd = os.getenv('GI_LDD') > +! if not ldd: > +! ldd = 'ldd' > +! args.extend([ldd, binary.args[0]]) > + proc = subprocess.Popen(args, stdout=subprocess.PIPE) > + patterns = {} > + for library in libraries: Please use "diff -u" to generate patches. And more precisely, for projects using Git as their version control system, we want a Git formatted patch, i.e the result of "git format-patch". > diff --git a/package/gobject-introspection/Config.in b/package/gobject-introspection/Config.in > new file mode 100644 > index 000000000..253c20b30 > --- /dev/null > +++ b/package/gobject-introspection/Config.in > @@ -0,0 +1,12 @@ > +config BR2_PACKAGE_GOBJECT_INTROSPECTION > + bool "gobject-introspection" Indentation with tabs. > + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_3 # ldd Not sure to see the relationship between ldd, which is provided by the C library, and the version of the compiler. > + select BR2_PACKAGE_HOST_QEMU > + select BR2_PACKAGE_PKGCONF You really need pkgconf on the target? Seems unlikely. > + select BR2_PACKAGE_LIBGLIB2 > + select BR2_PACKAGE_PYTHON > + Unneeded empty line. > + help > + gobject-introspection We want a better description here. > + > + http://wiki.gnome.org/ And a better upstream URL here. > +GOBJECT_INTROSPECTION_VERSION = 1.51.1 Only one space before and after '=' sign. > +GOBJECT_INTROSPECTION_SITE = $(call github,GNOME,gobject-introspection,$(GOBJECT_INTROSPECTION_VERSION)) Not tarball releases? > +GOBJECT_INTROSPECTION_LICENSE = GPLv2+ LGPLv2.1 Licenses should be comma separated, and you should preferably indicate what is under LGPLv2.1 and what is under GPLv2+. > +GOBJECT_INTROSPECTION_LICENSE_FILES = COPYING COPYING.GPL COPYING.LGPL COPYING.lib COPYING.tools > +GOBJECT_INTROSPECTION_INSTALL_STAGING = YES > +GOBJECT_INTROSPECTION_MAKE_OPTS = GI_CROSS_LAUNCHER="$(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH)" GI_LDD=$(STAGING_DIR)/usr/bin/ldd-cross INTROSPECTION_SCANNER=g-ir-scanner INTROSPECTION_COMPILER=g-ir-compiler Please split long lines. > +GOBJECT_INTROSPECTION_DEPENDENCIES = pkgconf python libglib2 host-qemu host-gobject-introspection > +HOST_GOBJECT_INTROSPECTION_DEPENDENCIES = host-pkgconf host-python host-libglib2 toolchain > + > +define GOBJECT_INTROSPECTION_BOOTSTRAP > + cd $(GOBJECT_INTROSPECTION_DIR) && env NOCONFIGURE=1 ./autogen.sh --host=$(GNU_TARGET_NAME) Please use <pkg>_AUTORECONF = YES. What you did is not correct because you will use autoconf/automake from the host machine instead of ones built by Buildroot. [... I did not yet review the rest of the .mk file....] > -LIBGLIB2_VERSION_MAJOR = 2.50 > -LIBGLIB2_VERSION = $(LIBGLIB2_VERSION_MAJOR).2 > +LIBGLIB2_VERSION_MAJOR = 2.51 This is a development/unstable version, so we typically don't package it in Buildroot. Best regards, Thomas
Hi Sam, Thanks for sharing your work. I'll add a few comments to Thomas'. This is not a full review. On Thu, Feb 23, 2017 at 11:54:54AM +1100, Sam Bobroff wrote: > Add gobject-introspection, built using host-qemu. > > Also updates libglib2 to a more recent version to satasify a dependency. > --- > package/Config.in | 1 + > .../0001-ldd-cross-launcher.patch | 22 +++++++++ > package/gobject-introspection/Config.in | 12 +++++ > package/gobject-introspection/create-ldd-cross.sh | 46 ++++++++++++++++++ > .../gobject-introspection/gobject-introspection.mk | 55 ++++++++++++++++++++++ A .hash file is missing. > package/libglib2/libglib2.hash | 4 +- > package/libglib2/libglib2.mk | 4 +- > 7 files changed, 140 insertions(+), 4 deletions(-) > create mode 100644 package/gobject-introspection/0001-ldd-cross-launcher.patch > create mode 100644 package/gobject-introspection/Config.in > create mode 100755 package/gobject-introspection/create-ldd-cross.sh > create mode 100644 package/gobject-introspection/gobject-introspection.mk [snip] > diff --git a/package/gobject-introspection/Config.in b/package/gobject-introspection/Config.in > new file mode 100644 > index 000000000..253c20b30 > --- /dev/null > +++ b/package/gobject-introspection/Config.in > @@ -0,0 +1,12 @@ > +config BR2_PACKAGE_GOBJECT_INTROSPECTION > + bool "gobject-introspection" > + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_3 # ldd > + select BR2_PACKAGE_HOST_QEMU > + select BR2_PACKAGE_PKGCONF > + select BR2_PACKAGE_LIBGLIB2 > + select BR2_PACKAGE_PYTHON Dependencies are missing. Here is what I came up with: depends on BR2_USE_WCHAR # glib2, python depends on BR2_TOOLCHAIN_HAS_THREADS # glib2, python depends on BR2_USE_MMU # glib2, python depends on !BR2_STATIC_LIBS # python Add to that also BR2_PACKAGE_HOST_QEMU arch dependencies. > + > + help > + gobject-introspection > + > + http://wiki.gnome.org/ https://wiki.gnome.org/Projects/GObjectIntrospection [snip] > +################################################################################ > +# > +# gobject-introspection > +# > +################################################################################ > + > +GOBJECT_INTROSPECTION_VERSION = 1.51.1 Latest version is 1.51.3. > +GOBJECT_INTROSPECTION_SITE = $(call github,GNOME,gobject-introspection,$(GOBJECT_INTROSPECTION_VERSION)) We prefer upstream provided tarballs when available. Get it from http://ftp.gnome.org/pub/GNOME/sources/gobject-introspection/. > +GOBJECT_INTROSPECTION_LICENSE = GPLv2+ LGPLv2.1 There are a few more. Here is what I found: GOBJECT_INTROSPECTION_LICENSE = LGPLv2+, GPLv2+, MIT, BSD-2c > +GOBJECT_INTROSPECTION_LICENSE_FILES = COPYING COPYING.GPL COPYING.LGPL COPYING.lib COPYING.tools There are some duplications here. My list in accordance with the above is: GOBJECT_INTROSPECTION_LICENSE_FILES = COPYING COPYING.GPL COPYING.LGPL \ giscanner/collections/ordereddict.py giscanner/scannerlexer.l baruch
Hello Sam, On Thursday 23 February 2017 11:54:54 CET Sam Bobroff wrote: [...] > diff --git a/package/gobject-introspection/create-ldd-cross.sh b/package/ gobject-introspection/create-ldd-cross.sh > new file mode 100755 > index 000000000..58d4281e8 > --- /dev/null > +++ b/package/gobject-introspection/create-ldd-cross.sh > @@ -0,0 +1,46 @@ > +#!/bin/bash > +set -e > + > +if [ $# -ne 2 ]; then > + echo "Usage: <full qemu path> <staging dir>" > + exit 2 > +fi > + > +QEMU="$1" > +STAGING_DIR="$2" > +OUT="$STAGING_DIR/usr/bin/ldd-cross" > +cat > "$OUT"<<'EOF' > +#! /bin/bash > + > +set -e > + > +EOF > +echo "QEMU=\"$QEMU\"" >> "$OUT" > +eval "$(grep ^RTLDLIST= $STAGING_DIR/usr/bin/ldd)" > + > +echo -n 'RTLDLIST="' >> "$OUT" > +for RTLD in $RTLDLIST; do > + echo -n "$STAGING_DIR$RTLD " >> "$OUT" > +done > +echo '"' >> "$OUT" > +echo >> "$OUT" > + > +cat >> "$OUT"<<'EOF' > +for file do > + case $file in > + */*) : > + ;; > + *) file=./$file > + ;; > + esac > + for RTLD in ${RTLDLIST}; do > + if test -x $RTLD; then > + "$QEMU" "$RTLD" --list "$file" Why do not call directly ldd? It does not give same result? In add, do uclibc and musl provide ldd? If you are looking for a cross-ldd, you can take a look to this script: https://gist.github.com/jerome-pouiller/c403786c1394f53f44a3b61214489e6f It come from crosstool-ng project (and written by Yann Morin). You may also have noticed that Yocto use "prelink"[1]. However, it seems overkill for our usage. [1] http://git.yoctoproject.org/cgit/cgit.cgi/prelink-cross/ > + fi > + done > +done > +EOF
On Thu, Feb 23, 2017 at 10:18:20AM +0100, Thomas Petazzoni wrote: > Hello, > > On Thu, 23 Feb 2017 11:54:54 +1100, Sam Bobroff wrote: > > Add gobject-introspection, built using host-qemu. > > Ouch, this is complicated dependency. Have you taken into account that > it's only available for some architectures? Also there's the major > issue of kernel headers version between host and target. See the > comment in qemu.mk: > > # The principle of qemu-user is that it emulates the instructions of > # the target architecture when running the binary, and then when this > # binary does a system call, it converts this system call into a > # system call on the host machine. This mechanism makes an assumption: > # that the target binary will not do system calls that do not exist on > # the host. This basically requires that the target binary should be > # built with kernel headers that are older or the same as the kernel > # version running on the host machine. > > This makes using host-qemu user mode emulation very unpractical, > because you must have target kernel headers older than the host kernel > headers, which is frequently not true. Yes, absolutely, and I really don't know how we should handle it. I searched fairly hard for another solution but the only other one seems to be to rewrite gobject-introspection completely and the other projects that have attempted this have used the same host-qemu approach (that's yocto and cygwin). I suppose what we want is to make this available but to make the constraints obvious so that noone is confused when it doesn't work. I thought about adding some text to the QEMU section of the menuconfig to explain the constraint but I couldn't work out how to make it useful, because gobject-introspection will select host-QEMU usermode by dependency so a user may never go to the QEMU section. I suppose we could remove the auto-selection and force users to go select QEMU manually. Maybe? Would a comment in the menuconfig section for gobject-introspection be any better? It's more likely to be seen but it's not really the "right" place -- you can run into the kernel version issue with QEMU usermode without using gobject-introspection (although not during the build!). But more generally, should buildroot accept this kind of solution at all? It works and is useful but it significantly more complex and therefore problem-prone than normal... > > Also updates libglib2 to a more recent version to satasify a dependency. > > Which dependency? Oh right, good point. It's gobject-introspection itself but I'll fix the comment. > Your SoB line is missing. Yes, and I also posted it as an RFC. I didn't want to misrepresent this as ready for inclusion, or even that it will ever be included, given what we're discussing up above about usermode QEMU. > > diff --git a/package/gobject-introspection/0001-ldd-cross-launcher.patch b/package/gobject-introspection/0001-ldd-cross-launcher.patch > > new file mode 100644 > > index 000000000..9129431cb > > --- /dev/null > > +++ b/package/gobject-introspection/0001-ldd-cross-launcher.patch > > @@ -0,0 +1,22 @@ > > +*** a/giscanner/shlibs.py 2016-06-01 13:42:52.559661299 +1000 > > +--- b/giscanner/shlibs.py 2016-06-01 13:44:43.028025807 +1000 > > +*************** > > +*** 103,109 **** > > + if platform_system == 'Darwin': > > + args.extend(['otool', '-L', binary.args[0]]) > > + else: > > +! args.extend(['ldd', binary.args[0]]) > > + proc = subprocess.Popen(args, stdout=subprocess.PIPE) > > + patterns = {} > > + for library in libraries: > > +--- 103,112 ---- > > + if platform_system == 'Darwin': > > + args.extend(['otool', '-L', binary.args[0]]) > > + else: > > +! ldd = os.getenv('GI_LDD') > > +! if not ldd: > > +! ldd = 'ldd' > > +! args.extend([ldd, binary.args[0]]) > > + proc = subprocess.Popen(args, stdout=subprocess.PIPE) > > + patterns = {} > > + for library in libraries: > > Please use "diff -u" to generate patches. And more precisely, for > projects using Git as their version control system, we want a Git > formatted patch, i.e the result of "git format-patch". Sorry I'll fix it for v2! (I generated that patch a long time ago). If we're able to get this to a point where we think we'll include it in buildroot, I'll try upstreaming it. It's upstream seems to be at least somewhat active. > > diff --git a/package/gobject-introspection/Config.in b/package/gobject-introspection/Config.in > > new file mode 100644 > > index 000000000..253c20b30 > > --- /dev/null > > +++ b/package/gobject-introspection/Config.in > > @@ -0,0 +1,12 @@ > > +config BR2_PACKAGE_GOBJECT_INTROSPECTION > > + bool "gobject-introspection" > > Indentation with tabs. Done. > > + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_3 # ldd > > Not sure to see the relationship between ldd, which is provided by the > C library, and the version of the compiler. I'll remove this. It is cruft from an earlier version where I was trying to pull ldd in from the toolchain (I think it's distributed with glibc) but I gave up on that approach because it's not installed when buildroot builds toolchains. > > + select BR2_PACKAGE_HOST_QEMU > > + select BR2_PACKAGE_PKGCONF > > You really need pkgconf on the target? Seems unlikely. Removed. > > > + select BR2_PACKAGE_LIBGLIB2 > > + select BR2_PACKAGE_PYTHON > > + > > Unneeded empty line. Removed. > > > + help > > + gobject-introspection > > We want a better description here. Will do. > > > + > > + http://wiki.gnome.org/ > > And a better upstream URL here. OK. > > > +GOBJECT_INTROSPECTION_VERSION = 1.51.1 > > Only one space before and after '=' sign. > > > +GOBJECT_INTROSPECTION_SITE = $(call github,GNOME,gobject-introspection,$(GOBJECT_INTROSPECTION_VERSION)) > > Not tarball releases? There wasn't one I could find, but I re-checked (since you asked) and it looks like there is one now! I'll switch it over. > > +GOBJECT_INTROSPECTION_LICENSE = GPLv2+ LGPLv2.1 > > Licenses should be comma separated, and you should preferably indicate > what is under LGPLv2.1 and what is under GPLv2+. Ah, OK. Will do (I think I can do it correctly now). > > +GOBJECT_INTROSPECTION_LICENSE_FILES = COPYING COPYING.GPL COPYING.LGPL COPYING.lib COPYING.tools > > +GOBJECT_INTROSPECTION_INSTALL_STAGING = YES > > +GOBJECT_INTROSPECTION_MAKE_OPTS = GI_CROSS_LAUNCHER="$(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH)" GI_LDD=$(STAGING_DIR)/usr/bin/ldd-cross INTROSPECTION_SCANNER=g-ir-scanner INTROSPECTION_COMPILER=g-ir-compiler > > Please split long lines. Done. > > +GOBJECT_INTROSPECTION_DEPENDENCIES = pkgconf python libglib2 host-qemu host-gobject-introspection > > +HOST_GOBJECT_INTROSPECTION_DEPENDENCIES = host-pkgconf host-python host-libglib2 toolchain > > + > > +define GOBJECT_INTROSPECTION_BOOTSTRAP > > + cd $(GOBJECT_INTROSPECTION_DIR) && env NOCONFIGURE=1 ./autogen.sh --host=$(GNU_TARGET_NAME) > > Please use <pkg>_AUTORECONF = YES. What you did is not correct because > you will use autoconf/automake from the host machine instead of ones > built by Buildroot. Ah, good point but AUTORECONF fails, or as least it did. I'll re-investigate and see if I can get it working. If I can't, I'll document it properly so that we know what the actual reason is. > [... I did not yet review the rest of the .mk file....] > > > -LIBGLIB2_VERSION_MAJOR = 2.50 > > -LIBGLIB2_VERSION = $(LIBGLIB2_VERSION_MAJOR).2 > > +LIBGLIB2_VERSION_MAJOR = 2.51 > > This is a development/unstable version, so we typically don't package it in Buildroot. Hmm, it may be the only version that can work but I'll see if I can get a proper release one working. > Best regards, > > Thomas Thanks! Sam.
Hello, On Fri, 24 Feb 2017 15:07:13 +1100, Sam Bobroff wrote: > Yes, absolutely, and I really don't know how we should handle it. I > searched fairly hard for another solution but the only other one seems > to be to rewrite gobject-introspection completely and the other > projects that have attempted this have used the same host-qemu approach > (that's yocto and cygwin). Can you summarize what is Qemu used for in the context of building gobject-introspection ? If it's to generate some architecture-specific file, can we pre-generate it, and bundle it with Buildroot ? > Would a comment in the menuconfig section for gobject-introspection be > any better? It's more likely to be seen but it's not really the "right" > place -- you can run into the kernel version issue with QEMU usermode > without using gobject-introspection (although not during the build!). > > But more generally, should buildroot accept this kind of solution at > all? It works and is useful but it significantly more complex and > therefore problem-prone than normal... I'd really like to avoid a qemu-based solution if possible. But of course, if there's really no other option and people want gobject-introspection, we won't have much choice :/ > > Please use <pkg>_AUTORECONF = YES. What you did is not correct because > > you will use autoconf/automake from the host machine instead of ones > > built by Buildroot. > > Ah, good point but AUTORECONF fails, or as least it did. I'll > re-investigate and see if I can get it working. OK. Depending on what their autogen script is doing, you might be able to fix it by using some kind of POST_PATCH_HOOKS. It really depends on what the actual issue is. Thanks, Thomas
On 24-02-17 09:25, Thomas Petazzoni wrote: > On Fri, 24 Feb 2017 15:07:13 +1100, Sam Bobroff wrote: > >> Yes, absolutely, and I really don't know how we should handle it. I >> searched fairly hard for another solution but the only other one seems >> to be to rewrite gobject-introspection completely and the other >> projects that have attempted this have used the same host-qemu approach >> (that's yocto and cygwin). > Can you summarize what is Qemu used for in the context of building > gobject-introspection ? > > If it's to generate some architecture-specific file, can we > pre-generate it, and bundle it with Buildroot ? As far as I can see, it just runs ldd under qemu, to find the paths to the libraries that will actually be used (i.e., the libraries that have to be introspected, I guess). That's why Jerome suggested to use his cross-ldd scriptlet [1], which would also work with musl and uClibc. However, as far as I understand, it really only needs to find a list of shared libraries used by a program. In our case, instead of going through ldd, we can just use readelf because there is normally only a single instance of each SONAME in the target. I'm not 100% sure, however; I also see some references to dlopen() though it's not clear to me if this is called during the build. >> Would a comment in the menuconfig section for gobject-introspection be >> any better? It's more likely to be seen but it's not really the "right" >> place -- you can run into the kernel version issue with QEMU usermode >> without using gobject-introspection (although not during the build!). >> >> But more generally, should buildroot accept this kind of solution at >> all? It works and is useful but it significantly more complex and >> therefore problem-prone than normal... > I'd really like to avoid a qemu-based solution if possible. But of > course, if there's really no other option and people want > gobject-introspection, we won't have much choice :/ At least, if we do go the qemu-user way, it should always work, so also with musl and uClibc, also if host headers are older than target... But really, cross-compilation doesn't work at all (i.e. qemu is needed), and anyway doesn't work on the unusual architectures where Debian is not available, then I don't think Buildroot is the right way. Regards, Arnout [1] https://gist.github.com/jerome-pouiller/c403786c1394f53f44a3b61214489e6f
Hi Arnout, On Mon, Feb 27, 2017 at 03:58:54PM +0100, Arnout Vandecappelle wrote: > On 24-02-17 09:25, Thomas Petazzoni wrote: > > On Fri, 24 Feb 2017 15:07:13 +1100, Sam Bobroff wrote: > > > >> Yes, absolutely, and I really don't know how we should handle it. I > >> searched fairly hard for another solution but the only other one seems > >> to be to rewrite gobject-introspection completely and the other > >> projects that have attempted this have used the same host-qemu approach > >> (that's yocto and cygwin). > > Can you summarize what is Qemu used for in the context of building > > gobject-introspection ? > > > > If it's to generate some architecture-specific file, can we > > pre-generate it, and bundle it with Buildroot ? > > As far as I can see, it just runs ldd under qemu, to find the paths to the > libraries that will actually be used (i.e., the libraries that have to be > introspected, I guess). That's why Jerome suggested to use his cross-ldd > scriptlet [1], which would also work with musl and uClibc. > > However, as far as I understand, it really only needs to find a list of shared > libraries used by a program. In our case, instead of going through ldd, we can > just use readelf because there is normally only a single instance of each SONAME > in the target. My understanding is that the process is more involved than that. Quoting the Yocto summary[1]: The data is generated when building such a library, by linking the library with a small executable binary that asks the library to describe itself, then executing the binary and processing its output. I didn't verify this myself yet. [1] https://lists.yoctoproject.org/pipermail/yocto/2016-April/029579.html baruch
Hello, On Mon, 27 Feb 2017 15:58:54 +0100, Arnout Vandecappelle wrote: > At least, if we do go the qemu-user way, it should always work, so also with > musl and uClibc, also if host headers are older than target... It simply cannot work with host headers older than the target headers, by principle. > But really, cross-compilation doesn't work at all (i.e. qemu is > needed), and anyway doesn't work on the unusual architectures where > Debian is not available, then I don't think Buildroot is the right > way. Do you meant "*if* cross-compilation doesn't work at all" ? Thomas
On Mon, Feb 27, 2017 at 04:58:59PM +0100, Thomas Petazzoni wrote: > Hello, > > On Mon, 27 Feb 2017 15:58:54 +0100, Arnout Vandecappelle wrote: > > > At least, if we do go the qemu-user way, it should always work, so also with > > musl and uClibc, also if host headers are older than target... > > It simply cannot work with host headers older than the target headers, > by principle. Right. I believe it's because what it's doing is translating system calls from one arch (and kernel version) to another (arch and kernel version) -- and if the target's headers are newer than the host kernel, there are new things that need to be translated that have no equivalent on the host. It's just impossible. > > But really, cross-compilation doesn't work at all (i.e. qemu is > > needed), and anyway doesn't work on the unusual architectures where > > Debian is not available, then I don't think Buildroot is the right > > way. > > Do you meant "*if* cross-compilation doesn't work at all" ? > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com
On Mon, Feb 27, 2017 at 05:06:37PM +0200, Baruch Siach wrote: > Hi Arnout, > > On Mon, Feb 27, 2017 at 03:58:54PM +0100, Arnout Vandecappelle wrote: > > On 24-02-17 09:25, Thomas Petazzoni wrote: > > > On Fri, 24 Feb 2017 15:07:13 +1100, Sam Bobroff wrote: > > > > > >> Yes, absolutely, and I really don't know how we should handle it. I > > >> searched fairly hard for another solution but the only other one seems > > >> to be to rewrite gobject-introspection completely and the other > > >> projects that have attempted this have used the same host-qemu approach > > >> (that's yocto and cygwin). > > > Can you summarize what is Qemu used for in the context of building > > > gobject-introspection ? > > > > > > If it's to generate some architecture-specific file, can we > > > pre-generate it, and bundle it with Buildroot ? > > > > As far as I can see, it just runs ldd under qemu, to find the paths to the > > libraries that will actually be used (i.e., the libraries that have to be > > introspected, I guess). That's why Jerome suggested to use his cross-ldd > > scriptlet [1], which would also work with musl and uClibc. > > > > However, as far as I understand, it really only needs to find a list of shared > > libraries used by a program. In our case, instead of going through ldd, we can > > just use readelf because there is normally only a single instance of each SONAME > > in the target. > > My understanding is that the process is more involved than that. Quoting the > Yocto summary[1]: > > The data is generated when building such a library, by linking the library > with a small executable binary that asks the library to describe itself, > then executing the binary and processing its output. > > I didn't verify this myself yet. > > [1] https://lists.yoctoproject.org/pipermail/yocto/2016-April/029579.html > > baruch That matches my understanding too, and I can add a little bit that I've deduced by trying to build it: It operates in two modes: one for libraries and one for binaries. For binaries, they must be compiled and linked against gobject-introspection and some function calls added to get access to argv[], then when the "scanner" is run it executes the target and passes some magic command line options which trigger those functions to output the introspection data. For libraries, it begins by running "ldd" on them and scraping the output but I'm not sure what happens after that. In both cases the data that's gathered is target-specific and can't be generated on a different arch. To complicate things gobject-introspection uses it's scanner on itself as part of it's own build process. So the problems aren't strictly do do with compiling, but with the build process as a whole. And the real fix is probably to re-design gobject-introspection so that it can "cross-inspect" data but the problem with any redesign is that a lot of packages use it so any change would require all of them to change. When I was digging around the 'net looking for solutions I found people complaining about this cross-build problem with it's design from years ago so it doesn't look like it's going to be changed any time soon. So what I've done is first build gobject-introspection for the host, unaltered. We don't need to run it on any host binaries but we need a gobject-introspection that can run on the host. We patch the binary so that we have a way to get it to use QEMU to run ldd (that's ldd-cross) or QEMU to run binaries directly. We don't use this to build the host version, however. (Oddly, upstream have accepted a patch to add a wrapper for binaries but not for libraries. Probably because running the target's 'ldd' directly under QEMU usermode doesn't work -- that's why I had to add the ldd-cross script.) Then I fake a staging install by manually creating scripts with the names of the scanner and other tools, and the scripts invoke the host tools with the wrappers activated, so they can analyse target binaries. Using this staging version we can build a native target version. We need this because target executables that use gobject-introspection have linked against it. We won't ever use the gi code in these target versions, on the target, it's just using during the cross building but they won't run later on the real target if they can't link. Now we can use the real target verion's libaries to link binaries and the staging wrapped cross-version to analyse them. As long as the house of cards stays up it works ;-) As I said before I feel quite conflicted about this overall -- it's necessary for so many packages but it's easy to break and when it does it can be very hard to debug (for example, I hit a bug in QEMU usermode and it's pretty hard to debug that deep inside cross building!). Apart from that, it has been so laborious just to get this much working that I feel a lot of empathy for the poor souls who run into cross compiling gobject-introspeciton themselves... So I want to help them if I can ;-) Cheers, Sam.
Hi; I know this post is a bit old; however, I would really like to see gobject-introspection added to Buildroot. It's preventing a fair amount of packages from being updated. Heck, python-gobject is stuck at 2.28.6 because of this, which is from 2011! Adding this dependency is beyond my capabilities; however I do see the importance of it, which is why I would like to see this patch series brought back to life. Thanks! Adam -- View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/RFC-PATCH-0-2-add-gobject-introspection-tp157754p173505.html Sent from the Buildroot (busybox) mailing list archive at Nabble.com.
Hi Adam, I experience the same difficulties because of lack of object-introspection. Cheers, Alex -- View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/RFC-PATCH-0-2-add-gobject-introspection-tp157754p173614.html Sent from the Buildroot (busybox) mailing list archive at Nabble.com.
On 17-08-17 08:44, Alexey Roslyakov wrote: > Hi Adam, > > I experience the same difficulties because of lack of object-introspection. I agree that gobject-introspection would definitely be nice to have, but it is complicated... And somebody will have to do the work! In addition to the comments made on Sam's patch, here are some additional sources of information. [1] describes a Linaro study of the possibilities of cross-compiling gobject-introspection. It's kind of wordy, but the bottom line is: almost impossible. It dates from 2011 but I don't think things have changed fundamentally. [2] shows the gobject-introspection architecture, which explains the different steps and how they fit together. Something like this should definitely be part of the commit message. From this, I gather that g-ir-compiler is the problem, because it's not g-ir-cross-compiler :-) [3] describes how it is solved in an OpenEmbedded layer. The interesting bit from this one is that they have 4 separate packages. I don't think we need 4, but 2 could be useful. Regards, Arnout [1] https://wiki.linaro.org/PeterPearse/GobjectIntrospection [2] https://wiki.gnome.org/Projects/GObjectIntrospection/Architecture [3] https://github.com/Guacamayo/meta-gir/blob/master/README.md
On Thu, Aug 17, 2017 at 6:39 PM, Arnout Vandecappelle <arnout@mind.be> wrote: > > > On 17-08-17 08:44, Alexey Roslyakov wrote: >> Hi Adam, >> >> I experience the same difficulties because of lack of object-introspection. > > I agree that gobject-introspection would definitely be nice to have, but it is > complicated... And somebody will have to do the work! > I don't disagree there! However; I started getting into it and I quickly realized it's going to be far above my pay-grade to integrate this beast into BuildRoot! > In addition to the comments made on Sam's patch, here are some additional > sources of information. > > [1] describes a Linaro study of the possibilities of cross-compiling > gobject-introspection. It's kind of wordy, but the bottom line is: almost > impossible. It dates from 2011 but I don't think things have changed fundamentally. > Well, in the Yocto project, there's a patch named: Revert-an-incomplete-upstream-attempt-at-cross-compi.patch (I assume it should be Revert an incomplete upstream attempt at cross-compile support). So upstream is starting to try and get cross-compiling support built into gobject-introspection it looks like! > [2] shows the gobject-introspection architecture, which explains the different > steps and how they fit together. Something like this should definitely be part > of the commit message. From this, I gather that g-ir-compiler is the problem, > because it's not g-ir-cross-compiler :-) > Probably, which is why qemu needs to be used. > [3] describes how it is solved in an OpenEmbedded layer. The interesting bit > from this one is that they have 4 separate packages. I don't think we need 4, > but 2 could be useful. > Is it possible to take most of the recipe from yocto and perhaps translate it into a make file that works for BuildRoot? > > Regards, > Arnout > > > [1] https://wiki.linaro.org/PeterPearse/GobjectIntrospection > [2] https://wiki.gnome.org/Projects/GObjectIntrospection/Architecture > [3] https://github.com/Guacamayo/meta-gir/blob/master/README.md > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot Thanks! Adam
On 22-08-17 12:57, Adam Duskett wrote: > On Thu, Aug 17, 2017 at 6:39 PM, Arnout Vandecappelle <arnout@mind.be> wrote: [snip] > Well, in the Yocto project, there's a patch named: > Revert-an-incomplete-upstream-attempt-at-cross-compi.patch > (I assume it should be Revert an incomplete upstream attempt at cross-compile > support). So upstream is starting to try and get cross-compiling support built > into gobject-introspection it looks like! Hm, could be interesting to work with upstream to really get cross-compilation to work... But still, someone needs to do it :-) >> [3] describes how it is solved in an OpenEmbedded layer. The interesting bit >> from this one is that they have 4 separate packages. I don't think we need 4, >> but 2 could be useful. >> > Is it possible to take most of the recipe from yocto and perhaps > translate it into > a make file that works for BuildRoot? Possibly, I haven't looked at the details. I also haven't looked at how they deal with the kernel ABI issue in qemu-user. Perhaps it works just with glibc and they assume that glibc is built with the very-old-kernel compatibility layer. Regards, Arnout
On Thu, Aug 24, 2017 at 7:03 PM, Arnout Vandecappelle <arnout@mind.be> wrote: > > > On 22-08-17 12:57, Adam Duskett wrote: >> On Thu, Aug 17, 2017 at 6:39 PM, Arnout Vandecappelle <arnout@mind.be> wrote: > [snip] >> Well, in the Yocto project, there's a patch named: >> Revert-an-incomplete-upstream-attempt-at-cross-compi.patch >> (I assume it should be Revert an incomplete upstream attempt at cross-compile >> support). So upstream is starting to try and get cross-compiling support built >> into gobject-introspection it looks like! > > Hm, could be interesting to work with upstream to really get cross-compilation > to work... But still, someone needs to do it :-) > I think upstream is probably happy with yocto/oe just using the scripts they made + qemu. ;) > > >>> [3] describes how it is solved in an OpenEmbedded layer. The interesting bit >>> from this one is that they have 4 separate packages. I don't think we need 4, >>> but 2 could be useful. >>> >> Is it possible to take most of the recipe from yocto and perhaps >> translate it into >> a make file that works for BuildRoot? > > Possibly, I haven't looked at the details. I also haven't looked at how they > deal with the kernel ABI issue in qemu-user. Perhaps it works just with glibc > and they assume that glibc is built with the very-old-kernel compatibility layer. > I highly doubt that considering OE/Yocto uses uclibc as well as glibc. They even have a bunch of patches to get SystemD working with uclibc. > Regards, > Arnout > > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF Adam
Hi, > I highly doubt that considering OE/Yocto uses uclibc as well as glibc. > They even > have a bunch of patches to get SystemD working with uclibc. OE no longer supports uclibc/uclibc-ng. Yocto just seems to carry an old recipe for uclibc. Latest release of uClibc-ng and systemd no longer requires any patches, so this shows how old the uclibc-ng support in OE might be, if there are still leftovers. best regards Waldemar
On Sun, Aug 27, 2017 at 1:11 AM, Waldemar Brodkorb <wbx@openadk.org> wrote: > Hi, > >> I highly doubt that considering OE/Yocto uses uclibc as well as glibc. >> They even >> have a bunch of patches to get SystemD working with uclibc. > > OE no longer supports uclibc/uclibc-ng. > Yocto just seems to carry an old recipe for uclibc. > Latest release of uClibc-ng and systemd no longer requires any patches, so this shows how old the uclibc-ng support in OE might be, if there are still leftovers. > best regards > Waldemar Ah fair enough. Either way, using qemu, it hopefully *should* work with uclibc. Thanks for the info on uclibc in OE! It's been a while since I was in that game! :) Adam
diff --git a/package/Config.in b/package/Config.in index deff0fe2c..bde5246e1 100644 --- a/package/Config.in +++ b/package/Config.in @@ -1348,6 +1348,7 @@ menu "Other" source "package/libffi/Config.in" source "package/libgee/Config.in" source "package/libglib2/Config.in" + source "package/gobject-introspection/Config.in" source "package/libglob/Config.in" source "package/libical/Config.in" source "package/libite/Config.in" diff --git a/package/gobject-introspection/0001-ldd-cross-launcher.patch b/package/gobject-introspection/0001-ldd-cross-launcher.patch new file mode 100644 index 000000000..9129431cb --- /dev/null +++ b/package/gobject-introspection/0001-ldd-cross-launcher.patch @@ -0,0 +1,22 @@ +*** a/giscanner/shlibs.py 2016-06-01 13:42:52.559661299 +1000 +--- b/giscanner/shlibs.py 2016-06-01 13:44:43.028025807 +1000 +*************** +*** 103,109 **** + if platform_system == 'Darwin': + args.extend(['otool', '-L', binary.args[0]]) + else: +! args.extend(['ldd', binary.args[0]]) + proc = subprocess.Popen(args, stdout=subprocess.PIPE) + patterns = {} + for library in libraries: +--- 103,112 ---- + if platform_system == 'Darwin': + args.extend(['otool', '-L', binary.args[0]]) + else: +! ldd = os.getenv('GI_LDD') +! if not ldd: +! ldd = 'ldd' +! args.extend([ldd, binary.args[0]]) + proc = subprocess.Popen(args, stdout=subprocess.PIPE) + patterns = {} + for library in libraries: diff --git a/package/gobject-introspection/Config.in b/package/gobject-introspection/Config.in new file mode 100644 index 000000000..253c20b30 --- /dev/null +++ b/package/gobject-introspection/Config.in @@ -0,0 +1,12 @@ +config BR2_PACKAGE_GOBJECT_INTROSPECTION + bool "gobject-introspection" + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_3 # ldd + select BR2_PACKAGE_HOST_QEMU + select BR2_PACKAGE_PKGCONF + select BR2_PACKAGE_LIBGLIB2 + select BR2_PACKAGE_PYTHON + + help + gobject-introspection + + http://wiki.gnome.org/ diff --git a/package/gobject-introspection/create-ldd-cross.sh b/package/gobject-introspection/create-ldd-cross.sh new file mode 100755 index 000000000..58d4281e8 --- /dev/null +++ b/package/gobject-introspection/create-ldd-cross.sh @@ -0,0 +1,46 @@ +#!/bin/bash +set -e + +if [ $# -ne 2 ]; then + echo "Usage: <full qemu path> <staging dir>" + exit 2 +fi + +QEMU="$1" +STAGING_DIR="$2" +OUT="$STAGING_DIR/usr/bin/ldd-cross" + +cat > "$OUT"<<'EOF' +#! /bin/bash + +set -e + +EOF +echo "QEMU=\"$QEMU\"" >> "$OUT" +eval "$(grep ^RTLDLIST= $STAGING_DIR/usr/bin/ldd)" + +echo -n 'RTLDLIST="' >> "$OUT" +for RTLD in $RTLDLIST; do + echo -n "$STAGING_DIR$RTLD " >> "$OUT" +done +echo '"' >> "$OUT" +echo >> "$OUT" + +cat >> "$OUT"<<'EOF' +for file do + case $file in + */*) : + ;; + *) file=./$file + ;; + esac + for RTLD in ${RTLDLIST}; do + if test -x $RTLD; then + "$QEMU" "$RTLD" --list "$file" + fi + done +done +EOF + +chmod +x "$OUT" +echo "Wrote: $OUT" diff --git a/package/gobject-introspection/gobject-introspection.mk b/package/gobject-introspection/gobject-introspection.mk new file mode 100644 index 000000000..2ee8fc8e0 --- /dev/null +++ b/package/gobject-introspection/gobject-introspection.mk @@ -0,0 +1,55 @@ +################################################################################ +# +# gobject-introspection +# +################################################################################ + +GOBJECT_INTROSPECTION_VERSION = 1.51.1 +GOBJECT_INTROSPECTION_SITE = $(call github,GNOME,gobject-introspection,$(GOBJECT_INTROSPECTION_VERSION)) +GOBJECT_INTROSPECTION_LICENSE = GPLv2+ LGPLv2.1 +GOBJECT_INTROSPECTION_LICENSE_FILES = COPYING COPYING.GPL COPYING.LGPL COPYING.lib COPYING.tools +GOBJECT_INTROSPECTION_INSTALL_STAGING = YES +GOBJECT_INTROSPECTION_MAKE_OPTS = GI_CROSS_LAUNCHER="$(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH)" GI_LDD=$(STAGING_DIR)/usr/bin/ldd-cross INTROSPECTION_SCANNER=g-ir-scanner INTROSPECTION_COMPILER=g-ir-compiler +GOBJECT_INTROSPECTION_DEPENDENCIES = pkgconf python libglib2 host-qemu host-gobject-introspection +HOST_GOBJECT_INTROSPECTION_DEPENDENCIES = host-pkgconf host-python host-libglib2 toolchain + +define GOBJECT_INTROSPECTION_BOOTSTRAP + cd $(GOBJECT_INTROSPECTION_DIR) && env NOCONFIGURE=1 ./autogen.sh --host=$(GNU_TARGET_NAME) +endef +GOBJECT_INTROSPECTION_POST_EXTRACT_HOOKS += GOBJECT_INTROSPECTION_BOOTSTRAP + +define HOST_GOBJECT_INTROSPECTION_BOOTSTRAP + cd $(HOST_GOBJECT_INTROSPECTION_DIR) && env NOCONFIGURE=1 ./autogen.sh +endef +HOST_GOBJECT_INTROSPECTION_POST_EXTRACT_HOOKS += HOST_GOBJECT_INTROSPECTION_BOOTSTRAP + +# We must hack staging twice: once after installing the host version in order to +# provide the cross-versions that can be used to build the target version +# and again after the target version has overwritten the staging files as it +# installs to staging. +HOST_GOBJECT_INTROSPECTION_POST_INSTALL_HOOKS += GOBJECT_INTROSPECTION_HACK_STAGING +GOBJECT_INTROSPECTION_POST_INSTALL_STAGING_HOOKS += GOBJECT_INTROSPECTION_HACK_STAGING_PKGCONFIG GOBJECT_INTROSPECTION_HACK_STAGING +# TODO: Use a better name than ldd-cross? Move it to a separate package? +define GOBJECT_INTROSPECTION_HACK_STAGING_PKGCONFIG + echo "HACKING STAGING PACKAGE CONFIG" + sed --in-place -e "s:^prefix=.*:prefix=$(STAGING_DIR)/usr:" $(STAGING_DIR)/usr/lib/pkgconfig/gobject-introspection-1.0.pc + sed --in-place -e "s:^exec_prefix=.*:exec_prefix=$(STAGING_DIR)/usr:" $(STAGING_DIR)/usr/lib/pkgconfig/gobject-introspection-1.0.pc +endef + +define GOBJECT_INTROSPECTION_HACK_STAGING + echo "CREATING STAGING g-ir-scanner" + echo "#!/bin/sh" > $(STAGING_DIR)/usr/bin/g-ir-scanner + echo "env GI_CROSS_LAUNCHER=\"$(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH)\" GI_LDD=ldd-cross $(HOST_DIR)/usr/bin/g-ir-scanner \""'$$'@"\"" >> $(STAGING_DIR)/usr/bin/g-ir-scanner + + echo "CREATING STAGING g-ir-compiler" + echo "#!/bin/sh" > $(STAGING_DIR)/usr/bin/g-ir-compiler + echo "env GI_CROSS_LAUNCHER=\"$(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH)\" GI_LDD=ldd-cross $(HOST_DIR)/usr/bin/g-ir-compiler \""'$$'"@\"" >> $(STAGING_DIR)/usr/bin/g-ir-compiler + + echo "CREATING STAGING ldd-cross" + $(GOBJECT_INTROSPECTION_PKGDIR)/create-ldd-cross.sh $(HOST_DIR)/usr/bin/qemu-$(HOST_QEMU_ARCH) $(STAGING_DIR) + echo "Created:" + cat $(STAGING_DIR)/usr/bin/ldd-cross +endef + +$(eval $(host-autotools-package)) +$(eval $(autotools-package)) diff --git a/package/libglib2/libglib2.hash b/package/libglib2/libglib2.hash index 94bf8700b..126b051b9 100644 --- a/package/libglib2/libglib2.hash +++ b/package/libglib2/libglib2.hash @@ -1,2 +1,2 @@ -# https://download.gnome.org/sources/glib/2.50/glib-2.50.2.sha256sum -sha256 be68737c1f268c05493e503b3b654d2b7f43d7d0b8c5556f7e4651b870acfbf5 glib-2.50.2.tar.xz +# https://download.gnome.org/sources/glib/2.51/glib-2.51.0.sha256sum +sha256 f113b7330f4b4a43e3e401fe7849e751831060d574bd936a63e979887137a74a glib-2.51.0.tar.xz diff --git a/package/libglib2/libglib2.mk b/package/libglib2/libglib2.mk index ddd1f3965..0e8c9410a 100644 --- a/package/libglib2/libglib2.mk +++ b/package/libglib2/libglib2.mk @@ -4,8 +4,8 @@ # ################################################################################ -LIBGLIB2_VERSION_MAJOR = 2.50 -LIBGLIB2_VERSION = $(LIBGLIB2_VERSION_MAJOR).2 +LIBGLIB2_VERSION_MAJOR = 2.51 +LIBGLIB2_VERSION = $(LIBGLIB2_VERSION_MAJOR).0 LIBGLIB2_SOURCE = glib-$(LIBGLIB2_VERSION).tar.xz LIBGLIB2_SITE = http://ftp.gnome.org/pub/gnome/sources/glib/$(LIBGLIB2_VERSION_MAJOR) LIBGLIB2_LICENSE = LGPLv2+