| Message ID | 20180221142801.28997-4-gael.portay@savoirfairelinux.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | None | expand |
Hello, +Peter, Yann, Arnout, there's a question below about a small change but has a big impact. On Wed, 21 Feb 2018 09:27:58 -0500, Gaël PORTAY wrote: > The host-pkg-config wrapper gets package data compiled for host. > > Signed-off-by: Gaël PORTAY <gael.portay@savoirfairelinux.com> > --- > package/pkgconf/host-pkg-config | 4 ++++ > package/pkgconf/pkgconf.mk | 6 ++++-- > 2 files changed, 8 insertions(+), 2 deletions(-) > create mode 100644 package/pkgconf/host-pkg-config > > diff --git a/package/pkgconf/host-pkg-config b/package/pkgconf/host-pkg-config > new file mode 100644 > index 0000000000..3362a3f927 > --- /dev/null > +++ b/package/pkgconf/host-pkg-config > @@ -0,0 +1,4 @@ > +#!/bin/sh > +PKGCONFDIR=$(dirname "$0") > +DEFAULT_PKG_CONFIG_PATH=${PKGCONFDIR}/../lib/pkgconfig:${PKGCONFDIR}/../share/pkgconfig > +PKG_CONFIG_PATH=${PKG_CONFIG_PATH:-${DEFAULT_PKG_CONFIG_PATH}} exec ${PKGCONFDIR}/pkgconf "$@" I've been thinking about this for a while, and I remember having a discussion about this with some other Buildroot developer a while ago. I think the most correct thing to do would be: $(HOST_DIR)/bin/pkg-config returns results valid for native compilation $(HOST_DIR)/bin/<tuple>-pkg-config returns results valid for cross-compilation I.e, the current pkg-config wrapper should be renamed <tuple>-pkg-config, and pkg-config should behave like a normal native pkg-config, except that it provides results for libraries located in $(HOST_DIR). The autoconf PKG_CHECK_MODULES() macro uses PKG_PROG_PKG_CONFIG(), which internally uses AC_PATH_TOOL(). And AC_PATH_TOOL() will first search for the program with the host machine tuple, and warn if the program cannot be found with this tuple. From https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Generic-Programs.html: """ When cross-compiling, this macro will issue a warning if no program prefixed with the host type could be found. For more information, see Specifying Target Triplets. """ I know this change will: - Potentially break a number of packages we have in Buildroot, which directly use pkg-config without first trying to use <tuple>-pkg-config - Potentially break a number of downstream users who are using pkg-config. However: - It would solve that once you add $(HOST_DIR)/bin to your PATH, you cannot anymore do native builds because "pkg-config" returns results that are not relevant for native builds. I already saw a number of people affected by this. - It would comply with the standard autoconf expectations. Best regards, Thomas
On 21-02-18 22:50, Thomas Petazzoni wrote: > Hello, > > +Peter, Yann, Arnout, there's a question below about a small change but > has a big impact. > > On Wed, 21 Feb 2018 09:27:58 -0500, Gaël PORTAY wrote: >> The host-pkg-config wrapper gets package data compiled for host. >> >> Signed-off-by: Gaël PORTAY <gael.portay@savoirfairelinux.com> >> --- >> package/pkgconf/host-pkg-config | 4 ++++ >> package/pkgconf/pkgconf.mk | 6 ++++-- >> 2 files changed, 8 insertions(+), 2 deletions(-) >> create mode 100644 package/pkgconf/host-pkg-config >> >> diff --git a/package/pkgconf/host-pkg-config b/package/pkgconf/host-pkg-config >> new file mode 100644 >> index 0000000000..3362a3f927 >> --- /dev/null >> +++ b/package/pkgconf/host-pkg-config >> @@ -0,0 +1,4 @@ >> +#!/bin/sh >> +PKGCONFDIR=$(dirname "$0") >> +DEFAULT_PKG_CONFIG_PATH=${PKGCONFDIR}/../lib/pkgconfig:${PKGCONFDIR}/../share/pkgconfig >> +PKG_CONFIG_PATH=${PKG_CONFIG_PATH:-${DEFAULT_PKG_CONFIG_PATH}} exec ${PKGCONFDIR}/pkgconf "$@" > > I've been thinking about this for a while, and I remember having a > discussion about this with some other Buildroot developer a while ago. > I think the most correct thing to do would be: > > $(HOST_DIR)/bin/pkg-config returns results valid for native compilation > > $(HOST_DIR)/bin/<tuple>-pkg-config returns results valid for cross-compilation This sounds like a good idea, but as you note below, could lead to breakage as well. An alternative would be (IIRC this idea was launched by ThomasDS (added in Cc)): $(HOST_DIR)/bin/ contains binaries valid for native compilation (i.e. no cross-compiler, cross-pkg-config, ...) $(HOST_DIR)/<tuple>/bin contains binaries valid for cross-compilation When building for host, PATH contains $(HOST_DIR)/bin When building for target, PATH contains $(HOST_DIR)/<tuple>/bin:$(HOST_DIR)/bin So let's analyse that option: > I.e, the current pkg-config wrapper should be renamed > <tuple>-pkg-config, and pkg-config should behave like a normal native > pkg-config, except that it provides results for libraries located in > $(HOST_DIR). > > The autoconf PKG_CHECK_MODULES() macro uses PKG_PROG_PKG_CONFIG(), > which internally uses AC_PATH_TOOL(). And AC_PATH_TOOL() will first > search for the program with the host machine tuple, and warn if the > program cannot be found with this tuple. From > https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Generic-Programs.html: > > """ > When cross-compiling, this macro will issue a warning if no program > prefixed with the host type could be found. For more information, see > Specifying Target Triplets. Weird, I checked a couple of log files and I could see no such warning. Ah, that's because we pass PKG_CONFIG=... in the environment. Is it the idea to remove that from the environment then? Anyway, the PATH-based alternative will not remove this warning, but that shouldn't be different from the current situation. > """ > > I know this change will: > > - Potentially break a number of packages we have in Buildroot, which > directly use pkg-config without first trying to use > <tuple>-pkg-config I.e. anything not using autotools? Well, most will probably heed the PKG_CONFIG=... in the environment. The PATH-based alternative doesn't have this potential breakage. > - Potentially break a number of downstream users who are using > pkg-config. The PATH-based alternative reduces this problem, downstream users just have to add $(HOST_DIR)/tuple/bin to their path (and Buildroot will do this for post-xxx scripts). > However: > > - It would solve that once you add $(HOST_DIR)/bin to your PATH, you > cannot anymore do native builds because "pkg-config" returns results > that are not relevant for native builds. I already saw a number of > people affected by this. The PATH-based alternative solves this as well. Note BTW that neither alternative solves the problem when building a host-tool during a target build (and yes, we have seen this problem already in some packages). AFAICS, autotools will also for host builds use the PKG_CONFIG passed in the environment or discovered through the tuple. Same for CMake. > - It would comply with the standard autoconf expectations. With the PATH-based alternative, it might make sense to have the cross-stuff both in $(HOST_DIR)/bin with the tuple prefix, and in $(HOST_DIR)/tuple/bin without the prefix. That way, we get the advantages of both: comply with autoconf expectations, and avoid breaking packages or downstream users. Regards, Arnout
Hello, Thanks for the feedback! On Thu, 22 Feb 2018 10:27:13 +0100, Arnout Vandecappelle wrote: > > I've been thinking about this for a while, and I remember having a > > discussion about this with some other Buildroot developer a while ago. > > I think the most correct thing to do would be: > > > > $(HOST_DIR)/bin/pkg-config returns results valid for native compilation > > > > $(HOST_DIR)/bin/<tuple>-pkg-config returns results valid for cross-compilation > > This sounds like a good idea, but as you note below, could lead to breakage as > well. Yes. > > An alternative would be (IIRC this idea was launched by ThomasDS (added in Cc)): > > $(HOST_DIR)/bin/ contains binaries valid for native compilation (i.e. no > cross-compiler, cross-pkg-config, ...) > > $(HOST_DIR)/<tuple>/bin contains binaries valid for cross-compilation Changing this will also lead to a lot of breakage. All our users who have scripts doing $(HOST_DIR)/bin/<tuple>-gcc will be broken. I would even qualify it as an even more radical change, breaking even more stuff. And more importantly, the result is not great. I think the most correct result is to have a single HOST_DIR/bin, with both native and cross compilation tools, with cross-compilation tools prefixed by <tuple>. > > I.e, the current pkg-config wrapper should be renamed > > <tuple>-pkg-config, and pkg-config should behave like a normal native > > pkg-config, except that it provides results for libraries located in > > $(HOST_DIR). > > > > The autoconf PKG_CHECK_MODULES() macro uses PKG_PROG_PKG_CONFIG(), > > which internally uses AC_PATH_TOOL(). And AC_PATH_TOOL() will first > > search for the program with the host machine tuple, and warn if the > > program cannot be found with this tuple. From > > https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Generic-Programs.html: > > > > """ > > When cross-compiling, this macro will issue a warning if no program > > prefixed with the host type could be found. For more information, see > > Specifying Target Triplets. > > Weird, I checked a couple of log files and I could see no such warning. Ah, > that's because we pass PKG_CONFIG=... in the environment. Yes, I believe it is because we pass PKG_CONFIG=. > Is it the idea to remove that from the environment then? Not necessarily, because PKG_CONFIG may be used by non-autoconf based packages. Some non-autotools packages do: FOO=`$PKG_CONFIG ...` > Anyway, the PATH-based alternative will not remove this warning, but that > shouldn't be different from the current situation. > > > """ > > > > I know this change will: > > > > - Potentially break a number of packages we have in Buildroot, which > > directly use pkg-config without first trying to use > > <tuple>-pkg-config > > I.e. anything not using autotools? Well, most will probably heed the > PKG_CONFIG=... in the environment. Most yes, but not all. Some of them do: FOO=`pkg-config ...` which currently works because PATH contains $(HOST_DIR)/bin, and pkg-config returns correct results thanks to us injecting the proper PKG_CONFIG_LIBDIR and PKG_CONFIG_SYSROOT_DIR depending on whether we're building a host or target package. > The PATH-based alternative doesn't have this potential breakage. > > > > - Potentially break a number of downstream users who are using > > pkg-config. > > The PATH-based alternative reduces this problem, downstream users just have to > add $(HOST_DIR)/tuple/bin to their path (and Buildroot will do this for post-xxx > scripts). Well, they still have to change to $(HOST_DIR)/tuple/bin, which means it breaks stuff for them anyway. > > However: > > > > - It would solve that once you add $(HOST_DIR)/bin to your PATH, you > > cannot anymore do native builds because "pkg-config" returns results > > that are not relevant for native builds. I already saw a number of > > people affected by this. > > The PATH-based alternative solves this as well. > > Note BTW that neither alternative solves the problem when building a host-tool > during a target build (and yes, we have seen this problem already in some > packages). AFAICS, autotools will also for host builds use the PKG_CONFIG passed > in the environment or discovered through the tuple. Same for CMake. Potentially autotools could be taught about PKG_CONFIG_FOR_BUILD, just like it does for CC_FOR_BUILD and al. But that's obviously not supported by packages today. > > - It would comply with the standard autoconf expectations. > > With the PATH-based alternative, it might make sense to have the cross-stuff > both in $(HOST_DIR)/bin with the tuple prefix, and in $(HOST_DIR)/tuple/bin > without the prefix. That way, we get the advantages of both: comply with > autoconf expectations, and avoid breaking packages or downstream users. I'm still not convinced about changing the HOST_DIR/ organization. It's a massive change, affecting everything, and not just pkg-config, and the outcome is less nice than what we have today, for my perspective. Best regards, Thomas
On 22-02-18 10:41, Thomas Petazzoni wrote: >> With the PATH-based alternative, it might make sense to have the cross-stuff >> both in $(HOST_DIR)/bin with the tuple prefix, and in $(HOST_DIR)/tuple/bin >> without the prefix. That way, we get the advantages of both: comply with >> autoconf expectations, and avoid breaking packages or downstream users. > > I'm still not convinced about changing the HOST_DIR/ organization. It's > a massive change, affecting everything, and not just pkg-config, and > the outcome is less nice than what we have today, for my perspective. Well, this extended PATH-based alternative (which is BTW what I think ThomasDS suggested to begin with) is not at all invasive: cross-things stay in host/bin as they are today, we just add host/tuple/bin which contains cross-things without the tuple-prefix, and we add this to the PATH. Then in a second step, we move pkg-config to tuple-pkg-config, BUT we also add it without prefix to host/tuple/bin. So it doesn't break anything at all, since for target builds, we have host/tuple/bin in PATH (and PKG_CONFIG=host/bin/tuple-pkg-config in the environment) - behaviour is the same as before. There is a bit of breakage, however: host/tuple/bin/gcc will be in PATH, so any package that calls gcc from PATH will now get target gcc instead of host gcc. On the other hand, apparently[1], CMake doesn't have any support for building host executables and requires a two-step approach instead; I guess most non-autotools packages will have a similar limitation. So yeah, adding host/tuple/bin to PATH may not be as uninvasive as I thought... Regards, Arnout [1] https://cmake.org/Wiki/CMake_Cross_Compiling#Using_executables_in_the_build_created_during_the_build
Hi, 2018-02-22 11:56 GMT+01:00 Arnout Vandecappelle <arnout@mind.be>: > > > On 22-02-18 10:41, Thomas Petazzoni wrote: >>> With the PATH-based alternative, it might make sense to have the cross-stuff >>> both in $(HOST_DIR)/bin with the tuple prefix, and in $(HOST_DIR)/tuple/bin >>> without the prefix. That way, we get the advantages of both: comply with >>> autoconf expectations, and avoid breaking packages or downstream users. >> >> I'm still not convinced about changing the HOST_DIR/ organization. It's >> a massive change, affecting everything, and not just pkg-config, and >> the outcome is less nice than what we have today, for my perspective. > > Well, this extended PATH-based alternative (which is BTW what I think ThomasDS > suggested to begin with) is not at all invasive: cross-things stay in host/bin > as they are today, we just add host/tuple/bin which contains cross-things > without the tuple-prefix, and we add this to the PATH. Then in a second step, we > move pkg-config to tuple-pkg-config, BUT we also add it without prefix to > host/tuple/bin. So it doesn't break anything at all, since for target builds, we > have host/tuple/bin in PATH (and PKG_CONFIG=host/bin/tuple-pkg-config in the > environment) - behaviour is the same as before. > > There is a bit of breakage, however: host/tuple/bin/gcc will be in PATH, so any > package that calls gcc from PATH will now get target gcc instead of host gcc. On > the other hand, apparently[1], CMake doesn't have any support for building host > executables and requires a two-step approach instead; I guess most non-autotools > packages will have a similar limitation. > > So yeah, adding host/tuple/bin to PATH may not be as uninvasive as I thought... > For reference, here is my mail: http://lists.busybox.net/pipermail/buildroot/2018-February/213629.html I think you misunderstood my question/proposal: what I'd need is a unique path to the cross tools (gcc, ...) that does _not_ contain the tuple anywhere in it (or is reachable via a symlink that does not contain the tuple). Where I wrote 'cross' in the example path: $(HOST_DIR)/bin/cross/{gcc,gdb,nm,readelf} I really meant the literal string 'cross', not the tuple (the exact string is of course something that can be discussed) The reason I need something like that is for scripts/build systems external to Buildroot. They do not know the tuple upfront, and would have to do tricks to determine it. All they typically know is a reference to the buildroot path and a defconfig name. So, for that use case, I do not need any changes to the PATH env variable. Just an extra shadow tree with symlinks to the necessary cross tools, but reachable without knowing the tuple. /Thomas
Thomas, All, On 2018-02-25 21:38 +0100, Thomas De Schampheleire spake thusly: [--SNIP--] > For reference, here is my mail: > http://lists.busybox.net/pipermail/buildroot/2018-February/213629.html > > I think you misunderstood my question/proposal: what I'd need is a > unique path to the cross tools (gcc, ...) that does _not_ contain the > tuple anywhere in it (or is reachable via a symlink that does not > contain the tuple). > Where I wrote 'cross' in the example path: > $(HOST_DIR)/bin/cross/{gcc,gdb,nm,readelf} > I really meant the literal string 'cross', not the tuple (the exact > string is of course something that can be discussed) > > The reason I need something like that is for scripts/build systems > external to Buildroot. They do not know the tuple upfront, and would > have to do tricks to determine it. All they typically know is a > reference to the buildroot path and a defconfig name. But then they can call buildroot to know the tuple: eval $(make -s printvars VARS=GNU_TARGET_NAME) and then gain access to the tuple with ${GNU_TARGET_NAME}. > So, for that use case, I do not need any changes to the PATH env > variable. Just an extra shadow tree with symlinks to the necessary > cross tools, but reachable without knowing the tuple. I wonder if we would want to support this case, especially since we can't have a sane way to keep it working. What worries me is that by doing what you suggest, we would be diverging *greatly* from established conventions, and any patch to packagees to support that, would not be upstreamable... Regards, Yann E. MORIN.
2018-02-25 21:53 GMT+01:00 Yann E. MORIN <yann.morin.1998@free.fr>: > Thomas, All, > > On 2018-02-25 21:38 +0100, Thomas De Schampheleire spake thusly: > [--SNIP--] >> For reference, here is my mail: >> http://lists.busybox.net/pipermail/buildroot/2018-February/213629.html >> >> I think you misunderstood my question/proposal: what I'd need is a >> unique path to the cross tools (gcc, ...) that does _not_ contain the >> tuple anywhere in it (or is reachable via a symlink that does not >> contain the tuple). >> Where I wrote 'cross' in the example path: >> $(HOST_DIR)/bin/cross/{gcc,gdb,nm,readelf} >> I really meant the literal string 'cross', not the tuple (the exact >> string is of course something that can be discussed) >> >> The reason I need something like that is for scripts/build systems >> external to Buildroot. They do not know the tuple upfront, and would >> have to do tricks to determine it. All they typically know is a >> reference to the buildroot path and a defconfig name. > > But then they can call buildroot to know the tuple: > eval $(make -s printvars VARS=GNU_TARGET_NAME) > > and then gain access to the tuple with ${GNU_TARGET_NAME}. Except that this is not always correct :-) For example, on a MIPS octeon based defconfig with external toolchain, I see: GNU_TARGET_NAME=mips64-buildroot-linux-gnu but: BR2_TOOLCHAIN_EXTERNAL_PREFIX=mips64-octeon-linux-gnu i.e. GNU_TARGET_NAME does not give the answer for all cases. Depending on whether you have internal or external toolchain, and in the latter case on whether the tool you need to access is provided by the external toolchain or built by Buildroot (like gdb), you need more logic. See also the logic and comment here: http://lists.busybox.net/pipermail/buildroot/2016-April/160055.html >> So, for that use case, I do not need any changes to the PATH env >> variable. Just an extra shadow tree with symlinks to the necessary >> cross tools, but reachable without knowing the tuple. > > I wonder if we would want to support this case, especially since we > can't have a sane way to keep it working. While I'm open to alternative solutions, I don't see the difficulty here. All existing logic remains, we'd just add a few extra symlinks from the shadow path pointing to outpt/host/bin/... > > What worries me is that by doing what you suggest, we would be diverging > *greatly* from established conventions, and any patch to packagees to > support that, would not be upstreamable... But none of the packages' source code would need any change for this. Everything remains exactly as it was. Only some additional symlinks are provided, which should normally not be needed by anything inside of Buildroot. /Thomas
diff --git a/package/pkgconf/host-pkg-config b/package/pkgconf/host-pkg-config new file mode 100644 index 0000000000..3362a3f927 --- /dev/null +++ b/package/pkgconf/host-pkg-config @@ -0,0 +1,4 @@ +#!/bin/sh +PKGCONFDIR=$(dirname "$0") +DEFAULT_PKG_CONFIG_PATH=${PKGCONFDIR}/../lib/pkgconfig:${PKGCONFDIR}/../share/pkgconfig +PKG_CONFIG_PATH=${PKG_CONFIG_PATH:-${DEFAULT_PKG_CONFIG_PATH}} exec ${PKGCONFDIR}/pkgconf "$@" diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk index 00b2d017ee..54b4570419 100644 --- a/package/pkgconf/pkgconf.mk +++ b/package/pkgconf/pkgconf.mk @@ -16,11 +16,13 @@ define PKGCONF_LINK_PKGCONFIG ln -sf pkgconf $(TARGET_DIR)/usr/bin/pkg-config endef -define HOST_PKGCONF_INSTALL_WRAPPER +define HOST_PKGCONF_INSTALL_WRAPPERS $(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \ $(HOST_DIR)/bin/pkg-config $(SED) 's,@STAGING_SUBDIR@,$(STAGING_SUBDIR),g' \ $(HOST_DIR)/bin/pkg-config + $(INSTALL) -m 0755 -D package/pkgconf/host-pkg-config \ + $(HOST_DIR)/bin/host-pkg-config endef define HOST_PKGCONF_STATIC @@ -32,7 +34,7 @@ define HOST_PKGCONF_SHARED endef PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG -HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER +HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPERS ifeq ($(BR2_STATIC_LIBS),y) HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_STATIC
The host-pkg-config wrapper gets package data compiled for host. Signed-off-by: Gaël PORTAY <gael.portay@savoirfairelinux.com> --- package/pkgconf/host-pkg-config | 4 ++++ package/pkgconf/pkgconf.mk | 6 ++++-- 2 files changed, 8 insertions(+), 2 deletions(-) create mode 100644 package/pkgconf/host-pkg-config