| Message ID | 20221024174911.75444-1-abrodkin@synopsys.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | package/numactl: point to target libs via LDFLAGS | expand |
Hello Alexey, On Mon, 24 Oct 2022 19:49:11 +0200 Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote: > In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") > we opted-out from unconditional linking with "-latomic" and made it > a conditional based on real availability of __atomic_fetch_and_1() > in the toolchain. I.e. for many targets "-latomic" was not used > any longer. > > And in those supposedly rare cases when libatomic was required it was > all OK until there's a LD_LIBRARY_PATH set with something where yet > another libatomic.so instance exists, especially if that's the one from > host (in my case I used a separately built host toolchain). Then we see: > ---------------------->8----------------------- > /bin/sh ./libtool --tag=CC --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o -latomic > > libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 > .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format > collect2: error: ld returned 1 exit status > ---------------------->8----------------------- > > That happens because libtool of numactl adds contents of the aforementioned > LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it > finds becomes the one used for linking. What's more our precios cross-built > libatomic.so never even gets considered because that first found libatomic.so > is used with full path, not as "-latomic" as otherwise it would have been used > without helpful libtool. Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH defined? Do you have a reproducer for this issue? > +NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \ > + LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib" This is really a red flag, $(STAGING_DIR)/lib is already in the default search path of the compiler, so that should never be needed. Thomas
Hello, Thomas! > On Mon, 24 Oct 2022 19:49:11 +0200 > Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote: > > > In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") > > we opted-out from unconditional linking with "-latomic" and made it > > a conditional based on real availability of __atomic_fetch_and_1() > > in the toolchain. I.e. for many targets "-latomic" was not used > > any longer. > > > > And in those supposedly rare cases when libatomic was required it was > > all OK until there's a LD_LIBRARY_PATH set with something where yet > > another libatomic.so instance exists, especially if that's the one from > > host (in my case I used a separately built host toolchain). Then we see: > > ---------------------->8----------------------- > > /bin/sh ./libtool --tag=CC --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o -latomic > > > > libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 > > .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format > > collect2: error: ld returned 1 exit status > > ---------------------->8----------------------- > > > > That happens because libtool of numactl adds contents of the aforementioned > > LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it > > finds becomes the one used for linking. What's more our precios cross-built > > libatomic.so never even gets considered because that first found libatomic.so > > is used with full path, not as "-latomic" as otherwise it would have been used > > without helpful libtool. > > Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH > defined? I meant LD_LIBRARY_PATH is defined before Buildroot's top-level make invocation, i.e. "LD_LIBRARY_PATH=_some_location_of_hosts_libatomic.so_ make". See below. > Do you have a reproducer for this issue? Yes, I have finally. It took me a bit longer than expected to figure out all the details, thus answering that late. So there're a couple of prerequsites for the reproduction even on vanilla "numactl" sources, I used 2.0.14 with the Buildroot's libatomic patch [1]: 1. For example in my Ubuntu 20.04 on x86_64 I installed ARM cross-tools with: ----------------------------->8----------------------------- apt install gcc-9-arm-linux-gnueabi ----------------------------->8----------------------------- 2. LD_LIBRARY_PATH which points to a location of a foreign (e.g. host or some other architecture) "libatomic.so", plus (that's really important) right there side-by-side with "libatomic.so" it's important to have "libatomic.la" (valid libtool archive, and not an empty file, see below my copy which works for the reproduction). I.e. ----------------------------->8----------------------------- export LD_LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/ ----------------------------->8----------------------------- 3. Configure "numactl" with `lt_cv_sys_lib_search_path_spec=""`, which we exactly do in Buildroot, see [2], i.e.: ----------------------------->8----------------------------- ./configure --host=arm-linux-gnueabi lt_cv_sys_lib_search_path_spec="" ----------------------------->8----------------------------- 4. Try to build now: ----------------------------->8----------------------------- make V=1 make all-am make[1]: Entering directory 'numactl' /bin/bash ./libtool --tag=CC --mode=link arm-linux-gnueabi-gcc -g -O2 -version-info 1:0:0 -Wl,--version-script,./versions.ldscript -Wl,-init,numa_init -Wl,-fini,numa_fini -o libnuma.la -rpath /usr/local/lib libnuma.lo syscall.lo distance.lo affinity.lo sysfs.lo rtnetlink.lo -latomic libtool: warning: library '/usr/lib/gcc/x86_64-linux-gnu/9//libatomic.la' was moved. libtool: link: arm-linux-gnueabi-gcc -shared -fPIC -DPIC .libs/libnuma.o .libs/syscall.o .libs/distance.o .libs/affinity.o .libs/sysfs.o .libs/rtnetlink.o -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ /usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so -g -O2 -Wl,--version-script -Wl,./versions.ldscript -Wl,-init -Wl,numa_init -Wl,-fini -Wl,numa_fini -pthread -Wl,-soname -Wl,libnuma.so.1 -o .libs/libnuma.so.1.0.0 /usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so: file not recognized: file format not recognized collect2: error: ld returned 1 exit status make[1]: *** [Makefile:889: libnuma.la] Error 1 make[1]: Leaving directory 'numactl' make: *** [Makefile:742: all] Error 2 ----------------------------->8----------------------------- So, what happens (3) removes compiler's default paths from libtool's "sys_lib_dlsearch_path", while (2) makes libtool think that libatomic.so from LD_LIBRARY_PATH is the right one. > > +NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \ > > + LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib" > > This is really a red flag, $(STAGING_DIR)/lib is already in the default > search path of the compiler, so that should never be needed. Well, if we don't mess with "lt_cv_sys_lib_search_path_spec" in the configure script options, then all works OK, but I guess Peter's change made in 2011 makes some sense and we don't want to revert it, or do we? Also, I accept the fact that my setup is a bit unusual at least and it might be a good enough reason to disregard all that discussion, but first I'd like to make sure we really don't care about situations like that: LD_LIBRARY_PATH pointing to a self-built toolchain. If we assume that situation as one of possible corner-cases, then probably we may want to troubleshoot such funny things which happen due to "libtool" smarts. -Alexey [1] https://github.com/numactl/numactl/commit/e0de0d9e981ddb53bdeb4a4b9dc43046c9ff4ff9 [2] https://git.buildroot.net/buildroot/commit/?id=6246704008ef219257f30a824903855d0e00cffb ----------------------------->8----------------------------- Contents of "/usr/lib/gcc/x86_64-linux-gnu/9/libatomic.la" ----------------------------->8----------------------------- # libatomic.la - a libtool library file # Generated by libtool (GNU libtool 1.3134 2009-11-29) 2.2.7a # # Please DO NOT delete this file! # It is necessary for linking the library. # The name that we can dlopen(3). dlname='libatomic.so.1' # Names of this library. library_names='libatomic.so.1.2.0 libatomic.so.1 libatomic.so' # The name of the static archive. old_library='libatomic.a' # Linker flags that can not go in dependency_libs. inherited_linker_flags=' -pthread' # Libraries that this one depends upon. dependency_libs='' # Names of additional weak libraries provided by this library weak_library_names='' # Version information for libatomic. current=3 age=2 revision=0 # Is this an already installed library? installed=yes # Should we warn about portability when linking against -modules? shouldnotlink=no # Files to dlopen/dlpreopen dlopen='' dlpreopen='' # Directory that this library needs to be installed in: libdir='/usr/lib/x86_64-linux-gnu/' ----------------------------->8-----------------------------
Hello Alexey, On Mon, 24 Oct 2022 19:49:11 +0200 Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote: > In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") > we opted-out from unconditional linking with "-latomic" and made it > a conditional based on real availability of __atomic_fetch_and_1() > in the toolchain. I.e. for many targets "-latomic" was not used > any longer. We finally looked at this patch today during the on-going Buildroot hackathon. We understand/recognize the problem, but we don't like the proposed solution as you can imagine (especially as the proposed solution is package specific). In addition, and as you have yourself expressed, your situation is quite a corner case. Here are the options that we see to resolve this: (1) Patch libtool so that it doesn't look at LD_LIBRARY_PATH. We are already patching libtool to work-around other issues. However, from a quick look at the libtool code base (which of course is not the most pleasant exercise on earth), it's not trivial to see where this logic is taking place. (2) Unset LD_LIBRARY_PATH entirely in the main Buildroot makefile to avoid interference with LD_LIBRARY_PATH being set. This would of course prevent people who have special/weird setups that require LD_LIBRARY_PATH from using Buildroot. (3) Warn/error out if LD_LIBRARY_PATH is set, which essentially is a variant of (2), but where we are explicit instead of silently ignoring LD_LIBRARY_PATH defined in the user's environment. Regarding the setting of lt_cv_sys_lib_search_path_spec="" that you have identified, the motivation for this was explained in 6246704008ef219257f30a824903855d0e00cffb: package/Makefile.in: ensure libtool doesn't search host dirs for libraries Commit 7e3e8ec040b (CFLAGS/LDFLAGS: don't add -I / -L args for STAGING_DIR) exposed a lingering libtool problem. Unless instructed otherwise (using -L) libtool will search its built in system path for libraries, and use those instead if found. The default search path is '/usr/lib, /lib, /usr/local/lib', which is no good for cross compilation. Fix it by setting the system search path to the empty string, effectively disabling this feature. Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk> So this seems to still be relevant. The reason why when you drop this in your case and it "works" is we believe the following one: due to -latomic, libtool will search for libatomic. It will find it in your system in say /lib or /usr/lib, and because that's a standard search path, libtool will keep -latomic instead of turning into a full absolute path. So the cross linker gets passed -latomic, which it properly resolves to the target/sysroot libatomic. When lt_cv_sys_lib_search_path_spec="" is there, libatomic from /lib or /usr/lib isn't found, so it finds the one in your LD_LIBRARY_PATH directory, which isn't a standard path, so our dear libtool replaces -latomic by the full absolute path to libatomic, and passes that to the cross-linker, which of course blows up. Best regards, Thomas
diff --git a/package/numactl/numactl.mk b/package/numactl/numactl.mk index cf9c759693..58b304d6e1 100644 --- a/package/numactl/numactl.mk +++ b/package/numactl/numactl.mk @@ -10,6 +10,7 @@ NUMACTL_LICENSE = LGPL-2.1 (libnuma), GPL-2.0 (programs) NUMACTL_LICENSE_FILES = README.md NUMACTL_INSTALL_STAGING = YES NUMACTL_AUTORECONF = YES -NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" +NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \ + LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib" $(eval $(autotools-package))
In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") we opted-out from unconditional linking with "-latomic" and made it a conditional based on real availability of __atomic_fetch_and_1() in the toolchain. I.e. for many targets "-latomic" was not used any longer. And in those supposedly rare cases when libatomic was required it was all OK until there's a LD_LIBRARY_PATH set with something where yet another libatomic.so instance exists, especially if that's the one from host (in my case I used a separately built host toolchain). Then we see: ---------------------->8----------------------- /bin/sh ./libtool --tag=CC --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o -latomic libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format collect2: error: ld returned 1 exit status ---------------------->8----------------------- That happens because libtool of numactl adds contents of the aforementioned LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it finds becomes the one used for linking. What's more our precios cross-built libatomic.so never even gets considered because that first found libatomic.so is used with full path, not as "-latomic" as otherwise it would have been used without helpful libtool. But with paths pointing to the staging location, these paths get added to "$searchdirs" and so the right libatomic.so gets found, which solves our problem. Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com> --- package/numactl/numactl.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)