| Message ID | 1474032631-21311-1-git-send-email-fpa@sbtaqua.com |
|---|---|
| State | Rejected |
| Headers | show |
Hello,
Please wrap the text in your commit messages to a reasonable line width
(72 characters).
On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote:
> The previous behaviour was to compile host-python3 with `--enable-shared` as is done for the target python installation. The problem is that if the host *already* has a python installation, then host-python3 will be called with the system `libpython3.5.so`. So even though the $(HOST_DIR) python executable is called, the system .so is used.
This should not be the case. All host binaries we build are built with
-Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up
libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib.
Can you double check what "readelf -a output/host/usr/bin/python" says
about this? readelf should show you this rpath encoded into the Python
binary. If that's not the case, then this is the problem.
Best regards,
Thomas
Hi Thomas Thanks again for reviewing. It appears that the host-python3 executable doesn't have an rpath entry. This would indeed explain the weird behaviour. I looked at the build log and found that "-Wl,-rpath" is indeed correctly passed on: -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib This is reflected in the resulting Makefile: CONFIGURE_LDFLAGS= -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/lib -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib -Wl,--enable-new-dtags So I don't know how it can be that the rpath is not set in the final binary. Do you also see this problem? Thanks again. *Frederik Aalund* Chief Information Officer, Co-owner SBT Aqua Copenhagen, Denmark Mobile: +45 30340086 fpa@sbtaqua.com sbtaqua.com On Fri, 16 Sep 2016 at 19:08 Thomas Petazzoni < thomas.petazzoni@free-electrons.com> wrote: > Hello, > > Please wrap the text in your commit messages to a reasonable line width > (72 characters). > > On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote: > > The previous behaviour was to compile host-python3 with > `--enable-shared` as is done for the target python installation. The > problem is that if the host *already* has a python installation, then > host-python3 will be called with the system `libpython3.5.so`. So even > though the $(HOST_DIR) python executable is called, the system .so is used. > > This should not be the case. All host binaries we build are built with > -Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up > libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib. > > Can you double check what "readelf -a output/host/usr/bin/python" says > about this? readelf should show you this rpath encoded into the Python > binary. If that's not the case, then this is the problem. > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com >
I noticed that while the host-python executable doesn't have an rpath value, it does have a runpath value: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libpython3.5m.so.1.0] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000001d (RUNPATH) Library runpath: [/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib] 0x000000000000000c (INIT) 0x4008c0 Turns out that I had LD_LIBRARY_PATH defined which overrides runpath (though not rpath). After undefining LD_LIBRARY_PATH, everything worked. Sorry for the confusion. *Frederik Aalund* Chief Information Officer, Co-owner SBT Aqua Copenhagen, Denmark Mobile: +45 30340086 fpa@sbtaqua.com sbtaqua.com On Tue, 20 Sep 2016 at 15:57 Frederik Peter Aalund <fpa@sbtaqua.com> wrote: > Hi Thomas > > Thanks again for reviewing. It appears that the host-python3 executable > doesn't have an rpath entry. This would indeed explain the weird behaviour. > > I looked at the build log and found that "-Wl,-rpath" is indeed correctly > passed on: > -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > > This is reflected in the resulting Makefile: > CONFIGURE_LDFLAGS= > -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/lib > -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > -Wl,--enable-new-dtags > > So I don't know how it can be that the rpath is not set in the final > binary. Do you also see this problem? > > Thanks again. > > *Frederik Aalund* > Chief Information Officer, Co-owner > > SBT Aqua > Copenhagen, Denmark > Mobile: +45 30340086 > fpa@sbtaqua.com > sbtaqua.com > > On Fri, 16 Sep 2016 at 19:08 Thomas Petazzoni < > thomas.petazzoni@free-electrons.com> wrote: > >> Hello, >> >> Please wrap the text in your commit messages to a reasonable line width >> (72 characters). >> >> On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote: >> > The previous behaviour was to compile host-python3 with >> `--enable-shared` as is done for the target python installation. The >> problem is that if the host *already* has a python installation, then >> host-python3 will be called with the system `libpython3.5.so`. So even >> though the $(HOST_DIR) python executable is called, the system .so is used. >> >> This should not be the case. All host binaries we build are built with >> -Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up >> libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib. >> >> Can you double check what "readelf -a output/host/usr/bin/python" says >> about this? readelf should show you this rpath encoded into the Python >> binary. If that's not the case, then this is the problem. >> >> Best regards, >> >> Thomas >> -- >> Thomas Petazzoni, CTO, Free Electrons >> Embedded Linux and Kernel engineering >> http://free-electrons.com >> >
Hello, On Fri, 23 Sep 2016 16:10:41 +0000, Frederik Peter Aalund wrote: > I noticed that while the host-python executable doesn't have an rpath > value, it does have a runpath value: > > Tag Type Name/Value > 0x0000000000000001 (NEEDED) Shared library: > [libpython3.5m.so.1.0] > 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] > 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] > 0x000000000000001d (RUNPATH) Library runpath: > [/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib] > 0x000000000000000c (INIT) 0x4008c0 Interesting. We are passing -Wl,-rpath, so it should be a rpath. Not sure why it gets turned into a runpath :/ > Turns out that I had LD_LIBRARY_PATH defined which overrides runpath > (though not rpath). After undefining LD_LIBRARY_PATH, everything worked. Ah, yes, indeed. But as said above, we're supposed to have a rpath in all binaries installed in output/host/usr/bin, so having a runpath here is not expected. However, it at least explains why you were seeing the problem, while many other people were using host-python3 without any problem. Thanks! Thomas
diff --git a/package/python3/python3.mk b/package/python3/python3.mk index 1b63f95..093f570 100644 --- a/package/python3/python3.mk +++ b/package/python3/python3.mk @@ -25,6 +25,7 @@ PYTHON3_LIBTOOL_PATCH = NO # third-party Python modules. HOST_PYTHON3_CONF_OPTS += \ + --disable-shared \ --without-ensurepip \ --without-cxx-main \ --disable-sqlite3 \
The previous behaviour was to compile host-python3 with `--enable-shared` as is done for the target python installation. The problem is that if the host *already* has a python installation, then host-python3 will be called with the system `libpython3.5.so`. So even though the $(HOST_DIR) python executable is called, the system .so is used. On my system, I have python 3.5.1. I was trying to python 3.5.2 for buildroot. I noticed druing the build that the host-python thought it was 3.5.1 (as on the system). This was caused by the above. In combination with the `PYTHON3_PYC_ONLY`, this caused all the *.pyc files to be compiled for 3.5.1 even though it should have been 3.5.2. I've changed the `python3.mk` file so that host-python3 is built statically. This way, such errors will not occur. Alternatively, one could make sure to always call host-python3 with an `LD_LIBRARY_PATH` which points to the `$(HOST_DIR)`. However, this requires a much more invasive change than simply compiling python statically. Signed-off-by: Frederik Aalund <fpa@sbtaqua.com> --- package/python3/python3.mk | 1 + 1 file changed, 1 insertion(+)