Patchwork [2/4] python: fix python-config for cross-compilation

login
register
mail settings
Submitter Thomas Petazzoni
Date Sept. 30, 2013, 9:20 p.m.
Message ID <1380576047-21001-2-git-send-email-thomas.petazzoni@free-electrons.com>
Download mbox | patch
Permalink /patch/279267/
State Superseded
Headers show

Comments

Thomas Petazzoni - Sept. 30, 2013, 9:20 p.m.
Some packages (trace-cmd) use python-config to find out which headers
and libraries should be used to link against the Python libraries. By
default, python-config returns paths that are inappropriate for
cross-compilation: this patch fixes this by prepending those paths by
the staging directory location.

We also fix one of the python*-config file to actually be a symbolic
link to the script, rather than a copy of it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/python/python.mk | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
Peter Korsgaard - Oct. 1, 2013, 9:58 p.m.
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 Thomas> Some packages (trace-cmd) use python-config to find out which
 Thomas> headers and libraries should be used to link against the Python
 Thomas> libraries. By default, python-config returns paths that are
 Thomas> inappropriate for cross-compilation: this patch fixes this by
 Thomas> prepending those paths by the staging directory location.

My first thought was that this should just use PYTHON_CONFIG_SCRIPTS,
but looking closer at it, I see that it's actually a python script so it
won't work. It would be good to add a comment about this.

The shebang reads:

#!/usr/bin/python2.7

Now that we go to the effort to build a host-python I think it makes
sense to fixup the shebang as well - But:

./output/host/usr/bin/python output/staging/usr/bin/python2.7-config --cflags
-I/home/peko/source/buildroot/output/host/usr/include/python2.7 -I/home/peko/source/buildroot/output/host/usr/include/python2.7 -fno-strict-aliasing -O2 -I/home/peko/source/buildroot/output/host/usr/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

E.G. your s/-I/-I$(STAGING_DIR)/ trick won't work (and it didn't work
before if the host python wasn't built with --prefix=/usr).

 Thomas> We also fix one of the python*-config file to actually be a symbolic
 Thomas> link to the script, rather than a copy of it.

This I don't see here:

ls -lah output/staging/usr/bin/python*-config
-rwxr-xr-x 1 peko peko 1.6K Oct  1 23:50 output/staging/usr/bin/python2.7-config
lrwxrwxrwx 1 peko peko   16 Oct  1 23:50 output/staging/usr/bin/python2-config -> python2.7-config
lrwxrwxrwx 1 peko peko   14 Oct  1 23:50 output/staging/usr/bin/python-config -> python2-config

Care to fix these issues and resubmit?
Thomas Petazzoni - Oct. 2, 2013, 7:13 a.m.
Dear Peter Korsgaard,

On Tue, 01 Oct 2013 23:58:32 +0200, Peter Korsgaard wrote:

> My first thought was that this should just use PYTHON_CONFIG_SCRIPTS,
> but looking closer at it, I see that it's actually a python script so it
> won't work. It would be good to add a comment about this.

Ok.

> The shebang reads:
> 
> #!/usr/bin/python2.7
> 
> Now that we go to the effort to build a host-python I think it makes
> sense to fixup the shebang as well - But:

I am not sure. We build host-python only to be able to build the Python
interpreter for the target. For all the rest, we already require a
Python interpreter to be installed on the host machine. The current
situation is a bit confusing: some packages that need a Python
interpreter at build time do depend on host-python, some other packages
do not depend on host-python, and instead rely on the fact that
Buildroot verifies that a Python interpreter is already available on
the build machine.

I believe this is something that should be clarified.

> ./output/host/usr/bin/python output/staging/usr/bin/python2.7-config --cflags
> -I/home/peko/source/buildroot/output/host/usr/include/python2.7 -I/home/peko/source/buildroot/output/host/usr/include/python2.7 -fno-strict-aliasing -O2 -I/home/peko/source/buildroot/output/host/usr/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
> 
> E.G. your s/-I/-I$(STAGING_DIR)/ trick won't work (and it didn't work
> before if the host python wasn't built with --prefix=/usr).

Right, from the above, I am not sure we want to change the shebang of
this script.

>  Thomas> We also fix one of the python*-config file to actually be a symbolic
>  Thomas> link to the script, rather than a copy of it.
> 
> This I don't see here:
> 
> ls -lah output/staging/usr/bin/python*-config
> -rwxr-xr-x 1 peko peko 1.6K Oct  1 23:50 output/staging/usr/bin/python2.7-config
> lrwxrwxrwx 1 peko peko   16 Oct  1 23:50 output/staging/usr/bin/python2-config -> python2.7-config
> lrwxrwxrwx 1 peko peko   14 Oct  1 23:50 output/staging/usr/bin/python-config -> python2-config
> 
> Care to fix these issues and resubmit?

Huh? Odd, I'll have to test this once again then. However, before
resending a new version, I'd like to see how we're going to clarify the
host-python thing vs. Python as a mandatory dependency.

Best regards,

Thomas
Peter Korsgaard - Oct. 2, 2013, 7:43 a.m.
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 >> The shebang reads:
 >> 
 >> #!/usr/bin/python2.7
 >> 
 >> Now that we go to the effort to build a host-python I think it makes
 >> sense to fixup the shebang as well - But:

 Thomas> I am not sure. We build host-python only to be able to build
 Thomas> the Python interpreter for the target. For all the rest, we
 Thomas> already require a Python interpreter to be installed on the
 Thomas> host machine. The current situation is a bit confusing: some
 Thomas> packages that need a Python interpreter at build time do depend
 Thomas> on host-python, some other packages do not depend on
 Thomas> host-python, and instead rely on the fact that Buildroot
 Thomas> verifies that a Python interpreter is already available on the
 Thomas> build machine.

 Thomas> I believe this is something that should be clarified.

Yes, I agree it's a bit of a mess.

As we already build host-python as a dependency of python, we presumably
might as well use it for anything building python modules as well?


 >> ./output/host/usr/bin/python output/staging/usr/bin/python2.7-config --cflags
 >> -I/home/peko/source/buildroot/output/host/usr/include/python2.7 -I/home/peko/source/buildroot/output/host/usr/include/python2.7 -fno-strict-aliasing -O2 -I/home/peko/source/buildroot/output/host/usr/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
 >> 
 >> E.G. your s/-I/-I$(STAGING_DIR)/ trick won't work (and it didn't work
 >> before if the host python wasn't built with --prefix=/usr).

 Thomas> Right, from the above, I am not sure we want to change the shebang of
 Thomas> this script.

Back when we dicussed host-python, somebody (I think it was Samuel)
mentioned that 'python' has become python 3.x on some distributions, and
as this script uses internal python stuff to spit out the needed CFLAGS
/ LDFLAGS, I think we NEED to run it with the correct python version.

Same for people with 'python' not in /usr.

The script apparently doesn't even work with python 3.x:

python3 /usr/bin/python2.7-config
  File "/usr/bin/python2.7-config", line 34
    print sysconfig.PREFIX
                  ^
SyntaxError: invalid syntax

Patch

diff --git a/package/python/python.mk b/package/python/python.mk
index df02b35..dfa0686 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -134,14 +134,27 @@  PYTHON_MAKE_ENV = \
 	PYTHON_MODULES_INCLUDE=$(STAGING_DIR)/usr/include \
 	PYTHON_MODULES_LIB="$(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib"
 
-# python distutils adds -L$LIBDIR when linking binary extensions, causing
-# trouble for cross compilation
-define PYTHON_FIXUP_LIBDIR
+# 1. python distutils adds -L$LIBDIR when linking binary extensions,
+# causing trouble for cross compilation.
+#
+# 2. adjust the python-config in the staging directory so that it
+# returns include paths that are valid on the build machine (i.e
+# prefixed by the STAGING_DIR).
+#
+# 3. make sure all python-*config are symbolic links to
+# python2.7-config, instead of being another file.
+#
+define PYTHON_FIXUP_DIRS
 	$(SED) 's|^LIBDIR=.*|LIBDIR= $(STAGING_DIR)/usr/lib|' \
 	   $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/config/Makefile
+	$(SED) "s|'-I'|'-I$(STAGING_DIR)'|" \
+	   $(STAGING_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR)-config
+	$(RM) -f $(STAGING_DIR)/usr/bin/python-config
+	ln -sf $(STAGING_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR)-config \
+		$(STAGING_DIR)/usr/bin/python-config
 endef
 
-PYTHON_POST_INSTALL_STAGING_HOOKS += PYTHON_FIXUP_LIBDIR
+PYTHON_POST_INSTALL_STAGING_HOOKS += PYTHON_FIXUP_DIRS
 
 #
 # Remove useless files. In the config/ directory, only the Makefile