diff mbox series

package/pkg-meson.mk: set Python sysconfigdata

Message ID 20190804220503.13534-1-arnout@mind.be
State Superseded
Headers show
Series package/pkg-meson.mk: set Python sysconfigdata | expand

Commit Message

Arnout Vandecappelle Aug. 4, 2019, 10:05 p.m. UTC 
meson has support for Python dependencies, to e.g. build Python modules.
It parses Python's sysconfig module to find the appropriate
configuration variables. However, the sysconfig module is the one for
the host, _not_ for the target. Therefore, wrong values may be set. Many
of them get overridden anyway by the cross-compilation.conf. One
particular one that is problematic is that it uses the python-XXX.pc
pkg-config file for the host instead of the target, because it
explicitly sets the PKG_CONFIG_LIBDIR found in sysconfig.

Use the same trick as for python package: pass PYTHONPATH,
PYTHONNOUSERSITE and _PYTHON_SYSCONFIGDATA_NAME in the environment to
point to staging instead of host. This ensures the sysconfig from the
target is used instead of the one from the host.

An alternative would be to patch meson to not use the PKG_CONFIG_LIBDIR
from sysconfig. However, it is likely that other problems will pop up at
some point because of using the wrong sysconfig.

Note that this approach will still break things when meson needs to
build something for the host during a target build.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Adam Duskett <Aduskett@gmail.com>
Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Cc: Peter Seiderer <ps.report@gmx.net>
Cc: James Hilliard <james.hilliard1@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
---
The setting of _PYTHON_SYSCONFIGDATA_NAME is different from the one in
pkg-python.mk. I don't understand how that definition could work: for
Python2, it is empty, so sysconfig tries to import a module with name
"". So apparently the call paths used by meson and by
setuptools/distutils are slightly different.
---
 package/pkg-meson.mk | 3 +++
 1 file changed, 3 insertions(+)

Comments

Adam Duskett Sept. 16, 2019, 6:41 p.m. UTC | #1 
All;


On Sun, Aug 4, 2019 at 3:05 PM Arnout Vandecappelle (Essensium/Mind)
<arnout@mind.be> wrote:
>
> meson has support for Python dependencies, to e.g. build Python modules.
> It parses Python's sysconfig module to find the appropriate
> configuration variables. However, the sysconfig module is the one for
> the host, _not_ for the target. Therefore, wrong values may be set. Many
> of them get overridden anyway by the cross-compilation.conf. One
> particular one that is problematic is that it uses the python-XXX.pc
> pkg-config file for the host instead of the target, because it
> explicitly sets the PKG_CONFIG_LIBDIR found in sysconfig.
>
> Use the same trick as for python package: pass PYTHONPATH,
> PYTHONNOUSERSITE and _PYTHON_SYSCONFIGDATA_NAME in the environment to
> point to staging instead of host. This ensures the sysconfig from the
> target is used instead of the one from the host.
>
> An alternative would be to patch meson to not use the PKG_CONFIG_LIBDIR
> from sysconfig. However, it is likely that other problems will pop up at
> some point because of using the wrong sysconfig.
>
> Note that this approach will still break things when meson needs to
> build something for the host during a target build.
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Adam Duskett <Aduskett@gmail.com>
> Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> Cc: Peter Seiderer <ps.report@gmx.net>
> Cc: James Hilliard <james.hilliard1@gmail.com>
> Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> The setting of _PYTHON_SYSCONFIGDATA_NAME is different from the one in
> pkg-python.mk. I don't understand how that definition could work: for
> Python2, it is empty, so sysconfig tries to import a module with name
> "". So apparently the call paths used by meson and by
> setuptools/distutils are slightly different.
> ---
>  package/pkg-meson.mk | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> index 0b811d1cc0..3f875c08ce 100644
> --- a/package/pkg-meson.mk
> +++ b/package/pkg-meson.mk
> @@ -81,6 +81,9 @@ define $(2)_CONFIGURE_CMDS
>             -e "s%@HOST_DIR@%$$(HOST_DIR)%g" \
>             package/meson/cross-compilation.conf.in \
>             > $$($$(PKG)_SRCDIR)/build/cross-compilation.conf
> +       _PYTHON_SYSCONFIGDATA_NAME="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
> +       PYTHONPATH="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PYTHON3_PATH),$$(PYTHON_PATH))" \
> +       PYTHONNOUSERSITE=1 \
>         PATH=$$(BR_PATH) $$($$(PKG)_CONF_ENV) $$(MESON) \
>                 --prefix=/usr \
>                 --libdir=lib \
> --
> 2.21.0
>
Not only does this work, it also fixes cross-compiled python libraries names
built by meson (such as the new versions of python-gobject)

Tested-by: Adam Duskett <Aduskett@gmail.com>
Adam Duskett Sept. 27, 2019, 3:26 a.m. UTC | #2 
All;

Any update on this patch? Now that libglib 2.62.0 is in Buildroot, I
will need to submit a new patch set, which requires this patch.

Thanks!

Adam

On Mon, Sep 16, 2019 at 11:41 AM Adam Duskett <aduskett@gmail.com> wrote:
>
> All;
>
>
> On Sun, Aug 4, 2019 at 3:05 PM Arnout Vandecappelle (Essensium/Mind)
> <arnout@mind.be> wrote:
> >
> > meson has support for Python dependencies, to e.g. build Python modules.
> > It parses Python's sysconfig module to find the appropriate
> > configuration variables. However, the sysconfig module is the one for
> > the host, _not_ for the target. Therefore, wrong values may be set. Many
> > of them get overridden anyway by the cross-compilation.conf. One
> > particular one that is problematic is that it uses the python-XXX.pc
> > pkg-config file for the host instead of the target, because it
> > explicitly sets the PKG_CONFIG_LIBDIR found in sysconfig.
> >
> > Use the same trick as for python package: pass PYTHONPATH,
> > PYTHONNOUSERSITE and _PYTHON_SYSCONFIGDATA_NAME in the environment to
> > point to staging instead of host. This ensures the sysconfig from the
> > target is used instead of the one from the host.
> >
> > An alternative would be to patch meson to not use the PKG_CONFIG_LIBDIR
> > from sysconfig. However, it is likely that other problems will pop up at
> > some point because of using the wrong sysconfig.
> >
> > Note that this approach will still break things when meson needs to
> > build something for the host during a target build.
> >
> > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> > Cc: Adam Duskett <Aduskett@gmail.com>
> > Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> > Cc: Peter Seiderer <ps.report@gmx.net>
> > Cc: James Hilliard <james.hilliard1@gmail.com>
> > Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> > ---
> > The setting of _PYTHON_SYSCONFIGDATA_NAME is different from the one in
> > pkg-python.mk. I don't understand how that definition could work: for
> > Python2, it is empty, so sysconfig tries to import a module with name
> > "". So apparently the call paths used by meson and by
> > setuptools/distutils are slightly different.
> > ---
> >  package/pkg-meson.mk | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> > index 0b811d1cc0..3f875c08ce 100644
> > --- a/package/pkg-meson.mk
> > +++ b/package/pkg-meson.mk
> > @@ -81,6 +81,9 @@ define $(2)_CONFIGURE_CMDS
> >             -e "s%@HOST_DIR@%$$(HOST_DIR)%g" \
> >             package/meson/cross-compilation.conf.in \
> >             > $$($$(PKG)_SRCDIR)/build/cross-compilation.conf
> > +       _PYTHON_SYSCONFIGDATA_NAME="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
> > +       PYTHONPATH="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PYTHON3_PATH),$$(PYTHON_PATH))" \
> > +       PYTHONNOUSERSITE=1 \
> >         PATH=$$(BR_PATH) $$($$(PKG)_CONF_ENV) $$(MESON) \
> >                 --prefix=/usr \
> >                 --libdir=lib \
> > --
> > 2.21.0
> >
> Not only does this work, it also fixes cross-compiled python libraries names
> built by meson (such as the new versions of python-gobject)
>
> Tested-by: Adam Duskett <Aduskett@gmail.com>
Adam Duskett Sept. 27, 2019, 5:38 p.m. UTC | #3 
Sorry;

I will need to submit a new gobject-introspection patch set; which
requires this patch.

Adam

On Thu, Sep 26, 2019 at 8:26 PM Adam Duskett <aduskett@gmail.com> wrote:
>
> All;
>
> Any update on this patch? Now that libglib 2.62.0 is in Buildroot, I
> will need to submit a new patch set, which requires this patch.
>
> Thanks!
>
> Adam
>
> On Mon, Sep 16, 2019 at 11:41 AM Adam Duskett <aduskett@gmail.com> wrote:
> >
> > All;
> >
> >
> > On Sun, Aug 4, 2019 at 3:05 PM Arnout Vandecappelle (Essensium/Mind)
> > <arnout@mind.be> wrote:
> > >
> > > meson has support for Python dependencies, to e.g. build Python modules.
> > > It parses Python's sysconfig module to find the appropriate
> > > configuration variables. However, the sysconfig module is the one for
> > > the host, _not_ for the target. Therefore, wrong values may be set. Many
> > > of them get overridden anyway by the cross-compilation.conf. One
> > > particular one that is problematic is that it uses the python-XXX.pc
> > > pkg-config file for the host instead of the target, because it
> > > explicitly sets the PKG_CONFIG_LIBDIR found in sysconfig.
> > >
> > > Use the same trick as for python package: pass PYTHONPATH,
> > > PYTHONNOUSERSITE and _PYTHON_SYSCONFIGDATA_NAME in the environment to
> > > point to staging instead of host. This ensures the sysconfig from the
> > > target is used instead of the one from the host.
> > >
> > > An alternative would be to patch meson to not use the PKG_CONFIG_LIBDIR
> > > from sysconfig. However, it is likely that other problems will pop up at
> > > some point because of using the wrong sysconfig.
> > >
> > > Note that this approach will still break things when meson needs to
> > > build something for the host during a target build.
> > >
> > > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> > > Cc: Adam Duskett <Aduskett@gmail.com>
> > > Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> > > Cc: Peter Seiderer <ps.report@gmx.net>
> > > Cc: James Hilliard <james.hilliard1@gmail.com>
> > > Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> > > ---
> > > The setting of _PYTHON_SYSCONFIGDATA_NAME is different from the one in
> > > pkg-python.mk. I don't understand how that definition could work: for
> > > Python2, it is empty, so sysconfig tries to import a module with name
> > > "". So apparently the call paths used by meson and by
> > > setuptools/distutils are slightly different.
> > > ---
> > >  package/pkg-meson.mk | 3 +++
> > >  1 file changed, 3 insertions(+)
> > >
> > > diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> > > index 0b811d1cc0..3f875c08ce 100644
> > > --- a/package/pkg-meson.mk
> > > +++ b/package/pkg-meson.mk
> > > @@ -81,6 +81,9 @@ define $(2)_CONFIGURE_CMDS
> > >             -e "s%@HOST_DIR@%$$(HOST_DIR)%g" \
> > >             package/meson/cross-compilation.conf.in \
> > >             > $$($$(PKG)_SRCDIR)/build/cross-compilation.conf
> > > +       _PYTHON_SYSCONFIGDATA_NAME="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
> > > +       PYTHONPATH="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PYTHON3_PATH),$$(PYTHON_PATH))" \
> > > +       PYTHONNOUSERSITE=1 \
> > >         PATH=$$(BR_PATH) $$($$(PKG)_CONF_ENV) $$(MESON) \
> > >                 --prefix=/usr \
> > >                 --libdir=lib \
> > > --
> > > 2.21.0
> > >
> > Not only does this work, it also fixes cross-compiled python libraries names
> > built by meson (such as the new versions of python-gobject)
> >
> > Tested-by: Adam Duskett <Aduskett@gmail.com>
Adam Duskett Oct. 12, 2019, 11:11 p.m. UTC | #4 
All;

I inadvertently found an issue with this patch.
If the user has not selected python or python3,
_PYTHON_SYSCONFIGDATA_NAME will be passed as an empty string, which
causes libglib2 to fail during the configuration step.
Apparently, all of my previous testings were done with python3
enabled, and I didn't notice!

Adam

On Fri, Sep 27, 2019 at 10:38 AM Adam Duskett <aduskett@gmail.com> wrote:
>
> Sorry;
>
> I will need to submit a new gobject-introspection patch set; which
> requires this patch.
>
> Adam
>
> On Thu, Sep 26, 2019 at 8:26 PM Adam Duskett <aduskett@gmail.com> wrote:
> >
> > All;
> >
> > Any update on this patch? Now that libglib 2.62.0 is in Buildroot, I
> > will need to submit a new patch set, which requires this patch.
> >
> > Thanks!
> >
> > Adam
> >
> > On Mon, Sep 16, 2019 at 11:41 AM Adam Duskett <aduskett@gmail.com> wrote:
> > >
> > > All;
> > >
> > >
> > > On Sun, Aug 4, 2019 at 3:05 PM Arnout Vandecappelle (Essensium/Mind)
> > > <arnout@mind.be> wrote:
> > > >
> > > > meson has support for Python dependencies, to e.g. build Python modules.
> > > > It parses Python's sysconfig module to find the appropriate
> > > > configuration variables. However, the sysconfig module is the one for
> > > > the host, _not_ for the target. Therefore, wrong values may be set. Many
> > > > of them get overridden anyway by the cross-compilation.conf. One
> > > > particular one that is problematic is that it uses the python-XXX.pc
> > > > pkg-config file for the host instead of the target, because it
> > > > explicitly sets the PKG_CONFIG_LIBDIR found in sysconfig.
> > > >
> > > > Use the same trick as for python package: pass PYTHONPATH,
> > > > PYTHONNOUSERSITE and _PYTHON_SYSCONFIGDATA_NAME in the environment to
> > > > point to staging instead of host. This ensures the sysconfig from the
> > > > target is used instead of the one from the host.
> > > >
> > > > An alternative would be to patch meson to not use the PKG_CONFIG_LIBDIR
> > > > from sysconfig. However, it is likely that other problems will pop up at
> > > > some point because of using the wrong sysconfig.
> > > >
> > > > Note that this approach will still break things when meson needs to
> > > > build something for the host during a target build.
> > > >
> > > > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> > > > Cc: Adam Duskett <Aduskett@gmail.com>
> > > > Cc: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> > > > Cc: Peter Seiderer <ps.report@gmx.net>
> > > > Cc: James Hilliard <james.hilliard1@gmail.com>
> > > > Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> > > > ---
> > > > The setting of _PYTHON_SYSCONFIGDATA_NAME is different from the one in
> > > > pkg-python.mk. I don't understand how that definition could work: for
> > > > Python2, it is empty, so sysconfig tries to import a module with name
> > > > "". So apparently the call paths used by meson and by
> > > > setuptools/distutils are slightly different.
> > > > ---
> > > >  package/pkg-meson.mk | 3 +++
> > > >  1 file changed, 3 insertions(+)
> > > >
> > > > diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> > > > index 0b811d1cc0..3f875c08ce 100644
> > > > --- a/package/pkg-meson.mk
> > > > +++ b/package/pkg-meson.mk
> > > > @@ -81,6 +81,9 @@ define $(2)_CONFIGURE_CMDS
> > > >             -e "s%@HOST_DIR@%$$(HOST_DIR)%g" \
> > > >             package/meson/cross-compilation.conf.in \
> > > >             > $$($$(PKG)_SRCDIR)/build/cross-compilation.conf
> > > > +       _PYTHON_SYSCONFIGDATA_NAME="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
> > > > +       PYTHONPATH="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PYTHON3_PATH),$$(PYTHON_PATH))" \
> > > > +       PYTHONNOUSERSITE=1 \
> > > >         PATH=$$(BR_PATH) $$($$(PKG)_CONF_ENV) $$(MESON) \
> > > >                 --prefix=/usr \
> > > >                 --libdir=lib \
> > > > --
> > > > 2.21.0
> > > >
> > > Not only does this work, it also fixes cross-compiled python libraries names
> > > built by meson (such as the new versions of python-gobject)
> > >
> > > Tested-by: Adam Duskett <Aduskett@gmail.com>
Arnout Vandecappelle Oct. 13, 2019, 4:43 p.m. UTC | #5 
On 13/10/2019 01:11, Adam Duskett wrote:
> All;
> 
> I inadvertently found an issue with this patch.
> If the user has not selected python or python3,
> _PYTHON_SYSCONFIGDATA_NAME will be passed as an empty string,

 That's weird...

+	_PYTHON_SYSCONFIGDATA_NAME="$$(if
$$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \

 If python3 is not selected, it should be set to _sysonfigdata... I don't
remember how extensively I tested that situation, but I definitely did take it
into account!


 Regards,
 Arnout

> which
> causes libglib2 to fail during the configuration step.
> Apparently, all of my previous testings were done with python3
> enabled, and I didn't notice!
[snip]
Adam Duskett Oct. 15, 2019, 5:52 p.m. UTC | #6 
If you look at the Yocto project, it seems like they intentionally
unset _PYTHON_SYSCONFIGDATA_NAME when compiling libglib2.


On Sun, Oct 13, 2019 at 9:43 AM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 13/10/2019 01:11, Adam Duskett wrote:
> > All;
> >
> > I inadvertently found an issue with this patch.
> > If the user has not selected python or python3,
> > _PYTHON_SYSCONFIGDATA_NAME will be passed as an empty string,
>
>  That's weird...
>
> +       _PYTHON_SYSCONFIGDATA_NAME="$$(if
> $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
>
>  If python3 is not selected, it should be set to _sysonfigdata... I don't
> remember how extensively I tested that situation, but I definitely did take it
> into account!
>
>
>  Regards,
>  Arnout
>
> > which
> > causes libglib2 to fail during the configuration step.
> > Apparently, all of my previous testings were done with python3
> > enabled, and I didn't notice!
> [snip]
diff mbox series

Patch

diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
index 0b811d1cc0..3f875c08ce 100644
--- a/package/pkg-meson.mk
+++ b/package/pkg-meson.mk
@@ -81,6 +81,9 @@  define $(2)_CONFIGURE_CMDS
 	    -e "s%@HOST_DIR@%$$(HOST_DIR)%g" \
 	    package/meson/cross-compilation.conf.in \
 	    > $$($$(PKG)_SRCDIR)/build/cross-compilation.conf
+	_PYTHON_SYSCONFIGDATA_NAME="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PKG_PYTHON_SYSCONFIGDATA_NAME),_sysconfigdata)" \
+	PYTHONPATH="$$(if $$(BR2_PACKAGE_PYTHON3),$$(PYTHON3_PATH),$$(PYTHON_PATH))" \
+	PYTHONNOUSERSITE=1 \
 	PATH=$$(BR_PATH) $$($$(PKG)_CONF_ENV) $$(MESON) \
 		--prefix=/usr \
 		--libdir=lib \