diff mbox series

[v2,2/2] package/gnuradio: fix gnuradio python libraries for cross-compile

Message ID 20230626070536.311268-2-gwenj@trabucayre.com
State Changes Requested
Headers show
Series [v2,1/2] package/gnuradio: fix build with python-pybind > 2.10.0 | expand

Commit Message

Gwenhael Goavec-Merou June 26, 2023, 7:05 a.m. UTC 
From: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>

By default, module libraries have a suffix based on cpython version + host
architecture: this is fine for a native compile when these libraries are used on
the same computer (or similar computers). But when target architecture is not
the same python is unable to find libraries due to the wrong suffix and produces
unclear errors messages:

# python3
Python 3.11.3 (main, Jun 19 2023, 14:15:44) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gnuradio import blocks  
Traceback (most recent call last):
  File "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py", line 18, in <module>
ModuleNotFoundError: No module named 'gnuradio.blocks.blocks_python'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py", line 22, in <module>
ModuleNotFoundError: No module named 'gnuradio.blocks.blocks_python'
>>>

By adding -DPYTHON_MODULE_EXTENSION=".so" at configure time, pybind11 will use
this option instead of using host-python to forge suffix.

Signed-off-by: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
---
Changes v1 -> v2:
- replaces patch by PYTHON_MODULE_EXTENSION option (result is the same
  but less "noisy")
---
 package/gnuradio/gnuradio.mk | 2 ++
 1 file changed, 2 insertions(+)

Comments

Arnout Vandecappelle July 4, 2023, 8:51 p.m. UTC | #1 
On 26/06/2023 09:05, Gwenhael Goavec-Merou wrote:
> From: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
> 
> By default, module libraries have a suffix based on cpython version + host
> architecture: this is fine for a native compile when these libraries are used on

  I think that this is solved in the python infrastructure by setting

_PYTHON_HOST_PLATFORM="$(PKG_PYTHON_HOST_PLATFORM)"

Or it may be some other environment variable that is set. For sure though, the 
same problem would exist in normal python packages, and it is resolved there. So 
ideally we should use the same solution for gnuradio.

  Regards,
  Arnout

> the same computer (or similar computers). But when target architecture is not
> the same python is unable to find libraries due to the wrong suffix and produces
> unclear errors messages:
> 
> # python3
> Python 3.11.3 (main, Jun 19 2023, 14:15:44) [GCC 11.4.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from gnuradio import blocks
> Traceback (most recent call last):
>    File "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py", line 18, in <module>
> ModuleNotFoundError: No module named 'gnuradio.blocks.blocks_python'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py", line 22, in <module>
> ModuleNotFoundError: No module named 'gnuradio.blocks.blocks_python'
>>>>
> 
> By adding -DPYTHON_MODULE_EXTENSION=".so" at configure time, pybind11 will use
> this option instead of using host-python to forge suffix.
> 
> Signed-off-by: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
> ---
> Changes v1 -> v2:
> - replaces patch by PYTHON_MODULE_EXTENSION option (result is the same
>    but less "noisy")
> ---
>   package/gnuradio/gnuradio.mk | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/package/gnuradio/gnuradio.mk b/package/gnuradio/gnuradio.mk
> index 1453b78493..d7b3b86c31 100644
> --- a/package/gnuradio/gnuradio.mk
> +++ b/package/gnuradio/gnuradio.mk
> @@ -119,6 +119,8 @@ GNURADIO_CONF_OPTS += -DENABLE_PYTHON=ON
>   # mandatory to avoid pybind11 to overwrite variables provided
>   # by gnuradio and buildroot
>   GNURADIO_CONF_OPTS += -DPYBIND11_PYTHONLIBS_OVERWRITE=OFF
> +# force libraries suffix (avoid to have libxxx.PYTHONVER-HOST_ARCH.so)
> +GNURADIO_CONF_OPTS += -DPYTHON_MODULE_EXTENSION=".so"
>   # mandatory to install python modules in site-packages and to use
>   # correct path for python libraries
>   GNURADIO_CONF_OPTS += -DGR_PYTHON_RELATIVE=ON \
Gwenhael Goavec-Merou July 7, 2023, 8:58 a.m. UTC | #2 
Arnout, all

On Tue, 4 Jul 2023 22:51:57 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:

> On 26/06/2023 09:05, Gwenhael Goavec-Merou wrote:
> > From: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
> > 
> > By default, module libraries have a suffix based on cpython version + host
> > architecture: this is fine for a native compile when these libraries are
> > used on  
> 
>   I think that this is solved in the python infrastructure by setting
> 
> _PYTHON_HOST_PLATFORM="$(PKG_PYTHON_HOST_PLATFORM)"
> 
> Or it may be some other environment variable that is set. For sure though,
> the same problem would exist in normal python packages, and it is resolved
> there. So ideally we should use the same solution for gnuradio.
> 
In fact I see I have missed to explain in my message this issue is related to
pybind.
If PYTHON_MODULE_EXTENSION is unset, .cmake uses a python script containing
sysconfig.get_config_var('EXT_SUFFIX')
This one return .cpython-311-x86_64-linux-gnu.so, because host-python is used.
>   Regards,
>   Arnout
> 
Regards,
Gwenhael

> > the same computer (or similar computers). But when target architecture is
> > not the same python is unable to find libraries due to the wrong suffix and
> > produces unclear errors messages:
> > 
> > # python3
> > Python 3.11.3 (main, Jun 19 2023, 14:15:44) [GCC 11.4.0] on linux
> > Type "help", "copyright", "credits" or "license" for more information.  
> >>>> from gnuradio import blocks  
> > Traceback (most recent call last):
> >    File
> > "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py",
> > line 18, in <module> ModuleNotFoundError: No module named
> > 'gnuradio.blocks.blocks_python'
> > 
> > During handling of the above exception, another exception occurred:
> > 
> > Traceback (most recent call last):
> >    File "<stdin>", line 1, in <module>
> >    File
> > "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py",
> > line 22, in <module> ModuleNotFoundError: No module named
> > 'gnuradio.blocks.blocks_python'  
> >>>>  
> > 
> > By adding -DPYTHON_MODULE_EXTENSION=".so" at configure time, pybind11 will
> > use this option instead of using host-python to forge suffix.
> > 
> > Signed-off-by: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
> > ---
> > Changes v1 -> v2:
> > - replaces patch by PYTHON_MODULE_EXTENSION option (result is the same
> >    but less "noisy")
> > ---
> >   package/gnuradio/gnuradio.mk | 2 ++
> >   1 file changed, 2 insertions(+)
> > 
> > diff --git a/package/gnuradio/gnuradio.mk b/package/gnuradio/gnuradio.mk
> > index 1453b78493..d7b3b86c31 100644
> > --- a/package/gnuradio/gnuradio.mk
> > +++ b/package/gnuradio/gnuradio.mk
> > @@ -119,6 +119,8 @@ GNURADIO_CONF_OPTS += -DENABLE_PYTHON=ON
> >   # mandatory to avoid pybind11 to overwrite variables provided
> >   # by gnuradio and buildroot
> >   GNURADIO_CONF_OPTS += -DPYBIND11_PYTHONLIBS_OVERWRITE=OFF
> > +# force libraries suffix (avoid to have libxxx.PYTHONVER-HOST_ARCH.so)
> > +GNURADIO_CONF_OPTS += -DPYTHON_MODULE_EXTENSION=".so"
> >   # mandatory to install python modules in site-packages and to use
> >   # correct path for python libraries
> >   GNURADIO_CONF_OPTS += -DGR_PYTHON_RELATIVE=ON \
Arnout Vandecappelle July 7, 2023, 9:44 a.m. UTC | #3 
On 07/07/2023 10:58, Gwenhael Goavec-Merou wrote:
> Arnout, all
> 
> On Tue, 4 Jul 2023 22:51:57 +0200
> Arnout Vandecappelle <arnout@mind.be> wrote:
> 
>> On 26/06/2023 09:05, Gwenhael Goavec-Merou wrote:
>>> From: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
>>>
>>> By default, module libraries have a suffix based on cpython version + host
>>> architecture: this is fine for a native compile when these libraries are
>>> used on
>>
>>    I think that this is solved in the python infrastructure by setting
>>
>> _PYTHON_HOST_PLATFORM="$(PKG_PYTHON_HOST_PLATFORM)"
>>
>> Or it may be some other environment variable that is set. For sure though,
>> the same problem would exist in normal python packages, and it is resolved
>> there. So ideally we should use the same solution for gnuradio.
>>
> In fact I see I have missed to explain in my message this issue is related to
> pybind.
> If PYTHON_MODULE_EXTENSION is unset, .cmake uses a python script containing
> sysconfig.get_config_var('EXT_SUFFIX')
> This one return .cpython-311-x86_64-linux-gnu.so, because host-python is used.

  Ah, in that case the environment variable you need is 
_PYTHON_SYSCONFIGDATA_NAME="$(PKG_PYTHON_SYSCONFIGDATA_NAME)"
This will make sure the sysconfig that gets used is the one for the target 
instead of for the host.

  It's quite likely that you need all of PKG_PYTHON_ENV to be present in the 
environment.

  Regards,
  Arnout

>>    Regards,
>>    Arnout
>>
> Regards,
> Gwenhael
> 
>>> the same computer (or similar computers). But when target architecture is
>>> not the same python is unable to find libraries due to the wrong suffix and
>>> produces unclear errors messages:
>>>
>>> # python3
>>> Python 3.11.3 (main, Jun 19 2023, 14:15:44) [GCC 11.4.0] on linux
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> from gnuradio import blocks
>>> Traceback (most recent call last):
>>>     File
>>> "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py",
>>> line 18, in <module> ModuleNotFoundError: No module named
>>> 'gnuradio.blocks.blocks_python'
>>>
>>> During handling of the above exception, another exception occurred:
>>>
>>> Traceback (most recent call last):
>>>     File "<stdin>", line 1, in <module>
>>>     File
>>> "/home/xxx/buildroot/output/build/gnuradio-3.10.4.0/gr-blocks/python/blocks/__init__.py",
>>> line 22, in <module> ModuleNotFoundError: No module named
>>> 'gnuradio.blocks.blocks_python'
>>>>>>   
>>>
>>> By adding -DPYTHON_MODULE_EXTENSION=".so" at configure time, pybind11 will
>>> use this option instead of using host-python to forge suffix.
>>>
>>> Signed-off-by: Gwenhael Goavec-Merou <gwenhael.goavec-merou@trabucayre.com>
>>> ---
>>> Changes v1 -> v2:
>>> - replaces patch by PYTHON_MODULE_EXTENSION option (result is the same
>>>     but less "noisy")
>>> ---
>>>    package/gnuradio/gnuradio.mk | 2 ++
>>>    1 file changed, 2 insertions(+)
>>>
>>> diff --git a/package/gnuradio/gnuradio.mk b/package/gnuradio/gnuradio.mk
>>> index 1453b78493..d7b3b86c31 100644
>>> --- a/package/gnuradio/gnuradio.mk
>>> +++ b/package/gnuradio/gnuradio.mk
>>> @@ -119,6 +119,8 @@ GNURADIO_CONF_OPTS += -DENABLE_PYTHON=ON
>>>    # mandatory to avoid pybind11 to overwrite variables provided
>>>    # by gnuradio and buildroot
>>>    GNURADIO_CONF_OPTS += -DPYBIND11_PYTHONLIBS_OVERWRITE=OFF
>>> +# force libraries suffix (avoid to have libxxx.PYTHONVER-HOST_ARCH.so)
>>> +GNURADIO_CONF_OPTS += -DPYTHON_MODULE_EXTENSION=".so"
>>>    # mandatory to install python modules in site-packages and to use
>>>    # correct path for python libraries
>>>    GNURADIO_CONF_OPTS += -DGR_PYTHON_RELATIVE=ON \
Thomas Petazzoni July 10, 2023, 6:25 p.m. UTC | #4 
Hello Gwenhael,

On Fri, 7 Jul 2023 11:44:21 +0200
Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:

> > In fact I see I have missed to explain in my message this issue is related to
> > pybind.
> > If PYTHON_MODULE_EXTENSION is unset, .cmake uses a python script containing
> > sysconfig.get_config_var('EXT_SUFFIX')
> > This one return .cpython-311-x86_64-linux-gnu.so, because host-python is used.  
> 
>   Ah, in that case the environment variable you need is 
> _PYTHON_SYSCONFIGDATA_NAME="$(PKG_PYTHON_SYSCONFIGDATA_NAME)"
> This will make sure the sysconfig that gets used is the one for the target 
> instead of for the host.
> 
>   It's quite likely that you need all of PKG_PYTHON_ENV to be present in the 
> environment.

Have you had the chance to test this suggestion from Arnout?

Thomas
Gwenhael Goavec-Merou July 17, 2023, 3:17 p.m. UTC | #5 
Thomas, Arnout, all
Sorry for delay, was away for a correct/stable connection.

On Mon, 10 Jul 2023 20:25:03 +0200
Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Hello Gwenhael,
> 
> On Fri, 7 Jul 2023 11:44:21 +0200
> Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> 
> > > In fact I see I have missed to explain in my message this issue is
> > > related to pybind.
> > > If PYTHON_MODULE_EXTENSION is unset, .cmake uses a python script
> > > containing sysconfig.get_config_var('EXT_SUFFIX')
> > > This one return .cpython-311-x86_64-linux-gnu.so, because host-python is
> > > used.    
> > 
> >   Ah, in that case the environment variable you need is 
> > _PYTHON_SYSCONFIGDATA_NAME="$(PKG_PYTHON_SYSCONFIGDATA_NAME)"
> > This will make sure the sysconfig that gets used is the one for the target 
> > instead of for the host.
> > 
> >   It's quite likely that you need all of PKG_PYTHON_ENV to be present in
> > the environment.  
> 
> Have you had the chance to test this suggestion from Arnout?
> 
Adding:
GNURADIO_CONF_ENV += _PYTHON_SYSCONFIGDATA_NAME="$(PKG_PYTHON_SYSCONFIGDATA_NAME)" \
    PYTHONPATH=$(PYTHON3_PATH)
Do the job.
Thanks. I have to redo my patch
> Thomas

Gwen
diff mbox series

Patch

diff --git a/package/gnuradio/gnuradio.mk b/package/gnuradio/gnuradio.mk
index 1453b78493..d7b3b86c31 100644
--- a/package/gnuradio/gnuradio.mk
+++ b/package/gnuradio/gnuradio.mk
@@ -119,6 +119,8 @@  GNURADIO_CONF_OPTS += -DENABLE_PYTHON=ON
 # mandatory to avoid pybind11 to overwrite variables provided
 # by gnuradio and buildroot
 GNURADIO_CONF_OPTS += -DPYBIND11_PYTHONLIBS_OVERWRITE=OFF
+# force libraries suffix (avoid to have libxxx.PYTHONVER-HOST_ARCH.so)
+GNURADIO_CONF_OPTS += -DPYTHON_MODULE_EXTENSION=".so"
 # mandatory to install python modules in site-packages and to use
 # correct path for python libraries
 GNURADIO_CONF_OPTS += -DGR_PYTHON_RELATIVE=ON \