| Message ID | 1392756013-27757-5-git-send-email-thomas.petazzoni@free-electrons.com |
|---|---|
| State | Accepted |
| Headers | show |
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes: > In Buildroot, we do not support installing both Python 2.x and Python > 3.x on the target. > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Committed to next, thanks.
Am Tue, 18 Feb 2014 21:39:57 +0100 schrieb Thomas Petazzoni: > In Buildroot, we do not support installing both Python 2.x and Python > 3.x on the target. Hi, sorry for warming up this old commit but I would like to ask whether it is possible at all to allow python2 and python3 together on the target? The topic was discussed on the python mailinglist (https:// mail.python.org/pipermail/python-list/2011-March/600724.html) and it did not mention any problems. I need python2 for Kodi, afaics[1] there are no current plans to support python3. For a new function[2] I would like to add to my buildroot-based system I need python3 on the target as well. Regards, Bernd [1] http://forum.kodi.tv/showthread.php?tid=216598 [2] https://github.com/itkach/slob/
Hello, On Wed, 05 Oct 2016 20:00:40 +0200, Bernd Kuhls wrote: > sorry for warming up this old commit but I would like to ask whether it > is possible at all to allow python2 and python3 together on the target? No, this is not possible, because we have no way in kconfig to say which Python modules should be installed for python2, and which Python modules should be installed for python3. So while having both interpreters and the Python standard library side-by-side would work, it doesn't work for all the external Python modules we have. > I need python2 for Kodi, afaics[1] there are no current plans to support > python3. I think that's the main problem. Staying on python2 only is really not a wise choice. Best regards, Thomas
Thomas, Bernd, All, On 2016-10-05 21:30 +0200, Thomas Petazzoni spake thusly: > On Wed, 05 Oct 2016 20:00:40 +0200, Bernd Kuhls wrote: > > sorry for warming up this old commit but I would like to ask whether it > > is possible at all to allow python2 and python3 together on the target? > > No, this is not possible, because we have no way in kconfig to say > which Python modules should be installed for python2, and which Python > modules should be installed for python3. > > So while having both interpreters and the Python standard library > side-by-side would work, it doesn't work for all the external Python > modules we have. For what is worth, I've also been thinking on how we could solve this in Buildroot. We've discussed this on IRC (with Romain or Samuel, I can't remember). Here's what we came up with... First, as Thomas said, we'd need to have the kcofnig-side sorted out. When looking only at external python modules for themselves, it is not very complex: we already have some modules depend on either python xor python3 for those modules that have an explicit dependency. Where it gets more complex is when a package needs a modules that works only with python and another module that works only with python3. I don't know if this situation is realistic, but we have to at least take it in consideration. Then, there is the problem of building modules for either or both of the pythons, when those modules need compiling (e.g. are not written entirely in python, but use C or whatever that needs compiling). When a module is only python xor python3, this is realtively easy; we just need to do a single build of that module. It does not change much from what we currently have for that module. But then, when the module is compatible with both python and python3, then we need to build it twice, which is currently not supported by our python-package infra, and is definitely not trivial to do. Finally, comes the installation phase. Again, modules that are python xor python3, this is mostly status-quo with what we currently have. Modules that are compatible with both python and python3 would need to be installed for both. Again, our python-package infra does not support that, and it is far from trivial to do. I've started looking into this, but I have absolutely no code to show. I think it *might* be possible to come up with something, but that may probably add quite a bit of complexity and uglyness in the python-package infra. I'm not sure it would be worth the effort... :-/ > > I need python2 for Kodi, afaics[1] there are no current plans to support > > python3. > > I think that's the main problem. Staying on python2 only is really not > a wise choice. Yes, this is a real pain... :-( Regards, Yann E. MORIN.
Hello, On Sat, 8 Oct 2016 14:24:34 +0200, Yann E. MORIN wrote: > First, as Thomas said, we'd need to have the kcofnig-side sorted out. > When looking only at external python modules for themselves, it is not > very complex: we already have some modules depend on either python xor > python3 for those modules that have an explicit dependency. > > Where it gets more complex is when a package needs a modules that works > only with python and another module that works only with python3. I > don't know if this situation is realistic, but we have to at least take > it in consideration. I don't understand how you solve the Kconfig side of the problem. How will you select which of the modules should be compiled for python2 only, for python3 only or for both ? Let's say you have three Python packages A, B and C. They are all compatible with both Python 2 and Python 3. And for the purpose of my system, I need A only with Python 2, I need B only for Python 3 and I need C for both Python 2 and Python 3. > I've started looking into this, but I have absolutely no code to show. I > think it *might* be possible to come up with something, but that may > probably add quite a bit of complexity and uglyness in the python-package > infra. I'm not sure it would be worth the effort... :-/ Please explain how to solve the Kconfig problem first. Until there is no solution for the Kconfig problem, there is really no point in even thinking about the build problems. Thomas
Thomas, All, On 2016-10-08 14:51 +0200, Thomas Petazzoni spake thusly: > On Sat, 8 Oct 2016 14:24:34 +0200, Yann E. MORIN wrote: > > First, as Thomas said, we'd need to have the kcofnig-side sorted out. > > When looking only at external python modules for themselves, it is not > > very complex: we already have some modules depend on either python xor > > python3 for those modules that have an explicit dependency. > > > > Where it gets more complex is when a package needs a modules that works > > only with python and another module that works only with python3. I > > don't know if this situation is realistic, but we have to at least take > > it in consideration. > > I don't understand how you solve the Kconfig side of the problem. How > will you select which of the modules should be compiled for python2 > only, for python3 only or for both ? > > Let's say you have three Python packages A, B and C. They are all > compatible with both Python 2 and Python 3. And for the purpose of my > system, I need A only with Python 2, I need B only for Python 3 and I > need C for both Python 2 and Python 3. Plain and simple: if a python module supports both python and python3, and both are enabled, that python module is built for both python and python3. If for your system you only need A for python and B for python3, then it is *your* responsibility to remove A for python3 and B for python in a post-build script. That's what we've been advertising in similar cases when people only wanted a subset of a package, and what you've been suggesting when we were arguing about the libudev stuff (i.e. build the full eudev and get rid of the udevd program in a post-build script). ;-) > > I've started looking into this, but I have absolutely no code to show. I > > think it *might* be possible to come up with something, but that may > > probably add quite a bit of complexity and uglyness in the python-package > > infra. I'm not sure it would be worth the effort... :-/ > > Please explain how to solve the Kconfig problem first. Until there is > no solution for the Kconfig problem, there is really no point in even > thinking about the build problems. I don't think there is a "kconfig problem" to start with. ;-) Regards, Yann E. MORIN.
Hello, On Sat, 8 Oct 2016 15:51:30 +0200, Yann E. MORIN wrote: > Plain and simple: if a python module supports both python and python3, > and both are enabled, that python module is built for both python and > python3. Well, that's horrible from a filesystem size point of view, and I'm not really sure this would solve Bernd's issue. I'm pretty sure Bernd would like to have a given set of modules built for Python 2, and another set of modules built for Python 3. > If for your system you only need A for python and B for python3, then it > is *your* responsibility to remove A for python3 and B for python in a > post-build script. That's what we've been advertising in similar cases > when people only wanted a subset of a package, and what you've been > suggesting when we were arguing about the libudev stuff (i.e. build the > full eudev and get rid of the udevd program in a post-build script). ;-) Well, yes, you could see it this way. But on the eudev thing, you still get it wrong: having libudev installed separately from udevd is not an option offered by upstream. You had to hack all around the place to make this possible. Building Python modules with just python2 or just python3 is perfectly fine from upstream's point of view. So please don't confuse things :) > > Please explain how to solve the Kconfig problem first. Until there is > > no solution for the Kconfig problem, there is really no point in even > > thinking about the build problems. > > I don't think there is a "kconfig problem" to start with. ;-) With your solution, indeed. However, I'm not sure it is worth the hassle supporting this mechanism. But it can certainly be done. Best regards, Thomas
Hi Thomas, hi Yann, Am Sat, 08 Oct 2016 16:02:28 +0200 schrieb Thomas Petazzoni: >> Plain and simple: if a python module supports both python and python3, >> and both are enabled, that python module is built for both python and >> python3. > > Well, that's horrible from a filesystem size point of view, and I'm not > really sure this would solve Bernd's issue. I'm pretty sure Bernd would > like to have a given set of modules built for Python 2, and another set > of modules built for Python 3. actually I solved the dependency problem by copying python2 and python3 to my target system ;) This is possible for me because I do not use a filesystem created by buildroot but use files from output/target/ stored as optional packages for the fli4l project, which then creates the filesystem image for the target machine by including these packages. To achieve this I am building with one .config which has python2 + kodi/ mwscrape and later another .config build is done with python3 + mwscrape2slob. To clarify: mwscrape[1] downloads articles from mediawiki-based sites, stores them on a couchdb server and has runtime dependencies to the following python modules: couchdb, futures, mwclient and pylru. Because mwclient is python2-only so is mwscrape. mwscrape2slob[2] downloads the stored mediawiki articles from the couchdb server and creates files in .slob format for use in the Android app aard2 [3], which is a offline-reader. It depends on the following python modules: couchdb, cssselect, cssutils, lxml and slob. The modules slob and mwscrape2slob are python3-only. Luckily no module with a python version restriction depends on a module with a different python version restriction :) So yes, there are some modules which I only need in python2, others only in python3, for this purpose I am using two different .config files and build python2 and python3 seperately. Filesystem space is not a problem for me, my target system is not an embedded device, therefore my solution may not be suitable for the general use in the buildroot project. I expect to send the mwscrape2slob patch series in the coming days after testing a self-build .slob file. Currently I am busy with downloading Wikipedia data ;) The CouchDB server built with buildroot runs rock-solid. Regards, Bernd [1] https://github.com/itkach/mwscrape [2] https://github.com/itkach/mwscrape2slob [3] https://github.com/itkach/aard2-android
Thomas, All, On 2016-10-08 16:02 +0200, Thomas Petazzoni spake thusly: > On Sat, 8 Oct 2016 15:51:30 +0200, Yann E. MORIN wrote: > > Plain and simple: if a python module supports both python and python3, > > and both are enabled, that python module is built for both python and > > python3. > > Well, that's horrible from a filesystem size point of view, and I'm not > really sure this would solve Bernd's issue. I'm pretty sure Bernd would > like to have a given set of modules built for Python 2, and another set > of modules built for Python 3. Well, short of duplicating (tripling? more?) the kconfig symbols, there is no easy solution. But again, I would argue that, for Buildroot, we would be just happy to at least have that situation: having both python2 and python3 on the target, even with duplicated and useless python modules in some cases. That would be an improvement against the current situation. If anything, that would make scenarii currently impossible, possible. *Then*, we could look at solving how to remove the useless duplicated modules, if at all possible in a sane way... > > If for your system you only need A for python and B for python3, then it > > is *your* responsibility to remove A for python3 and B for python in a > > post-build script. That's what we've been advertising in similar cases > > when people only wanted a subset of a package, and what you've been > > suggesting when we were arguing about the libudev stuff (i.e. build the > > full eudev and get rid of the udevd program in a post-build script). ;-) > > Well, yes, you could see it this way. > > But on the eudev thing, you still get it wrong: having libudev > installed separately from udevd is not an option offered by upstream. > You had to hack all around the place to make this possible. Hmm... Don't get me started on this topic (woops, I did start it!). There are things not supporrted by upstream for which we patch some stuff; and in some situations, I believe this is insane. The best (worst!) example being the patch to build openssl in parallel; that one is insane and dangerous: openssl is a tricky and critical piece of code, and I'm not sure patching it with a patch that was not blessed (and was even refused!) by upstream is sane... :-/ But OK, I'll first push upstream eudev to enable building only libudev. I already started that, but time being always in short supplies, this hasn't progressed too much for now... :-/ Topic closed. ;-) > With your solution, indeed. However, I'm not sure it is worth the > hassle supporting this mechanism. But it can certainly be done. On contrary, I do believe this is worth the hassle. But until I have some code, I'll close the topic... ;-) Regards, Yann E. MORIN.
On Sat, Oct 8, 2016 at 5:57 PM, Bernd Kuhls <bernd.kuhls@t-online.de> wrote: > Hi Thomas, hi Yann, > > Am Sat, 08 Oct 2016 16:02:28 +0200 schrieb Thomas Petazzoni: > >>> Plain and simple: if a python module supports both python and python3, >>> and both are enabled, that python module is built for both python and >>> python3. >> >> Well, that's horrible from a filesystem size point of view, and I'm not >> really sure this would solve Bernd's issue. I'm pretty sure Bernd would >> like to have a given set of modules built for Python 2, and another set >> of modules built for Python 3. > > actually I solved the dependency problem by copying python2 and python3 > to my target system ;) This is possible for me because I do not use a > filesystem created by buildroot but use files from output/target/ stored > as optional packages for the fli4l project, which then creates the > filesystem image for the target machine by including these packages. > > To achieve this I am building with one .config which has python2 + kodi/ > mwscrape and later another .config build is done with python3 + > mwscrape2slob. > > To clarify: > mwscrape[1] downloads articles from mediawiki-based sites, stores them on > a couchdb server and has runtime dependencies to the following python > modules: couchdb, futures, mwclient and pylru. Because mwclient is > python2-only so is mwscrape. isn't mwclient Python 3 compatible? See [1] [1] https://github.com/mwclient/mwclient/blob/master/setup.py#L48 Yegor > mwscrape2slob[2] downloads the stored mediawiki articles from the couchdb > server and creates files in .slob format for use in the Android app aard2 > [3], which is a offline-reader. It depends on the following python > modules: couchdb, cssselect, cssutils, lxml and slob. The modules slob > and mwscrape2slob are python3-only. > > Luckily no module with a python version restriction depends on a module > with a different python version restriction :) > > So yes, there are some modules which I only need in python2, others only > in python3, for this purpose I am using two different .config files and > build python2 and python3 seperately. Filesystem space is not a problem > for me, my target system is not an embedded device, therefore my solution > may not be suitable for the general use in the buildroot project. > > I expect to send the mwscrape2slob patch series in the coming days after > testing a self-build .slob file. Currently I am busy with downloading > Wikipedia data ;) The CouchDB server built with buildroot runs rock-solid. > > Regards, Bernd > > [1] https://github.com/itkach/mwscrape > [2] https://github.com/itkach/mwscrape2slob > [3] https://github.com/itkach/aard2-android > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Am Sat, 08 Oct 2016 20:23:09 +0200 schrieb Yegor Yefremov: > isn't mwclient Python 3 compatible? See [1] > > [1] https://github.com/mwclient/mwclient/blob/master/setup.py#L48 Hi Yegor, using this defconfig BR2_TOOLCHAIN_BUILDROOT_WCHAR=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_MWCLIENT=y this error occurs with plain make: ------------------------------------------------- >>> Finalizing target directory PYTHONPATH="/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/ sysconfigdata/:/home/buildroot/br3_couchdb/output/target/usr/lib/ python3.5/site-packages/" /home/buildroot/br3_couchdb/output/host/usr/bin/ python3.5 support/scripts/pycompile.py /home/buildroot/br3_couchdb/output/ target/usr/lib/python3.5 Traceback (most recent call last): File "/home/buildroot/br3_couchdb/output/host/usr/lib/python3.5/ py_compile.py", line 125, in compile _optimize=optimize) File "<frozen importlib._bootstrap_external>", line 700, in source_to_code File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/site- packages/mwclient/ex.py", line 15 exec _file in globals(), predata ^ SyntaxError: invalid syntax ------------------------------------------------- This error does not occur with python2. Regards, Bernd
On Sun, Oct 9, 2016 at 9:55 AM, Bernd Kuhls <bernd.kuhls@t-online.de> wrote: > Am Sat, 08 Oct 2016 20:23:09 +0200 schrieb Yegor Yefremov: > >> isn't mwclient Python 3 compatible? See [1] >> >> [1] https://github.com/mwclient/mwclient/blob/master/setup.py#L48 > > Hi Yegor, > > using this defconfig > > BR2_TOOLCHAIN_BUILDROOT_WCHAR=y > BR2_PACKAGE_PYTHON3=y > BR2_PACKAGE_PYTHON_MWCLIENT=y > > this error occurs with plain make: > ------------------------------------------------- >>>> Finalizing target directory > PYTHONPATH="/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/ > sysconfigdata/:/home/buildroot/br3_couchdb/output/target/usr/lib/ > python3.5/site-packages/" /home/buildroot/br3_couchdb/output/host/usr/bin/ > python3.5 support/scripts/pycompile.py /home/buildroot/br3_couchdb/output/ > target/usr/lib/python3.5 > Traceback (most recent call last): > File "/home/buildroot/br3_couchdb/output/host/usr/lib/python3.5/ > py_compile.py", line 125, in compile > _optimize=optimize) > File "<frozen importlib._bootstrap_external>", line 700, in > source_to_code > File "<frozen importlib._bootstrap>", line 222, in > _call_with_frames_removed > File "/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/site- > packages/mwclient/ex.py", line 15 > exec _file in globals(), predata > ^ > SyntaxError: invalid syntax > ------------------------------------------------- > This error does not occur with python2. OK. I'll try to take a look at this issue. Yegor
On Sun, Oct 9, 2016 at 11:44 AM, Yegor Yefremov <yegorslists@googlemail.com> wrote: > On Sun, Oct 9, 2016 at 9:55 AM, Bernd Kuhls <bernd.kuhls@t-online.de> wrote: >> Am Sat, 08 Oct 2016 20:23:09 +0200 schrieb Yegor Yefremov: >> >>> isn't mwclient Python 3 compatible? See [1] >>> >>> [1] https://github.com/mwclient/mwclient/blob/master/setup.py#L48 >> >> Hi Yegor, >> >> using this defconfig >> >> BR2_TOOLCHAIN_BUILDROOT_WCHAR=y >> BR2_PACKAGE_PYTHON3=y >> BR2_PACKAGE_PYTHON_MWCLIENT=y >> >> this error occurs with plain make: >> ------------------------------------------------- >>>>> Finalizing target directory >> PYTHONPATH="/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/ >> sysconfigdata/:/home/buildroot/br3_couchdb/output/target/usr/lib/ >> python3.5/site-packages/" /home/buildroot/br3_couchdb/output/host/usr/bin/ >> python3.5 support/scripts/pycompile.py /home/buildroot/br3_couchdb/output/ >> target/usr/lib/python3.5 >> Traceback (most recent call last): >> File "/home/buildroot/br3_couchdb/output/host/usr/lib/python3.5/ >> py_compile.py", line 125, in compile >> _optimize=optimize) >> File "<frozen importlib._bootstrap_external>", line 700, in >> source_to_code >> File "<frozen importlib._bootstrap>", line 222, in >> _call_with_frames_removed >> File "/home/buildroot/br3_couchdb/output/target/usr/lib/python3.5/site- >> packages/mwclient/ex.py", line 15 >> exec _file in globals(), predata >> ^ >> SyntaxError: invalid syntax >> ------------------------------------------------- >> This error does not occur with python2. > > OK. I'll try to take a look at this issue. Applying this upstream patch [1] solves the issue. [1] https://github.com/mwclient/mwclient/commit/5de1bb82465d39962e26175c62f644a3e423d030 Yegor
diff --git a/package/python3/Config.in b/package/python3/Config.in index 6580cfd..12928f6 100644 --- a/package/python3/Config.in +++ b/package/python3/Config.in @@ -1,5 +1,6 @@ config BR2_PACKAGE_PYTHON3 bool "python3" + depends on !BR2_PACKAGE_PYTHON depends on BR2_USE_WCHAR # uses fork() depends on BR2_USE_MMU
In Buildroot, we do not support installing both Python 2.x and Python 3.x on the target. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> --- package/python3/Config.in | 1 + 1 file changed, 1 insertion(+)