| Message ID | 1425918264-1912-1-git-send-email-s.martin49@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
On Mon, Mar 9, 2015 at 5:24 PM, Samuel Martin <s.martin49@gmail.com> wrote: > This patch generates all *.pyc files from *.py located in > <target>/usr/python*/site-packages, before stripping the rootfs. > > This prevents modules from packages that do not compile the *.py > files from disappearing. > > Reported-by: Yegor Yefremov <yegorslists@googlemail.com> > Cc: Yegor Yefremov <yegorslists@googlemail.com> > Signed-off-by: Samuel Martin <s.martin49@gmail.com> Tested-by: Yegor Yefremov <yegorslists@googlemail.com> > --- > Makefile | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/Makefile b/Makefile > index af043a3..ec0b923 100644 > --- a/Makefile > +++ b/Makefile > @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize > > target-finalize: $(TARGETS) > @$(call MESSAGE,"Finalizing target directory") > +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),) > + PYTHONPATH="$(PYTHON_PATH)" \ > + $(HOST_DIR)/usr/bin/python -c "import compileall; \ > + compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')" > +endif > $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) > rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ > $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ > -- > 2.1.0 >
Dear Samuel Martin, On Mon, 9 Mar 2015 17:24:23 +0100, Samuel Martin wrote: > This patch generates all *.pyc files from *.py located in > <target>/usr/python*/site-packages, before stripping the rootfs. > > This prevents modules from packages that do not compile the *.py > files from disappearing. > > Reported-by: Yegor Yefremov <yegorslists@googlemail.com> > Cc: Yegor Yefremov <yegorslists@googlemail.com> > Signed-off-by: Samuel Martin <s.martin49@gmail.com> > --- > Makefile | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/Makefile b/Makefile > index af043a3..ec0b923 100644 > --- a/Makefile > +++ b/Makefile > @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize > > target-finalize: $(TARGETS) > @$(call MESSAGE,"Finalizing target directory") > +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),) > + PYTHONPATH="$(PYTHON_PATH)" \ > + $(HOST_DIR)/usr/bin/python -c "import compileall; \ > + compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')" > +endif Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files? Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the issue that the hook is the same between python and python3, so we wouldn't know where to put it. Thomas
Hi Thomas, all, On Mon, Mar 9, 2015 at 6:54 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Dear Samuel Martin, > > On Mon, 9 Mar 2015 17:24:23 +0100, Samuel Martin wrote: >> This patch generates all *.pyc files from *.py located in >> <target>/usr/python*/site-packages, before stripping the rootfs. >> >> This prevents modules from packages that do not compile the *.py >> files from disappearing. >> >> Reported-by: Yegor Yefremov <yegorslists@googlemail.com> >> Cc: Yegor Yefremov <yegorslists@googlemail.com> >> Signed-off-by: Samuel Martin <s.martin49@gmail.com> >> --- >> Makefile | 5 +++++ >> 1 file changed, 5 insertions(+) >> >> diff --git a/Makefile b/Makefile >> index af043a3..ec0b923 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize >> >> target-finalize: $(TARGETS) >> @$(call MESSAGE,"Finalizing target directory") >> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),) >> + PYTHONPATH="$(PYTHON_PATH)" \ >> + $(HOST_DIR)/usr/bin/python -c "import compileall; \ >> + compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')" >> +endif > > Why not, but then don't we want to do something in the Python package > infra, and in the Python 2/3 packages themselves to avoid spending time > biulding the .pyc files? Indeed, it'd be better to avoid wasting time building *.pyc during per-package build, though it does not seem really easy to do at first glance. Well, after hacking around, it is not that hard to do it! :-) I cleanup my hacks and will submit them. > > Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the > issue that the hook is the same between python and python3, so we > wouldn't know where to put it. The hook is simple enough to be moved in pkg-python.mk as a helper, then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS. However, if we decide to not go using a global TARGET_FINALIZE_HOOKS, but per-package target-install hook, the helper can be easily added to the packages that need it; the biggest remaining task being identifying and updating all packages not using the pkg-python infra but providing a python module. Regards,
On 09/03/15 18:54, Thomas Petazzoni wrote: > Dear Samuel Martin, > > On Mon, 9 Mar 2015 17:24:23 +0100, Samuel Martin wrote: >> This patch generates all *.pyc files from *.py located in >> <target>/usr/python*/site-packages, before stripping the rootfs. >> >> This prevents modules from packages that do not compile the *.py >> files from disappearing. >> >> Reported-by: Yegor Yefremov <yegorslists@googlemail.com> >> Cc: Yegor Yefremov <yegorslists@googlemail.com> >> Signed-off-by: Samuel Martin <s.martin49@gmail.com> >> --- >> Makefile | 5 +++++ >> 1 file changed, 5 insertions(+) >> >> diff --git a/Makefile b/Makefile >> index af043a3..ec0b923 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize >> >> target-finalize: $(TARGETS) >> @$(call MESSAGE,"Finalizing target directory") >> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),) It should also be conditional on !BR2_PACKAGE_PYTHON_PY_ONLY >> + PYTHONPATH="$(PYTHON_PATH)" \ >> + $(HOST_DIR)/usr/bin/python -c "import compileall; \ >> + compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')" I believe the canonical way to call compileall from the command line is using the -m option: $(HOST_DIR)/usr/bin/python -m compileall \ $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages >> +endif > > Why not, but then don't we want to do something in the Python package > infra, and in the Python 2/3 packages themselves to avoid spending time > biulding the .pyc files? compileall will check timestamps to avoid recompiling unnecessarily. So you just have the tree walk to consider, which I think is a minor issue. > Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the > issue that the hook is the same between python and python3, so we > wouldn't know where to put it. In fact, we already have a PYTHON_FINALIZE_TARGET (and similar for PYTHON3) that does the removal of the .py files, so the compileall should be added there. And that immediately handles the BR2_PACKAGE_PYTHON_PY_ONLY condition as well. Regards, Arnout
Dear Samuel Martin, On Mon, 9 Mar 2015 22:30:02 +0100, Samuel Martin wrote: > Indeed, it'd be better to avoid wasting time building *.pyc during > per-package build, though it does not seem really easy to do at first > glance. > Well, after hacking around, it is not that hard to do it! :-) > I cleanup my hacks and will submit them. Ok, cool. > The hook is simple enough to be moved in pkg-python.mk as a helper, > then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS. Indeed. > However, if we decide to not go using a global TARGET_FINALIZE_HOOKS, > but per-package target-install hook, the helper can be easily added to > the packages that need it; the biggest remaining task being > identifying and updating all packages not using the pkg-python infra > but providing a python module. Well, as it is currently implemented, the hook byte-compiles *all* .py files. So if you call this hook multiple times, you're going to byte-compile all .py files multiple times. Doesn't seem like the most efficient thing to do. Thomas
Dear Arnout Vandecappelle, On Mon, 09 Mar 2015 22:42:04 +0100, Arnout Vandecappelle wrote: > > Why not, but then don't we want to do something in the Python package > > infra, and in the Python 2/3 packages themselves to avoid spending time > > biulding the .pyc files? > > compileall will check timestamps to avoid recompiling unnecessarily. So you > just have the tree walk to consider, which I think is a minor issue. Ah, good to know. Then indeed it's useless to do any special trick: let's just call compileall as a target finalize hook. Thomas
On Mon, Mar 9, 2015 at 10:46 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Dear Arnout Vandecappelle, > > On Mon, 09 Mar 2015 22:42:04 +0100, Arnout Vandecappelle wrote: > >> > Why not, but then don't we want to do something in the Python package >> > infra, and in the Python 2/3 packages themselves to avoid spending time >> > biulding the .pyc files? >> >> compileall will check timestamps to avoid recompiling unnecessarily. So you >> just have the tree walk to consider, which I think is a minor issue. > > Ah, good to know. Then indeed it's useless to do any special trick: > let's just call compileall as a target finalize hook. What's the current state of this issue? It would be great, if this could be fixed for the upcoming release. Yegor
diff --git a/Makefile b/Makefile index af043a3..ec0b923 100644 --- a/Makefile +++ b/Makefile @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize target-finalize: $(TARGETS) @$(call MESSAGE,"Finalizing target directory") +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),) + PYTHONPATH="$(PYTHON_PATH)" \ + $(HOST_DIR)/usr/bin/python -c "import compileall; \ + compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')" +endif $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
This patch generates all *.pyc files from *.py located in <target>/usr/python*/site-packages, before stripping the rootfs. This prevents modules from packages that do not compile the *.py files from disappearing. Reported-by: Yegor Yefremov <yegorslists@googlemail.com> Cc: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Samuel Martin <s.martin49@gmail.com> --- Makefile | 5 +++++ 1 file changed, 5 insertions(+)