| Message ID | 20240204062645.3616072-1-abiliojr@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | package/micropython: add support for manifest.py in the configuration | expand |
Abilio, All, On 2024-02-03 22:26 -0800, Abilio Marques spake thusly: > Micropython can embed packages an modules as frozen bytecode. What code > gets built this way can be defined by means of a "manifest.py" file. > > This commit exposes the variable FROZEN_MANIFEST to Buildroot users > through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST. > > Please check Micropython's documentation for more information: > https://docs.micropython.org/en/latest/reference/manifest.html Minor nit: this blurb should have been in the help text of the new option, and I could have done that when applying, but I have a much more involved question, see below. > Signed-off-by: Abilio Marques <abiliojr@gmail.com> > --- > package/micropython/Config.in | 8 ++++++++ > package/micropython/micropython.mk | 5 +++++ > 2 files changed, 13 insertions(+) > > diff --git a/package/micropython/Config.in b/package/micropython/Config.in > index 26a00baab0..52717d815a 100644 > --- a/package/micropython/Config.in > +++ b/package/micropython/Config.in > @@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB > help > Core Python libraries ported to MicroPython. > > +config BR2_PACKAGE_MICROPYTHON_MANIFEST > + string "Path to a manifest.py file" > + help > + MicroPython allows Python code to be “frozen” as bytecode > + into its binary, as an alternative to loading code from > + the filesystem. See MicroPython's documentation for more > + information. > + > endif # BR2_PACKAGE_MICROPYTHON > > comment "micropython needs a toolchain w/ threads, dynamic library" > diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk > index 125a0edcfb..5a2c136547 100644 > --- a/package/micropython/micropython.mk > +++ b/package/micropython/micropython.mk > @@ -43,6 +43,11 @@ else > MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0 > endif > > +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"") > +MICROPYTHON_MAKE_OPTS += \ > + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST) > +endif So, as I understand this, micropython will grab the files (from modules, files, etc...) listed from the manifest and bundle them in the micropython executable. It seems the paths in the manifest can be either absolute, or relative; in the latter case, it not documented what they would be relative to; additionally there are placeholders (see below) that can be used but are absolute paths. So this has a few implications: - the files must be available before micropython is built, so some dependency order is needed to ensure that: if files are provided in a package, then we need a dependency and it is going to be a little bit difficult to do and we need a way to address that; - the paths can't be hard-coded to absolute paths in the manifest file itself, because we don't know beforehand where the buildroot build directory will be, so we need a way to shoehorn $(TOPDIR) or some such variable in the manifest; - absolute paths must be fixed to accommodate the per-package option, because those paths change for eaach package. From the micropython doc, that you pointed above, there are a few placeholders that will be replaced in the manifest, but I don't think we can reuse those: Any paths used in manifest files can include the following variables. These all resolve to absolute paths. * $(MPY_DIR) – path to the micropython repo. * $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to use require(). * $(PORT_DIR) – path to the current port (e.g. ports/stm32) * $(BOARD_DIR) – path to the current board (e.g. ports/stm32/boards/PYBV11) So we need a bit more information how this feature is to be used, and an example (or a few) would probably help a lot in this respect. We also have a runtime test for micropython, in: support/testing/tests/package/test_micropython.py It would be nice to extend that runtime test as well. Regards, Yann E. MORIN. > define MICROPYTHON_BUILD_CMDS > $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross > $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \ > -- > 2.43.0 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Yann,
One of the uses of manifest.py is to specify which modules of
micropython-lib should be frozen within the binary. For those modules you
don't need to include the path. e.g.,
package('os')
That's the application I'm going for. I know it seems limited but it's
really useful, and probably welcomed by other people using Micropython in
the next release of Buildroot.
I have ideas on how to solve the path and dependency order problem while
still using the "official" manifest.py concept. The biggest challenge is
that currently there are no other 3rd party Micropython modules available
for Buildroot, so that makes it all very theoretical. All solutions require
a bigger amount of work than the one needed for this patch. Also, I would
like to discuss them before actually presenting a patch that allows 3rd
party modules to be frozen.
I always try to go for an incremental approach, where I get the bigger bang
for the buck. I believe that allowing people to freeze the official
Micropython modules is already a big step forward. But at the same time,
I'm new to the Buildroot project, so please advice on the approach.
Kind regards,
Abilio
On Mon, Feb 5, 2024 at 5:24 AM Yann E. MORIN <yann.morin.1998@free.fr>
wrote:
> Abilio, All,
>
> On 2024-02-03 22:26 -0800, Abilio Marques spake thusly:
> > Micropython can embed packages an modules as frozen bytecode. What code
> > gets built this way can be defined by means of a "manifest.py" file.
> >
> > This commit exposes the variable FROZEN_MANIFEST to Buildroot users
> > through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST.
> >
> > Please check Micropython's documentation for more information:
> > https://docs.micropython.org/en/latest/reference/manifest.html
>
> Minor nit: this blurb should have been in the help text of the new
> option, and I could have done that when applying, but I have a much
> more involved question, see below.
>
> > Signed-off-by: Abilio Marques <abiliojr@gmail.com>
> > ---
> > package/micropython/Config.in | 8 ++++++++
> > package/micropython/micropython.mk | 5 +++++
> > 2 files changed, 13 insertions(+)
> >
> > diff --git a/package/micropython/Config.in
> b/package/micropython/Config.in
> > index 26a00baab0..52717d815a 100644
> > --- a/package/micropython/Config.in
> > +++ b/package/micropython/Config.in
> > @@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB
> > help
> > Core Python libraries ported to MicroPython.
> >
> > +config BR2_PACKAGE_MICROPYTHON_MANIFEST
> > + string "Path to a manifest.py file"
> > + help
> > + MicroPython allows Python code to be “frozen” as bytecode
> > + into its binary, as an alternative to loading code from
> > + the filesystem. See MicroPython's documentation for more
> > + information.
> > +
> > endif # BR2_PACKAGE_MICROPYTHON
> >
> > comment "micropython needs a toolchain w/ threads, dynamic library"
> > diff --git a/package/micropython/micropython.mk b/package/micropython/
> micropython.mk
> > index 125a0edcfb..5a2c136547 100644
> > --- a/package/micropython/micropython.mk
> > +++ b/package/micropython/micropython.mk
> > @@ -43,6 +43,11 @@ else
> > MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0
> > endif
> >
> > +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"")
> > +MICROPYTHON_MAKE_OPTS += \
> > + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST)
> > +endif
>
> So, as I understand this, micropython will grab the files (from modules,
> files, etc...) listed from the manifest and bundle them in the
> micropython executable. It seems the paths in the manifest can be either
> absolute, or relative; in the latter case, it not documented what they
> would be relative to; additionally there are placeholders (see below)
> that can be used but are absolute paths.
>
> So this has a few implications:
>
> - the files must be available before micropython is built, so some
> dependency order is needed to ensure that: if files are provided in
> a package, then we need a dependency and it is going to be a little
> bit difficult to do and we need a way to address that;
>
> - the paths can't be hard-coded to absolute paths in the manifest file
> itself, because we don't know beforehand where the buildroot build
> directory will be, so we need a way to shoehorn $(TOPDIR) or some
> such variable in the manifest;
>
> - absolute paths must be fixed to accommodate the per-package option,
> because those paths change for eaach package.
>
> From the micropython doc, that you pointed above, there are a few
> placeholders that will be replaced in the manifest, but I don't think
> we can reuse those:
>
> Any paths used in manifest files can include the following variables.
> These all resolve to absolute paths.
> * $(MPY_DIR) – path to the micropython repo.
> * $(MPY_LIB_DIR) – path to the micropython-lib submodule. Prefer to
> use require().
> * $(PORT_DIR) – path to the current port (e.g. ports/stm32)
> * $(BOARD_DIR) – path to the current board (e.g.
> ports/stm32/boards/PYBV11)
>
> So we need a bit more information how this feature is to be used, and an
> example (or a few) would probably help a lot in this respect. We also
> have a runtime test for micropython, in:
> support/testing/tests/package/test_micropython.py
>
> It would be nice to extend that runtime test as well.
>
> Regards,
> Yann E. MORIN.
>
> > define MICROPYTHON_BUILD_CMDS
> > $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross
> > $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
> > --
> > 2.43.0
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
> --
>
> .-----------------.--------------------.------------------.--------------------.
> | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___
> |
> | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is
> no |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v
> conspiracy. |
>
> '------------------------------^-------^------------------^--------------------'
>
Abilio, All, [Please, don't top-post, but reply in-line] On 2024-02-11 09:15 -0800, Abilio Marques spake thusly: > One of the uses of manifest.py is to specify which modules of > micropython-lib should be frozen within the binary. For those modules you > don't need to include the path. e.g., > package('os') > > That's the application I'm going for. I know it seems limited but it's > really useful, OK, so maybe this can be explained in the help text of the new option, like: Note: in Buildroot, only modules provided with micropython-lib can be frozen with a manifest; freezing arbitrary files is not supported. along with a little blurb in the commit log, statng something like; We only support freezing of modules from micropython-lib. Freezing arbitrary modules would require some handling of absoulte paths, and a guarantee that the files be available before micropython is built, which is considered a bit too complex; an interested party can provide such support in the future. And thus, would it be possible to sanity-check the manifest to ensure that it indeed only references modules from icropython-lib, so that people do not get the impression it works when in fact it does not? Like, grep that only require(...) are used? (Note: you mentioned package(...) but that is not limited to files from micropython-lib; require(...) however is.) > and probably welcomed by other people using Micropython in > the next release of Buildroot. One question I was wondering about: if a module is frozen in the binary, then it is no longer needed in the filesystem, right? If so, should we have a way to remove them? > I have ideas on how to solve the path and dependency order problem while > still using the "official" manifest.py concept. The biggest challenge is > that currently there are no other 3rd party Micropython modules available > for Buildroot, so that makes it all very theoretical. All solutions require > a bigger amount of work than the one needed for this patch. Also, I would > like to discuss them before actually presenting a patch that allows 3rd > party modules to be frozen. Indeed, without a few actual examples, it's going to be difficult to see a common pattern and abstract that away. Are there any pulicly available? > I always try to go for an incremental approach, where I get the bigger bang > for the buck. I believe that allowing people to freeze the official > Micropython modules is already a big step forward. But at the same time, > I'm new to the Buildroot project, so please advice on the approach. The incremental path is totally OK; I even prefer it. Of course, any limitation (such as those we are discussing) should be explained in the commit log. Given all the feedback in this thread, can you respin a v2 taking the comments into account? As for your ideas for lifting those limitations, you can just explain them in a reply in this thread. Usually, a patchset doing the job is also a good first step to start the dicsussion. Regards, Yann E. MORIN.
On Sun, Feb 11, 2024 at 9:54 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Abilio, All, > > [Please, don't top-post, but reply in-line] > > On 2024-02-11 09:15 -0800, Abilio Marques spake thusly: > > One of the uses of manifest.py is to specify which modules of > > micropython-lib should be frozen within the binary. For those modules you > > don't need to include the path. e.g., > > package('os') > > > > That's the application I'm going for. I know it seems limited but it's > > really useful, > > OK, so maybe this can be explained in the help text of the new option, > like: > > Note: in Buildroot, only modules provided with micropython-lib > can be frozen with a manifest; freezing arbitrary files is not > supported. > > along with a little blurb in the commit log, statng something like; > > We only support freezing of modules from micropython-lib. Freezing > arbitrary modules would require some handling of absoulte paths, and > a guarantee that the files be available before micropython is built, > which is considered a bit too complex; an interested party can > provide such support in the future. > > And thus, would it be possible to sanity-check the manifest to ensure > that it indeed only references modules from icropython-lib, so that > people do not get the impression it works when in fact it does not? > > Like, grep that only require(...) are used? Not sure how to do it correctly. The manifest is a regular python file, and I'm not sure how to do it in a way that does the correct the thing. e.g., the user might want to add comment lines or an include to a second part of the manifest. > > (Note: you mentioned package(...) but that is not limited to files from > micropython-lib; require(...) however is.) > > > and probably welcomed by other people using Micropython in > > the next release of Buildroot. > > One question I was wondering about: if a module is frozen in the binary, > then it is no longer needed in the filesystem, right? If so, should we > have a way to remove them? Good idea. Do you mind if I add this in a separate patch? > > > I have ideas on how to solve the path and dependency order problem while > > still using the "official" manifest.py concept. The biggest challenge is > > that currently there are no other 3rd party Micropython modules available > > for Buildroot, so that makes it all very theoretical. All solutions require > > a bigger amount of work than the one needed for this patch. Also, I would > > like to discuss them before actually presenting a patch that allows 3rd > > party modules to be frozen. > > Indeed, without a few actual examples, it's going to be difficult to see > a common pattern and abstract that away. Are there any pulicly > available? > > > I always try to go for an incremental approach, where I get the bigger bang > > for the buck. I believe that allowing people to freeze the official > > Micropython modules is already a big step forward. But at the same time, > > I'm new to the Buildroot project, so please advice on the approach. > > The incremental path is totally OK; I even prefer it. Of course, any > limitation (such as those we are discussing) should be explained in the > commit log. > > Given all the feedback in this thread, can you respin a v2 taking the > comments into account? I did, but just found out that I submitted a patch with a whitespace problem, in this line "can be frozen with a manifest; freezing arbitrary files is" Could you please correct it? > > As for your ideas for lifting those limitations, you can just explain > them in a reply in this thread. Usually, a patchset doing the job is > also a good first step to start the dicsussion. Will try a few things and submit one. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
diff --git a/package/micropython/Config.in b/package/micropython/Config.in index 26a00baab0..52717d815a 100644 --- a/package/micropython/Config.in +++ b/package/micropython/Config.in @@ -17,6 +17,14 @@ config BR2_PACKAGE_MICROPYTHON_LIB help Core Python libraries ported to MicroPython. +config BR2_PACKAGE_MICROPYTHON_MANIFEST + string "Path to a manifest.py file" + help + MicroPython allows Python code to be “frozen” as bytecode + into its binary, as an alternative to loading code from + the filesystem. See MicroPython's documentation for more + information. + endif # BR2_PACKAGE_MICROPYTHON comment "micropython needs a toolchain w/ threads, dynamic library" diff --git a/package/micropython/micropython.mk b/package/micropython/micropython.mk index 125a0edcfb..5a2c136547 100644 --- a/package/micropython/micropython.mk +++ b/package/micropython/micropython.mk @@ -43,6 +43,11 @@ else MICROPYTHON_MAKE_OPTS += MICROPY_PY_FFI=0 endif +ifneq ($(BR2_PACKAGE_MICROPYTHON_MANIFEST),"") +MICROPYTHON_MAKE_OPTS += \ + FROZEN_MANIFEST=$(BR2_PACKAGE_MICROPYTHON_MANIFEST) +endif + define MICROPYTHON_BUILD_CMDS $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/mpy-cross $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/ports/unix \
Micropython can embed packages an modules as frozen bytecode. What code gets built this way can be defined by means of a "manifest.py" file. This commit exposes the variable FROZEN_MANIFEST to Buildroot users through a new variable called BR2_PACKAGE_MICROPYTHON_MANIFEST. Please check Micropython's documentation for more information: https://docs.micropython.org/en/latest/reference/manifest.html Signed-off-by: Abilio Marques <abiliojr@gmail.com> --- package/micropython/Config.in | 8 ++++++++ package/micropython/micropython.mk | 5 +++++ 2 files changed, 13 insertions(+)