| Message ID | 20230821175828.42827-1-ckhardin@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [1/1] package/python-pykwalify: add required package for zephyr cmake | expand |
Hello Charles, On Mon, 21 Aug 2023 10:58:28 -0700 Charles Hardin <ckhardin@gmail.com> wrote: > When building zephyr based firmware, the cmake integrates with > python scripts that can be run using the buildroot host-python > dependencies. This is useful for the sdk setups and associated > build steps for the firmware. In this use-case, a buildroot > external project can be setup with recipes to build firmware > images using eabi toolchains and the linux controller software > to talk to those firmware images using one SDK. This might not > be overtly obvious from the patches. Could you clarify a bit what you call "zephyr cmake" and the use case for this? > Add the required host packages and additional pykwalify This needs to be split into multiple patches, one per package affected. But I think we also need to understand the bigger picture of what this is useful for. Thanks! Thomas
inline On Mon, Aug 21, 2023 at 3:13 PM Thomas Petazzoni < thomas.petazzoni@bootlin.com> wrote: > Hello Charles, > > On Mon, 21 Aug 2023 10:58:28 -0700 > Charles Hardin <ckhardin@gmail.com> wrote: > > > When building zephyr based firmware, the cmake integrates with > > python scripts that can be run using the buildroot host-python > > dependencies. This is useful for the sdk setups and associated > > build steps for the firmware. In this use-case, a buildroot > > external project can be setup with recipes to build firmware > > images using eabi toolchains and the linux controller software > > to talk to those firmware images using one SDK. This might not > > be overtly obvious from the patches. > > Could you clarify a bit what you call "zephyr cmake" and the use case > for this? > https://github.com/zephyrproject-rtos The build is cmake derived but relies on calling into some python scripts to generate some build dependencies https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt > > > Add the required host packages and additional pykwalify > > This needs to be split into multiple patches, one per package affected. > But I think we also need to understand the bigger picture of what this > is useful for. > So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-pi Assume a reference to https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add the gcc eabi toolchain has been done in an external package, with the patches above we can actually build both firmware and the drivers in buildroot from one tree with external recipes like below env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake \ -DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake \ -DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" \ -DCMAKE_BUILD_TYPE=Debug \ ../buildroot-external make VERBOSE=1 -j$(nproc) > > Thanks! > > Thomas > -- > Thomas Petazzoni, co-owner and CEO, Bootlin > Embedded Linux and Kernel engineering and training > https://bootlin.com >
Hello Charles, On Mon, 21 Aug 2023 16:45:23 -0700 Charles Hardin <ckhardin@gmail.com> wrote: > https://github.com/zephyrproject-rtos Yes, I know what Zephyr is :-) > The build is cmake derived but relies on calling into some python scripts to > generate some build dependencies > > https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt OK. > So, we have various RTOS MCUs that are hooked up into linux controllers, > and this is just a simple hook so that the same SDK can be handled to build > both firmware for something like an STM32 and a Raspberry-pi If I rephrase this: you want to use Buildroot to build the Zephyr firmware for your STM32 micro-controller, alongside with the Linux system running on the RaspberryPi? > Assume a reference to > https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add > the gcc eabi toolchain has been done in an external package, with the > patches above we can actually build both firmware and the drivers in > buildroot from one tree with external recipes like below Note: this gcc eabi toolchain is now in Buildroot upstream, no need to add it as an external package. > env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake > \ > > -DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake > \ > > > -DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" > \ > > -DCMAKE_BUILD_TYPE=Debug \ > > ../buildroot-external Not sure to really follow in which projects this cmake command is used? To build Zephyr? Best regards, Thomas
Thomas, > On Aug 22, 2023, at 10:57 AM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Hello Charles, > >> On Mon, 21 Aug 2023 16:45:23 -0700 >> Charles Hardin <ckhardin@gmail.com> wrote: >> >> https://github.com/zephyrproject-rtos > > Yes, I know what Zephyr is :-) > >> The build is cmake derived but relies on calling into some python scripts to >> generate some build dependencies >> >> https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt > > OK. > >> So, we have various RTOS MCUs that are hooked up into linux controllers, >> and this is just a simple hook so that the same SDK can be handled to build >> both firmware for something like an STM32 and a Raspberry-pi > > If I rephrase this: you want to use Buildroot to build the Zephyr > firmware for your STM32 micro-controller, alongside with the Linux > system running on the RaspberryPi? > Yes, and we are doing this today with these patches. >> Assume a reference to >> https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add >> the gcc eabi toolchain has been done in an external package, with the >> patches above we can actually build both firmware and the drivers in >> buildroot from one tree with external recipes like below > > Note: this gcc eabi toolchain is now in Buildroot upstream, no need to > add it as an external package. > Thanks >> env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake >> \ >> >> -DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake >> \ >> >> >> -DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" >> \ >> >> -DCMAKE_BUILD_TYPE=Debug \ >> >> ../buildroot-external > > Not sure to really follow in which projects this cmake command is used? > To build Zephyr? > There is a custom cmake file to do the zephyr build, so make does the buildroot build and we can use cmake for the zephyr build. It is not out of the box but pretty simple glue to make it work. > Best regards, > > Thomas > -- > Thomas Petazzoni, co-owner and CEO, Bootlin > Embedded Linux and Kernel engineering and training > https://bootlin.com
On 22/08/2023 20:02, Charles Hardin wrote: > Thomas, > >> On Aug 22, 2023, at 10:57 AM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: >> >> Hello Charles, >> >>> On Mon, 21 Aug 2023 16:45:23 -0700 >>> Charles Hardin <ckhardin@gmail.com> wrote: [snip] >>> So, we have various RTOS MCUs that are hooked up into linux controllers, >>> and this is just a simple hook so that the same SDK can be handled to build >>> both firmware for something like an STM32 and a Raspberry-pi >> >> If I rephrase this: you want to use Buildroot to build the Zephyr >> firmware for your STM32 micro-controller, alongside with the Linux >> system running on the RaspberryPi? >> > > Yes, and we are doing this today with these patches. Honestly, I don't know if we want to go in this direction. If Zephyr doesn't give you the possibility to do "from scratch" builds, then in this day and age it's more appropriate to create a container image for your build environment IMHO. There's something to be said for including host packages that are likely to be used in post-build/post-image scripts, but for building a completely different OS, that is going a little far IMHO. Specifically for Python packages, I also think it's much more appropriate to use poetry to control them for the host. That way we just have to provide host-python-poetry, and the post-build/post-image scripts can take care of installing all the needed python tools at the exact version you need, controlled by a lock file. In my opinion, even including those arm-none-eabi etc. compilers in Buildroot was going a bit too far already. But I guess that that ship has sailed :-) Regards, Arnout [snip]
Arnout, Charles, All, On 2023-09-21 22:03 +0200, Arnout Vandecappelle via buildroot spake thusly: > On 22/08/2023 20:02, Charles Hardin wrote: > >>On Aug 22, 2023, at 10:57 AM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > >>>On Mon, 21 Aug 2023 16:45:23 -0700 > >>>Charles Hardin <ckhardin@gmail.com> wrote: > >>>So, we have various RTOS MCUs that are hooked up into linux controllers, > >>>and this is just a simple hook so that the same SDK can be handled to build > >>>both firmware for something like an STM32 and a Raspberry-pi > >>If I rephrase this: you want to use Buildroot to build the Zephyr > >>firmware for your STM32 micro-controller, alongside with the Linux > >>system running on the RaspberryPi? > >Yes, and we are doing this today with these patches. > Honestly, I don't know if we want to go in this direction. If Zephyr > doesn't give you the possibility to do "from scratch" builds, then in this > day and age it's more appropriate to create a container image for your build > environment IMHO. There's something to be said for including host packages > that are likely to be used in post-build/post-image scripts, but for > building a completely different OS, that is going a little far IMHO. I concur. I believe in this case that external projects are better suited to handle the nitty-gritty details of running those buildsystems, even if, as Arnout suggests, that is wrapped in a docker image. With proper CI/CD, it is easy to construct pipelines that build those "other OS" stuff, export their artefacts for followup Buildroot-based jobs that integrate those artefacts (either as wrapping package that jsut instlla binary blob) or as rootfs overlays, or from post-build scripts. I have marked the series as rejected in patchwork. > Specifically for Python packages, I also think it's much more appropriate > to use poetry to control them for the host. That way we just have to provide > host-python-poetry, and the post-build/post-image scripts can take care of > installing all the needed python tools at the exact version you need, > controlled by a lock file. > > In my opinion, even including those arm-none-eabi etc. compilers in > Buildroot was going a bit too far already. But I guess that that ship has > sailed :-) I also heartily agree. Regards, Yann E. MORIN.
diff --git a/package/Config.in b/package/Config.in index 54cddc3914..8eb6c2bf37 100644 --- a/package/Config.in +++ b/package/Config.in @@ -1225,6 +1225,7 @@ menu "External python modules" source "package/python-pyicu/Config.in" source "package/python-pyinotify/Config.in" source "package/python-pyjwt/Config.in" + source "package/python-pykwalify/Config.in" source "package/python-pylibfdt/Config.in" source "package/python-pylibftdi/Config.in" source "package/python-pylru/Config.in" diff --git a/package/python-dateutil/python-dateutil.mk b/package/python-dateutil/python-dateutil.mk index b75f510bce..d45a047ba8 100644 --- a/package/python-dateutil/python-dateutil.mk +++ b/package/python-dateutil/python-dateutil.mk @@ -10,5 +10,7 @@ PYTHON_DATEUTIL_SETUP_TYPE = setuptools PYTHON_DATEUTIL_LICENSE = BSD-3-Clause PYTHON_DATEUTIL_LICENSE_FILES = LICENSE PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm +HOST_PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm host-python-six $(eval $(python-package)) +$(eval $(host-python-package)) diff --git a/package/python-docopt/python-docopt.mk b/package/python-docopt/python-docopt.mk index f87d2f85a6..d326f4f0aa 100644 --- a/package/python-docopt/python-docopt.mk +++ b/package/python-docopt/python-docopt.mk @@ -11,3 +11,4 @@ PYTHON_DOCOPT_LICENSE_FILES = LICENSE-MIT PYTHON_DOCOPT_SETUP_TYPE = setuptools $(eval $(python-package)) +$(eval $(host-python-package)) diff --git a/package/python-pykwalify/Config.in b/package/python-pykwalify/Config.in new file mode 100644 index 0000000000..03afe31865 --- /dev/null +++ b/package/python-pykwalify/Config.in @@ -0,0 +1,9 @@ +config BR2_PACKAGE_PYTHON_PYKWALIFY + bool "python-pykwalify" + select BR2_PACKAGE_PYTHON_DATEUTIL # runtime + select BR2_PACKAGE_PYTHON_DOCOPT # runtime + select BR2_PACKAGE_PYTHON_RUAMEL_YAML # runtime + help + Python lib/cli for JSON/YAML schema validation. + + http://github.com/grokzen/pykwalify diff --git a/package/python-pykwalify/python-pykwalify.hash b/package/python-pykwalify/python-pykwalify.hash new file mode 100644 index 0000000000..6ddfb6306a --- /dev/null +++ b/package/python-pykwalify/python-pykwalify.hash @@ -0,0 +1,6 @@ +# md5, sha256 from https://pypi.org/pypi/pykwalify/json +md5 153197598f8508a49aefc0f44156c2b8 pykwalify-1.8.0.tar.gz +sha256 796b2ad3ed4cb99b88308b533fb2f559c30fa6efb4fa9fda11347f483d245884 pykwalify-1.8.0.tar.gz +# Locally computed sha256 checksums +sha256 626fafd952ab0345edc227f2d565835fe1fd14067ae0d635ee3e47289b773dc4 LICENSE +sha256 f89f247ddd2b9fdff655668148da4b6feaa9175fa44dd244d233006d0f551cd5 docs/license.rst diff --git a/package/python-pykwalify/python-pykwalify.mk b/package/python-pykwalify/python-pykwalify.mk new file mode 100644 index 0000000000..8a0bc6a227 --- /dev/null +++ b/package/python-pykwalify/python-pykwalify.mk @@ -0,0 +1,23 @@ +################################################################################ +# +# python-pykwalify +# +################################################################################ + +PYTHON_PYKWALIFY_VERSION = 1.8.0 +PYTHON_PYKWALIFY_SOURCE = pykwalify-$(PYTHON_PYKWALIFY_VERSION).tar.gz +PYTHON_PYKWALIFY_SITE = https://files.pythonhosted.org/packages/d5/77/2d6849510dbfce5f74f1f69768763630ad0385ad7bb0a4f39b55de3920c7 +PYTHON_PYKWALIFY_SETUP_TYPE = setuptools +PYTHON_PYKWALIFY_LICENSE = MIT +PYTHON_PYKWALIFY_LICENSE_FILES = LICENSE docs/license.rst +PYTHON_PYKWALIFY_DEPENDENCIES = \ + python-dateutil \ + python-docopt \ + python-ruamel-yaml +HOST_PYTHON_PYKWALIFY_DEPENDENCIES = \ + host-python-dateutil \ + host-python-docopt \ + host-python-ruamel-yaml + +$(eval $(python-package)) +$(eval $(host-python-package)) diff --git a/package/python-ruamel-yaml/python-ruamel-yaml.mk b/package/python-ruamel-yaml/python-ruamel-yaml.mk index a6fed3bc2c..4f5691c981 100644 --- a/package/python-ruamel-yaml/python-ruamel-yaml.mk +++ b/package/python-ruamel-yaml/python-ruamel-yaml.mk @@ -14,3 +14,4 @@ PYTHON_RUAMEL_YAML_CPE_ID_VENDOR = ruamel.yaml_project PYTHON_RUAMEL_YAML_CPE_ID_PRODUCT = ruamel.yaml $(eval $(python-package)) +$(eval $(host-python-package))
When building zephyr based firmware, the cmake integrates with python scripts that can be run using the buildroot host-python dependencies. This is useful for the sdk setups and associated build steps for the firmware. In this use-case, a buildroot external project can be setup with recipes to build firmware images using eabi toolchains and the linux controller software to talk to those firmware images using one SDK. This might not be overtly obvious from the patches. Add the required host packages and additional pykwalify Signed-off-by: Charles Hardin <ckhardin@gmail.com> --- package/Config.in | 1 + package/python-dateutil/python-dateutil.mk | 2 ++ package/python-docopt/python-docopt.mk | 1 + package/python-pykwalify/Config.in | 9 ++++++++ .../python-pykwalify/python-pykwalify.hash | 6 +++++ package/python-pykwalify/python-pykwalify.mk | 23 +++++++++++++++++++ .../python-ruamel-yaml/python-ruamel-yaml.mk | 1 + 7 files changed, 43 insertions(+) create mode 100644 package/python-pykwalify/Config.in create mode 100644 package/python-pykwalify/python-pykwalify.hash create mode 100644 package/python-pykwalify/python-pykwalify.mk