| Message ID | 20220406081837.2222008-1-sbabic@denx.de |
|---|---|
| Headers | show |
| Series | Support to call functions inside sw-description | expand |
Hi Stefano, Am 06.04.22 um 10:18 schrieb Stefano Babic: > sw-description is often generated from a template. There are several help functions > to assign automatically values to attributes, like the generation of sha256 hash. > Some other functions were added and more will be added in future, and this is > done creating quirks and some unlogical matching rules when sw-description is > parsed in the classes. This was done with '@', or '@SWU_VERSION', etc. > > To avoid to create any time new rules that are starting to conflict, implement > a generic method to call a user defined function returning an attribute value. > The syntax is: > > <attribute name > = "$<function name>(<function parameters>)"; > > Example: > sha256 = "$swupdate_get_sha256(rootfs)"; > What is the inspiration for this solution? This looks uncommon. Does this solution detects any changes in the meta data like function changes in a class or value changes of variables used inside a function? Why don't you use a variable with a list of variables which are forwarded to a dict and a python format function call? Regards Stefan
Hi Stefan, On 06.04.22 19:34, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 06.04.22 um 10:18 schrieb Stefano Babic: >> sw-description is often generated from a template. There are several >> help functions >> to assign automatically values to attributes, like the generation of >> sha256 hash. >> Some other functions were added and more will be added in future, and >> this is >> done creating quirks and some unlogical matching rules when >> sw-description is >> parsed in the classes. This was done with '@', or '@SWU_VERSION', etc. >> >> To avoid to create any time new rules that are starting to conflict, >> implement >> a generic method to call a user defined function returning an >> attribute value. >> The syntax is: >> >> <attribute name > = "$<function name>(<function parameters>)"; >> >> Example: >> sha256 = "$swupdate_get_sha256(rootfs)"; >> > > What is the inspiration for this solution? This looks uncommon. Does > this solution detects any changes in the meta data like function changes > in a class or value changes of variables used inside a function? > > Why don't you use a variable with a list of variables which are > forwarded to a dict and a python format function call? > Well, it is sometimes useful to look at new solutions ;-). But it is not so new and uncommon. If a function is changed in a class, it is not different as a change in any class in OE, from a do_rootfs() or whatever function is builtin. Parameters to the function itself are set inside sw-description, then a change means sw-description was changed and it is surely detected. Add more variables is more common in OE, but this can lead to a lot of variables that are then more difficult to document and combine together. And at any time, it seems there is a new quirk to get a specific behavior required by a project / user, but more difficult to generalize. Really, it is not that such as functionality was not present: anyone can set variables in the own recipe and add new functions and because meta-swupdate replaces any variable in sw-description surrounded by "@@", it was already possible if functions / complexity are moved inside the own image recipe. But this is not what most users want. In the last times, several automatism were introduced, and all of them require some tricks by parsing sw-description, thought as template. Starting from a single "@" for sha256, then there is a hack to reuse sha256() together with the "version" attribute (well, and new hacks can be then introduced if someone else want to use the hash together with another attribute), then @SWU_VERSION, then signing rootfs, then.... So my idea is to simplify swupdate-common and drop all hacks we have now, and let the possibility for users to attach own functions. A user can create an own class, inherit in the own image recipe, and make usage in sw-description without hacking the parsing of sw-description as it should be now. New introduced functions can merged easier, because they won't change sw-description parsing and they can be just collected into swupdate-lib.class. It depends then on the user which functions he need, but I am quite sure that the main functionality is never broken and this simplify regression tests, too. Syntax is not very new, too: it rearrange something like "${@bb.utils...." used in recipe, even if I know it differs. But for the generation of a SWU, the master is sw-description and not the recipe. Best regards, Stefano
Hi Stefano, Am 07.04.22 um 08:16 schrieb Stefano Babic: > Hi Stefan, > > On 06.04.22 19:34, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 06.04.22 um 10:18 schrieb Stefano Babic: >>> sw-description is often generated from a template. There are several >>> help functions >>> to assign automatically values to attributes, like the generation of >>> sha256 hash. >>> Some other functions were added and more will be added in future, and >>> this is >>> done creating quirks and some unlogical matching rules when >>> sw-description is >>> parsed in the classes. This was done with '@', or '@SWU_VERSION', etc. >>> >>> To avoid to create any time new rules that are starting to conflict, >>> implement >>> a generic method to call a user defined function returning an >>> attribute value. >>> The syntax is: >>> >>> <attribute name > = "$<function name>(<function parameters>)"; >>> >>> Example: >>> sha256 = "$swupdate_get_sha256(rootfs)"; >>> >> >> What is the inspiration for this solution? This looks uncommon. Does >> this solution detects any changes in the meta data like function >> changes in a class or value changes of variables used inside a function? >> >> Why don't you use a variable with a list of variables which are >> forwarded to a dict and a python format function call? >> > > Well, it is sometimes useful to look at new solutions ;-). But new solutions need special care and meta-swupdate have already uncommon solution which leads to problems and misunderstanding. Are there any examples inside the core which use source files (ex. defconfig and sw-description) during recipe parsing? > But it is not > so new and uncommon. Do you have an example which specify the used functions outside of recipes or classes? > If a function is changed in a class, it is not > different as a change in any class in OE, from a do_rootfs() or whatever > function is builtin. Bitbake has features to model dependencies and thereby to track function changes and this feature is used inside the classes. > Parameters to the function itself are set inside > sw-description, then a change means sw-description was changed and it is > surely detected. A change inside a function doesn't require a change in sw-description if the signature isn't changed. > Add more variables is more common in OE, but this can lead to a lot of > variables that are then more difficult to document and combine together. > And at any time, it seems there is a new quirk to get a specific > behavior required by a project / user, but more difficult to generalize. What do you mean by this? A possible solution is a SWUPDATE_VARS variable with a list of variables and this variables and its value could be forwarded to a Template substitute(**args) call. Alternative a format call but this requires the escape of { and } characters. > Really, it is not that such as functionality was not present: anyone can > set variables in the own recipe and add new functions and because > meta-swupdate replaces any variable in sw-description surrounded by > "@@", it was already possible if functions / complexity are moved inside > the own image recipe. But this is not what most users want. > > In the last times, several automatism were introduced, and all of them > require some tricks by parsing sw-description, thought as template. > Starting from a single "@" for sha256, then there is a hack to reuse > sha256() together with the "version" attribute (well, and new hacks can > be then introduced if someone else want to use the hash together with > another attribute), then @SWU_VERSION, then signing rootfs, then.... I see the requirement for a solution but this solution looks uncommon and need special care to keep the change detection working. I think it is uncommon for a recipe or class to depend on the content of source files and we still have problems with this solution in other places. > So my idea is to simplify swupdate-common and drop all hacks we have > now, and let the possibility for users to attach own functions. A user > can create an own class, inherit in the own image recipe, and make usage > in sw-description without hacking the parsing of sw-description as it > should be now. New introduced functions can merged easier, because they > won't change sw-description parsing and they can be just collected into > swupdate-lib.class. It depends then on the user which functions he need, > but I am quite sure that the main functionality is never broken and this > simplify regression tests, too. My concern is only the function list inside the sw-description. The common solution is a variable with the function list because this could be added to the do_rootfs[vardeps] and the sw-description could be handled like any other source file. > Syntax is not very new, too: it rearrange something like > "${@bb.utils...." used in recipe, even if I know it differs. You use a different syntax as bitbake. ${...} is used to mark variables and ${@...} to mark python code not only functions. > But for the > generation of a SWU, the master is sw-description and not the recipe. That's the main problem. The recipe is the master for bitbake and the sw-description is a source file. Source files could be manipulated by bitbake tasks and are not valid at parse time. Regards Stefan
Hi Stefan, On 10.04.22 13:57, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 07.04.22 um 08:16 schrieb Stefano Babic: >> Hi Stefan, >> >> On 06.04.22 19:34, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 06.04.22 um 10:18 schrieb Stefano Babic: >>>> sw-description is often generated from a template. There are several >>>> help functions >>>> to assign automatically values to attributes, like the generation of >>>> sha256 hash. >>>> Some other functions were added and more will be added in future, >>>> and this is >>>> done creating quirks and some unlogical matching rules when >>>> sw-description is >>>> parsed in the classes. This was done with '@', or '@SWU_VERSION', etc. >>>> >>>> To avoid to create any time new rules that are starting to conflict, >>>> implement >>>> a generic method to call a user defined function returning an >>>> attribute value. >>>> The syntax is: >>>> >>>> <attribute name > = "$<function name>(<function parameters>)"; >>>> >>>> Example: >>>> sha256 = "$swupdate_get_sha256(rootfs)"; >>>> >>> >>> What is the inspiration for this solution? This looks uncommon. Does >>> this solution detects any changes in the meta data like function >>> changes in a class or value changes of variables used inside a function? >>> >>> Why don't you use a variable with a list of variables which are >>> forwarded to a dict and a python format function call? >>> >> >> Well, it is sometimes useful to look at new solutions ;-). > > But new solutions need special care and meta-swupdate have already > uncommon solution which leads to problems and misunderstanding. > > Are there any examples inside the core which use source files (ex. > defconfig and sw-description) during recipe parsing? Ok, I understand now what you mean - Regarding dependencies, I have searched and did not find - so the parsing of defconfig in meta-swupdate is quite unique. I do not know or I have not searched enough for the second case. IMO the two case are distinct, because dependencies are set during parsing of recipes, but the new use parses sw-description in an own task. > >> But it is not so new and uncommon. > > Do you have an example which specify the used functions outside of > recipes or classes? > >> If a function is changed in a class, it is not different as a change >> in any class in OE, from a do_rootfs() or whatever function is builtin. > > Bitbake has features to model dependencies and thereby to track function > changes and this feature is used inside the classes. ok - so if for example swupdate_get_aes256() changes, this is automatically tracked and SWU is rebuilt. Worse is if each function is reading variables, and they are not tracked in vardeps. > > >> Parameters to the function itself are set inside sw-description, then >> a change means sw-description was changed and it is surely detected. > > A change inside a function doesn't require a change in sw-description if > the signature isn't changed. > >> Add more variables is more common in OE, but this can lead to a lot of >> variables that are then more difficult to document and combine >> together. And at any time, it seems there is a new quirk to get a >> specific behavior required by a project / user, but more difficult to >> generalize. > > What do you mean by this? > > A possible solution is a SWUPDATE_VARS variable with a list of variables > and this variables and its value could be forwarded to a Template > substitute(**args) call. Alternative a format call but this requires the > escape of { and } characters. So putting the name of used funtions inside SWUPDATE_VARS and checking before evaluating the function that it belongs the list should be enough ? SWUPDATE_VARS is nadded to vardeps, and if function does not belong to list, build fails. User should update the list, but a default can be set in swupdate-common for the mainlined functions. > >> Really, it is not that such as functionality was not present: anyone >> can set variables in the own recipe and add new functions and because >> meta-swupdate replaces any variable in sw-description surrounded by >> "@@", it was already possible if functions / complexity are moved >> inside the own image recipe. But this is not what most users want. >> >> In the last times, several automatism were introduced, and all of them >> require some tricks by parsing sw-description, thought as template. >> Starting from a single "@" for sha256, then there is a hack to reuse >> sha256() together with the "version" attribute (well, and new hacks >> can be then introduced if someone else want to use the hash together >> with another attribute), then @SWU_VERSION, then signing rootfs, then.... > > I see the requirement for a solution but this solution looks uncommon > and need special care to keep the change detection working. > > I think it is uncommon for a recipe or class to depend on the content of > source files and we still have problems with this solution in other places. ok, agree - but in this case, clss contains a bunch of python functions that are just called to create the SWU. The class itself does not depend on SWU, or I do not see the connection. > >> So my idea is to simplify swupdate-common and drop all hacks we have >> now, and let the possibility for users to attach own functions. A user >> can create an own class, inherit in the own image recipe, and make >> usage in sw-description without hacking the parsing of sw-description >> as it should be now. New introduced functions can merged easier, >> because they won't change sw-description parsing and they can be just >> collected into swupdate-lib.class. It depends then on the user which >> functions he need, but I am quite sure that the main functionality is >> never broken and this simplify regression tests, too. > > My concern is only the function list inside the sw-description. The > common solution is a variable with the function list because this could > be added to the do_rootfs[vardeps] and the sw-description could be > handled like any other source file. > >> Syntax is not very new, too: it rearrange something like >> "${@bb.utils...." used in recipe, even if I know it differs. > > You use a different syntax as bitbake. ${...} is used to mark variables > and ${@...} to mark python code not only functions. Yes, right, but '@' was already used for sha256 - let's say I am using a "similar" syntax :-) > > >> But for the generation of a SWU, the master is sw-description and not >> the recipe. > > That's the main problem. The recipe is the master for bitbake and the > sw-description is a source file. Source files could be manipulated by > bitbake tasks and are not valid at parse time. Anyway, even if OE is quite dominant and used in many projects, it is not the only one. I have projects based on other technologies (buildroot and Debian), too. The common denominator for all buildsystems is sw-description, and this decides how the SWU is generated. At the moment, while the SWU build is well supported in OE, it is poor or missing with other technologies. This leads to a spread of project specific hacks, with bunches of scripts that are not reusable, and I am trying to address this, too, and my plan is to factorize al lof them. Nevertheless, the generation of a SWU strictly depends on the sw-description: the list of artifacts is really defined in sw-description, but it is defined in recipe for OE because this is the way OE expects (with SWUPDATE_IMAGES). Regards, Stefano