| Message ID | 20211222140401.780-2-sam.van.den.berge@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | None | expand |
Hello, Would it be possible to get some feedback on those 2 patches? Thanks in advance. Kr, Sam.
Hi Sam, On 07.01.22 09:03, Sam Van Den Berge wrote: > Hello, > > Would it be possible to get some feedback on those 2 patches? > I am not currently in office, I will check your changes in next days. Best regards, Stefano Babic > Thanks in advance. > > Kr, > Sam. >
Hi Sam, On 22.12.21 09:04, Sam Van Den Berge wrote: > The same principle from automatic sha256 insertion is applied for > automatic decrypted-size insertion: > > images: ( > { > filename = "image.ubifs"; > volume = "root_r"; > sha256 = "@image.ubifs"; > encrypted = true; > properties: { > decrypted-size = "@image.ubifs"; > } > }, > ); > > This is usefull especially for *encrypted* UBI images which need the > "decrypted-size" property when the image is installed. > Same is for "decompressed-size", when a ubifs is not encrypted but compressed. Anyway, it looks to me that adding a specific regex for each attribute ("decrypted-size" vs "decompressed-size") does not scale well. I would split the issue : one topic is to add a function (that is something like swupdate_decrypted_size()) that can derive a value from an artifact. The other topic and my concern is how to bind the function to the attribute in sw-description. sha256 was the first function added to automate the generation of sw-description from a template. After that, full variable substitution was added. What I would like to see but I never investigated is to add a general way to bind results from function to attributes, like an extended variable substitution. This is already supported in OE, just see for example "${@bb.utils..". If we have the following mechanismus: decrypted-size = "${@os.path.getsize(@@IMAGE@@))"; we do not need any further rule to regexp, and it scales much more better. Can you check if this can be done ? > Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com> > --- > classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass > index e6e5a5f..79f81df 100644 > --- a/classes/swupdate-common.bbclass > +++ b/classes/swupdate-common.bbclass > @@ -91,6 +91,29 @@ def swupdate_write_sha256(s): > for line in write_lines: > f.write(line) > > +def swupdate_write_decrypted_size(s, src): > + import re > + write_lines = [] > + > + decrypted_size = os.path.getsize(src) > + > + with open(os.path.join(s, "sw-description"), 'r') as f: > + for line in f: > + m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line) > + if m: > + filename = m.group('filename') > + if os.path.basename(src) == filename: > + bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size)) > + write_lines.append(line.replace("@%s" % (filename), str(decrypted_size))) > + else: > + write_lines.append(line) > + else: > + write_lines.append(line) > + > + with open(os.path.join(s, "sw-description"), 'w+') as f: > + for line in write_lines: > + f.write(line) > + > def swupdate_expand_bitbake_variables(d, s): > write_lines = [] > > @@ -313,6 +336,7 @@ def swupdate_add_src_uri(d, list_for_cpio): > if not key or not iv: > bb.fatal("Encryption required, but no key found") > swupdate_encrypt_file(local, dst, key, iv) > + swupdate_write_decrypted_size(s, local) > else: > shutil.copyfile(local, dst) > list_for_cpio.append(filename) > @@ -329,6 +353,7 @@ def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio): > key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True)) > bb.note("Encryption requested for %s" %(imagename)) > swupdate_encrypt_file(src, dst, key, iv) > + swupdate_write_decrypted_size(s, src) > else: > shutil.copyfile(src, dst) > list_for_cpio.append(target_imagename) > Best regards, Stefano Babic
On Mon, Jan 17, 2022 at 04:30:42PM +0100, Stefano Babic wrote: > Hi Sam, > > On 22.12.21 09:04, Sam Van Den Berge wrote: > > The same principle from automatic sha256 insertion is applied for > > automatic decrypted-size insertion: > > > > images: ( > > { > > filename = "image.ubifs"; > > volume = "root_r"; > > sha256 = "@image.ubifs"; > > encrypted = true; > > properties: { > > decrypted-size = "@image.ubifs"; > > } > > }, > > ); > > > > This is usefull especially for *encrypted* UBI images which need the > > "decrypted-size" property when the image is installed. > > > > Same is for "decompressed-size", when a ubifs is not encrypted but > compressed. Anyway, it looks to me that adding a specific regex for each > attribute ("decrypted-size" vs "decompressed-size") does not scale well. > > I would split the issue : one topic is to add a function (that is something > like swupdate_decrypted_size()) that can derive a value from an artifact. > The other topic and my concern is how to bind the function to the attribute > in sw-description. sha256 was the first function added to automate the > generation of sw-description from a template. After that, full variable > substitution was added. > > What I would like to see but I never investigated is to add a general way to > bind results from function to attributes, like an extended variable > substitution. This is already supported in OE, just see for example > "${@bb.utils..". > > If we have the following mechanismus: > > decrypted-size = "${@os.path.getsize(@@IMAGE@@))"; > > we do not need any further rule to regexp, and it scales much more better. > Can you check if this can be done ? Sorry for the late answer. If I understand correctly we would have an sw-description file which contains lines like decrypted-size = "${@os.path.getsize(@@IMAGE@@))"; These lines then contain function calls to methods we can implement ourselves. At the time when an image is created, the meta-swupdate layer would read the sw-description file and fill in the necessary parts by calling the appropriate functions. After doing some (quick) investigation I came to the conclusion that it won't be that easy. At first F-strings seemed like a good candidate. It supports variable substitution and calling functions. However F-strings is merely a syntax thing. I didn't find any easy way to convert a regular string into an F-string. The latter would be needed because we would have to read the 'sw-description' file and then convert it into an F-string. Another alternative would be Template strings[1]. However this only supports variable substitution and not calling functions. Same goes for the Format String Syntax [2]. This one however supports accessing an object's arguments. Theoretically we could use this to call a function. E.g. "{image.sha256sum}".format(image) where image is a class and sha256sum is a function of that class decorated with the @property decorator. We would have to implement such a function for every 'automatic' attribute we want (decrypted-size, decompressed-size, sha256sum, ...) This all feels a bit cumbersome. Another possible solution is to use regex + eval. The regex part we are doing at the moment and we could extend this with a bit of eval() code. However this also doesn't seem like a good plan to me. Finally there exist templating engines which do support what you suggest (e.g. jinja2) however that is an additional dependency and also that doesn't look like a good idea to me. I'm not aware of any other solution but on the other hand I'm also not a python expert so it could verry well be that I missed something. Kr, Sam. [1] https://docs.python.org/3/library/string.html#template-strings [2] https://docs.python.org/3/library/string.html#format-string-syntax [3] https://docs.python.org/3/library/functions.html#property > > Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com> > > --- > > classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++ > > 1 file changed, 25 insertions(+) > > > > diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass > > index e6e5a5f..79f81df 100644 > > --- a/classes/swupdate-common.bbclass > > +++ b/classes/swupdate-common.bbclass > > @@ -91,6 +91,29 @@ def swupdate_write_sha256(s): > > for line in write_lines: > > f.write(line) > > +def swupdate_write_decrypted_size(s, src): > > + import re > > + write_lines = [] > > + > > + decrypted_size = os.path.getsize(src) > > + > > + with open(os.path.join(s, "sw-description"), 'r') as f: > > + for line in f: > > + m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line) > > + if m: > > + filename = m.group('filename') > > + if os.path.basename(src) == filename: > > + bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size)) > > + write_lines.append(line.replace("@%s" % (filename), str(decrypted_size))) > > + else: > > + write_lines.append(line) > > + else: > > + write_lines.append(line) > > + > > + with open(os.path.join(s, "sw-description"), 'w+') as f: > > + for line in write_lines: > > + f.write(line) > > + > > def swupdate_expand_bitbake_variables(d, s): > > write_lines = [] > > @@ -313,6 +336,7 @@ def swupdate_add_src_uri(d, list_for_cpio): > > if not key or not iv: > > bb.fatal("Encryption required, but no key found") > > swupdate_encrypt_file(local, dst, key, iv) > > + swupdate_write_decrypted_size(s, local) > > else: > > shutil.copyfile(local, dst) > > list_for_cpio.append(filename) > > @@ -329,6 +353,7 @@ def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio): > > key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True)) > > bb.note("Encryption requested for %s" %(imagename)) > > swupdate_encrypt_file(src, dst, key, iv) > > + swupdate_write_decrypted_size(s, src) > > else: > > shutil.copyfile(src, dst) > > list_for_cpio.append(target_imagename) > > > > Best regards, > Stefano Babic > > -- > ===================================================================== > DENX Software Engineering GmbH, Managing Director: Wolfgang Denk > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany > Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de > =====================================================================
diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass index e6e5a5f..79f81df 100644 --- a/classes/swupdate-common.bbclass +++ b/classes/swupdate-common.bbclass @@ -91,6 +91,29 @@ def swupdate_write_sha256(s): for line in write_lines: f.write(line) +def swupdate_write_decrypted_size(s, src): + import re + write_lines = [] + + decrypted_size = os.path.getsize(src) + + with open(os.path.join(s, "sw-description"), 'r') as f: + for line in f: + m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line) + if m: + filename = m.group('filename') + if os.path.basename(src) == filename: + bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size)) + write_lines.append(line.replace("@%s" % (filename), str(decrypted_size))) + else: + write_lines.append(line) + else: + write_lines.append(line) + + with open(os.path.join(s, "sw-description"), 'w+') as f: + for line in write_lines: + f.write(line) + def swupdate_expand_bitbake_variables(d, s): write_lines = [] @@ -313,6 +336,7 @@ def swupdate_add_src_uri(d, list_for_cpio): if not key or not iv: bb.fatal("Encryption required, but no key found") swupdate_encrypt_file(local, dst, key, iv) + swupdate_write_decrypted_size(s, local) else: shutil.copyfile(local, dst) list_for_cpio.append(filename) @@ -329,6 +353,7 @@ def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio): key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True)) bb.note("Encryption requested for %s" %(imagename)) swupdate_encrypt_file(src, dst, key, iv) + swupdate_write_decrypted_size(s, src) else: shutil.copyfile(src, dst) list_for_cpio.append(target_imagename)
The same principle from automatic sha256 insertion is applied for automatic decrypted-size insertion: images: ( { filename = "image.ubifs"; volume = "root_r"; sha256 = "@image.ubifs"; encrypted = true; properties: { decrypted-size = "@image.ubifs"; } }, ); This is usefull especially for *encrypted* UBI images which need the "decrypted-size" property when the image is installed. Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com> --- classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)